This is a followup discussion of #38439. We fix #38439 by restoring ThreadLocalState and distributed autograd context separately. Creating this issue to track discussion on whether distributed autograd context id belongs to ThreadLocalState.h or we should create RpcThreadLocalState.h or else. Below are some concerns and notes from an offline discussion with @xush6528 @ilia-cher and @pritamdamania87,

• the reason we didn't put ctx id in ThreadLocalState was because there was a concern that it would get passed to the backward pass and might trigger autograd recording for RPC calls in the backward pass.
• There is not preferred to add another flag to ThreadLocalState.h to set/skip autograd context id.
• If we add context id into ThreadLocalState.h, we should also dedup the similar logic added for RPC's TorchScript support. See [DistAutograd x JIT] Capture global state, dist autograd current context id, before thread switching triggered by JIT future.wait() #36395

cc @suo @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @rohan-varma @xush6528 @jjlilley @osalpekar