Skip to content

unregister onExit event handler on disposing interactive host#24058

Merged
ivanbasov merged 19 commits intodotnet:dev15.7.xfrom
ivanbasov:interactive
Jan 26, 2018
Merged

unregister onExit event handler on disposing interactive host#24058
ivanbasov merged 19 commits intodotnet:dev15.7.xfrom
ivanbasov:interactive

Conversation

@ivanbasov
Copy link
Copy Markdown
Contributor

@ivanbasov ivanbasov commented Jan 5, 2018

Customer scenario

Customer shuts down Visual Studio. There is a race condition when closing the Interactive Window and the corresponding process.

When the process is closed before the window (within this or another scenario), it sends a message to display it in the interactive window. If the window was closed somewhere between checks the it is open and executing a command to output the message, an failure happens.

Bugs this fixes

VSO 514822 and corresponding: #24006

Workarounds, if any

None

Risk

Low

Performance impact

None

Is this a regression from a previous update?

No

How was the bug found?

Watson hits

@ivanbasov ivanbasov added Area-Interactive Tenet-Reliability Customer telemetry indicates that the product is failing in a crash/hang/dataloss manner. labels Jan 5, 2018
@ivanbasov ivanbasov requested review from a team, jasonmalinowski and tmat January 5, 2018 00:18
Copy link
Copy Markdown
Member

@jasonmalinowski jasonmalinowski Jan 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the local method now vs. just a regular good old fashioned method? I realize it was nice when it was localized to this, but that ship has sailed. #Resolved

Copy link
Copy Markdown
Member

@jasonmalinowski jasonmalinowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think tossing the local method now makes this easier to follow.

Copy link
Copy Markdown
Member

@tmat tmat Jan 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if [](start = 16, length = 2)

This doesn't guarantee that ProcessExitedHandler does not call _host.OnProcesExited after we return from Dispose.

There is a race condition:

  1. ProcessExitedHandler has already been called and in it _disposing already tested, yet OnProcessExited has not been called yet
  2. Dispose is called
  3. Dispose returns
  4. ProcessExitedHandler is resumed and calls OnProcessExited #Resolved

Copy link
Copy Markdown
Member

@tmat tmat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🕐

Copy link
Copy Markdown
Member

@tmat tmat Jan 10, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the benefit of using DisposableWait, vs. just _disposeLock.Wait(); _disposeLock.Release();? #Resolved

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, we should probably not wait when running in finalizer (disposing == false).


In reply to: 160765822 [](ancestors = 160765822)

Copy link
Copy Markdown
Member

@tmat tmat Jan 10, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more thing: Since we access _disposeLock in destructor we should guard against it being null, in case the constructor of this class doesn't run to completion.

var semaphore = _disposeSemaphore;
if (semaphore != null && disposing)
{
  semaphore.Wait();
  semaphore.Dispose();
}

In reply to: 160766053 [](ancestors = 160766053,160765822)

Copy link
Copy Markdown
Member

@tmat tmat Jan 10, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bool joinThreads, bool disposing [](start = 29, length = 32)

I think we should remove joinThreads and use disposing instead on line 250. We don't want to be waiting during finalization (on threads to join). #Resolved

Copy link
Copy Markdown
Member

@tmat tmat Jan 10, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GC.SuppressFinalize(this); [](start = 20, length = 26)

Let's move GC.SuppressFinalize(this) to the end of Dispose() to follow the common pattern. #Resolved

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What pattern do you mean? I see various position of GC.SuppressFinalize(this) within Dispose(): in the beginning, in the middle and in the end.
What is the purpose to move it to the end?


In reply to: 160767337 [](ancestors = 160767337)

Copy link
Copy Markdown
Member

@tmat tmat Jan 10, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the object is taken out of the finalizer queue and then dispose fails, it's never gonna be finalized. So the idea is that SuppressFinalize should be the last call of Dispose in order to avoid that. #Resolved

Copy link
Copy Markdown
Member

@tmat tmat Jan 10, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Member

@tmat tmat Jan 10, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_disposeLock [](start = 39, length = 12)

Let's call this _disposeSemaphore, since it's not actually a lock. #Resolved

@ivanbasov
Copy link
Copy Markdown
Contributor Author

ivanbasov commented Jan 16, 2018

@tmat, could you please review? #Resolved

Copy link
Copy Markdown
Member

@tmat tmat Jan 17, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using (await _disposeSemaphore.DisposableWaitAsync().ConfigureAwait(false)) [](start = 28, length = 75)

I'm not sure ho is this supposed to work. Should we just signal (call _disposeSemaphore.Release()) when this event is finished? #Resolved

Copy link
Copy Markdown
Member

@tmat tmat Jan 17, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For maintain some consistency I'd also signal the semaphore here (call _disposeSemaphore.Release()) and add a comment on that semaphore that it is signaled when the ProcessExited handler is removed and can no longer be executed. #Resolved

Copy link
Copy Markdown
Member

@tmat tmat Jan 17, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 [](start = 91, length = 1)

Shouldn't this start in non-signaled state (initialCount: 0)?
Actually, since we only need two states we should use ManualResetEventSlim instead of SemaphoreSlim. #Resolved

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more thing: i think this should be moved to the RemoteService instance and used in its Dispose method. The InteractiveHost itself doesn't really need to know about this -- it's all about disposing the RemoteService.


In reply to: 162140846 [](ancestors = 162140846)

Copy link
Copy Markdown
Member

@tmat tmat Jan 17, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's move this to the RemoteService.Dispose() method.
#Resolved

Copy link
Copy Markdown
Member

@tmat tmat Jan 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_host != null [](start = 43, length = 13)

Good catch here, but there is still race. _host?.OnProcessExited(...) would be better. #Resolved

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, since _disposing must be true before _host is set to null, _host can't be null at this point.

Make _disposing volatile field though, so that the JIT doesn't reorder.


In reply to: 162224558 [](ancestors = 162224558)

Copy link
Copy Markdown
Member

@tmat tmat Jan 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

object _disposeLock = new object(); #Resolved

Copy link
Copy Markdown
Member

@tmat tmat Jan 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if (Interlocked.Exchange(ref _processExitHandling, ProcessExitHandled) == ProcessExitHooked) [](start = 20, length = 92)

bool invokeEvent = false;

lock (_disposeLock) 
{
  if (_processExitHandling == ProcessExitHooked) 
  {
    Process.Exited -= ProcessExitedHandler;
     invokeEvent  =!_disposing;
    _processExitHandling = ProcessExitHandled;
  }
}

if (invokeEvent)
{
await _host.OnProcessExited(Process).ConfigureAwait(false);
}
``` #Resolved

Copy link
Copy Markdown
Member

@tmat tmat Jan 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lock (_disposeLock)
{
  _disposing = true;

  if (_processExitHandling  == ProcessExitHooked) {
    Process.Exited -= ProcessExitedHandler;
    _processExitHandling = ProcessExitHandled;
  }
}
``` #Resolved

Copy link
Copy Markdown
Member

@tmat tmat Jan 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lock (_disposeLock)
{
if (_processExitHandling == 0)
{
_processExitHandling = ProcessExitHooked;
Process.Exited += ProcessExitedHandler;
}
}
#Resolved

Copy link
Copy Markdown
Member

@tmat tmat Jan 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_disposing = true; [](start = 16, length = 18)

This should also be inside the using. #Resolved

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually we don't need _disposing variable anymore.


In reply to: 162233741 [](ancestors = 162233741)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. For consistency, it is better to put it inside using.


In reply to: 162233741 [](ancestors = 162233741)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not just for consistency. Since this is now shared variable it would need to be under the lock. But we don't need it at all actually.


In reply to: 162234150 [](ancestors = 162234150,162233741)

Copy link
Copy Markdown
Member

@tmat tmat Jan 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if (!_disposing) [](start = 28, length = 16)

I think this check is not needed now. Once Dispose sets _processExitHandling to ProcessExitHandled we won't get here. #Resolved

@ivanbasov ivanbasov added PR For Personal Review Only The PR doesn’t require anyone other than the developer to review it. and removed PR For Personal Review Only The PR doesn’t require anyone other than the developer to review it. labels Jan 19, 2018
@ivanbasov
Copy link
Copy Markdown
Contributor Author

ivanbasov commented Jan 23, 2018

@tmat, @jinujoseph, @jasonmalinowski, please review. I would like to push this fix into 15.6 #Resolved

@tmat
Copy link
Copy Markdown
Member

tmat commented Jan 23, 2018

@ivanbasov Does it meet the bar for escrow? I think moving it to 15.7 would be ok. #Resolved

@ivanbasov ivanbasov changed the base branch from master to dev15.7.x January 23, 2018 22:14
@ivanbasov
Copy link
Copy Markdown
Contributor Author

ivanbasov commented Jan 24, 2018

@tmat, could you please review? I hope I addressed all the comments. Thanks! #Resolved

private Thread _readErrorOutputThread; // nulled on dispose
private InteractiveHost _host; // nulled on dispose
private bool _disposing; // set to true on dispose
private volatile int _processExitHandling; // set to ProcessExitHandled on dispose
Copy link
Copy Markdown
Member

@tmat tmat Jan 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

int [](start = 29, length = 3)

Sorry, one more thing: now that we don't use Interlocked we could define the ProcessExitXXX values as an enum (ProcessExitHandlerStatus { Uninitialized = 0, Initialized = 1, Executed = 2 } ). Would make the code more readable.
#Resolved

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea! Thank you!


In reply to: 163422405 [](ancestors = 163422405)

Copy link
Copy Markdown
Member

@tmat tmat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

}
Process.Exited -= ProcessExitedHandler;
_processExitHandlerStatus = ProcessExitHandlerStatus.Handled;
// Should set _processExitHandling before calling OnProcessExited to avoid deadlocks.
Copy link
Copy Markdown
Member

@tmat tmat Jan 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_processExitHandling [](start = 42, length = 20)

nit: wrong name #Resolved

@ivanbasov
Copy link
Copy Markdown
Contributor Author

Tagging @MattGertz to approve merge into dev15.7.x

@ivanbasov
Copy link
Copy Markdown
Contributor Author

ivanbasov commented Jan 25, 2018

@jasonmalinowski please unblock #Resolved

Copy link
Copy Markdown
Member

@jasonmalinowski jasonmalinowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ivanbasov I admit I still have some questions. I think there's some room for a better comment, maybe?

{
IpcServerChannel serverChannel = process._ServerChannel;
process.Dispose(joinThreads: true);
process.Dispose(disposing: true);
Copy link
Copy Markdown
Member

@jasonmalinowski jasonmalinowski Jan 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just call the public overload? #Resolved

if (joinThreads)
// There can be a call from host initiated from OnProcessExit.
// This check on the beginning helps to avoid a reentrancy.
if (_processExitHandlerStatus == ProcessExitHandlerStatus.Hooked)
Copy link
Copy Markdown
Member

@jasonmalinowski jasonmalinowski Jan 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the problem here that this can deadlock because if the _host.OnProcessExited() call is happening above and that's still holding the lock, and it tries to call back into Dispose(), that will deadlock? Why can't we call _host.OnProcessExited() outside the lock?

I guess a better comment here would be appreciated. The use of volatile here is generally scary. I think it works, but that doesn't make it less scary. #Resolved

Copy link
Copy Markdown
Member

@jasonmalinowski jasonmalinowski Jan 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(this question is best answered by just adding more comments so the code is clearer vs. answering my question in GitHub) #Resolved

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, Jason! I added more comments to clarify the scenario.

Maybe there can be an option to avoid using volatile and removing the _host.OnProcessExited() outside of the lock. We could not see these options so far. The problem we fixed that the host is being disposed during _host.OnProcessExited() execution. So, the fix is to protect it from the dispose.


In reply to: 163994639 [](ancestors = 163994639)

@ivanbasov ivanbasov merged commit 958c99e into dotnet:dev15.7.x Jan 26, 2018
@ivanbasov ivanbasov deleted the interactive branch January 26, 2018 00:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Approved to merge Area-Interactive Tenet-Reliability Customer telemetry indicates that the product is failing in a crash/hang/dataloss manner.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants