Skip to content

Don't count internal ApplicationCallbackExecCtx against ExcCtx count#17973

Merged
ericgribkoff merged 2 commits intogrpc:masterfrom
ericgribkoff:prefork_handler_bugfix
Feb 9, 2019
Merged

Don't count internal ApplicationCallbackExecCtx against ExcCtx count#17973
ericgribkoff merged 2 commits intogrpc:masterfrom
ericgribkoff:prefork_handler_bugfix

Conversation

@ericgribkoff
Copy link
Copy Markdown
Contributor

This was originally done for ExcCtx in
#15825. This avoids a hang in our
pre-fork handler, where grpc_core::Executor::RunClosures attempts to
increment the thread count from an invocation of grpc_prefork().

The stack trace of the hang without this change:

#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffff6717492 in gpr_cv_wait (cv=cv@entry=0x555555bcbb20, mu=mu@entry=0x555555bcbaf8, abs_deadline=...) at src/core/lib/gpr/sync_posix.cc:172
#2  0x00007ffff67184fb in grpc_core::internal::ExecCtxState::IncExecCtxCount (this=0x555555bcbaf0) at src/core/lib/gprpp/fork.cc:67
#3  grpc_core::Fork::IncExecCtxCount () at src/core/lib/gprpp/fork.cc:215
#4  0x00007ffff67308ea in grpc_core::ApplicationCallbackExecCtx::ApplicationCallbackExecCtx (this=0x7fffffff98f0) at ./src/core/lib/iomgr/exec_ctx.h:236
#5  grpc_core::Executor::RunClosures (executor_name=0x7ffff68c61a2 "default-executor", list=...) at src/core/lib/iomgr/executor.cc:119
#6  0x00007ffff6730e3f in grpc_core::Executor::SetThreading (this=0x5555562071c0, threading=threading@entry=false) at src/core/lib/iomgr/executor.cc:203
#7  0x00007ffff673194c in grpc_core::Executor::SetThreadingAll (enable=enable@entry=false) at src/core/lib/iomgr/executor.cc:458
#8  0x00007ffff6731dee in grpc_prefork () at src/core/lib/iomgr/fork_posix.cc:74

At least one other option besides the GRPC_APP_CALLBACK_EXEC_CTX_FLAG_IS_INTERNAL_THREAD flag introduced here is to look at the existing excctx and inherit whether to increment/decrement the thread count based on its setting. But it's not clear to me that an ApplicationCallbackExecCtx will always be created with an associated ExecCtx - in which case the increment/decrement calls may be suitable to delete altogether.

As for why this wasn't caught by our test infrastructure: this was an oversight on my part, and I'm reviving #16513 to make sure we have test coverage for this in the future.

This was originally done for ExcCtx in
grpc#15825. This avoids a hang in our
pre-fork handler, where grpc_core::Executor::RunClosures attempts to
increment the thread count from an invocation of grpc_prefork():
static void GlobalShutdown(void) { gpr_tls_destroy(&callback_exec_ctx_); }

private:
uintptr_t flags_;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

flags_{0} will solve your MSAN issues

@ericgribkoff ericgribkoff merged commit b3ed1f4 into grpc:master Feb 9, 2019
@ericgribkoff ericgribkoff deleted the prefork_handler_bugfix branch February 9, 2019 01:47
@ericgribkoff
Copy link
Copy Markdown
Contributor Author

Thanks @vjpai!

@lock lock bot locked as resolved and limited conversation to collaborators May 10, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

lang/core priority/P0/RELEASE BLOCKER release notes: no Indicates if PR should not be in release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants