[Ruby] v1.57.0-alpha SEGV when calling GRPC.postfork_child #33787
Description
What version of gRPC and what language are you using?
Ruby 3.2.2
grpc 1.57.0-alpha (exact ref is https://github.com/Shopify/grpc/tree/v1.57.0-alpha2-keep-symbol to have symbols for debugging)
What operating system (Linux, Windows,...) and version?
Linux / Ubuntu 22.04
What runtime / compiler are you using (e.g. python version or version of gcc)
gcc.
What did you do?
I'll try to reduce this to a simpler repro, but our application is huge so it may take a while to isolate what may trigger this, so I'm reporting what I see first in case the bug might be evident with the backtrace.
To make sure I never forget to call the hooks, I use the Ruby 3.1+ fork hook interface:
module GRPCForkHook
def _fork
PanamaClient.before_fork
if (pid = super)
PanamaClient.after_fork_parent
else
PanamaClient.after_fork_child
end
pid
end
end
Process.singleton_class.prepend(GRPCForkHook)Here PanamaClient calls are thing wrapper around GRPC to do some extra things before or after the hooks:
def before_fork
INSTANCES.each_value(&:before_fork)
GRPC.prefork if GRPC_FORK_SUPPORT
end
def after_fork_child
GRPC.postfork_child if GRPC_FORK_SUPPORT
INSTANCES.each_value(&:after_fork)
end
def after_fork_parent
GRPC.postfork_parent if GRPC_FORK_SUPPORT
INSTANCES.each_value(&:after_fork)
endWhat did you expect to see?
I'd expect the program to continue
What did you see instead?
/tmp/bundle/ruby/3.2.0/gems/panama_client-2.4.0/lib/panama_client/client/fork_safety.rb:24: [BUG] Segmentation fault at 0x0000000000000038
ruby 3.2.2 (2023-07-19 revision 36486ba854) [x86_64-linux]
-- Control frame information -----------------------------------------------
c:0016 p:---- s:0083 e:000082 CFUNC :postfork_child
c:0015 p:0008 s:0079 e:000078 METHOD /tmp/bundle/ruby/3.2.0/gems/panama_client-2.4.0/lib/panama_client/client/fork_safety.rb:24
c:0014 p:0004 s:0075 e:000074 METHOD /tmp/bundle/ruby/3.2.0/gems/panama_client-2.4.0/lib/panama_client.rb:25
c:0013 p:0020 s:0071 e:000070 METHOD /app/components/platform/essentials/app/utils/shopify/pitchfork.rb:30
c:0012 p:0057 s:0065 e:000064 METHOD /app/components/platform/essentials/test/support/helpers/pitchfork_test_helper.rb:28
c:0011 p:0008 s:0058 e:000057 METHOD /tmp/bundle/ruby/3.2.0/gems/minitest-5.18.1/lib/minitest.rb:1065
c:0010 p:0010 s:0051 e:000050 BLOCK /tmp/bundle/ruby/3.2.0/gems/ci-queue-0.31.0/lib/minitest/queue.rb:179
c:0009 p:0007 s:0048 e:000047 METHOD /tmp/bundle/ruby/3.2.0/gems/ci-queue-0.31.0/lib/minitest/queue.rb:168
c:0008 p:0004 s:0042 e:000041 METHOD /tmp/bundle/ruby/3.2.0/gems/ci-queue-0.31.0/lib/minitest/queue.rb:178
c:0007 p:0005 s:0038 e:000037 BLOCK /tmp/bundle/ruby/3.2.0/gems/ci-queue-0.31.0/lib/minitest/queue.rb:229
c:0006 p:0027 s:0031 e:000030 METHOD /tmp/bundle/ruby/3.2.0/gems/ci-queue-0.31.0/lib/ci/queue/redis/worker.rb:55
c:0005 p:0006 s:0026 e:000025 METHOD /tmp/bundle/ruby/3.2.0/gems/ci-queue-0.31.0/lib/minitest/queue.rb:228
c:0004 p:0012 s:0020 e:000019 METHOD /tmp/bundle/ruby/3.2.0/gems/ci-queue-0.31.0/lib/minitest/queue.rb:213
c:0003 p:0133 s:0015 e:000014 METHOD /tmp/bundle/ruby/3.2.0/gems/minitest-5.18.1/lib/minitest.rb:159
c:0002 p:0045 s:0008 E:000540 BLOCK /tmp/bundle/ruby/3.2.0/gems/minitest-5.18.1/lib/minitest.rb:83 [FINISH]
c:0001 p:0000 s:0003 E:0016c0 DUMMY [FINISH]
-- Ruby level backtrace information ----------------------------------------
/tmp/bundle/ruby/3.2.0/gems/minitest-5.18.1/lib/minitest.rb:83:in `block in autorun'
/tmp/bundle/ruby/3.2.0/gems/minitest-5.18.1/lib/minitest.rb:159:in `run'
/tmp/bundle/ruby/3.2.0/gems/ci-queue-0.31.0/lib/minitest/queue.rb:213:in `__run'
/tmp/bundle/ruby/3.2.0/gems/ci-queue-0.31.0/lib/minitest/queue.rb:228:in `run_from_queue'
/tmp/bundle/ruby/3.2.0/gems/ci-queue-0.31.0/lib/ci/queue/redis/worker.rb:55:in `poll'
/tmp/bundle/ruby/3.2.0/gems/ci-queue-0.31.0/lib/minitest/queue.rb:229:in `block in run_from_queue'
/tmp/bundle/ruby/3.2.0/gems/ci-queue-0.31.0/lib/minitest/queue.rb:178:in `run'
/tmp/bundle/ruby/3.2.0/gems/ci-queue-0.31.0/lib/minitest/queue.rb:168:in `with_timestamps'
/tmp/bundle/ruby/3.2.0/gems/ci-queue-0.31.0/lib/minitest/queue.rb:179:in `block in run'
/tmp/bundle/ruby/3.2.0/gems/minitest-5.18.1/lib/minitest.rb:1065:in `run_one_method'
/app/components/platform/essentials/test/support/helpers/pitchfork_test_helper.rb:28:in `run'
/app/components/platform/essentials/app/utils/shopify/pitchfork.rb:30:in `after_worker_fork'
/tmp/bundle/ruby/3.2.0/gems/panama_client-2.4.0/lib/panama_client.rb:25:in `after_fork_child'
/tmp/bundle/ruby/3.2.0/gems/panama_client-2.4.0/lib/panama_client/client/fork_safety.rb:24:in `after_fork_child'
/tmp/bundle/ruby/3.2.0/gems/panama_client-2.4.0/lib/panama_client/client/fork_safety.rb:24:in `postfork_child'
-- Machine register context ------------------------------------------------
RIP: 0x00007f31b829abae RBP: 0x00007f31b86ee8e0 RSP: 0x00007ffee73bf2f8
RAX: 0x0000000000000000 RBX: 0x0000000000000078 RCX: 0x00007f31b61f5140
RDX: 0x00007f31cc7ea630 RDI: 0x000000000000000a RSI: 0x00007f31b9820710
R8: 0x00007ffee73bf320 R9: 0x00007f31cc400980 R10: 0x00007f31cc408af0
R11: 0x0000000000000001 R12: 0x0000000000000000 R13: 0x0000000000000000
R14: 0x00007f31b86ee8e0 R15: 0x00007f31b86d0560 EFL: 0x0000000000010246
-- C level backtrace information -------------------------------------------
The process exited with status 43.
/usr/local/ruby/bin/ruby(rb_print_backtrace+0x14) [0x55e8b111971f] /tmp/ruby-build.20230719070129.391.cSc12N/ruby-3.2.0-36486ba8544a5aed5ff97b24c082ceada532ccdb/vm_dump.c:785
/usr/local/ruby/bin/ruby(rb_vm_bugreport) /tmp/ruby-build.20230719070129.391.cSc12N/ruby-3.2.0-36486ba8544a5aed5ff97b24c082ceada532ccdb/vm_dump.c:1080
/usr/local/ruby/bin/ruby(rb_bug_for_fatal_signal+0xec) [0x55e8b126819c] /tmp/ruby-build.20230719070129.391.cSc12N/ruby-3.2.0-36486ba8544a5aed5ff97b24c082ceada532ccdb/error.c:813
/usr/local/ruby/bin/ruby(sigsegv+0x4d) [0x55e8b10655fd] /tmp/ruby-build.20230719070129.391.cSc12N/ruby-3.2.0-36486ba8544a5aed5ff97b24c082ceada532ccdb/signal.c:964
/lib/x86_64-linux-gnu/libc.so.6(0x7f31cca57520) [0x7f31cca57520]
/tmp/bundle/ruby/3.2.0/bundler/gems/grpc-7d6c64d4f85b/src/ruby/lib/grpc/grpc_c.so(grpc_pollset_size+0xe) [0x7f31b829abae] src/core/lib/iomgr/pollset.cc:56
/tmp/bundle/ruby/3.2.0/bundler/gems/grpc-7d6c64d4f85b/src/ruby/lib/grpc/grpc_c.so(grpc_completion_queue_create_internal+0x1c3) [0x7f31b7ff57a3] src/core/lib/surface/completion_queue.cc:540
/tmp/bundle/ruby/3.2.0/bundler/gems/grpc-7d6c64d4f85b/src/ruby/lib/grpc/grpc_c.so(0x7f31b7ff7cef) [0x7f31b7ff7cef]
/tmp/bundle/ruby/3.2.0/bundler/gems/grpc-7d6c64d4f85b/src/ruby/lib/grpc/grpc_c.so(grpc_completion_queue_create_for_next) src/core/lib/surface/completion_queue_factory.cc:69
/tmp/bundle/ruby/3.2.0/bundler/gems/grpc-7d6c64d4f85b/src/ruby/lib/grpc/grpc_c.so(grpc_rb_channel_polling_thread_start+0x5d) [0x7f31b7ee58ed] /tmp/bundle/ruby/3.2.0/bundler/gems/grpc-7d6c64d4f85b/src/ruby/ext/grpc/rb_channel.c:820
/tmp/bundle/ruby/3.2.0/bundler/gems/grpc-7d6c64d4f85b/src/ruby/lib/grpc/grpc_c.so(grpc_rb_postfork_child+0xbe) [0x7f31b7ee7f7e] /tmp/bundle/ruby/3.2.0/bundler/gems/grpc-7d6c64d4f85b/src/ruby/ext/grpc/rb_grpc.c:338
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame+0x137) [0x55e8b10f7d97] /tmp/ruby-build.20230719070129.391.cSc12N/ruby-3.2.0-36486ba8544a5aed5ff97b24c082ceada532ccdb/vm_insnhelper.c:3270
/usr/local/ruby/bin/ruby(vm_sendish+0x9c) [0x55e8b110b884] /tmp/ruby-build.20230719070129.391.cSc12N/ruby-3.2.0-36486ba8544a5aed5ff97b24c082ceada532ccdb/vm_insnhelper.c:5082
/usr/local/ruby/bin/ruby(vm_exec_core) /tmp/ruby-build.20230719070129.391.cSc12N/ruby-3.2.0-36486ba8544a5aed5ff97b24c082ceada532ccdb/insns.def:820
/usr/local/ruby/bin/ruby(rb_vm_exec+0xa95) [0x55e8b10fb565] /tmp/ruby-build.20230719070129.391.cSc12N/ruby-3.2.0-36486ba8544a5aed5ff97b24c082ceada532ccdb/vm.c:2384
/usr/local/ruby/bin/ruby(rb_vm_invoke_proc+0x5f) [0x55e8b11024af] /tmp/ruby-build.20230719070129.391.cSc12N/ruby-3.2.0-36486ba8544a5aed5ff97b24c082ceada532ccdb/vm.c:1604
/usr/local/ruby/bin/ruby(rb_proc_call_kw+0x67) [0x55e8b0ff9375] /tmp/ruby-build.20230719070129.391.cSc12N/ruby-3.2.0-36486ba8544a5aed5ff97b24c082ceada532ccdb/proc.c:995
/usr/local/ruby/bin/ruby(rb_proc_call) /tmp/ruby-build.20230719070129.391.cSc12N/ruby-3.2.0-36486ba8544a5aed5ff97b24c082ceada532ccdb/proc.c:1005
/usr/local/ruby/bin/ruby(exec_end_procs_chain+0x44) [0x55e8b0f056bd] /tmp/ruby-build.20230719070129.391.cSc12N/ruby-3.2.0-36486ba8544a5aed5ff97b24c082ceada532ccdb/eval_jump.c:105
/usr/local/ruby/bin/ruby(rb_ec_exec_end_proc) /tmp/ruby-build.20230719070129.391.cSc12N/ruby-3.2.0-36486ba8544a5aed5ff97b24c082ceada532ccdb/eval_jump.c:120
/usr/local/ruby/bin/ruby(rb_ec_teardown+0xc7) [0x55e8b0f058b7] /tmp/ruby-build.20230719070129.391.cSc12N/ruby-3.2.0-36486ba8544a5aed5ff97b24c082ceada532ccdb/eval.c:159
/usr/local/ruby/bin/ruby(rb_ec_cleanup+0x14a) [0x55e8b0f05aba] /tmp/ruby-build.20230719070129.391.cSc12N/ruby-3.2.0-36486ba8544a5aed5ff97b24c082ceada532ccdb/eval.c:212
/usr/local/ruby/bin/ruby(ruby_run_node+0x57) [0x55e8b0f06447] /tmp/ruby-build.20230719070129.391.cSc12N/ruby-3.2.0-36486ba8544a5aed5ff97b24c082ceada532ccdb/eval.c:330
/usr/local/ruby/bin/ruby(rb_main+0x21) [0x55e8b0f01007] ./main.c:38
/usr/local/ruby/bin/ruby(main) ./main.c:57
/lib/x86_64-linux-gnu/libc.so.6(0x7f31cca3ed90) [0x7f31cca3ed90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f31cca3ee40]
[0x55e8b0f01055]
cc @apolcyn