Skip to content

feat: drop Python 3.9#3816

Merged
ianna merged 8 commits intomainfrom
henryiii/feat/drop39
Jan 27, 2026
Merged

feat: drop Python 3.9#3816
ianna merged 8 commits intomainfrom
henryiii/feat/drop39

Conversation

@henryiii
Copy link
Copy Markdown
Member

@henryiii henryiii commented Jan 22, 2026

(builds on #3815, since I needed that to test locally)

Dropping 3.9, and adding strict=, trying to use True everywhere the tests would let me.

  • chore: fix noxfile on Intel macOS
  • chore: drop Python 3.9
  • chore: remove Python < 3.6 compat line
  • fix: add strict= to zip

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 22, 2026

The documentation preview is ready to be viewed at http://preview.awkward-array.org.s3-website.us-east-1.amazonaws.com/PR3816

henryiii and others added 5 commits January 26, 2026 10:29
@henryiii
chore: drop Python 3.9
fb039c7 
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
@henryiii
chore: remove Python < 3.6 compat line
a91c2d5 
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
@henryiii
fix: add strict= to zip
8f3b344 
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
@pre-commit-ci @henryiii
style: pre-commit fixes
983df23
@henryiii
ci: remove 3.9
a0fbe2a 
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
@henryiii henryiii force-pushed the henryiii/feat/drop39 branch from e3d6dfe to a0fbe2a Compare January 26, 2026 15:33
@codecov
Copy link
Copy Markdown

codecov bot commented Jan 26, 2026 •
edited
Loading

Codecov Report

❌ Patch coverage is 82.60870% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.65%. Comparing base (680ab6e) to head (8556723).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/awkward/_connect/cling.py 0.00% 1 Missing ⚠️
src/awkward/_connect/hist.py 0.00% 1 Missing ⚠️
src/awkward/_connect/numba/arrayview.py 0.00% 1 Missing ⚠️
src/awkward/_kernels.py 66.66% 1 Missing ⚠️
src/awkward/contents/content.py 0.00% 1 Missing ⚠️
src/awkward/forms/form.py 50.00% 1 Missing ⚠️
src/awkward/forms/unionform.py 0.00% 1 Missing ⚠️
src/awkward/types/_awkward_datashape_parser.py 75.00% 1 Missing ⚠️
Additional details and impacted files
Files with missing lines Coverage Δ
src/awkward/_behavior.py 95.19% <ø> (ø)
src/awkward/_broadcasting.py 91.94% <100.00%> (ø)
src/awkward/_connect/jax/trees.py 100.00% <100.00%> (ø)
src/awkward/_connect/numexpr.py 88.88% <ø> (+7.40%) ⬆️
src/awkward/_connect/pyarrow/table_conv.py 88.88% <100.00%> (ø)
src/awkward/_namedaxis.py 91.28% <100.00%> (ø)
src/awkward/_nplikes/array_module.py 95.25% <ø> (+0.31%) ⬆️
src/awkward/_nplikes/typetracer.py 77.59% <100.00%> (+1.70%) ⬆️
src/awkward/_nplikes/virtual.py 90.86% <100.00%> (ø)
src/awkward/_slicing.py 87.09% <100.00%> (ø)
... and 25 more

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ikrommyd
Copy link
Copy Markdown
Collaborator

ikrommyd commented Jan 26, 2026

@henryiii from searching, it looks like the python 3.10 was added between numpy 1.21.2 and 1.22.0 with 1.21.2 providing only linux wheels for 3.10. What should we pin the minimal requirements to? 1.21.3 looks like it has macos and windows wheels https://pypi.org/project/numpy/1.21.3/#files. Should we do that or 1.22?

@henryiii
Copy link
Copy Markdown
Member Author

henryiii commented Jan 26, 2026

If there's something needed from 1.22, we could do that, but 1.21.3 is the most compatible option. I don't think splitting this up for 1.21.2 is worth it. :)

@ikrommyd
Copy link
Copy Markdown
Collaborator

ikrommyd commented Jan 26, 2026

Just look at release notes and decide https://numpy.org/devdocs/release/1.21.3-notes.html. There's just a couple of wheel caveats.

@ikrommyd
Copy link
Copy Markdown
Collaborator

ikrommyd commented Jan 26, 2026

Ah, this also needs pandas==1.3.4 in the minimal case.

@henryiii
fix: update min numpy version to 1.21.3, pandas 1.3.4
5833b6e 
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
@henryiii henryiii force-pushed the henryiii/feat/drop39 branch from 14867f4 to 5833b6e Compare January 26, 2026 16:19
@henryiii
Copy link
Copy Markdown
Member Author

henryiii commented Jan 26, 2026

Do you know what the free-threading fail is? Assuming I didn't really touch the 3.14t code much?

@ikrommyd
Copy link
Copy Markdown
Collaborator

ikrommyd commented Jan 26, 2026

I can't reproduce those locally and I haven't hit those in any other CI so far.

@ikrommyd
Copy link
Copy Markdown
Collaborator

ikrommyd commented Jan 26, 2026 •
edited
Loading

I've hit rerun to see if we get it again but this test defines some arrays outside of the test functions

awkward/tests/test_1490_jax_reducers_combinations.py

Lines 19 to 31 in 680ab6e

test_regulararray = ak.Array(
[[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]], backend="jax"
)
test_regulararray_tangent = ak.Array(
[[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]], backend="jax"
)
test_regulararray_jax = jax.numpy.array(
[[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]], dtype=np.float64
)
test_regulararray_tangent_jax = jax.numpy.array(
[[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]], dtype=np.float64
)
which in pytest-run-parallel will be shared across threads. So if there is some kind of mutation, I can see how things could go wrong.

This is the error we hit in jax https://github.com/jax-ml/jax/blob/0d34e25aecf6466f9014186a2175f56f52106163/jax/_src/array.py#L498-L502 so something caused sharding of the array?

Edit:
no error on rerun

@ikrommyd
Copy link
Copy Markdown
Collaborator

ikrommyd commented Jan 26, 2026 •
edited
Loading

There has to be something hanging in the Linux-ROOT tests. It just gets cancelled.
It stops at

tests/test_1374_to_rdataframe.py::test_two_columns PASSED                [ 29%]

@ikrommyd
Copy link
Copy Markdown
Collaborator

ikrommyd commented Jan 26, 2026

I can get the hang locally on macos even. Here is the backtrace when the process has hanged

Click to show full backtrace 
❯ lldb -p 74410 -o "bt all" -o "quit"
(lldb) process attach --pid 74410
Process 74410 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    frame #0: 0x000000013866901c libImt.6.36.06.so`ROOT::Internal::RSlotStack::GetSlot() + 72
libImt.6.36.06.so`ROOT::Internal::RSlotStack::GetSlot:
->  0x13866901c <+72>: b      0x13866901c    ; <+72>
    0x138669020 <+76>: mov    x0, x8
    0x138669024 <+80>: ret

libImt.6.36.06.so`ROOT::Internal::TExecutor::TExecutor:
    0x138669028 <+0>:  stp    x24, x23, [sp, #-0x40]!
Target 0: (python3.10) stopped.
Executable binary set to "/Users/iason/micromamba/envs/awkward/bin/python3.10".
Architecture set to: arm64-apple-macosx-.
(lldb) bt all
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x000000013866901c libImt.6.36.06.so`ROOT::Internal::RSlotStack::GetSlot() + 72
    frame #1: 0x00000003a7649880 libROOTDataFrame.6.36.06.so`ROOT::Detail::RDF::RLoopManager::DataSourceThreadTask(std::__1::pair<unsigned long long, unsigned long long> const&, ROOT::Internal::RSlotStack&, std::__1::atomic<unsigned long long>&) + 64
    frame #2: 0x000000013866c644 libImt.6.36.06.so`void tbb::detail::d1::dynamic_grainsize_mode<tbb::detail::d1::adaptive_mode<tbb::detail::d1::auto_partition_type>>::work_balance<tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned int>, tbb::detail::d1::parallel_for_body_wrapper<std::__1::function<void (unsigned int)>, unsigned int>, tbb::detail::d1::auto_partitioner const>, tbb::detail::d1::blocked_range<unsigned int>>(tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned int>, tbb::detail::d1::parallel_for_body_wrapper<std::__1::function<void (unsigned int)>, unsigned int>, tbb::detail::d1::auto_partitioner const>&, tbb::detail::d1::blocked_range<unsigned int>&, tbb::detail::d1::execution_data&) + 1812
    frame #3: 0x000000013866bef8 libImt.6.36.06.so`void tbb::detail::d1::partition_type_base<tbb::detail::d1::auto_partition_type>::execute<tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned int>, tbb::detail::d1::parallel_for_body_wrapper<std::__1::function<void (unsigned int)>, unsigned int>, tbb::detail::d1::auto_partitioner const>, tbb::detail::d1::blocked_range<unsigned int>>(tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned int>, tbb::detail::d1::parallel_for_body_wrapper<std::__1::function<void (unsigned int)>, unsigned int>, tbb::detail::d1::auto_partitioner const>&, tbb::detail::d1::blocked_range<unsigned int>&, tbb::detail::d1::execution_data&) + 396
    frame #4: 0x000000013866bc04 libImt.6.36.06.so`tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned int>, tbb::detail::d1::parallel_for_body_wrapper<std::__1::function<void (unsigned int)>, unsigned int>, tbb::detail::d1::auto_partitioner const>::execute(tbb::detail::d1::execution_data&) + 180
    frame #5: 0x0000000138c22478 libtbb.12.17.dylib`tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter>(tbb::detail::d1::task*, tbb::detail::r1::external_waiter&) + 960
    frame #6: 0x0000000138c1e6e4 libtbb.12.17.dylib`tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) + 180
    frame #7: 0x000000013866bae8 libImt.6.36.06.so`tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned int>, tbb::detail::d1::parallel_for_body_wrapper<std::__1::function<void (unsigned int)>, unsigned int>, tbb::detail::d1::auto_partitioner const>::run(tbb::detail::d1::blocked_range<unsigned int> const&, tbb::detail::d1::parallel_for_body_wrapper<std::__1::function<void (unsigned int)>, unsigned int> const&, tbb::detail::d1::auto_partitioner const&) + 248
    frame #8: 0x000000013866b9b4 libImt.6.36.06.so`tbb::detail::d1::task_arena_function<ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::__1::function<void (unsigned int)> const&)::$_0::operator()() const::'lambda'(), void>::operator()() const + 120
    frame #9: 0x0000000138c10208 libtbb.12.17.dylib`tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, long) + 88
    frame #10: 0x000000013866b908 libImt.6.36.06.so`tbb::detail::d1::task_arena_function<ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::__1::function<void (unsigned int)> const&)::$_0, void>::operator()() const + 68
    frame #11: 0x0000000138c0f53c libtbb.12.17.dylib`tbb::detail::r1::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) + 704
    frame #12: 0x000000013866b5a0 libImt.6.36.06.so`ROOT::TThreadExecutor::ParallelFor(unsigned int, unsigned int, unsigned int, std::__1::function<void (unsigned int)> const&) + 332
    frame #13: 0x00000003a75edb70 libROOTDataFrame.6.36.06.so`ROOT::RDF::RDataSource::ProcessMT(ROOT::Detail::RDF::RLoopManager&) + 220
    frame #14: 0x00000003a7643fe4 libROOTDataFrame.6.36.06.so`ROOT::Detail::RDF::RLoopManager::RunDataSourceMT() + 88
    frame #15: 0x00000003a7645104 libROOTDataFrame.6.36.06.so`ROOT::Detail::RDF::RLoopManager::Run(bool) + 508
    frame #16: 0x00000003a74d85bc
    frame #17: 0x00000003a74d80e0
    frame #18: 0x000000012ffcfa30 libcppyy_backend.6.36.06.so`WrapperCall(long, unsigned long, void*, void*, void*) + 2008
    frame #19: 0x000000013f17df28 libcppyy.6.36.06.so`CPyCppyy::(anonymous namespace)::VoidExecutor::Execute(long, void*, CPyCppyy::CallContext*) + 140
    frame #20: 0x000000013f16374c libcppyy.6.36.06.so`CPyCppyy::CPPMethod::ExecuteFast(void*, long, CPyCppyy::CallContext*) + 76
    frame #21: 0x000000013f163cb0 libcppyy.6.36.06.so`CPyCppyy::CPPMethod::ExecuteProtected(void*, long, CPyCppyy::CallContext*) + 424
    frame #22: 0x000000013f163fe0 libcppyy.6.36.06.so`CPyCppyy::CPPMethod::Call(CPyCppyy::CPPInstance*&, _object* const*, unsigned long, _object*, CPyCppyy::CallContext*) + 544
    frame #23: 0x000000013f1653a4 libcppyy.6.36.06.so`CPyCppyy::(anonymous namespace)::mp_vectorcall(CPyCppyy::CPPOverload*, _object* const*, unsigned long, _object*) + 200
    frame #24: 0x000000013f19d7a4 libcppyy.6.36.06.so`CPyCppyy::CallMethodImp(CPyCppyy::TemplateProxy*, _object*&, _object* const*, unsigned long, _object*, bool, unsigned long long) + 304
    frame #25: 0x000000013f19d074 libcppyy.6.36.06.so`CPyCppyy::tpp_vectorcall(CPyCppyy::TemplateProxy*, _object* const*, unsigned long, _object*) + 1972
    frame #26: 0x00000001007f4bfc python3.10`call_function + 584
    frame #27: 0x00000001007eec8c python3.10`_PyEval_EvalFrameDefault + 11892
    frame #28: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #29: 0x00000001007ef438 python3.10`_PyEval_EvalFrameDefault + 13856
    frame #30: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #31: 0x00000001007ef438 python3.10`_PyEval_EvalFrameDefault + 13856
    frame #32: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #33: 0x00000001007f4bfc python3.10`call_function + 584
    frame #34: 0x00000001007eecac python3.10`_PyEval_EvalFrameDefault + 11924
    frame #35: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #36: 0x0000000100707aa4 python3.10`method_vectorcall + 164
    frame #37: 0x00000001007f4bfc python3.10`call_function + 584
    frame #38: 0x00000001007eecac python3.10`_PyEval_EvalFrameDefault + 11924
    frame #39: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #40: 0x00000001007045ec python3.10`_PyObject_FastCallDictTstate + 160
    frame #41: 0x00000001007052a4 python3.10`_PyObject_Call_Prepend + 172
    frame #42: 0x000000010077a53c python3.10`slot_tp_call + 116
    frame #43: 0x00000001007043d8 python3.10`_PyObject_MakeTpCall + 604
    frame #44: 0x00000001007f4c90 python3.10`call_function + 732
    frame #45: 0x00000001007f0080 python3.10`_PyEval_EvalFrameDefault + 17000
    frame #46: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #47: 0x00000001007f4bfc python3.10`call_function + 584
    frame #48: 0x00000001007eec8c python3.10`_PyEval_EvalFrameDefault + 11892
    frame #49: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #50: 0x00000001007ef438 python3.10`_PyEval_EvalFrameDefault + 13856
    frame #51: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #52: 0x00000001007f4bfc python3.10`call_function + 584
    frame #53: 0x00000001007eecac python3.10`_PyEval_EvalFrameDefault + 11924
    frame #54: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #55: 0x0000000100707aa4 python3.10`method_vectorcall + 164
    frame #56: 0x00000001007f4bfc python3.10`call_function + 584
    frame #57: 0x00000001007eecac python3.10`_PyEval_EvalFrameDefault + 11924
    frame #58: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #59: 0x00000001007045ec python3.10`_PyObject_FastCallDictTstate + 160
    frame #60: 0x00000001007052a4 python3.10`_PyObject_Call_Prepend + 172
    frame #61: 0x000000010077a53c python3.10`slot_tp_call + 116
    frame #62: 0x0000000100704dfc python3.10`_PyObject_Call + 148
    frame #63: 0x00000001007ef438 python3.10`_PyEval_EvalFrameDefault + 13856
    frame #64: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #65: 0x00000001007f4bfc python3.10`call_function + 584
    frame #66: 0x00000001007f0efc python3.10`_PyEval_EvalFrameDefault + 20708
    frame #67: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #68: 0x0000000100707aa4 python3.10`method_vectorcall + 164
    frame #69: 0x00000001007f4bfc python3.10`call_function + 584
    frame #70: 0x00000001007f0080 python3.10`_PyEval_EvalFrameDefault + 17000
    frame #71: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #72: 0x00000001007f4bfc python3.10`call_function + 584
    frame #73: 0x00000001007f0efc python3.10`_PyEval_EvalFrameDefault + 20708
    frame #74: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #75: 0x00000001007f4bfc python3.10`call_function + 584
    frame #76: 0x00000001007f0080 python3.10`_PyEval_EvalFrameDefault + 17000
    frame #77: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #78: 0x00000001007ef438 python3.10`_PyEval_EvalFrameDefault + 13856
    frame #79: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #80: 0x00000001007f4bfc python3.10`call_function + 584
    frame #81: 0x00000001007eecac python3.10`_PyEval_EvalFrameDefault + 11924
    frame #82: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #83: 0x0000000100707aa4 python3.10`method_vectorcall + 164
    frame #84: 0x00000001007f4bfc python3.10`call_function + 584
    frame #85: 0x00000001007eecac python3.10`_PyEval_EvalFrameDefault + 11924
    frame #86: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #87: 0x00000001007045ec python3.10`_PyObject_FastCallDictTstate + 160
    frame #88: 0x00000001007052a4 python3.10`_PyObject_Call_Prepend + 172
    frame #89: 0x000000010077a53c python3.10`slot_tp_call + 116
    frame #90: 0x00000001007043d8 python3.10`_PyObject_MakeTpCall + 604
    frame #91: 0x00000001007f4c90 python3.10`call_function + 732
    frame #92: 0x00000001007f0080 python3.10`_PyEval_EvalFrameDefault + 17000
    frame #93: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #94: 0x00000001007ef438 python3.10`_PyEval_EvalFrameDefault + 13856
    frame #95: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #96: 0x00000001007f4bfc python3.10`call_function + 584
    frame #97: 0x00000001007eecac python3.10`_PyEval_EvalFrameDefault + 11924
    frame #98: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #99: 0x0000000100707aa4 python3.10`method_vectorcall + 164
    frame #100: 0x00000001007f4bfc python3.10`call_function + 584
    frame #101: 0x00000001007eecac python3.10`_PyEval_EvalFrameDefault + 11924
    frame #102: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #103: 0x00000001007045ec python3.10`_PyObject_FastCallDictTstate + 160
    frame #104: 0x00000001007052a4 python3.10`_PyObject_Call_Prepend + 172
    frame #105: 0x000000010077a53c python3.10`slot_tp_call + 116
    frame #106: 0x00000001007043d8 python3.10`_PyObject_MakeTpCall + 604
    frame #107: 0x00000001007f4c90 python3.10`call_function + 732
    frame #108: 0x00000001007f0080 python3.10`_PyEval_EvalFrameDefault + 17000
    frame #109: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #110: 0x00000001007f4bfc python3.10`call_function + 584
    frame #111: 0x00000001007f0efc python3.10`_PyEval_EvalFrameDefault + 20708
    frame #112: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #113: 0x00000001007f4bfc python3.10`call_function + 584
    frame #114: 0x00000001007f0efc python3.10`_PyEval_EvalFrameDefault + 20708
    frame #115: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #116: 0x00000001007ef438 python3.10`_PyEval_EvalFrameDefault + 13856
    frame #117: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #118: 0x00000001007f4bfc python3.10`call_function + 584
    frame #119: 0x00000001007eecac python3.10`_PyEval_EvalFrameDefault + 11924
    frame #120: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #121: 0x0000000100707aa4 python3.10`method_vectorcall + 164
    frame #122: 0x00000001007f4bfc python3.10`call_function + 584
    frame #123: 0x00000001007eecac python3.10`_PyEval_EvalFrameDefault + 11924
    frame #124: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #125: 0x00000001007045ec python3.10`_PyObject_FastCallDictTstate + 160
    frame #126: 0x00000001007052a4 python3.10`_PyObject_Call_Prepend + 172
    frame #127: 0x000000010077a53c python3.10`slot_tp_call + 116
    frame #128: 0x00000001007043d8 python3.10`_PyObject_MakeTpCall + 604
    frame #129: 0x00000001007f4c90 python3.10`call_function + 732
    frame #130: 0x00000001007f0080 python3.10`_PyEval_EvalFrameDefault + 17000
    frame #131: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #132: 0x00000001007f4bfc python3.10`call_function + 584
    frame #133: 0x00000001007f0efc python3.10`_PyEval_EvalFrameDefault + 20708
    frame #134: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #135: 0x00000001007f4bfc python3.10`call_function + 584
    frame #136: 0x00000001007eecac python3.10`_PyEval_EvalFrameDefault + 11924
    frame #137: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #138: 0x00000001007eac8c python3.10`PyEval_EvalCode + 120
    frame #139: 0x00000001007e68c0 python3.10`builtin_exec + 844
    frame #140: 0x0000000100757d04 python3.10`cfunction_vectorcall_FASTCALL + 200
    frame #141: 0x00000001007f4bfc python3.10`call_function + 584
    frame #142: 0x00000001007f0efc python3.10`_PyEval_EvalFrameDefault + 20708
    frame #143: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #144: 0x00000001007f4bfc python3.10`call_function + 584
    frame #145: 0x00000001007f0efc python3.10`_PyEval_EvalFrameDefault + 20708
    frame #146: 0x00000001007eb4d8 python3.10`_PyEval_Vector + 2076
    frame #147: 0x00000001008697a0 python3.10`pymain_run_module + 272
    frame #148: 0x0000000100868d8c python3.10`Py_RunMain + 1088
    frame #149: 0x000000010086a5e0 python3.10`pymain_main + 1216
    frame #150: 0x00000001006adb48 python3.10`main + 56
    frame #151: 0x0000000185ab9d54 dyld`start + 7184
  thread #2
    frame #0: 0x0000000185e409dc libsystem_kernel.dylib`__workq_kernreturn + 8
  thread #3
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000118f73ec8 interpreter.cpython-310-darwin.so`th_worker(void*) + 1040
    frame #3: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #4
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000118f73ec8 interpreter.cpython-310-darwin.so`th_worker(void*) + 1040
    frame #3: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #5
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000118f73ec8 interpreter.cpython-310-darwin.so`th_worker(void*) + 1040
    frame #3: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #6
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000118f73ec8 interpreter.cpython-310-darwin.so`th_worker(void*) + 1040
    frame #3: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #7
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000118f73ec8 interpreter.cpython-310-darwin.so`th_worker(void*) + 1040
    frame #3: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #8
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000118f73ec8 interpreter.cpython-310-darwin.so`th_worker(void*) + 1040
    frame #3: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #9
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000118f73ec8 interpreter.cpython-310-darwin.so`th_worker(void*) + 1040
    frame #3: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #10
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000118f73ec8 interpreter.cpython-310-darwin.so`th_worker(void*) + 1040
    frame #3: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #11
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000118f73ec8 interpreter.cpython-310-darwin.so`th_worker(void*) + 1040
    frame #3: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #12
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000118f73ec8 interpreter.cpython-310-darwin.so`th_worker(void*) + 1040
    frame #3: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #13
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000118f73ec8 interpreter.cpython-310-darwin.so`th_worker(void*) + 1040
    frame #3: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #14
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000118f73ec8 interpreter.cpython-310-darwin.so`th_worker(void*) + 1040
    frame #3: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #15
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000118f73ec8 interpreter.cpython-310-darwin.so`th_worker(void*) + 1040
    frame #3: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #16
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000118f73ec8 interpreter.cpython-310-darwin.so`th_worker(void*) + 1040
    frame #3: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #17
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #18
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #19
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #20
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #21
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #22
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #23
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #24
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #25
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #26
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #27
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #28
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #29
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #30
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #31
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #32
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #33
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #34
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #35
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #36
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #37
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #38
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #39
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #40
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #41
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #42
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #43
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #44
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #45
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x00000003547e85c8 libjax_common.dylib`absl::lts_20230802::synchronization_internal::PthreadWaiter::Wait(absl::lts_20230802::synchronization_internal::KernelTimeout) + 152
    frame #3: 0x00000003547e84b8 libjax_common.dylib`AbslInternalPerThreadSemWait_lts_20230802 + 76
    frame #4: 0x0000000354e08134 libjax_common.dylib`absl::lts_20230802::CondVar::WaitCommon(absl::lts_20230802::Mutex*, absl::lts_20230802::synchronization_internal::KernelTimeout) + 184
    frame #5: 0x0000000354e082b0 libjax_common.dylib`absl::lts_20230802::CondVar::WaitWithDeadline(absl::lts_20230802::Mutex*, absl::lts_20230802::Time) + 28
    frame #6: 0x000000034fbf5a34 libjax_common.dylib`xla::SlowOperationAlarm::AlarmLoop() + 548
    frame #7: 0x000000034fbf65ac libjax_common.dylib`void absl::lts_20230802::internal_any_invocable::LocalInvoker<false, void, xla::SlowOperationAlarm::ScheduleAlarm(xla::SlowOperationAlarm*)::$_0::operator()() const::'lambda'()&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) + 12
    frame #8: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #9: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #46
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #47
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #48
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #49
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #50
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #51
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #52
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #53
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #54
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #55
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #56
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #57
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #58
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #59
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x0000000350a48fcc libjax_common.dylib`Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 220
    frame #4: 0x00000003546d73fc libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) + 780
    frame #5: 0x00000003546d6a04 libjax_common.dylib`Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) + 296
    frame #6: 0x00000003546d67d8 libjax_common.dylib`std::__1::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>::type std::__1::invoke[abi:nn190102]<tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()&) + 80
    frame #7: 0x00000003546c885c libjax_common.dylib`tsl::(anonymous namespace)::PThread::ThreadFn(void*) + 116
    frame #8: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #60
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x000000017b1d386c libarrow.2300.dylib`void* std::__1::__thread_proxy[abi:ue170006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_8>>(void*) + 756
    frame #4: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #61
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x000000017b1d386c libarrow.2300.dylib`void* std::__1::__thread_proxy[abi:ue170006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_8>>(void*) + 756
    frame #4: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #62
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x000000017b1d386c libarrow.2300.dylib`void* std::__1::__thread_proxy[abi:ue170006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_8>>(void*) + 756
    frame #4: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #63
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x000000017b1d386c libarrow.2300.dylib`void* std::__1::__thread_proxy[abi:ue170006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_8>>(void*) + 756
    frame #4: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
  thread #64
    frame #0: 0x0000000185e424f8 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x0000000185e820dc libsystem_pthread.dylib`_pthread_cond_wait + 984
    frame #2: 0x0000000185db2754 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 32
    frame #3: 0x000000017b1d386c libarrow.2300.dylib`void* std::__1::__thread_proxy[abi:ue170006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_8>>(void*) + 756
    frame #4: 0x0000000185e81c08 libsystem_pthread.dylib`_pthread_start + 136
(lldb) quit

@ianna
Copy link
Copy Markdown
Member

ianna commented Jan 26, 2026

I can get the hang locally on macos even. Here is the backtrace when the process has hanged

Click to show full backtrace

Do you actually have 64 physical or logical cores?

@ikrommyd
Copy link
Copy Markdown
Collaborator

ikrommyd commented Jan 26, 2026 •
edited
Loading

I can get the hang locally on macos even. Here is the backtrace when the process has hanged
Click to show full backtrace

Do you actually have 64 physical or logical cores?

No these are threads spawned by third party libraries like Jax, arrow, root etc. Running the tests was called in a single threaded way “pytest tests” basically.

I didn’t look at it a more, but it looks like some kind of deadlock to me. It may be an upstream bug because the CI caught it in Linux and I can get it on arm64 macOS. In both operating systems it works with 3.9 but we get the hang with 3.10

@ianna
Copy link
Copy Markdown
Member

ianna commented Jan 27, 2026 •
edited
Loading

I can get the hang locally on macos even. Here is the backtrace when the process has hanged
Click to show full backtrace

Do you actually have 64 physical or logical cores?

No these are threads spawned by third party libraries like Jax, arrow, root etc. Running the tests was called in a single threaded way “pytest tests” basically.

I didn’t look at it a more, but it looks like some kind of deadlock to me. It may be an upstream bug because the CI caught it in Linux and I can get it on arm64 macOS. In both operating systems it works with 3.9 but we get the hang with 3.10

Try to limit the number of threads used by ROOT and JAX. Given that controlling the JAX threads may not be possible (jax-ml/jax#22739) IMHO, we should not run ROOT and JAX tests simultaneously.

@ikrommyd
Copy link
Copy Markdown
Collaborator

ikrommyd commented Jan 27, 2026

I found that the error is root related too. With 3.9 you can get root 6.34 max from conda-forge but python 3.10 gets you 6.36. If I manually request 6.34 with python 3.10, it doesn’t hang for me. I also get the same problem with root 6.36 in all python versions till 3.14 even.

@ikrommyd
Copy link
Copy Markdown
Collaborator

ikrommyd commented Jan 27, 2026

jax and pyarrow do not play a role in this. I can hit the problem with both not in the environment.
I'm able to run the tests without hanging if I remove all the ROOT.ROOT.EnableImplicitMT(1) we do in the tests currently.
This seems to be an issue with ROOT's multithreading or how we use it.

@ianna
Copy link
Copy Markdown
Member

ianna commented Jan 27, 2026

jax and pyarrow do not play a role in this. I can hit the problem with both not in the environment. I'm able to run the tests without hanging if I remove all the ROOT.ROOT.EnableImplicitMT(1) we do in the tests currently. This seems to be an issue with ROOT's multithreading or how we use it.

it has been discussed in here.

@ikrommyd
Copy link
Copy Markdown
Collaborator

ikrommyd commented Jan 27, 2026 •
edited
Loading

jax and pyarrow do not play a role in this. I can hit the problem with both not in the environment. I'm able to run the tests without hanging if I remove all the ROOT.ROOT.EnableImplicitMT(1) we do in the tests currently. This seems to be an issue with ROOT's multithreading or how we use it.

it has been discussed in here.

You are pointing to the named axes thread safety discussion? This has nothing to do with it. The problem is reproducible with awkward versions before that PR and also the tests do not run in multi-threaded mode in the "Linux-ROOT" tests. I get the exact same hang in awkward 2.7.X. It's just that we have never tested with ROOT 6.36 until this point.

@ianna
Copy link
Copy Markdown
Member

ianna commented Jan 27, 2026

jax and pyarrow do not play a role in this. I can hit the problem with both not in the environment. I'm able to run the tests without hanging if I remove all the ROOT.ROOT.EnableImplicitMT(1) we do in the tests currently. This seems to be an issue with ROOT's multithreading or how we use it.

it has been discussed in here.

You are pointing to the named axes thread safety discussion? This has nothing to do with it. The problem is reproducible with awkward versions before that PR and also the tests do not run in multi-threaded mode in the "Linux-ROOT" tests. I get the exact same hang in awkward 2.7.X. It's just that we have never tested with ROOT 6.36 until this point.

Ditto -- the discussion is relevant in the context of native threads and mixed thread contexts, not the specific code.

@ikrommyd
Copy link
Copy Markdown
Collaborator

ikrommyd commented Jan 27, 2026 •
edited
Loading

Btw, all the ROOT tests do

ROOT = pytest.importorskip("ROOT")

ROOT.ROOT.EnableImplicitMT(1)

at module level. Meaning this will when pytest does the imports. Things have been magically running so far. pytest tests/test_1473_from_rdataframe.py for example just hangs for me.

@ikrommyd
Copy link
Copy Markdown
Collaborator

ikrommyd commented Jan 27, 2026

Ditto -- the discussion is relevant in the context of native threads and mixed thread contexts, not the specific code.

I see, yeah indeed although in this case we run the tests from one python thread with the GIL enabled and an external library (ROOT) spawns native threads. Other libraries like jax and torch to that too but we haven't had an issue so far.

@ianna
Copy link
Copy Markdown
Member

ianna commented Jan 27, 2026

Ditto -- the discussion is relevant in the context of native threads and mixed thread contexts, not the specific code.

I see, yeah indeed although in this case we run the tests from one python thread with the GIL enabled and an external library (ROOT) spawns native threads. Other libraries like jax and torch to that too but we haven't had an issue so far.

Only by luck

@ikrommyd
Copy link
Copy Markdown
Collaborator

ikrommyd commented Jan 27, 2026 •
edited
Loading

I went through all of the tests that import ROOT one by one. I first ran them individually one by one and I have removed ROOT.ROOT.EnableImplicitMT(1) from the ones that hang.
Only 3 tests remained that didn't hang with ROOT.ROOT.EnableImplicitMT(1). In those tests, we were still doing ROOT.ROOT.EnableImplicitMT(1) at the module level, meaning it would actually be enabled for all tests when we run them together. I have moved this into a module-level fixture.

@pytest.fixture(scope="module", autouse=True)
def manage_imt():
    ROOT.ROOT.EnableImplicitMT(1)
    yield
    ROOT.ROOT.DisableImplicitMT()

Now all the tests should pass together and individually. This is the resulting commit a3cb1a3.
It works for me locally but let's see what CI has to say too.

@ikrommyd
Copy link
Copy Markdown
Collaborator

ikrommyd commented Jan 27, 2026

Yay, the Linux-ROOT CI is happy. I have also tested this locally with 3.10, 3.13 and 3.14 and the tests (including ROOT) pass. This should be ready now right @henryiii @ianna ?

@ianna
Copy link
Copy Markdown
Member

ianna commented Jan 27, 2026

Yay, the Linux-ROOT CI is happy. I have also tested this locally with 3.10, 3.13 and 3.14 and the tests (including ROOT) pass. This should be ready now right @henryiii @ianna ?

Looks good to me. Thanks.

@henryiii henryiii marked this pull request as ready for review January 27, 2026 14:42
ikrommyd
ikrommyd reviewed Jan 27, 2026
View reviewed changes
Copy link
Copy Markdown
Collaborator

@ikrommyd ikrommyd left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awkward/README.md

Line 7 in 680ab6e

[![Python 3.9‒3.14](https://img.shields.io/badge/python-3.9%E2%80%923.14-blue)](https://www.python.org)

The only thing I see is this last mention of 3.9 in the README badge. I got this one. I don't think there's any other place where something was forgotten.

ikrommyd
ikrommyd approved these changes Jan 27, 2026
View reviewed changes
Copy link
Copy Markdown
Collaborator

@ikrommyd ikrommyd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I searched for 3.9/py39 patterns and do not see any outstanding places that need changing.

ianna
ianna approved these changes Jan 27, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@ianna ianna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@henryiii - This is great! Thank you! I'll enable auto-merge. Thanks.

@ianna ianna enabled auto-merge (squash) January 27, 2026 15:00
@ikrommyd ikrommyd mentioned this pull request Jan 27, 2026
Merged
@ikrommyd
Copy link
Copy Markdown
Collaborator

ikrommyd commented Jan 27, 2026

We probably need to do something (not here of course) about the annoying docs failure that happens very often

ClientResponseError: 503, message='Service Unavailable', url='https://zenodo.org/records/14537442/files/chicago-taxi.parquet'

This error occurred while calling

    ak.metadata_from_parquet(
        'https://zenodo.org/records/14537442/files/chicago-taxi.pa'...
    )

@ianna ianna merged commit 53744b2 into main Jan 27, 2026
45 of 46 checks passed
@ianna ianna deleted the henryiii/feat/drop39 branch January 27, 2026 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ianna ianna ianna approved these changes

@ikrommyd ikrommyd ikrommyd approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@henryiii @ikrommyd @ianna