I noticed this error when running some code with significantly higher number of thunks.
So at a lower number of thunks used this code runs ok (~1000 thunks), but at ~10000 thunks this error pops up every time (but it finishes successfuly anyway and provides a good result).
I can share the code later and prepare an example.
But I would like to know first where this is a Dagger issue or Distributed issue?
error in running finalizer: ErrorException("task switch not allowed from inside gc finalizer")
jl_error at /opt/julia/src/rtutils.c:41
jl_switch at /opt/julia/src/task.c:501
try_yieldto at ./task.jl:700
wait at ./task.jl:769 [inlined]
wait at ./condition.jl:106
lock at ./lock.jl:100
lock at ./lock.jl:185
lock at ./weakkeydict.jl:87 [inlined]
delete! at ./weakkeydict.jl:166 [inlined]
finalize_ref at /opt/julia/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:92
jl_apply at /opt/julia/src/julia.h:1703 [inlined]
run_finalizer at /opt/julia/src/gc.c:278
jl_gc_run_finalizers_in_list at /opt/julia/src/gc.c:365
run_finalizers at /opt/julia/src/gc.c:394 [inlined]
run_finalizers at /opt/julia/src/gc.c:372
jl_gc_collect at /opt/julia/src/gc.c:3260
maybe_collect at /opt/julia/src/gc.c:880 [inlined]
jl_gc_pool_alloc at /opt/julia/src/gc.c:1204
jl_gc_alloc_ at /opt/julia/src/julia_internal.h:285 [inlined]
jl_gc_alloc at /opt/julia/src/gc.c:3283
_new_array_ at /opt/julia/src/array.c:132
jl_array_copy at /opt/julia/src/array.c:1206
copy at ./array.jl:349 [inlined]
copy at /home/krynju/.julia/packages/Arrow/viADU/src/arraytypes/primitive.jl:37
unknown function (ip: 0x7f95921a8181)
getindex at /home/krynju/.julia/packages/DataFrames/vQokV/src/dataframe/dataframe.jl:500
#407 at /home/krynju/.julia/dev/Dagger/src/table/operations.jl:108
macro expansion at /home/krynju/.julia/dev/Dagger/src/processor.jl:154 [inlined]
#51 at ./threadingconstructs.jl:169
unknown function (ip: 0x7f959007a4fc)
jl_apply at /opt/julia/src/julia.h:1703 [inlined]
start_task at /opt/julia/src/task.c:839
unknown function (ip: (nil))
I noticed this error when running some code with significantly higher number of thunks.
So at a lower number of thunks used this code runs ok (~1000 thunks), but at ~10000 thunks this error pops up every time (but it finishes successfuly anyway and provides a good result).
I can share the code later and prepare an example.
But I would like to know first where this is a Dagger issue or Distributed issue?