make extension not load twice on workers#49441
Conversation
|
The alternative would look something like function run_extension_callbacks(pkgid::PkgId)
+ distributed_id = PkgId(UUID((0x8ba89e20_285c_5b6f, 0x9357_94700520ee1b)), "
Distributed")
+ Distributed = get(loaded_modules, distributed_id, nothing)
+ if Distributed !== nothing && isdefined(Distributed, :myid)
+ id = @invokelatest Distributed.myid()
+ # Do not trigger extensions to load on workers, they will be loaded by the callback Distributed inserts
+ id != 1 && return
+ end |
|
Thanks for this quick fix. The solution in the PR seems right to me because at least based on your description / my basic understanding (ie I havent actually tested), it will handle this correctly: using Foo, Distributed
addprocs(2)
using Barwhich, based on current rules, should load Foo, Bar, and BarFooExt on master, and only Bar on workers. Conversely, letting master trigger the worker-loads I don't know if it could really handle that easily. |
vtjnash
left a comment
There was a problem hiding this comment.
Shouldn't Base.toplevel_load[] be false already, since the extension is not the toplevel load?
I don't know the exact definition of that but adding @show Base.toplevel_load[]
@show modto julia/stdlib/Distributed/src/Distributed.jl Lines 74 to 85 in f84fb5b shows: |
|
I think an extension loading like this is arguably a top level load, no; it doesn't get loaded into any other package. The "extension callback" here gets run after we have loaded all the triggers and at that point we are at toplevel again calling |
|
I'll go with this for now, if someone has a cleaner solution it can always be tweaked. |
(cherry picked from commit b1c0eac)
Fixes #49437
The issue is the following:
An alternative to this PR could be to never trigger the extension loading on workers and always let the callback handle it but I don't know how to identify a worker (
myid) in Base.