Skip to content

Implement an interface for Distributed-like libraries#871

Merged
timholy merged 3 commits intotimholy:masterfrom
JamesWrigley:distributed-extension
Jan 2, 2025
Merged

Implement an interface for Distributed-like libraries#871
timholy merged 3 commits intotimholy:masterfrom
JamesWrigley:distributed-extension

Conversation

@JamesWrigley
Copy link
Copy Markdown
Contributor

@JamesWrigley JamesWrigley commented Dec 10, 2024

This moves support for Distributed into a package extension and adds an interface for other distributed worker libraries (i.e. DistributedNext.jl) to use to support Revise for their workers. Admittedly the interface is quite... clunky... so I'm open to other ideas. This also required bumping the minimum Julia version to 1.9 to use package extensions.

As a bonus the load times are slightly improved:

# v3.6.4
julia> @time_imports using Revise
      2.7 ms  OrderedCollections
      0.2 ms  UUIDs
      3.8 ms  CodeTracking
               ┌ 0.2 ms JuliaInterpreter.__init__() 
    136.9 ms  JuliaInterpreter 72.95% compilation time
     35.6 ms  LoweredCodeUtils
      0.7 ms  Serialization
               ┌ 0.0 ms Distributed.__init__() 
     31.7 ms  Distributed 71.09% compilation time
               ┌ 0.0 ms NetworkOptions.__init__() 
      2.2 ms  NetworkOptions
      0.4 ms  Printf
               ┌ 0.2 ms MbedTLS_jll.__init__() 
      2.4 ms  MbedTLS_jll
               ┌ 0.1 ms LibSSH2_jll.__init__() 
      2.1 ms  LibSSH2_jll
               ┌ 0.1 ms LibGit2_jll.__init__() 
      2.0 ms  LibGit2_jll
      7.5 ms  LibGit2
               ┌ 0.1 ms Revise.__init__() 
     79.2 ms  Revise

# This PR
julia> @time_imports using Revise
      3.3 ms  OrderedCollections
      3.4 ms  CodeTracking
               ┌ 0.2 ms JuliaInterpreter.__init__() 
    145.1 ms  JuliaInterpreter 73.80% compilation time
     36.3 ms  LoweredCodeUtils
               ┌ 0.1 ms Revise.__init__() 
     80.9 ms  Revise

@JamesWrigley JamesWrigley mentioned this pull request Dec 10, 2024
Merged
@JamesWrigley JamesWrigley force-pushed the distributed-extension branch 2 times, most recently from a0bc479 to 6d879e4 Compare December 10, 2024 23:41
timholy
timholy reviewed Dec 28, 2024
View reviewed changes
Copy link
Copy Markdown
Owner

@timholy timholy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm strongly supportive of anything that reduces dependency on Distributed, especially if the replacements are less vulnerable to invalidations. The overall design of this PR seems OK.

@JamesWrigley
Bump minumum Julia version to 1.10
67ee77c 
Mostly so that we can use package extensions.
@JamesWrigley
Implement an interface for Distributed-like libraries
5955e0e 
This moves support for Distributed into a package extension and adds an
interface for other distributed worker libraries to use to support Revise for
their workers.
@timholy timholy merged commit c83fa3d into timholy:master Jan 2, 2025
@timholy
Copy link
Copy Markdown
Owner

timholy commented Jan 2, 2025

Thanks! Coming out momentarily in Revise 3.7.0.

@JamesWrigley JamesWrigley deleted the distributed-extension branch January 2, 2025 18:25
@affans
Copy link
Copy Markdown

affans commented Jan 2, 2025

How will this work? Right now, I basically do

using Revise
includet("main.jl")

with main.jl as follows

using Distributed 
addprocs(2, exeflags="--project=.") # for local usage
# if running Revise, all worker processes will be updated with new code
@everywhere using Revise
@everywhere includet("model.jl")

and now if I update model.jl, the code is updated on all workers.

Does this workflow need to be changed?

@JamesWrigley
Copy link
Copy Markdown
Contributor Author

JamesWrigley commented Jan 2, 2025

No, the behaviour should be identical to previous versions. The only change in the PR is to make loading Distributed optional and allow support for other libraries to hook into Revise for their workers.

@timholy
Copy link
Copy Markdown
Owner

timholy commented Jan 2, 2025

@affans , Distributed badly needs re-engineering. This should make it easier to update or replace it.

@dpinol dpinol mentioned this pull request Jan 10, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@timholy timholy timholy approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@JamesWrigley @timholy @affans