Skip to content

feat: wandb beta sync --live#11079

Merged
timoffex merged 1 commit intomainfrom
timoffex/12-08-runsync_live
Dec 31, 2025
Merged

feat: wandb beta sync --live#11079
timoffex merged 1 commit intomainfrom
timoffex/12-08-runsync_live

Conversation

@timoffex
Copy link
Copy Markdown
Contributor

@timoffex timoffex commented Dec 16, 2025 •
edited by dmitryduev
Loading

Fixes WB-27693 and #2887.

Adds the --live option to wandb beta sync to allow syncing a run while it is being logged.

This only uploads data as it appears in the run's .wandb file, which has some consequences:

  • There will be some delay due to polling
  • The .wandb file is flushed once every 32 KiB, so the sync may lag behind by an arbitrary amount of time for runs that don't log a lot of data
  • Nothing is uploaded at all until the first 32 KiB is written to disk

This behavior can be improved by flushing more frequently.

For now, this should be considered an interactive mode: it may hang if the run stops updating its transaction log before it is done, requiring manually killing the process with Ctrl+C. This can be improved by having all runs create and update a heartbeat file to allow crash detection.

@timoffex Graphite App
Copy link
Copy Markdown
Contributor Author

timoffex commented Dec 16, 2025 •
edited
Loading

This stack of pull requests is managed by Graphite. Learn more about stacking.

This was referenced Dec 16, 2025
Merged
Merged
Merged
@codecov
Copy link
Copy Markdown

codecov Bot commented Dec 16, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 93.33333% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
core/internal/runsync/runreader.go 88.23% 1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@timoffex timoffex force-pushed the timoffex/12-08-runsync_live branch from 6dad490 to 64d0075 Compare December 17, 2025 00:51
@timoffex timoffex force-pushed the timoffex/12-08-transactionlog_retry branch from 4d0801b to d43372e Compare December 17, 2025 00:51
@timoffex timoffex force-pushed the timoffex/12-08-runsync_live branch from 64d0075 to e11cc68 Compare December 17, 2025 00:52
@timoffex timoffex force-pushed the timoffex/12-08-transactionlog_retry branch from d43372e to 1c8dfc1 Compare December 17, 2025 00:52
@timoffex timoffex force-pushed the timoffex/12-08-runsync_live branch from e11cc68 to 2d1f334 Compare December 17, 2025 01:01
@timoffex timoffex force-pushed the timoffex/12-08-transactionlog_retry branch from 1c8dfc1 to 101dc65 Compare December 17, 2025 01:01
@timoffex timoffex changed the base branch from timoffex/12-08-transactionlog_retry to graphite-base/11079 December 19, 2025 02:23
This was referenced Dec 19, 2025
Closed
Merged
@timoffex timoffex changed the title runsync live feat: wandb beta sync --live Dec 22, 2025
@timoffex timoffex force-pushed the timoffex/12-08-runsync_live branch from 2d1f334 to b1d7892 Compare December 22, 2025 22:53
@timoffex timoffex force-pushed the graphite-base/11079 branch from 101dc65 to a45bba6 Compare December 22, 2025 22:53
@timoffex timoffex changed the base branch from graphite-base/11079 to timoffex/12-18-leet_transactionlog December 22, 2025 22:53
@timoffex timoffex marked this pull request as ready for review December 22, 2025 23:07
@timoffex timoffex requested a review from a team as a code owner December 22, 2025 23:07
@timoffex timoffex force-pushed the timoffex/12-08-runsync_live branch from b1d7892 to 33e69dc Compare December 22, 2025 23:07
@timoffex timoffex force-pushed the timoffex/12-18-leet_transactionlog branch from a45bba6 to ee72016 Compare December 23, 2025 01:28
@timoffex timoffex force-pushed the timoffex/12-08-runsync_live branch from 33e69dc to ecfd29f Compare December 23, 2025 01:28
@timoffex timoffex force-pushed the timoffex/12-18-leet_transactionlog branch 2 times, most recently from 62a641b to 5247dc4 Compare December 23, 2025 21:16
@timoffex timoffex force-pushed the timoffex/12-08-runsync_live branch from ecfd29f to 0ae4be1 Compare December 23, 2025 21:16
@timoffex timoffex force-pushed the timoffex/12-18-leet_transactionlog branch 2 times, most recently from 28adc90 to 7e4072b Compare December 23, 2025 21:31
@timoffex timoffex force-pushed the timoffex/12-08-runsync_live branch 2 times, most recently from 1d9c89b to 20a829c Compare December 23, 2025 22:49
@timoffex timoffex changed the base branch from graphite-base/11079 to timoffex/12-18-leet_transactionlog December 29, 2025 20:26
@timoffex timoffex mentioned this pull request Dec 29, 2025
Merged
@timoffex timoffex changed the base branch from timoffex/12-18-leet_transactionlog to graphite-base/11079 December 29, 2025 20:47
@timoffex timoffex force-pushed the graphite-base/11079 branch from 5623e85 to e60e6b2 Compare December 29, 2025 23:30
@timoffex timoffex force-pushed the timoffex/12-08-runsync_live branch from 8094f62 to bb30aa2 Compare December 29, 2025 23:30
@timoffex timoffex changed the base branch from graphite-base/11079 to timoffex/12-18-leet_transactionlog December 29, 2025 23:30
@timoffex timoffex force-pushed the timoffex/12-18-leet_transactionlog branch from e60e6b2 to 8254816 Compare December 29, 2025 23:49
@timoffex timoffex requested a review from a team as a code owner December 29, 2025 23:49
@timoffex timoffex force-pushed the timoffex/12-18-leet_transactionlog branch from 8254816 to 635934f Compare December 29, 2025 23:49
@timoffex timoffex force-pushed the timoffex/12-08-runsync_live branch from bb30aa2 to e855a3e Compare December 29, 2025 23:50
jacobromero
jacobromero approved these changes Dec 30, 2025
View reviewed changes
Comment thread core/internal/runsync/runreader.go Outdated
@timoffex timoffex force-pushed the timoffex/12-18-leet_transactionlog branch from 635934f to c4b45fd Compare December 30, 2025 21:50
@timoffex timoffex force-pushed the timoffex/12-08-runsync_live branch from e855a3e to a61b235 Compare December 30, 2025 21:50
@timoffex timoffex mentioned this pull request Dec 30, 2025
Merged
@timoffex timoffex force-pushed the timoffex/12-08-runsync_live branch from a61b235 to c88d631 Compare December 31, 2025 00:06
@timoffex timoffex force-pushed the timoffex/12-18-leet_transactionlog branch 3 times, most recently from 22e7a4f to 52cd2d9 Compare December 31, 2025 00:22
@timoffex timoffex force-pushed the timoffex/12-08-runsync_live branch from c88d631 to d62137f Compare December 31, 2025 00:22
@timoffex timoffex changed the base branch from timoffex/12-18-leet_transactionlog to graphite-base/11079 December 31, 2025 00:36
@timoffex timoffex force-pushed the timoffex/12-08-runsync_live branch from d62137f to 30e4b54 Compare December 31, 2025 00:49
@timoffex timoffex force-pushed the graphite-base/11079 branch from 52cd2d9 to 89e2351 Compare December 31, 2025 00:49
@graphite-app graphite-app Bot changed the base branch from graphite-base/11079 to main December 31, 2025 00:50
@timoffex timoffex force-pushed the timoffex/12-08-runsync_live branch from 30e4b54 to d232bdf Compare December 31, 2025 00:50
@timoffex timoffex merged commit 47df405 into main Dec 31, 2025
33 of 34 checks passed
@timoffex Graphite App
Copy link
Copy Markdown
Contributor Author

timoffex commented Dec 31, 2025

Merge activity

@timoffex timoffex deleted the timoffex/12-08-runsync_live branch December 31, 2025 01:02
@timoffex timoffex mentioned this pull request Dec 31, 2025
Merged
@dmitryduev dmitryduev mentioned this pull request Feb 2, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jacobromero jacobromero jacobromero approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@timoffex @jacobromero