Skip to content

Track total processed bytes and total transferred bytes#328

Merged
hoytak merged 2 commits intomainfrom
hoytak/250514-tracking-process-and-transfer
May 19, 2025
Merged

Track total processed bytes and total transferred bytes#328
hoytak merged 2 commits intomainfrom
hoytak/250514-tracking-process-and-transfer

Conversation

@hoytak
Copy link
Collaborator

@hoytak hoytak commented May 15, 2025

With deduplication, just tracking the total processed bytes can give a false impression of actual progress when uploading files; the user would often see a huge jump when hitting a deduplicated part. This PR allows us to report bytes uploaded or downloaded separately from the bytes processed, which would allow us to correctly surface network utilization.

For example, one possible way to show this would be:

Data Processed      #####################╺━━━━━━━━━━━━━━━━  58% • 1.8/3.1 GB     • 731.6 MB/s • 0:00:10
New Data Uploaded   ######╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   9% • 65.9/731.7 MB  • 131.0 MB/s         • -:--:--
data_up/d2.dat      #############################╸━━━━━━━━  67% • 67.1/100.0 MB  • 568.1 MB/s • 0:00:01

@hoytak hoytak requested review from assafvayner and seanses May 15, 2025 17:13
@hoytak hoytak force-pushed the hoytak/250514-tracking-process-and-transfer branch from 84e5b36 to 2e888c6 Compare May 15, 2025 17:18
assafvayner added a commit that referenced this pull request May 15, 2025
fix XET-567

This PR adds the retry mechanism on downloading a range from CF/S3 for
when an error occurs reading or parsing the body.
The upload is handled by reverting a change to reintroduce retries on a
clonable body PR #328.
@hoytak hoytak force-pushed the hoytak/250514-tracking-process-and-transfer branch from 381e82a to 30f141d Compare May 16, 2025 02:58
@hoytak hoytak merged commit bef6a9a into main May 19, 2025
4 checks passed
@hoytak hoytak deleted the hoytak/250514-tracking-process-and-transfer branch May 19, 2025 18:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants