Skip to content

Make data collection aggregated throughout#362

Merged
harshavardhana merged 14 commits intominio:masterfrom
klauspost:live-aggregate
Jan 30, 2025
Merged

Make data collection aggregated throughout#362
harshavardhana merged 14 commits intominio:masterfrom
klauspost:live-aggregate

Conversation

@klauspost
Copy link
Collaborator

@klauspost klauspost commented Jan 30, 2025

This PR will change all data collection to be automatically aggregated as requests happen.

This means individual requests will no longer be recorded unless -full is specified.

This will allow warp to run for days without consuming significant amount of memory. The only downside is that there are no full-run median request values, but they are instead grouped and averaged.

  • Replaces output display with nicer one.
  • Displays live aggregated values while running.
  • Allows stopping running benchmarks and save result.
  • Displays remote data live.
  • Allows --autoterm for remote running benchmarks as well.
  • -analyze.v does not do anything (all details shown by default)
  • Adds (hidden) --stdout to print operations to stdout.
  • New output format is a zstd compressed JSON file.

Obviously a significant rewrite, so there will likely be uncovered corner cases. But testing checks out fine now.

Pending:

  • Update documentation.
  • Update YAML files.
  • Update warp cmp. Not strictly needed for release.
  • Update warp merge. Rarely used.

# Conflicts:
#	cli/benchclient.go
#	cli/flags.go
#	go.mod
#	go.sum
#	pkg/bench/benchmark.go
#	pkg/bench/delete.go
#	pkg/bench/get.go
#	pkg/bench/list.go
#	pkg/bench/mixed.go
#	pkg/bench/multipart.go
#	pkg/bench/ops.go
#	pkg/bench/retention.go
#	pkg/bench/s3zip.go
#	pkg/bench/select.go
#	pkg/bench/stat.go
#	pkg/bench/versioned.go
@harshavardhana
Copy link
Member

Can you share the https://asciinema.org/ of the new UI?

@klauspost
Copy link
Collaborator Author

@harshavardhana If you port it to Windows ;)

Let's see if this works: https://github.com/user-attachments/assets/c68b74d1-1748-4e1b-89f4-bbb455c4baff

@harshavardhana
Copy link
Member

harshavardhana commented Jan 30, 2025

@harshavardhana If you port it to Windows ;)

Let's see if this works: https://github.com/user-attachments/assets/c68b74d1-1748-4e1b-89f4-bbb455c4baff

why do you have a more verbose output for default results when the test is over? Can we have the same output as it is today, which mainly talks about the higher-level points?

@klauspost
Copy link
Collaborator Author

@harshavardhana Just seemed unnecessary, but I can hook it up. Most infrastructure needed should be there.

(except on `warp analyze`, where it is always enabled)
@klauspost
Copy link
Collaborator Author

klauspost commented Jan 30, 2025

@harshavardhana Brought back the simple view - thinking of it - it was only on warp analyze it annoyed me, so I kept the "always full detail" there.

Report: Total. Concurrency: 20. Ran: 38s
 * Average: 2.96 MiB/s, 516.66 obj/s

Throughput by host:
 * http://127.0.0.1:9001: Avg: 0.75 MiB/s, 129.50 obj/s
 * http://127.0.0.1:9002: Avg: 0.74 MiB/s, 129.93 obj/s
 * http://127.0.0.1:9003: Avg: 0.75 MiB/s, 131.86 obj/s
 * http://127.0.0.1:9004: Avg: 0.71 MiB/s, 124.94 obj/s

Throughput, split into 38 x 1s:
 * Fastest: 3.5MiB/s, 611.64 obj/s
 * 50% Median: 3.0MiB/s, 518.73 obj/s
 * Slowest: 2047.2KiB/s, 349.55 obj/s

@harshavardhana harshavardhana merged commit 483f844 into minio:master Jan 30, 2025
6 checks passed
@klauspost klauspost deleted the live-aggregate branch January 31, 2025 10:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants