[train] Cleanups for training ingest benchmark#53684
Merged
justinvyu merged 26 commits intoray-project:masterfrom Jun 19, 2025
Merged
[train] Cleanups for training ingest benchmark#53684justinvyu merged 26 commits intoray-project:masterfrom
justinvyu merged 26 commits intoray-project:masterfrom
Conversation
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
This reverts commit fd8e8a5. Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR refactors the training ingest benchmark by introducing task-level configurations, consolidating and deduplicating image classification factories, and reorganizing where dataloader settings live.
- Add
TaskConfigandImageClassificationConfigto centralize per-task settings and remove per-variant tasks. - Move batch‐size and row‐limit fields into
DataLoaderConfigsubclasses and update factories to callget_dataloader_config(). - Deduplicate image‐classification factories (JPEG/Parquet) under a single
ImageClassificationFactoryusing injecteddata_dirs.
Reviewed Changes
Copilot reviewed 22 out of 22 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| release/train_tests/benchmark/runner.py | Pass dataset_creation_time into get_metrics and update imports |
| release/train_tests/benchmark/recsys/recsys_factory.py | Fix import path for BenchmarkFactory |
| release/train_tests/benchmark/ray_dataloader_factory.py | Add abstract get_ray_datasets and get_ray_data_config |
| release/train_tests/benchmark/image_classification/parquet/factory.py | Inject data_dirs and use get_dataloader_config().limit_* |
| release/train_tests/benchmark/image_classification/localfs_image_classification_jpeg/factory.py | Remove obsolete localfs‐JPEG factory |
| release/train_tests/benchmark/image_classification/localfs_image_classification_jpeg/init.py | Remove empty module docstring |
| release/train_tests/benchmark/image_classification/jpeg/factory.py | Inject data_dirs, remove hardcoded dirs, and use limits |
| release/train_tests/benchmark/image_classification/imagenet.py | Add IMAGENET_LOCALFS_SPLIT_DIRS and import DatasetKey |
| release/train_tests/benchmark/image_classification/factory.py | New unified ImageClassificationFactory and helper get_imagenet_data_dirs |
| release/train_tests/benchmark/dataloader_factory.py | Remove unused stub methods |
| release/train_tests/benchmark/config.py | Add TaskConfig types and move row‐limit fields to DataLoaderConfig |
| release/train_tests/benchmark/benchmark_factory.py | Remove deprecated dataset methods |
| release/release_tests.yaml | Update test scripts for new task and flag names |
…hmark_minimal_cleanup
srinathk10
approved these changes
Jun 18, 2025
Contributor
srinathk10
left a comment
There was a problem hiding this comment.
Nice. Thanks for the restructure.
justinvyu
commented
Jun 18, 2025
| datasets = {} | ||
| data_config = None | ||
|
|
||
| factory.set_dataset_creation_time(time.perf_counter() - start_time) |
Contributor
Author
There was a problem hiding this comment.
I think the dataset creation time previously did not capture the actual Ray Dataset construction (in get_ray_datasets).
I updated it to capture the range. Just want to double check that this is accurate.
Contributor
There was a problem hiding this comment.
Ah ok. Good catch! Hope that get_ray_datasets call is negligible (sub-second).
srinathk10
added a commit
that referenced
this pull request
Jun 20, 2025
This reverts commit 1957ce2.
minerharry
pushed a commit
to minerharry/ray
that referenced
this pull request
Jun 27, 2025
This PR does some cleanup for the training benchmark: * Introduces task level configs so that we don't need to create a new task per variant of the image classification task. * Moves some configuration setting to logical places (ex: grouping all Ray Data configs in one place). * Deduplicates some of the redundant "benchmark factories" that were created for the image classification data format / data storage variants. * Misc. file/directory renames for conciseness. --------- Signed-off-by: Justin Yu <justinvyu@anyscale.com>
elliot-barn
pushed a commit
that referenced
this pull request
Jul 2, 2025
This PR does some cleanup for the training benchmark: * Introduces task level configs so that we don't need to create a new task per variant of the image classification task. * Moves some configuration setting to logical places (ex: grouping all Ray Data configs in one place). * Deduplicates some of the redundant "benchmark factories" that were created for the image classification data format / data storage variants. * Misc. file/directory renames for conciseness. --------- Signed-off-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR does some cleanup for the training benchmark: