<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

This change is a follow-up for
#56105.

Now dataset size estimation is based on listed file sizes. However,
encoding ratio was still based on the file size estimates derived from
the uncompressed data size obtained from Parquet metadata.

This change is addressing that by:

- Rebasing encoding ratio to relate estimated in-memory size to the
listed file size
 - Cleaning up unused abstractions (like `ParquetMetadataProvider`)

## Related issue number

<!-- For example: "Closes #1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

…-project#56105)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

Historically, `ParquetDatasource` have been fetching individual files
parquet footer metadata to obtain granular metadata about the dataset.

While laudable in principle, it's really inefficient in practice and
manifests itself in extremely poor performance on very large datasets
(10s to 100s Ks of files).

This change revisits this approach by

- Removing metadata fetching as a step (and deprecating involved
components)
- Cleaning up and streamlining some of the other utilities further
optimizing performance

<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
Signed-off-by: sampan <sampan@anyscale.com>

…ct#56268)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

This change is a follow-up for
ray-project#56105.

Now dataset size estimation is based on listed file sizes. However,
encoding ratio was still based on the file size estimates derived from
the uncompressed data size obtained from Parquet metadata.

This change is addressing that by:

- Rebasing encoding ratio to relate estimated in-memory size to the
listed file size
 - Cleaning up unused abstractions (like `ParquetMetadataProvider`)

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
Signed-off-by: sampan <sampan@anyscale.com>

…-project#56105)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

Historically, `ParquetDatasource` have been fetching individual files
parquet footer metadata to obtain granular metadata about the dataset.

While laudable in principle, it's really inefficient in practice and
manifests itself in extremely poor performance on very large datasets
(10s to 100s Ks of files).

This change revisits this approach by

- Removing metadata fetching as a step (and deprecating involved
components)
- Cleaning up and streamlining some of the other utilities further
optimizing performance

<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
Signed-off-by: jugalshah291 <shah.jugal291@gmail.com>

…ct#56268)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

This change is a follow-up for
ray-project#56105.

Now dataset size estimation is based on listed file sizes. However,
encoding ratio was still based on the file size estimates derived from
the uncompressed data size obtained from Parquet metadata.

This change is addressing that by:

- Rebasing encoding ratio to relate estimated in-memory size to the
listed file size
 - Cleaning up unused abstractions (like `ParquetMetadataProvider`)

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
Signed-off-by: jugalshah291 <shah.jugal291@gmail.com>

…-project#56105)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

Historically, `ParquetDatasource` have been fetching individual files
parquet footer metadata to obtain granular metadata about the dataset.

While laudable in principle, it's really inefficient in practice and
manifests itself in extremely poor performance on very large datasets
(10s to 100s Ks of files).

This change revisits this approach by

- Removing metadata fetching as a step (and deprecating involved
components)
- Cleaning up and streamlining some of the other utilities further
optimizing performance

<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
Signed-off-by: yenhong.wong <yenhong.wong@grabtaxi.com>

…ct#56268)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

This change is a follow-up for
ray-project#56105.

Now dataset size estimation is based on listed file sizes. However,
encoding ratio was still based on the file size estimates derived from
the uncompressed data size obtained from Parquet metadata.

This change is addressing that by:

- Rebasing encoding ratio to relate estimated in-memory size to the
listed file size
 - Cleaning up unused abstractions (like `ParquetMetadataProvider`)

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
Signed-off-by: yenhong.wong <yenhong.wong@grabtaxi.com>

)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

Historically, `ParquetDatasource` have been fetching individual files
parquet footer metadata to obtain granular metadata about the dataset.

While laudable in principle, it's really inefficient in practice and
manifests itself in extremely poor performance on very large datasets
(10s to 100s Ks of files).

This change revisits this approach by 

- Removing metadata fetching as a step (and deprecating involved
components)
- Cleaning up and streamlining some of the other utilities further
optimizing performance


<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number

<!-- For example: "Closes #1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

This change is a follow-up for
#56105.

Now dataset size estimation is based on listed file sizes. However,
encoding ratio was still based on the file size estimates derived from
the uncompressed data size obtained from Parquet metadata.

This change is addressing that by:

- Rebasing encoding ratio to relate estimated in-memory size to the
listed file size
 - Cleaning up unused abstractions (like `ParquetMetadataProvider`)

## Related issue number

<!-- For example: "Closes #1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

…ct#56268)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

This change is a follow-up for
ray-project#56105.

Now dataset size estimation is based on listed file sizes. However,
encoding ratio was still based on the file size estimates derived from
the uncompressed data size obtained from Parquet metadata.

This change is addressing that by:

- Rebasing encoding ratio to relate estimated in-memory size to the
listed file size
 - Cleaning up unused abstractions (like `ParquetMetadataProvider`)

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
Signed-off-by: zac <zac@anyscale.com>

)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

Historically, `ParquetDatasource` have been fetching individual files
parquet footer metadata to obtain granular metadata about the dataset.

While laudable in principle, it's really inefficient in practice and
manifests itself in extremely poor performance on very large datasets
(10s to 100s Ks of files).

This change revisits this approach by

- Removing metadata fetching as a step (and deprecating involved
components)
- Cleaning up and streamlining some of the other utilities further
optimizing performance

<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number

<!-- For example: "Closes #1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
Signed-off-by: Douglas Strodtman <douglas@anyscale.com>

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

This change is a follow-up for
#56105.

Now dataset size estimation is based on listed file sizes. However,
encoding ratio was still based on the file size estimates derived from
the uncompressed data size obtained from Parquet metadata.

This change is addressing that by:

- Rebasing encoding ratio to relate estimated in-memory size to the
listed file size
 - Cleaning up unused abstractions (like `ParquetMetadataProvider`)

## Related issue number

<!-- For example: "Closes #1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
Signed-off-by: Douglas Strodtman <douglas@anyscale.com>

…ct#56268)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

This change is a follow-up for
ray-project#56105.

Now dataset size estimation is based on listed file sizes. However,
encoding ratio was still based on the file size estimates derived from
the uncompressed data size obtained from Parquet metadata.

This change is addressing that by:

- Rebasing encoding ratio to relate estimated in-memory size to the
listed file size
 - Cleaning up unused abstractions (like `ParquetMetadataProvider`)

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

This change is a follow-up for
ray-project/ray#56105.

Now dataset size estimation is based on listed file sizes. However,
encoding ratio was still based on the file size estimates derived from
the uncompressed data size obtained from Parquet metadata.

This change is addressing that by:

- Rebasing encoding ratio to relate estimated in-memory size to the
listed file size
 - Cleaning up unused abstractions (like `ParquetMetadataProvider`)

## Related issue number

<!-- For example: "Closes #1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Original PR #56105 by alexeykudinkin
Original: ray-project/ray#56105

…ParquetDatasource`

Merged from original PR #56105
Original: ray-project/ray#56105

…se test

This test became broken after the removal of Parquet metadata fetching
tasks in #56105. The test relies on specific autoscaling behavior that
no longer works as expected without metadata fetching.

Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>

…ease test (#58048)

## Summary

This PR removes the `image_classification_chaos_no_scale_back` release
test and its associated setup script
(`setup_cluster_compute_config_updater.py`). This test has become
non-functional and is no longer providing useful signal.

## Background

The `image_classification_chaos_no_scale_back` release test was designed
to validate Ray Data's fault tolerance when many nodes abruptly get
preempted at the same time.

The test worked by:
1. Running on an autoscaling cluster with 1-10 nodes
2. Updating the compute config mid-test to downscale to 5 nodes
3. Asserting that there are dead nodes as a sanity check

## Why This Test Is Broken

After the removal of Parquet metadata fetching in #56105 (September 2,
2025), the autoscaling behavior changed significantly:

- **Before metadata removal**: The cluster would autoscale more
aggressively because metadata fetching created additional tasks that
triggered faster scale-up. The cluster would scale past 5 nodes, then
downscale, leaving dead nodes that the test could detect.

- **After metadata removal**: Without the metadata fetching tasks, the
cluster doesn't scale up fast enough to get past 5 nodes before the
downscale happens. This means there are no dead nodes to detect, causing
the test to fail.

## Why We're Removing It

1. **Test is fundamentally broken**: The test's assumptions about
autoscaling behavior are no longer valid after the metadata fetching
removal
2. **Not actively monitored**: The test is marked as unstable and isn't
closely watched

## Changes

- Removed `image_classification_chaos_no_scale_back` test from
`release/release_data_tests.yaml`
- Deleted
`release/nightly_tests/setup_cluster_compute_config_updater.py` (only
used by this test)

## Related

See #56105

Fixes #56528

Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>

…ease test (ray-project#58048)

## Summary

This PR removes the `image_classification_chaos_no_scale_back` release
test and its associated setup script
(`setup_cluster_compute_config_updater.py`). This test has become
non-functional and is no longer providing useful signal.

## Background

The `image_classification_chaos_no_scale_back` release test was designed
to validate Ray Data's fault tolerance when many nodes abruptly get
preempted at the same time.

The test worked by:
1. Running on an autoscaling cluster with 1-10 nodes
2. Updating the compute config mid-test to downscale to 5 nodes
3. Asserting that there are dead nodes as a sanity check

## Why This Test Is Broken

After the removal of Parquet metadata fetching in ray-project#56105 (September 2,
2025), the autoscaling behavior changed significantly:

- **Before metadata removal**: The cluster would autoscale more
aggressively because metadata fetching created additional tasks that
triggered faster scale-up. The cluster would scale past 5 nodes, then
downscale, leaving dead nodes that the test could detect.

- **After metadata removal**: Without the metadata fetching tasks, the
cluster doesn't scale up fast enough to get past 5 nodes before the
downscale happens. This means there are no dead nodes to detect, causing
the test to fail.

## Why We're Removing It

1. **Test is fundamentally broken**: The test's assumptions about
autoscaling behavior are no longer valid after the metadata fetching
removal
2. **Not actively monitored**: The test is marked as unstable and isn't
closely watched

## Changes

- Removed `image_classification_chaos_no_scale_back` test from
`release/release_data_tests.yaml`
- Deleted
`release/nightly_tests/setup_cluster_compute_config_updater.py` (only
used by this test)

## Related

See ray-project#56105

Fixes ray-project#56528

Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Signed-off-by: xgui <xgui@anyscale.com>

…-project#56105)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

Historically, `ParquetDatasource` have been fetching individual files
parquet footer metadata to obtain granular metadata about the dataset.

While laudable in principle, it's really inefficient in practice and
manifests itself in extremely poor performance on very large datasets
(10s to 100s Ks of files).

This change revisits this approach by 

- Removing metadata fetching as a step (and deprecating involved
components)
- Cleaning up and streamlining some of the other utilities further
optimizing performance


<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

…ct#56268)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

This change is a follow-up for
ray-project#56105.

Now dataset size estimation is based on listed file sizes. However,
encoding ratio was still based on the file size estimates derived from
the uncompressed data size obtained from Parquet metadata.

This change is addressing that by:

- Rebasing encoding ratio to relate estimated in-memory size to the
listed file size
 - Cleaning up unused abstractions (like `ParquetMetadataProvider`)

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

…ease test (ray-project#58048)

## Summary

This PR removes the `image_classification_chaos_no_scale_back` release
test and its associated setup script
(`setup_cluster_compute_config_updater.py`). This test has become
non-functional and is no longer providing useful signal.

## Background

The `image_classification_chaos_no_scale_back` release test was designed
to validate Ray Data's fault tolerance when many nodes abruptly get
preempted at the same time.

The test worked by:
1. Running on an autoscaling cluster with 1-10 nodes
2. Updating the compute config mid-test to downscale to 5 nodes
3. Asserting that there are dead nodes as a sanity check

## Why This Test Is Broken

After the removal of Parquet metadata fetching in ray-project#56105 (September 2,
2025), the autoscaling behavior changed significantly:

- **Before metadata removal**: The cluster would autoscale more
aggressively because metadata fetching created additional tasks that
triggered faster scale-up. The cluster would scale past 5 nodes, then
downscale, leaving dead nodes that the test could detect.

- **After metadata removal**: Without the metadata fetching tasks, the
cluster doesn't scale up fast enough to get past 5 nodes before the
downscale happens. This means there are no dead nodes to detect, causing
the test to fail.

## Why We're Removing It

1. **Test is fundamentally broken**: The test's assumptions about
autoscaling behavior are no longer valid after the metadata fetching
removal
2. **Not actively monitored**: The test is marked as unstable and isn't
closely watched

## Changes

- Removed `image_classification_chaos_no_scale_back` test from
`release/release_data_tests.yaml`
- Deleted
`release/nightly_tests/setup_cluster_compute_config_updater.py` (only
used by this test)

## Related

See ray-project#56105

Fixes ray-project#56528

Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>

…ease test (ray-project#58048)

## Summary

This PR removes the `image_classification_chaos_no_scale_back` release
test and its associated setup script
(`setup_cluster_compute_config_updater.py`). This test has become
non-functional and is no longer providing useful signal.

## Background

The `image_classification_chaos_no_scale_back` release test was designed
to validate Ray Data's fault tolerance when many nodes abruptly get
preempted at the same time.

The test worked by:
1. Running on an autoscaling cluster with 1-10 nodes
2. Updating the compute config mid-test to downscale to 5 nodes
3. Asserting that there are dead nodes as a sanity check

## Why This Test Is Broken

After the removal of Parquet metadata fetching in ray-project#56105 (September 2,
2025), the autoscaling behavior changed significantly:

- **Before metadata removal**: The cluster would autoscale more
aggressively because metadata fetching created additional tasks that
triggered faster scale-up. The cluster would scale past 5 nodes, then
downscale, leaving dead nodes that the test could detect.

- **After metadata removal**: Without the metadata fetching tasks, the
cluster doesn't scale up fast enough to get past 5 nodes before the
downscale happens. This means there are no dead nodes to detect, causing
the test to fail.

## Why We're Removing It

1. **Test is fundamentally broken**: The test's assumptions about
autoscaling behavior are no longer valid after the metadata fetching
removal
2. **Not actively monitored**: The test is marked as unstable and isn't
closely watched

## Changes

- Removed `image_classification_chaos_no_scale_back` test from
`release/release_data_tests.yaml`
- Deleted
`release/nightly_tests/setup_cluster_compute_config_updater.py` (only
used by this test)

## Related

See ray-project#56105

Fixes ray-project#56528

Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Signed-off-by: Aydin Abiar <aydin@anyscale.com>

…ease test (ray-project#58048)

## Summary

This PR removes the `image_classification_chaos_no_scale_back` release
test and its associated setup script
(`setup_cluster_compute_config_updater.py`). This test has become
non-functional and is no longer providing useful signal.

## Background

The `image_classification_chaos_no_scale_back` release test was designed
to validate Ray Data's fault tolerance when many nodes abruptly get
preempted at the same time.

The test worked by:
1. Running on an autoscaling cluster with 1-10 nodes
2. Updating the compute config mid-test to downscale to 5 nodes
3. Asserting that there are dead nodes as a sanity check

## Why This Test Is Broken

After the removal of Parquet metadata fetching in ray-project#56105 (September 2,
2025), the autoscaling behavior changed significantly:

- **Before metadata removal**: The cluster would autoscale more
aggressively because metadata fetching created additional tasks that
triggered faster scale-up. The cluster would scale past 5 nodes, then
downscale, leaving dead nodes that the test could detect.

- **After metadata removal**: Without the metadata fetching tasks, the
cluster doesn't scale up fast enough to get past 5 nodes before the
downscale happens. This means there are no dead nodes to detect, causing
the test to fail.

## Why We're Removing It

1. **Test is fundamentally broken**: The test's assumptions about
autoscaling behavior are no longer valid after the metadata fetching
removal
2. **Not actively monitored**: The test is marked as unstable and isn't
closely watched

## Changes

- Removed `image_classification_chaos_no_scale_back` test from
`release/release_data_tests.yaml`
- Deleted
`release/nightly_tests/setup_cluster_compute_config_updater.py` (only
used by this test)

## Related

See ray-project#56105

Fixes ray-project#56528

Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Signed-off-by: Future-Outlier <eric901201@gmail.com>

…ease test (ray-project#58048)

## Summary

This PR removes the `image_classification_chaos_no_scale_back` release
test and its associated setup script
(`setup_cluster_compute_config_updater.py`). This test has become
non-functional and is no longer providing useful signal.

## Background

The `image_classification_chaos_no_scale_back` release test was designed
to validate Ray Data's fault tolerance when many nodes abruptly get
preempted at the same time.

The test worked by:
1. Running on an autoscaling cluster with 1-10 nodes
2. Updating the compute config mid-test to downscale to 5 nodes
3. Asserting that there are dead nodes as a sanity check

## Why This Test Is Broken

After the removal of Parquet metadata fetching in ray-project#56105 (September 2,
2025), the autoscaling behavior changed significantly:

- **Before metadata removal**: The cluster would autoscale more
aggressively because metadata fetching created additional tasks that
triggered faster scale-up. The cluster would scale past 5 nodes, then
downscale, leaving dead nodes that the test could detect.

- **After metadata removal**: Without the metadata fetching tasks, the
cluster doesn't scale up fast enough to get past 5 nodes before the
downscale happens. This means there are no dead nodes to detect, causing
the test to fail.

## Why We're Removing It

1. **Test is fundamentally broken**: The test's assumptions about
autoscaling behavior are no longer valid after the metadata fetching
removal
2. **Not actively monitored**: The test is marked as unstable and isn't
closely watched

## Changes

- Removed `image_classification_chaos_no_scale_back` test from
`release/release_data_tests.yaml`
- Deleted
`release/nightly_tests/setup_cluster_compute_config_updater.py` (only
used by this test)

## Related

See ray-project#56105

Fixes ray-project#56528

Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>

…ease test (ray-project#58048)

## Summary

This PR removes the `image_classification_chaos_no_scale_back` release
test and its associated setup script
(`setup_cluster_compute_config_updater.py`). This test has become
non-functional and is no longer providing useful signal.

## Background

The `image_classification_chaos_no_scale_back` release test was designed
to validate Ray Data's fault tolerance when many nodes abruptly get
preempted at the same time.

The test worked by:
1. Running on an autoscaling cluster with 1-10 nodes
2. Updating the compute config mid-test to downscale to 5 nodes
3. Asserting that there are dead nodes as a sanity check

## Why This Test Is Broken

After the removal of Parquet metadata fetching in ray-project#56105 (September 2,
2025), the autoscaling behavior changed significantly:

- **Before metadata removal**: The cluster would autoscale more
aggressively because metadata fetching created additional tasks that
triggered faster scale-up. The cluster would scale past 5 nodes, then
downscale, leaving dead nodes that the test could detect.

- **After metadata removal**: Without the metadata fetching tasks, the
cluster doesn't scale up fast enough to get past 5 nodes before the
downscale happens. This means there are no dead nodes to detect, causing
the test to fail.

## Why We're Removing It

1. **Test is fundamentally broken**: The test's assumptions about
autoscaling behavior are no longer valid after the metadata fetching
removal
2. **Not actively monitored**: The test is marked as unstable and isn't
closely watched

## Changes

- Removed `image_classification_chaos_no_scale_back` test from
`release/release_data_tests.yaml`
- Deleted
`release/nightly_tests/setup_cluster_compute_config_updater.py` (only
used by this test)

## Related

See ray-project#56105

Fixes ray-project#56528

Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Signed-off-by: peterxcli <peterxcli@gmail.com>