Skip to content

[ci/release] Trigger Ray release by running a Bazel binary#52962

Merged
aslonnie merged 9 commits intomasterfrom
khluu/bazelize_release
Jun 13, 2025
Merged

[ci/release] Trigger Ray release by running a Bazel binary#52962
aslonnie merged 9 commits intomasterfrom
khluu/bazelize_release

Conversation

@khluu
Copy link
Copy Markdown
Contributor

@khluu khluu commented May 13, 2025

  • Add a flag to specify release image used for release tests
  • If image is specified, create/reuse a custom cluster environment and use its ID to trigger release tests with
  • If cluster env ID is already specified, the pipeline would not create another cluster env but choose to reuse the custom one we created
  • Add data deps so this script can be run with Bazel
  • A lot of the code in create_cluster_env_from_image is reused from minimal.py

@khluu
p
ad8bde6 
Signed-off-by: kevin <kevin@anyscale.com>
@khluu khluu requested a review from a team as a code owner May 13, 2025 09:04
@khluu khluu requested a review from aslonnie May 13, 2025 09:04
@khluu
Copy link
Copy Markdown
Contributor Author

khluu commented May 13, 2025

@aslonnie Instead of creating a custom cluster env and inject its ID, I think I can also intercept the part where it creates the cluster env in minimal.py. It might require adding a few more args in cluster_manager.py minimal.py and glue.py.

However, bazel run //release:run_release_test -- tune_scalability_durable_trainable.aws --image 029272617770.dkr.ecr.us-west-2.amazonaws.com/anyscale/ray:66b19d-py39-cpu with the right env added (those related to Anyscale) was able to submit a job to Anyscale that runs the release test for me

khluu
khluu commented May 13, 2025
View reviewed changes
release/ray_release/glue.py
try:
cluster_manager.cluster_env_id = cluster_env_id
cluster_manager.build_cluster_env()
cluster_manager.fetch_build_info()
Copy link
Copy Markdown
Contributor Author

@khluu khluu May 13, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this function doesn't even exist anymore

@khluu
Copy link
Copy Markdown
Contributor Author

khluu commented May 13, 2025

Sample job run triggered by Bazel script: https://console.anyscale-staging.com/cld_kvedZWag2qA8i5BjxUevf5i7/prj_92c7b71w55flm6gv6imv4m6vqg/jobs/prodjob_nnlk1fbbn2j1s81l6alvlik7fi?job-tab=overview&job-logs-section-tabs=application_logs

khluu and others added 2 commits May 14, 2025 08:02
@khluu
p
4db0b3f 
Signed-off-by: kevin <kevin@anyscale.com>
@aslonnie
Merge branch 'master' into khluu/bazelize_release
d8419cd
aslonnie
aslonnie reviewed May 19, 2025
View reviewed changes
release/ray_release/util.py
return f"test_env_{image_normalized}_{test_name}"


def create_cluster_env_from_image(
Copy link
Copy Markdown
Collaborator

@aslonnie aslonnie May 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I had a look. this is too similar to the one in minimal.py. these two should be merged.

Copy link
Copy Markdown
Contributor Author

@khluu khluu Jun 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Copy Markdown
Collaborator

@aslonnie aslonnie Jun 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

where is it?

Copy link
Copy Markdown
Contributor Author

@khluu khluu Jun 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh I didn't realize the commits were not pushed yet... pushed now

@aslonnie aslonnie added the go add ONLY when ready to merge, run all tests label May 28, 2025
@aslonnie aslonnie self-requested a review May 28, 2025 22:40
aslonnie
aslonnie reviewed May 29, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@aslonnie aslonnie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the two functions are still not merged.

@khluu
Copy link
Copy Markdown
Contributor Author

khluu commented Jun 9, 2025

2 functions are merged and the one in minimal.py is reusing the new one in util.py now. I just triggered a job with bazel here: https://console.anyscale-staging.com/cld_kvedZWag2qA8i5BjxUevf5i7/prj_92c7b71w55flm6gv6imv4m6vqg/jobs/prodjob_wxywgranubg8m92wbn13m3q76l?job-tab=overview&job-logs-section-tabs=application_logs

@khluu khluu requested a review from aslonnie June 9, 2025 23:27
@khluu
Copy link
Copy Markdown
Contributor Author

khluu commented Jun 9, 2025

Triggered a test from release pipeline to make sure the code in minimal.py still works: https://buildkite.com/ray-project/release/builds/43451

@aslonnie aslonnie removed the go add ONLY when ready to merge, run all tests label Jun 10, 2025
khluu and others added 5 commits June 10, 2025 09:01
@khluu
p
153e7a6 
Signed-off-by: kevin <kevin@anyscale.com>
@khluu
p
99ded7d 
Signed-off-by: kevin <kevin@anyscale.com>
@khluu
p
447d840 
Signed-off-by: kevin <kevin@anyscale.com>
@aslonnie
Merge branch 'master' into khluu/bazelize_release
8f30712
@khluu
p
4e2e15e 
Signed-off-by: kevin <kevin@anyscale.com>
@khluu
Copy link
Copy Markdown
Contributor Author

khluu commented Jun 13, 2025

Release test is still passing: https://buildkite.com/ray-project/release/builds/45209#01975b3f-99ac-4b91-b368-088def773260

@aslonnie aslonnie added the go add ONLY when ready to merge, run all tests label Jun 13, 2025
@aslonnie aslonnie merged commit e35c2c8 into master Jun 13, 2025
6 checks passed
@aslonnie aslonnie deleted the khluu/bazelize_release branch June 13, 2025 21:07
elliot-barn pushed a commit that referenced this pull request Jun 18, 2025
@khluu @aslonnie @elliot-barn
[ci/release] Trigger Ray release by running a Bazel binary (#52962)
054d8ba 
- Add a flag to specify release image used for release tests
- If image is specified, create/reuse a custom cluster environment and
use its ID to trigger release tests with
- If cluster env ID is already specified, the pipeline would not create
another cluster env but choose to reuse the custom one we created
- Add data deps so this script can be run with Bazel
- A lot of the code in `create_cluster_env_from_image` is reused from
`minimal.py`

---------

Signed-off-by: kevin <kevin@anyscale.com>
Co-authored-by: Lonnie Liu <95255098+aslonnie@users.noreply.github.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
elliot-barn pushed a commit that referenced this pull request Jul 2, 2025
@khluu @aslonnie @elliot-barn
[ci/release] Trigger Ray release by running a Bazel binary (#52962)
0ec9d39 
- Add a flag to specify release image used for release tests
- If image is specified, create/reuse a custom cluster environment and
use its ID to trigger release tests with
- If cluster env ID is already specified, the pipeline would not create
another cluster env but choose to reuse the custom one we created
- Add data deps so this script can be run with Bazel
- A lot of the code in `create_cluster_env_from_image` is reused from
`minimal.py`

---------

Signed-off-by: kevin <kevin@anyscale.com>
Co-authored-by: Lonnie Liu <95255098+aslonnie@users.noreply.github.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@aslonnie aslonnie aslonnie approved these changes

Assignees

No one assigned

Labels

go add ONLY when ready to merge, run all tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@khluu @aslonnie @hainesmichaelc