Conversation
-e Signed-off-by: machichima <nary12321@gmail.com>
|
Maybe next time, @MortalHappiness, you could ask contributors to run the tests ten times before approving the PR, in case some of us are not familiar with the doc tests and might miss something during review. |
|
@kevin85421 Okay no problem. @machichima Could you post your screenshot of running ten times? Thanks. |
| "# Step 4.3: List all Pods in the `default` namespace.\n", | ||
| "# The Pod created by the Kubernetes Job will be terminated after the Kubernetes Job finishes.\n", | ||
| "kubectl get pods --sort-by=.metadata.name" | ||
| "kubectl get pods --sort-by='.metadata.creationTimestamp'" |
There was a problem hiding this comment.
@machichima, can you help me understand where the logic for validating this cell's output is located? Thanks!
There was a problem hiding this comment.
I think it's because the last 5 characters are randomized, so if we sort by name, sometimes rayjob-sample-xxxxx comes before rayjob-sample-raycluster-yyyyy-head-zzzzz.
There was a problem hiding this comment.
Thank you for the explanation!
Not necessarily for this PR. The test is flaky in CI and blocks multiple PRs. My point is that we can improve the review process by requesting contributors to run the tests multiple times locally to avoid this situation in the future. Thank @MortalHappiness for the review and @machichima for the quick fix! |
I will run more times locally and see if I can spot the reason for this flaky behaviour |
|
I tried multiple times (7 times) and they all passed... I don't actually know what might be the case, As the CI log seems to logging more info then I did. I am wondering if it might caused by using different version of something? Attached my running log here: logging.log UPDATE: I got kubectl version UPDATE: I use versions same as CI
and tried 7 times locally and they all passed. I couldn't reproduce the flaky behaviour locally. New logging is here: logging-new.log |
Signed-off-by: Steve Han <stevehan2001@gmail.com>
Why are these changes needed?
As mentioned here: #51756 (comment), the doc test for rayjob is flaky.
Sort get pods output with create time to prevent flaky. Ensure the doc test passed locally twice.
Doc link: https://anyscale-ray--52170.com.readthedocs.build/en/52170/cluster/kubernetes/getting-started/rayjob-quick-start.html
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.