[ci] move sccache stats off RDS by suo · Pull Request #79366 · pytorch/pytorch

suo · 2022-06-12T04:57:33Z

Stack from ghstack (oldest at bottom):

We want to decommission RDS, so upload sccache stats to S3/GHA as
appropriate.

We want to decommission RDS, so upload sccache stats to S3/GHA as appropriate. [ghstack-poisoned]

facebook-github-bot · 2022-06-12T04:57:56Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/79366
📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓Need help or want to give feedback on the CI? Visit our office hours

✅ No Failures (0 Pending)

As of commit b39b476 (more details on the Dr. CI page):

Expand to see more

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

We want to decommission RDS, so upload sccache stats to S3/GHA as appropriate. [ghstack-poisoned]

We want to decommission RDS, so upload sccache stats to S3/GHA as appropriate. ghstack-source-id: 238ab57 Pull Request resolved: #79366

We want to decommission RDS, so upload sccache stats to S3/GHA as appropriate. [ghstack-poisoned]

With #79366, the only remaining use case for RDS is import latency stats. Afaik we do not look at these, so removing, which eliminates our last dependency on RDS. [ghstack-poisoned]

We want to decommission RDS, so upload sccache stats to S3/GHA as appropriate. [ghstack-poisoned]

With #79366, the only remaining use case for RDS is import latency stats. Afaik we do not look at these, so removing, which eliminates our last dependency on RDS. [ghstack-poisoned]

We want to decommission RDS, so upload sccache stats to S3/GHA as appropriate. [ghstack-poisoned]

With #79366, the only remaining use case for RDS is import latency stats. Afaik we do not look at these, so removing, which eliminates our last dependency on RDS. [ghstack-poisoned]

We want to decommission RDS, so upload sccache stats to S3/GHA as appropriate. [ghstack-poisoned]

With #79366, the only remaining use case for RDS is import latency stats. Afaik we do not look at these, so removing, which eliminates our last dependency on RDS. [ghstack-poisoned]

With #79366, the only remaining use case for RDS is import latency stats. Afaik we do not look at these, so removing, which eliminates our last dependency on RDS. ghstack-source-id: a1f9fcb Pull Request resolved: #79370

We want to decommission RDS, so upload sccache stats to S3/GHA as appropriate. [ghstack-poisoned]

With #79366, the only remaining use case for RDS is import latency stats. Afaik we do not look at these, so removing, which eliminates our last dependency on RDS. [ghstack-poisoned]

We want to decommission RDS, so upload sccache stats to S3/GHA as appropriate. [ghstack-poisoned]

With #79366, the only remaining use case for RDS is import latency stats. Afaik we do not look at these, so removing, which eliminates our last dependency on RDS. [ghstack-poisoned]

With #79366, the only remaining use case for RDS is import latency stats. Afaik we do not look at these, so removing, which eliminates our last dependency on RDS. ghstack-source-id: 2654da6 Pull Request resolved: #79370

janeyx99 · 2022-06-13T15:35:14Z

          PR_LABELS: ${{ toJson(github.event.pull_request.labels.*.name) }}
          TORCH_CUDA_ARCH_LIST: 5.2
          DOCKER_IMAGE: ${{ steps.calculate-docker-image.outputs.docker-image }}
+          OUR_GITHUB_JOB_ID: ${{ steps.get-job-id.outputs.job-id }}


is the OUR_ just to not collide with GITHUB_JOB_ID which we know is not what we care about?

can't decide if we should add a comment to clarify that

Yeah, just in general we shouldn't trample on the GITHUB_ prefix because GITHUB may add env vars in the future

janeyx99 · 2022-06-13T15:39:35Z

+    return name.replace(" ", "_").replace("-", "_").lower()
+
+
+STAT_NAMES = {


what happens if these names change sometime in the future (which is probably rare)? will we rely on alerts on the data in rockset to ensure reporting stability here?

we probably can just make this code more generic, I can take it as a followup

suo · 2022-06-13T16:08:33Z

@pytorchbot merge

pytorchmergebot · 2022-06-13T16:09:43Z

@pytorchbot successfully started a merge job. Check the current status here

github-actions · 2022-06-13T16:10:31Z

Hey @suo.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

With #79366, the only remaining use case for RDS is import latency stats. Afaik we do not look at these, so removing, which eliminates our last dependency on RDS. [ghstack-poisoned]

With #79366, the only remaining use case for RDS is import latency stats. Afaik we do not look at these, so removing, which eliminates our last dependency on RDS. Pull Request resolved: #79370 Approved by: https://github.com/janeyx99

Allows use of .jenkins/pytorch/build.sh without assuming OUR_GITHUB_JOB_ID is set. This is a regression caused by #79366. Pull Request resolved: #79541 Approved by: https://github.com/suo, https://github.com/seemethere

Summary: We want to decommission RDS, so upload sccache stats to S3/GHA as appropriate. Pull Request resolved: #79366 Approved by: https://github.com/janeyx99 Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/3771331b2a692399c228ce0aa2290a9b03010018 Reviewed By: osalpekar Differential Revision: D37119148 Pulled By: suo fbshipit-source-id: 0000b03cbc8eabc8acc16da5d38b1764e8e8cab9

Summary: With #79366, the only remaining use case for RDS is import latency stats. Afaik we do not look at these, so removing, which eliminates our last dependency on RDS. Pull Request resolved: #79370 Approved by: https://github.com/janeyx99 Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/964d5059587c017b02f1d7d62587722a38683c63 Reviewed By: malfet Differential Revision: D37153120 Pulled By: suo fbshipit-source-id: 6a2e1fb827a88e0828b1f05fdbe62f07ec61da87

…) (#79541) Summary: Allows use of .jenkins/pytorch/build.sh without assuming OUR_GITHUB_JOB_ID is set. This is a regression caused by #79366. Pull Request resolved: #79541 Approved by: https://github.com/suo, https://github.com/seemethere Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/adaafeedb1f66c4c877a446a6082f2a761878883 Reviewed By: malfet Differential Revision: D37156933 fbshipit-source-id: a9309facfccfb8b4aa4f8fb7a727929fe760d95c

We want to decommission RDS, so upload sccache stats to S3/GHA as appropriate. Pull Request resolved: pytorch#79366 Approved by: https://github.com/janeyx99

With pytorch#79366, the only remaining use case for RDS is import latency stats. Afaik we do not look at these, so removing, which eliminates our last dependency on RDS. Pull Request resolved: pytorch#79370 Approved by: https://github.com/janeyx99

…rch#79541) Allows use of .jenkins/pytorch/build.sh without assuming OUR_GITHUB_JOB_ID is set. This is a regression caused by pytorch#79366. Pull Request resolved: pytorch#79541 Approved by: https://github.com/suo, https://github.com/seemethere

[ci] move sccache stats off RDS

2f4ebce

We want to decommission RDS, so upload sccache stats to S3/GHA as appropriate. [ghstack-poisoned]

suo requested a review from a team as a code owner June 12, 2022 04:57

facebook-github-bot added the cla signed label Jun 12, 2022

suo added the ciflow/trunk Trigger trunk jobs on your pull request label Jun 12, 2022

Update on "[ci] move sccache stats off RDS"

1c09586

We want to decommission RDS, so upload sccache stats to S3/GHA as appropriate. [ghstack-poisoned]

suo mentioned this pull request Jun 12, 2022

[ci] Pull out generic upload_test_stats functionality #79367

Closed

suo added a commit that referenced this pull request Jun 12, 2022

[ci] move sccache stats off RDS

67b175b

We want to decommission RDS, so upload sccache stats to S3/GHA as appropriate. ghstack-source-id: 238ab57 Pull Request resolved: #79366

Update on "[ci] move sccache stats off RDS"

e5495e8

We want to decommission RDS, so upload sccache stats to S3/GHA as appropriate. [ghstack-poisoned]

suo mentioned this pull request Jun 12, 2022

[ci] remove remaining RDS dependency #79370

Closed

suo added a commit that referenced this pull request Jun 12, 2022

[ci] remove remaining RDS dependency

7c86fc6

With #79366, the only remaining use case for RDS is import latency stats. Afaik we do not look at these, so removing, which eliminates our last dependency on RDS. [ghstack-poisoned]

suo mentioned this pull request Jun 12, 2022

[ci] unpin get-workflow-job-id #79369

Closed

Update on "[ci] move sccache stats off RDS"

cc18f1d

We want to decommission RDS, so upload sccache stats to S3/GHA as appropriate. [ghstack-poisoned]

Update on "[ci] move sccache stats off RDS"

d841aae

We want to decommission RDS, so upload sccache stats to S3/GHA as appropriate. [ghstack-poisoned]

suo mentioned this pull request Jun 12, 2022

[ci] remove TD + test specification code from run_test.py #79372

Closed

Update on "[ci] move sccache stats off RDS"

eec79ac

We want to decommission RDS, so upload sccache stats to S3/GHA as appropriate. [ghstack-poisoned]

Update on "[ci] move sccache stats off RDS"

822bc59

We want to decommission RDS, so upload sccache stats to S3/GHA as appropriate. [ghstack-poisoned]

Update on "[ci] move sccache stats off RDS"

b39b476

We want to decommission RDS, so upload sccache stats to S3/GHA as appropriate. [ghstack-poisoned]

suo mentioned this pull request Jun 13, 2022

[ci] convert empty s3 artifacts from error to warning #79397

Closed

janeyx99 reviewed Jun 13, 2022

View reviewed changes

janeyx99 approved these changes Jun 13, 2022

View reviewed changes

pytorchmergebot added the Merged label Jun 13, 2022

pytorchmergebot closed this in 3771331 Jun 13, 2022

jeffdaily mentioned this pull request Jun 14, 2022

do not write sccache stats to json if missing OUR_GITHUB_JOB_ID #79541

Closed

facebook-github-bot deleted the gh/suo/557/head branch June 17, 2022 14:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ci] move sccache stats off RDS#79366

[ci] move sccache stats off RDS#79366
suo wants to merge 9 commits intogh/suo/557/basefrom
gh/suo/557/head

suo commented Jun 12, 2022 •

edited

Loading

Uh oh!

facebook-github-bot commented Jun 12, 2022 •

edited

Loading

Uh oh!

janeyx99 Jun 13, 2022

Uh oh!

janeyx99 Jun 13, 2022

Uh oh!

suo Jun 13, 2022

Uh oh!

janeyx99 Jun 13, 2022

Uh oh!

suo Jun 13, 2022

Uh oh!

suo commented Jun 13, 2022

Uh oh!

pytorchmergebot commented Jun 13, 2022

Uh oh!

github-actions Bot commented Jun 13, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		return name.replace(" ", "_").replace("-", "_").lower()


		STAT_NAMES = {

Conversation

suo commented Jun 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Jun 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

✅ No Failures (0 Pending)

Uh oh!

janeyx99 Jun 13, 2022

Choose a reason for hiding this comment

Uh oh!

janeyx99 Jun 13, 2022

Choose a reason for hiding this comment

Uh oh!

suo Jun 13, 2022

Choose a reason for hiding this comment

Uh oh!

janeyx99 Jun 13, 2022

Choose a reason for hiding this comment

Uh oh!

suo Jun 13, 2022

Choose a reason for hiding this comment

Uh oh!

suo commented Jun 13, 2022

Uh oh!

pytorchmergebot commented Jun 13, 2022

Uh oh!

github-actions Bot commented Jun 13, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

suo commented Jun 12, 2022 •

edited

Loading

facebook-github-bot commented Jun 12, 2022 •

edited

Loading