Skip to content

[CircleCI] Store build artifacts for python docs#37658

Closed
ShawnZhong wants to merge 10 commits intopytorch:masterfrom
ShawnZhong:python_doc_artifacts
Closed

[CircleCI] Store build artifacts for python docs#37658
ShawnZhong wants to merge 10 commits intopytorch:masterfrom
ShawnZhong:python_doc_artifacts

Conversation

@ShawnZhong
Copy link
Copy Markdown
Contributor

@ShawnZhong ShawnZhong commented May 1, 2020

This PR allows the build artifacts for python docs to be stored on CircieCI, which helps the reviewer to preview doc changes before merging.

The artifacts can be found in the ARTIFACTS tab of the test pytorch_cpp_doc_push, and the website is served at https://5320907-65600975-gh.circle-artifacts.com/0/docs/index.html

This PR is inspired by @rgommers's comment under #37419 (comment)

There's a CircleCI job pytorch_python_doc_push that builds the docs, however it doesn't store any artifacts for PRs. Controlled by .circleci/scripts/python_doc_push_script.sh. I think that's the only doc build (?). Not sure why it doesn't store the artifacts, that would be useful.

@ShawnZhong ShawnZhong marked this pull request as ready for review May 1, 2020 13:02
@ShawnZhong ShawnZhong changed the title Store build artifacts for python docs [CircleCI] Store build artifacts for python docs May 1, 2020
Copy link
Copy Markdown
Collaborator

@rgommers rgommers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks @ShawnZhong!

This works, I have a small possible improvement suggestion:

    - store_artifacts: 
        path: ~/workspace/build_artifacts/master
        destination: docs

should change the full path of all files under the artifacts tab from ~/workspace/build_artifacts/master/* to docs/* so it's easier to read.

Copy link
Copy Markdown
Collaborator

@rgommers rgommers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, good to merge.

@rgommers rgommers requested a review from ezyang May 2, 2020 11:21
@ezyang
Copy link
Copy Markdown
Contributor

ezyang commented May 4, 2020

This is nice, I will merge it. However, it is not very discoverable right now. Is there some way we can give more information to users when they modify docs that their docs are viewable at some spot?

@kostmo maybe this is something Dr. CI could do?

Copy link
Copy Markdown
Contributor

@facebook-github-bot facebook-github-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ezyang is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@rgommers
Copy link
Copy Markdown
Collaborator

rgommers commented May 4, 2020

it is not very discoverable right now. Is there some way we can give more information to users when they modify docs that their docs are viewable at some spot?

SciPy has a little GitHub app (https://github.com/apps/circleci-artifacts-redirector/) that adds a direct link as a separate CI result after the docs build job has completed:

image

It does require some permissions to install on this repo of course, not sure if that's desirable.

Otherwise it may be a matter of just teaching people. I'm personally not a fan of posting a comment after each CI run, that's way too noisy.

@ezyang
Copy link
Copy Markdown
Contributor

ezyang commented May 4, 2020

Unfortunately if this adds a link for every circleci artifact, we'll probably need to do some tweaking, as I think we do a lot of these right now

@rgommers
Copy link
Copy Markdown
Collaborator

rgommers commented May 4, 2020

Unfortunately if this adds a link for every circleci artifact, we'll probably need to do some tweaking, as I think we do a lot of these right now

Ah yes, SciPy uses CircleCI only to build the docs. That app may need a tweak to work here, to take into account the job name. And since it's a Heroku app, it may need to be cloned and tweaked for PyTorch. Too much trouble at this point probably.

Permissions not too bad:
image

@ShawnZhong ShawnZhong deleted the python_doc_artifacts branch May 4, 2020 14:33
@facebook-github-bot
Copy link
Copy Markdown
Contributor

@ezyang merged this pull request in 66a20c2.

@kostmo
Copy link
Copy Markdown
Member

kostmo commented May 4, 2020

[...] Is there some way we can give more information to users when they modify docs that their docs are viewable at some spot?

@kostmo maybe this is something Dr. CI could do?

@ezyang , yes, sounds doable. The necessary ingredients would be associations of file glob patterns with jobs names, to determine which job artifacts link(s) we should surface through Dr. CI based on the files modified in the diff.

We could either store these associations as a text file in the pytorch repo or hard-code them in the Dr. CI app.

@rgommers
Copy link
Copy Markdown
Collaborator

rgommers commented May 4, 2020

The necessary ingredients would be associations of file glob patterns with jobs names, to determine which job artifacts link(s)

  • Job name: pytorch_python_doc_push
  • Artifact link: https://xxxxxxx-xxxxxxxx-gh.circle-artifacts.com/0/docs/index.html

@kostmo
Copy link
Copy Markdown
Member

kostmo commented May 4, 2020

The necessary ingredients would be associations of file glob patterns with jobs names, to determine which job artifacts link(s)

  • Job name: pytorch_python_doc_push
  • Artifact link: https://xxxxxxx-xxxxxxxx-gh.circle-artifacts.com/0/docs/index.html

Is there a simple way to determine which files edited constitute changes to documentation? Is it as straightforward as any file that ends in the *.rst extension? Or perhaps any file underneath a certain directory in the repo?

ShawnZhong added a commit to ShawnZhong/pytorch that referenced this pull request May 5, 2020
Summary:
This PR allows the build artifacts for python docs to be stored on CircieCI, which helps the reviewer to preview doc changes before merging.

The artifacts can be found in the [`ARTIFACTS` tab](  https://app.circleci.com/pipelines/github/pytorch/pytorch/162986/workflows/a969f256-3243-414f-8a02-1234b9dac149/jobs/5320907/artifacts) of the test **pytorch_cpp_doc_push**, and the website is served at https://5320907-65600975-gh.circle-artifacts.com/0/docs/index.html

This PR is inspired by rgommers's comment under pytorch#37419 (comment)

> There's a CircleCI job pytorch_python_doc_push that builds the docs, however it doesn't store any artifacts for PRs. Controlled by .circleci/scripts/python_doc_push_script.sh. I think that's the only doc build (?). Not sure why it doesn't store the artifacts, that would be useful.
Pull Request resolved: pytorch#37658

Differential Revision: D21380094

Pulled By: ezyang

fbshipit-source-id: 1dd44bf836ebc74454f4444ae9321807dccdb313
@ShawnZhong
Copy link
Copy Markdown
Contributor Author

Is there a simple way to determine which files edited constitute changes to documentation? Is it as straightforward as any file that ends in the *.rst extension? Or perhaps any file underneath a certain directory in the repo?

Like all the *.rst files under docs and all the *.py files under torch?

@rgommers
Copy link
Copy Markdown
Collaborator

rgommers commented May 5, 2020

I don't think there's a good way to do that. Touching .py files may or may not result in changes to the docs (docstrings get extracted), and can result in changes in multiple places in the docs (e.g. in the autosummary table and in the page for the function(s) touched).

I'd just always link to the top-level index.html and let the reviewer figure it out. Saving an extra couple of clicks here isn't worth a complicated and possible incomplete link-generation exercise imho.

bharatr21 pushed a commit to bharatr21/pytorch that referenced this pull request May 5, 2020
Summary:
This PR allows the build artifacts for python docs to be stored on CircieCI, which helps the reviewer to preview doc changes before merging.

The artifacts can be found in the [`ARTIFACTS` tab](  https://app.circleci.com/pipelines/github/pytorch/pytorch/162986/workflows/a969f256-3243-414f-8a02-1234b9dac149/jobs/5320907/artifacts) of the test **pytorch_cpp_doc_push**, and the website is served at https://5320907-65600975-gh.circle-artifacts.com/0/docs/index.html

This PR is inspired by rgommers's comment under pytorch#37419 (comment)

> There's a CircleCI job pytorch_python_doc_push that builds the docs, however it doesn't store any artifacts for PRs. Controlled by .circleci/scripts/python_doc_push_script.sh. I think that's the only doc build (?). Not sure why it doesn't store the artifacts, that would be useful.
Pull Request resolved: pytorch#37658

Differential Revision: D21380094

Pulled By: ezyang

fbshipit-source-id: 1dd44bf836ebc74454f4444ae9321807dccdb313
@ezyang
Copy link
Copy Markdown
Contributor

ezyang commented May 5, 2020

Yes, I agree. If it rolls into an existing comment the CI bot makes, it's NBD.

laurentdupin pushed a commit to laurentdupin/pytorch that referenced this pull request Apr 24, 2026
Summary:
This PR allows the build artifacts for python docs to be stored on CircieCI, which helps the reviewer to preview doc changes before merging.

The artifacts can be found in the [`ARTIFACTS` tab](  https://app.circleci.com/pipelines/github/pytorch/pytorch/162986/workflows/a969f256-3243-414f-8a02-1234b9dac149/jobs/5320907/artifacts) of the test **pytorch_cpp_doc_push**, and the website is served at https://5320907-65600975-gh.circle-artifacts.com/0/docs/index.html

This PR is inspired by rgommers's comment under pytorch#37419 (comment)

> There's a CircleCI job pytorch_python_doc_push that builds the docs, however it doesn't store any artifacts for PRs. Controlled by .circleci/scripts/python_doc_push_script.sh. I think that's the only doc build (?). Not sure why it doesn't store the artifacts, that would be useful.
Pull Request resolved: pytorch#37658

Differential Revision: D21380094

Pulled By: ezyang

fbshipit-source-id: 1dd44bf836ebc74454f4444ae9321807dccdb313
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants