Skip to content

Sphinx parallel build#38785

Closed
ShawnZhong wants to merge 1 commit intopytorch:masterfrom
ShawnZhong:parallel_docs
Closed

Sphinx parallel build#38785
ShawnZhong wants to merge 1 commit intopytorch:masterfrom
ShawnZhong:parallel_docs

Conversation

@ShawnZhong
Copy link
Copy Markdown
Contributor

@ShawnZhong ShawnZhong commented May 20, 2020

See https://www.sphinx-doc.org/en/master/man/sphinx-build.html#cmdoption-sphinx-build-j

Distribute the build over N processes in parallel, to make building on multiprocessor machines more effective. Note that not all parts and not all builders of Sphinx can be parallelized. If auto argument is given, Sphinx uses the number of CPUs as N.

  • Timing results
    • Python doc build on a 40-core machine: 9:34 down to 1:29
    • pytorch_cpp_doc_push: ~1h 10m down to 47m
    • pytorch_python_doc_push: 34m down to 32m

@ShawnZhong ShawnZhong changed the title [WIP] Sphinx parallel build Sphinx parallel build May 20, 2020
@ShawnZhong ShawnZhong marked this pull request as ready for review May 20, 2020 16:35
@dr-ci
Copy link
Copy Markdown

dr-ci Bot commented May 20, 2020

💊 CI failures summary and remediations

As of commit 466e1e6 (more details on the Dr. CI page):


  • 1/1 failures possibly* introduced in this PR
    • 1/1 non-CircleCI failure(s)

ci.pytorch.org: 1 failed


This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker.

See how this bot performed.

This comment has been revised 1 time.

@ailzhang ailzhang requested a review from zou3519 May 21, 2020 16:39
@ailzhang ailzhang added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label May 21, 2020
Copy link
Copy Markdown
Contributor

@zou3519 zou3519 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is pretty sweet, thank you!

Copy link
Copy Markdown
Contributor

@facebook-github-bot facebook-github-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zou3519 is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

@zou3519 merged this pull request in 481838f.

@ShawnZhong ShawnZhong deleted the parallel_docs branch May 23, 2020 00:02
laurentdupin pushed a commit to laurentdupin/pytorch that referenced this pull request Apr 24, 2026
Summary:
See https://www.sphinx-doc.org/en/master/man/sphinx-build.html#cmdoption-sphinx-build-j

> Distribute the build over N processes in parallel, to make building on multiprocessor machines more effective. Note that not all parts and not all builders of Sphinx can be parallelized. If auto argument is given, Sphinx uses the number of CPUs as N.

- Timing results
  - Python doc build on a 40-core machine: 9:34 down to 1:29
  - pytorch_cpp_doc_push: ~1h 10m down to 47m
  - pytorch_python_doc_push: 34m down to 32m
Pull Request resolved: pytorch#38785

Differential Revision: D21691991

Pulled By: zou3519

fbshipit-source-id: cfc5e8cd13414640f82edfd2ad1ce4d9c7afce12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Merged open source triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants