Skip to content

[3.8] bpo-42398: Fix "make regen-all" race condition (GH-23362) (GH-23367)#23371

Merged
miss-islington merged 1 commit intopython:3.8from
miss-islington:backport-c53c3f4-3.8
Nov 18, 2020
Merged

[3.8] bpo-42398: Fix "make regen-all" race condition (GH-23362) (GH-23367)#23371
miss-islington merged 1 commit intopython:3.8from
miss-islington:backport-c53c3f4-3.8

Conversation

@miss-islington
Copy link
Copy Markdown
Contributor

@miss-islington miss-islington commented Nov 18, 2020

Fix a race condition in "make regen-all" when make -jN option is used
to run jobs in parallel. The clinic.py script now only use atomic
write to write files. Moveover, generated files are now left
unchanged if the content does not change, to not change the file
modification time.

The "make regen-all" command runs "make clinic" and "make
regen-importlib" targets:

  • "make regen-importlib" builds object files (ex: Modules/_weakref.o)
    from source files (ex: Modules/_weakref.c) and clinic files (ex:
    Modules/clinic/_weakref.c.h)
  • "make clinic" always rewrites all clinic files
    (ex: Modules/clinic/_weakref.c.h)

Since there is no dependency between "clinic" and "regen-importlib"
Makefile targets, these two targets can be run in parallel. Moreover,
half of clinic.py file writes are not atomic and so there is a race
condition when "make regen-all" runs jobs in parallel using make -jN
option (which can be passed in MAKEFLAGS environment variable).

Fix clinic.py to make all file writes atomic:

  • Add write_file() function to ensure that all file writes are
    atomic: write into a temporary file and then use os.replace().
  • Moreover, write_file() doesn't recreate or modify the file if the
    content does not change to avoid modifying the file modification
    file.
  • Update test_clinic to verify these assertions with a functional
    test.
  • Remove Clinic.force attribute which was no longer used, whereas
    Clinic.verify remains useful.

(cherry picked from commit 8fba952)
(cherry picked from commit c53c3f4)

Co-authored-by: Victor Stinner vstinner@python.org

https://bugs.python.org/issue42398

…onGH-23367)

Fix a race condition in "make regen-all" when make -jN option is used
to run jobs in parallel. The clinic.py script now only use atomic
write to write files. Moveover, generated files are now left
unchanged if the content does not change, to not change the file
modification time.

The "make regen-all" command runs "make clinic" and "make
regen-importlib" targets:

* "make regen-importlib" builds object files (ex: Modules/_weakref.o)
  from source files (ex: Modules/_weakref.c) and clinic files (ex:
  Modules/clinic/_weakref.c.h)
* "make clinic" always rewrites all clinic files
  (ex: Modules/clinic/_weakref.c.h)

Since there is no dependency between "clinic" and "regen-importlib"
Makefile targets, these two targets can be run in parallel. Moreover,
half of clinic.py file writes are not atomic and so there is a race
condition when "make regen-all" runs jobs in parallel using make -jN
option (which can be passed in MAKEFLAGS environment variable).

Fix clinic.py to make all file writes atomic:

* Add write_file() function to ensure that all file writes are
  atomic: write into a temporary file and then use os.replace().
* Moreover, write_file() doesn't recreate or modify the file if the
  content does not change to avoid modifying the file modification
  file.
* Update test_clinic to verify these assertions with a functional
  test.
* Remove Clinic.force attribute which was no longer used, whereas
  Clinic.verify remains useful.

(cherry picked from commit 8fba952)
(cherry picked from commit c53c3f4)

Co-authored-by: Victor Stinner <vstinner@python.org>
@miss-islington
Copy link
Copy Markdown
Contributor Author

@vstinner: Status check is done, and it's a success ✅ .

Copy link
Copy Markdown
Member

@vstinner vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, good bot.

@miss-islington
Copy link
Copy Markdown
Contributor Author

@vstinner: Status check is done, and it's a success ✅ .

@miss-islington miss-islington merged commit 66dd533 into python:3.8 Nov 18, 2020
@miss-islington miss-islington deleted the backport-c53c3f4-3.8 branch November 18, 2020 16:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants