Summary
There seems to be a race condition in the parallel gem installation code, that's especially likely to be seen on Windows. Sorry, it didn't show up in the initial CI runs, but has cropped up a couple of times since...
Basically, two gem processes are trying to write to (different files in) the same folder at the same time. Then they try to read all the files in that folder. If one of the processes is still working on writing a file when the other process reads it, it gets corrupt data and errors.
This isn't a major issue on Linux/MacOS, mostly as, on these platforms, gem uses write-and-rename semantics to atomically update the file. It seems that Windows isn't so good at this.
Sadly, all of my stress testing of the change was on Linux, with only the CI runs to check the Windows behaviour.
One option is obviously to back out the parallel gem support, or to default it to run in serial mode rather than parallel. But I've also got a PR for a hardened approach which I'll raise in a moment.
Willing to submit a PR?
Platform
windows-latest CI runners
Version
prek 0.3.4
.pre-commit-config.yaml
repos:
- repo: local
hooks:
- id: test-gem-require
name: test-gem-require
language: ruby
entry: ruby test_script.rb
language_version: system
additional_dependencies: ["rspec"]
pass_filenames: false
always_run: true
- id: test-gem-require-versioned
name: test-gem-require-versioned
language: ruby
entry: ruby test_script.rb
language_version: system
additional_dependencies: ["rspec:3.12.0"]
pass_filenames: false
always_run: true
- id: test-gem-require-missing
name: test-gem-require-missing
language: ruby
entry: ruby test_script.rb
language_version: system
pass_filenames: false
always_run: trueLog file
FAIL [ 13.336s] ( 6/12) prek::languages ruby::additional_gem_dependencies
stdout ───
running 1 test
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Snapshot Summary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Snapshot: additional_gem_dependencies
Source: D:\a\prek\prek:325
───────────────────────────────────────────────────────────────────────────────
program: prek
args:
- run
- "-v"
env:
GIT_CONFIG_COUNT: "1"
GIT_CONFIG_KEY_0: core.autocrlf
GIT_CONFIG_VALUE_0: "false"
PREK_HOME: "C:\\Users\\runneradmin\\AppData\\Roaming\\prek\\tests\\.tmpK7mJux\\home"
PREK_INTERNAL__SORT_FILENAMES: "1"
───────────────────────────────────────────────────────────────────────────────
-old snapshot
+new results
────────────┬──────────────────────────────────────────────────────────────────
1 1 │ success: false
2 │-exit_code: 1
2 │+exit_code: 2
3 3 │ ----- stdout -----
4 │-test-gem-require.........................................................Passed
5 │-- hook id: test-gem-require
6 │-- duration: [TIME]
7 4 │
8 │- X.Y.Z
9 │-test-gem-require-versioned...............................................Passed
10 │-- hook id: test-gem-require-versioned
11 │-- duration: [TIME]
5 │+----- stderr -----
6 │+error: Failed to install hook `test-gem-require-versioned`
7 │+ caused by: Failed to install gems
8 │+ caused by: Command `gem install rspec-core` exited with an error:
12 9 │
13 │- 3.12.0
14 │-test-gem-require-missing.................................................Failed
15 │-- hook id: test-gem-require-missing
16 │-- duration: [TIME]
17 │-- exit code: 1
10 │+[status]
11 │+exit code: 1
18 12 │
19 │- <internal:[RUBY_LIB]>:[X]:in 'Kernel#require': cannot load such file -- rspec (LoadError)
20 │- from <internal:[RUBY_LIB]>:[X]:in 'Kernel#require'
21 │- from test_script.rb:1:in '<main>'
22 │-
23 │------ stderr -----
13 │+[stderr]
14 │+[[HOME]/hooks/ruby-Y5MeMjekczJiCQVjcag1/gems/specifications/rspec-expectations-3.12.4.gemspec] isn't a Gem::Specification (NilClass instead).
15 │+ERROR: While executing gem ... (NoMethodError)
16 │+undefined method 'full_name' for nil
17 │+if existing.find {|s| s.full_name == spec.full_name }
18 │+^^^^^^^^^^
19 │+C:/hostedtoolcache/windows/Ruby/3.4.8/x64/lib/ruby/3.4.0/rubygems/request_set.rb:276:in 'block (2 levels) in Gem::RequestSet#install_into'
20 │+C:/hostedtoolcache/windows/Ruby/3.4.8/x64/lib/ruby/3.4.0/rubygems/request_set.rb:276:in 'Array#each'
21 │+C:/hostedtoolcache/windows/Ruby/3.4.8/x64/lib/ruby/3.4.0/rubygems/request_set.rb:276:in 'Enumerable#find'
22 │+C:/hostedtoolcache/windows/Ruby/3.4.8/x64/lib/ruby/3.4.0/rubygems/request_set.rb:276:in 'block in Gem::RequestSet#install_into'
23 │+C:/hostedtoolcache/windows/Ruby/3.4.8/x64/lib/ruby/3.4.0/rubygems/request_set.rb:273:in 'Array#each'
24 │+C:/hostedtoolcache/windows/Ruby/3.4.8/x64/lib/ruby/3.4.0/rubygems/request_set.rb:273:in 'Gem::RequestSet#install_into'
25 │+C:/hostedtoolcache/windows/Ruby/3.4.8/x64/lib/ruby/3.4.0/rubygems/request_set.rb:148:in 'Gem::RequestSet#install'
26 │+C:/hostedtoolcache/windows/Ruby/3.4.8/x64/lib/ruby/3.4.0/rubygems/commands/install_command.rb:207:in 'Gem::Commands::InstallCommand#install_gem'
27 │+C:/hostedtoolcache/windows/Ruby/3.4.8/x64/lib/ruby/3.4.0/rubygems/commands/install_command.rb:223:in 'block in Gem::Commands::InstallCommand#install_gems'
28 │+C:/hostedtoolcache/windows/Ruby/3.4.8/x64/lib/ruby/3.4.0/rubygems/commands/install_command.rb:216:in 'Array#each'
29 │+C:/hostedtoolcache/windows/Ruby/3.4.8/x64/lib/ruby/3.4.0/rubygems/commands/install_command.rb:216:in 'Gem::Commands::InstallCommand#install_gems'
30 │+C:/hostedtoolcache/windows/Ruby/3.4.8/x64/lib/ruby/3.4.0/rubygems/commands/install_command.rb:162:in 'Gem::Commands::InstallCommand#execute'
31 │+C:/hostedtoolcache/windows/Ruby/3.4.8/x64/lib/ruby/3.4.0/rubygems/command.rb:326:in 'Gem::Command#invoke_with_build_args'
32 │+C:/hostedtoolcache/windows/Ruby/3.4.8/x64/lib/ruby/3.4.0/rubygems/command_manager.rb:253:in 'Gem::CommandManager#invoke_command'
33 │+C:/hostedtoolcache/windows/Ruby/3.4.8/x64/lib/ruby/3.4.0/rubygems/command_manager.rb:194:in 'Gem::CommandManager#process_args'
34 │+C:/hostedtoolcache/windows/Ruby/3.4.8/x64/lib/ruby/3.4.0/rubygems/command_manager.rb:152:in 'Gem::CommandManager#run'
35 │+C:/hostedtoolcache/windows/Ruby/3.4.8/x64/lib/ruby/3.4.0/rubygems/gem_runner.rb:57:in 'Gem::GemRunner#run'
36 │+C:/hostedtoolcache/windows/Ruby/3.4.8/x64/bin/gem:12:in '<main>'
────────────┴──────────────────────────────────────────────────────────────────
Stopped on the first failure. Run `cargo insta test` to run all snapshots.
test ruby::additional_gem_dependencies ... FAILED
failures:
failures:
ruby::additional_gem_dependencies
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 69 filtered out; finished in 13.27s
Summary
There seems to be a race condition in the parallel gem installation code, that's especially likely to be seen on Windows. Sorry, it didn't show up in the initial CI runs, but has cropped up a couple of times since...
Basically, two
gemprocesses are trying to write to (different files in) the same folder at the same time. Then they try to read all the files in that folder. If one of the processes is still working on writing a file when the other process reads it, it gets corrupt data and errors.This isn't a major issue on Linux/MacOS, mostly as, on these platforms,
gemuses write-and-rename semantics to atomically update the file. It seems that Windows isn't so good at this.Sadly, all of my stress testing of the change was on Linux, with only the CI runs to check the Windows behaviour.
One option is obviously to back out the parallel gem support, or to default it to run in serial mode rather than parallel. But I've also got a PR for a hardened approach which I'll raise in a moment.
Willing to submit a PR?
Platform
windows-latest CI runners
Version
prek 0.3.4
.pre-commit-config.yaml
Log file