Fix generator issues with newer rubygems and bundler by ahukkanen · Pull Request #9849 · decidim/decidim

ahukkanen · 2022-09-22T09:01:50Z

🎩 What? Why?

With rubygems 3.3.x which ships with bundler 2.3.x we have currently issues using the decidim generators or actually any executables we provide (which is generally the decidim executable).

My assumption is that the key feature related to this issue that was added in the newer version of bundler is that it recognizes the bundler version based on the Gemfile.lock and installs that particular bundler version in case it was not already installed and installs the gems under that bundler version.

Apparently this approach has still some problems as the bundler versions get mixed. If you only have one version of bundler installed, there is no problem running the decidim executables but if you have more than one version (2.3.x with an older version such as 2.2.x), this issue appears.

To fix this issue, I am proposing that we remove the automatic copying of the Gemfile.lock files into the newly generated Decidim application to get around this issue. Let's just let bundler generate the Gemfile.lock locally.

In fact, I don't even understand why we wanted to do this in the first place so if anyone has an idea why this is done, feel free to enlighten me.

As an extra, we can finally get rid of these messages (showed up after Ruby 3.1 upgrade) in the console once this is merged:

Calling `DidYouMean::SPELL_CHECKERS.merge!(error_name => spell_checker)' has been deprecated. Please call `DidYouMean.correct_error(error_name, spell_checker)' instead.

📌 Related Issues

Fixes Uninitialized constant Gem::BundlerVersionFinder error during gem generation #9839
Related issues at RubyGems
- Add missing requires ruby/rubygems#2196
- remove explicit require for auto-loaded constant ruby/rubygems#3751

Testing

See the replication instructions from #9839.

Also, see that the CI is green. We have many failing build examples e.g. at #9766 where I tried to initially update the bundler to newer one but I believe it failed because of this same reason.

ahukkanen · 2022-09-22T10:32:08Z

It seems that I still have exactly the same problems that I had at #9766, so marking this as draft for now until investigating further.

Bottom line is that my initial assumption seems incorrect.

ahukkanen · 2022-09-22T12:38:31Z

Tested this also locally and the proposed fix would actually solve #9839 but now the problem is with the generators spec, which are also broken if you run them locally under the same conditions.

This is something I haven't figured out yet...

ahukkanen · 2022-09-22T14:08:51Z

Note to myself regarding that last CI problem:

The problem seems to be if the gem environment already has different versions of some gems required what Decidim requires. In the particular case that I was debugging I found out that:

Decidim has been installed with rack version 2.2.4
When the generators test is trying to install the decidim gems, it checks if rack is already installed
The gem environment has already loaded rack version 2.2.3.1 which does not match the current requirement at Gemfile.lock
Therefore the rack gem is interpreted as uninstalled in this particular case
When the gem is interpreted as "uninstalled" rubygems tries to install it here:
https://github.com/rubygems/rubygems/blob/409a5c78f975d794036dc040dfc857c7b2b9007f/lib/rubygems/request_set.rb#L194-L196
But if the gem specification is a Gem::Resolver::InstalledSpecification the install method will just yield and return nil here
https://github.com/rubygems/rubygems/blob/409a5c78f975d794036dc040dfc857c7b2b9007f/lib/rubygems/resolver/installed_specification.rb#L17-L19
When the install method returns nil, the rest of the process will break as RubyGems is not expecting any nil values in the installed gems list

So the solution to this would be to always have exactly the same versions of gems installed as are loaded to RubyGems.

I need to investigate a bit further what is causing the version mis-match, i.e. is it the same rack gem at the CI as locally on my machine and what is causing the different version of that gem to be already installed when interpreting the Decidim requirements.

- Fix psych open ended dependency - Update minitest, parallel_tests, parser, rack, regexp_parser, rubocop-ast, tzinfo

Multiple versions of bundler cause the generators specs to fail with newer bundler versions. Yet to be investigated in which version this breaks exactly.

andreslucena · 2022-11-09T10:16:18Z

@ahukkanen can you merge with develop and also update the bundler version from 2.3.7 to 2.3.25 🙏🏽 ?

andreslucena · 2022-11-10T10:33:01Z

I mentioned it on our internal chat, but I think that is good to also leave it here for the people from the future:

The error that we're seeing (uninitialized constant Gem::BundlerVersionFinder exception) was already present on Decidim for these specs. Examples: #3654 (comment) #3532 (comment)

It was solved on rubygems ruby/rubygems#2196. My hypothesis on why it broke again is because they started removing these kinds of requires as there was an autoload related bug on JRuby ruby/rubygems#3751. They mention that these shouldn't be necessary, but maybe something that we're doing in the specs/generator produces a change in the Bundler/rubygems autoload?

I didn't have the chance (yet) to try out if the naive solution (adding again the requires on the necessary files in my local installation) could solve this issue, but when I have time I'd want to try that.

deivid-rodriguez · 2022-12-18T20:31:28Z

For what it's worth, we got this problem reported at ruby/rubygems#6145 and I plan to ship the fix with RubyGems 3.4.0 next week.

andreslucena · 2022-12-20T07:06:11Z

Awesome @deivid-rodriguez, thank you very much!

We didn't have time to allocate to this issue, and couldn't pinpoint exactly the reason why this was happening and make a proof of concept to report to rubygems. Thank you for letting it us know.

We'll keep track of that issue and let you know if we have any problem with the fix once its released.

deivid-rodriguez · 2022-12-20T08:49:02Z

No problem! I don't really know why it's happening either, a Ruby bug I guess. But the workaround for us is easy :)

Keep up the good work!

deivid-rodriguez · 2022-12-24T08:33:34Z

RubyGems 3.4.0 is out now with this fix. Happy holidays! :)

ahukkanen · 2023-01-13T14:06:39Z

Thanks for the fix @deivid-rodriguez, really nice Christmas gift from you!

Turns out we had a bug in Decidim that I submitted a new PR for #10220.

But your fix made it way easier to debug and fix the problem as the stacktrace made sense all of a sudden after upgrading RubyGems and Bundler.

I will try upgrading RubyGems again after we have merged #10220.

Happy new year!

deivid-rodriguez · 2023-01-13T14:21:41Z

Awesome, really glad our releases made things easier! Keep up the good work 💪

ahukkanen · 2023-02-01T09:58:05Z

Superseded by #10323.

Closing this one.

ahukkanen added 3 commits September 22, 2022 11:48

bundle update --bundler

d8f9b46

Do not force the Gemfile.lock in generated Decidim applications

b1b9e57

Add gem cleanup and update of system gems to the generators workflow

7eca45e

ahukkanen marked this pull request as draft September 22, 2022 10:31

ahukkanen added 15 commits September 23, 2022 09:54

Add gem list to the generators CI to try to identify double gems

cded154

Run gem list after bundle install too

4d69876

Update psych and make sure rails is at least at 6.1.5

782b8c8

Add aliases: true to the YAML.load_file calls

91de2d7

Fix some dependencies

984a4f9

- Fix psych open ended dependency - Update minitest, parallel_tests, parser, rack, regexp_parser, rubocop-ast, tzinfo

Lock the correct versions of the default gems

baa5a59

Update diffy gem

da0e994

Lock io-wait and minitest to the default versions

e02f0de

Remove locking of minitest and io-wait

4800cf7

Update several gems

fd8466f

Try without updating the system gems

71bf563

Downgrade bundler back to 2.3.7 to avoid multiple versions

346527c

Multiple versions of bundler cause the generators specs to fail with newer bundler versions. Yet to be investigated in which version this breaks exactly.

Try disabling bundler cache

fdd4ee1

Try gem cleanup before bundle install

e869ec2

Update system gems (again)

475a042

ahukkanen mentioned this pull request Oct 25, 2022

Bump versions on install docs #9916

Merged

ahukkanen added 3 commits November 9, 2022 12:26

Merge branch 'develop' into fix/modern-rubygems-executable-issues

cd2a350

Update locked bundler version

8f5aa52

Merge branch 'develop' into fix/modern-rubygems-executable-issues

6c0c0cb

ahukkanen mentioned this pull request Dec 19, 2022

Uninitialized constant Gem::BundlerVersionFinder error during gem generation #9839

Closed

ahukkanen mentioned this pull request Jan 13, 2023

Fix dependency resolver trying to fetch gem paths from lazy specifications #10220

Merged

ahukkanen mentioned this pull request Feb 1, 2023

Update locked Bundler version #10323

Merged

ahukkanen closed this Feb 1, 2023

ahukkanen deleted the fix/modern-rubygems-executable-issues branch February 1, 2023 09:58

ahukkanen mentioned this pull request Feb 1, 2023

Update psych to 4.X and bump minimum Rails version to 6.1.5 #10325

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix generator issues with newer rubygems and bundler#9849

Fix generator issues with newer rubygems and bundler#9849
ahukkanen wants to merge 21 commits intodecidim:developfrom
mainio:fix/modern-rubygems-executable-issues

ahukkanen commented Sep 22, 2022 •

edited

Loading

Uh oh!

ahukkanen commented Sep 22, 2022

Uh oh!

ahukkanen commented Sep 22, 2022

Uh oh!

ahukkanen commented Sep 22, 2022

Uh oh!

andreslucena commented Nov 9, 2022

Uh oh!

andreslucena commented Nov 10, 2022 •

edited

Loading

Uh oh!

deivid-rodriguez commented Dec 18, 2022

Uh oh!

andreslucena commented Dec 20, 2022

Uh oh!

deivid-rodriguez commented Dec 20, 2022

Uh oh!

deivid-rodriguez commented Dec 24, 2022

Uh oh!

ahukkanen commented Jan 13, 2023

Uh oh!

deivid-rodriguez commented Jan 13, 2023

Uh oh!

ahukkanen commented Feb 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ahukkanen commented Sep 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎩 What? Why?

📌 Related Issues

Testing

Uh oh!

ahukkanen commented Sep 22, 2022

Uh oh!

ahukkanen commented Sep 22, 2022

Uh oh!

ahukkanen commented Sep 22, 2022

Uh oh!

andreslucena commented Nov 9, 2022

Uh oh!

andreslucena commented Nov 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deivid-rodriguez commented Dec 18, 2022

Uh oh!

andreslucena commented Dec 20, 2022

Uh oh!

deivid-rodriguez commented Dec 20, 2022

Uh oh!

deivid-rodriguez commented Dec 24, 2022

Uh oh!

ahukkanen commented Jan 13, 2023

Uh oh!

deivid-rodriguez commented Jan 13, 2023

Uh oh!

ahukkanen commented Feb 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ahukkanen commented Sep 22, 2022 •

edited

Loading

andreslucena commented Nov 10, 2022 •

edited

Loading