@ahukkanen I can't add you as a reviewer, but it would be great if you can check this PR. Thanks! ❤️

@leio10 I think a better approach would be to release the package.json to NPM e.g. with name @decidim/essentials as I mentioned at #8094. Or is there some specific reason why you'd want to use the one stored in the repository/gem?

If you want to go without releasing the dependency to NPM, I think the changes look fine.

@leio10 I think a better approach would be to release the package.json to NPM e.g. with name @decidim/essentials as I mentioned at #8094. Or is there some specific reason why you'd want to use the one stored in the repository/gem?

If you want to go without releasing the dependency to NPM, I think the changes look fine.

My concern is related to versioning if that list of dependencies changes: if you are running Decidim 0.25.1, but the stable version of Decidim is 0.26 you don't want to install dependencies from that one, so you should do something like npm i @decidim/essentials@0.25.1. And even more, if you are adding an NPM dependency to decidim in a new branch, you will want to load the updated list of dependencies in the test app. At the end, you need to use the exact same version of the decidim code you are using. This PR tries to implement that.

Regarding the NPM package, it is possible to do it with a new PR over this one, but it won't be needed if you create a tag with every release. I like the idea to publish an NPM package with each version, but it would imply extra work during the release process, so we can discuss about it to decide if it worth it or not.

My concern is related to versioning if that list of dependencies changes

OK, understood the reasoning.

Regarding the NPM package, it is possible to do it with a new PR over this one, but it won't be needed if you create a tag with every release. I like the idea to publish an NPM package with each version, but it would imply extra work during the release process, so we can discuss about it to decide if it worth it or not.

The only thing that I'm worried about with the git repository approach is that it is extremely slow as it copies the whole repository under node_modules. With the approach you took with the installed gem packages, I don't see a problem because it is referring the package.json directly from the gem folder which is obviously fast.

That's the only reason why I'd like to have a faster installation method.

But yes, I understand it is a balance for the ease of maintenance. I'm just seeing that eventually something would have to be released to NPM. E.g. at #8115 I implemented something in the config/webpack/base.js that I would much rather pull as a dependency from a Decidim NPM package.

@ahukkanen ok, speed is a excelent reason to publish an NPM package, so we should include it on the release process 👍

@leio10 maybe a Github action like this would help a bit to publish the package automatically? It looks like it's able to read the version properly

@leio10 Please also see my proposed changes regarding the NPM packages at #8121.

@ahukkanen I like your approach (I'd just name the votings package as elections), but I still don't see a clear path if we don't find a fix for npm/cli#2339. I spent the whole afternoon trying to do it without any luck 😞.
For example, in your PR you needed to add a reference to your repo, and I guess that it will happen every time we change any of those packages. I don't like it, but maybe we can live with that at the moment. What do you think?

@leio10

I'd just name the votings package as elections

It's changed now.

I still don't see a clear path if we don't find a fix for npm/cli#2339

With NPM, there is no solution right now other than rolling back to NPM v6 as mentioned in the issue. They discussed about it in the open RFC meeting (linked in that issue) but there was no resolution to this.

I also tried with Yarn 2.1+ which does support installing single packages from a Yarn workspace at GitHub but it has the same problem that it does not install the subsequent dependencies of the workspace packages.

For example, in your PR you needed to add a reference to your repo, and I guess that it will happen every time we change any of those packages. I don't like it, but maybe we can live with that at the moment. What do you think?

@leio10 Regarding this point, yes I think that would be the case with pull requests. But as mentioned at #8121, I'm not trying to propose moving all assets to NPM, just the dependencies right now and possibly some support stuff (e.g. that would be useful in #8115).

I think the situation would be comparable to the current situation if someone wants to update anything in the package.json.

@ahukkanen I was able to make this work using npm pack to create a Decidim package locally, as the local reference was not working at all. This is only used when the gem is used from a local folder, for dev and test apps. I don't like that is even slower than before, but it could be a starting point. What do you think?

@leio10 LGTM, just left one comment above.

I think I figured out a way to avoid packing the NPM packages at #8121 but let's merge this one first. In #8121 I applied the same principle that you applied here:

• Install the local packages if Decidim is installed through a path (no pack needed)
• Install the git NPM packages for development gem bundles and for git installations
• Install the released NPM packages for released gem bundles from RubyGems

With the local reference, the "linked" package's package.json needs to live under the directory where you are installing it from. Only then it installs also the linked package's dependencies. If they live in some other directory outside of the application folder (where the main package.json lives at), it won't install the linked package's dependencies.

With the local reference, the "linked" package's package.json needs to live under the directory where you are installing it from. Only then it installs also the linked package's dependencies. If they live in some other directory outside of the application folder (where the main package.json lives at), it won't install the linked package's dependencies.

Mm, ok, so one option would be to copy it into an app subfolder as decidim-npm-deps, right?

Mm, ok, so one option would be to copy it into an app subfolder as decidim-npm-deps, right?

Yes, that's exactly what I did at #8121. The downside is still that this needs to be replicated to the design app too because of the root webpacker spec that ensures the package.json and package-lock.json match with the ones at the root. It's not a major problem, just a small nuisance.

I really hope they would find a solution to npm/cli#2339 which would allow avoiding such problems. It's a real headache with the local references.

@ahukkanen I've checked #8121 and I think it would be much better to close this one and merge yours directly.

I only find two issues:

• I still see static references to the decidim/decidim repo in the subackages, that could break something if you are using a different repo or branch. There's any way to avoid this?
• I would like to run the decidim:webpacker:installer during the decidim:upgrade task. This would force us to change the install generator to avoid doing it twice during the creation of the app.

It would be nice to address both before merging it, but we can work on them later in future PRs.

I've checked #8121 and I think it would be much better to close this one and merge yours directly.

Either way is fine. I just thought your PR adds some important stuff (like the upgrade task changes), so it would be a good starting point. It's not much work to merge it.

There are some things that I would like to change with #8121 but I don't believe there's much we can do until they fix the issues at NPM 7... There's also no visibility when they would fix the related NPM issue, so I guess we have to live with it for the time being.

• I still see static references to the decidim/decidim repo in the subackages, that could break something if you are using a different repo or branch. There's any way to avoid this?

Those references are used when external apps install Decidim from GitHub. It is possible to avoid if we copy the whole packages folder (where the NPM packages live) under the application but I see that would be against what we are trying to do. This would mean that the packages folder inside the application would have to be updated every time Decidim is upgraded, hoping the maintainer has not done any changes in those files. I would prefer the Git approach with Git installations of Decidim.

The NPM packages should be released to NPM and the actual Decidim applications should refer them directly from NPM by running e.g. npm i @decidim/all@~0.25.0. So with normal installations, there won't be any git references.

With the CI and Decidim development environment, the references are local, so there won't be any git references either.

Git references are needed only for git installations of Decidim.

• I would like to run the decidim:webpacker:installer during the decidim:upgrade task. This would force us to change the install generator to avoid doing it twice during the creation of the app.

Sure, this can be done, no problem.

Closed in favor of #8121

@leio10 I applied the changes regarding the decidim:upgrade task at #8121 and also some refactoring of the decidim:webpacker namespace you had done in this PR.