@nizamial09 I'm guessing this draft is to test the recent make check failure? I raised a tracker for it here: https://tracker.ceph.com/issues/62082

Are you testing a fix here? If so, feel free to add it to the tracker.

@nizamial09 I'm guessing this draft is to test the recent make check failure? I raised a tracker for it here: https://tracker.ceph.com/issues/62082

Are you testing a fix here? If so, feel free to add it to the tracker.

thanks @ljflores will update the tracker.

I don't know if this is helpful or not but it seems like the failures are in the src/tools/setup-virtualenv.sh script and that maybe a dependency changed and now it can't find the packages (honestly I find this script way under-documented). I was just looking at this script because it had a side effect due to the plan on adding dependencies to the cephadm binary in #52492. So if you want to bounce ideas off someone I'm happy to help. Thanks!

thanks @phlogistonjohn and yes please any kind of help is appreciated. I am kind of stuck with this now and not able to test it. I tested this partly in a ubuntu container where i didn't find this issue but I am trying to check this in jenkins env. TBH, not an expert in this area!

maybe a dependency changed and now it can't find the packages

do you think the dependency is on the ceph or something changed from pip?

Assuming I'm right about src/tools/setup-virtualenv.sh what do you think about just starting by adding a set -x to the top of the script so we can see exactly what command is failing?

I ended up studying how the dependencies are installed on the make-checks. Basically the install-deps script installs all the python dependencies and store them in the wheelhouse folder. So when the tox requires a dependency, its fetched from the wheelhouse folder rather than going online. But for some reason, the jsonschema, cryptography, mypy... deps are not getting saved to the wheelhouse folder when these --use-feature=fast-deps --use-deprecated=legacy-resolver flags are set. Since it was introduced as a workaround in pip20.3, which was useful upto pip 21 I think (I read that somewhere but I lost the link, maybe if I go through history I can find it again).

So I removed this flags which resolves this issue. Not sure if its the best/right approach to the problem.

This run was successful with the setup-venv's. But it got aborted when I pushed again. So maybe it'll pass again in the current run.

I ended up studying how the dependencies are installed on the make-checks. Basically the install-deps script installs all the python dependencies and store them in the wheelhouse folder. So when the tox requires a dependency, its fetched from the wheelhouse folder rather than going online. But for some reason, the jsonschema, cryptography, mypy... deps are not getting saved to the wheelhouse folder when these --use-feature=fast-deps --use-deprecated=legacy-resolver flags are set. Since it was introduced as a workaround in pip20.3, which was useful upto pip 21 I think (I read that somewhere but I lost the link, maybe if I go through history I can find it again).

So I removed this flags which resolves this issue. Not sure if its the best/right approach to the problem.

This run was successful with the setup-venv's. But it got aborted when I pushed again. So maybe it'll pass again in the current run.

I found the same issue with /wheelhouse, and I was able to fix it by removing a line in install-deps.sh that updates pip. (See PR #52558). A new version of pip was released a few days ago, which is I think what caused the change.

Up to you which method you want to use, but it looks like both fix the issue.

Approving, but take a look at #52558 and let me know what you think works best.

I added my thought there! I am a bit uncertain which will work the best. I'll let others take a look at these.

@badone Could you please sanity check this against what lead to #43221? If install-deps.sh runs into a similar issue with the "new" resolver in the future, how would we notice that it's happening in Jenkins?

Perhaps we should consider wrapping pip install with a timeout?

Since it was introduced as a workaround in pip20.3, which was useful upto pip 21 I think (I read that somewhere but I lost the link, maybe if I go through history I can find it again).

@nizamial09 It would be very nice to add that to the commit message. We are removing a workaround that was in place for almost two years, having an explanation for why it's no longer needed with a link is miles better than "Not sure if this is still valid."

Since it was introduced as a workaround in pip20.3, which was useful upto pip 21 I think (I read that somewhere but I lost the link, maybe if I go through history I can find it again).

@nizamial09 It would be very nice to add that to the commit message. We are removing a workaround that was in place for almost two years, having an explanation for why it's no longer needed with a link is miles better than "Not sure if this is still valid."

@idryomov I couldn't find the url. But there is this documentation that explains why we should be using the new dependency resolver rather than using the legacy resolver. And moreover, legacy-resolver looks like something that could be deprecated when the community decides. And the more viable solution is to be more strict with our pip dependencies.

I'll update the commit with these links.

@nizamial09 want to merge this?

@nizamial09 want to merge this?

@ljflores I think we can. Please do the honors. I am connected through phone now.

@nizamial09 should be backported to Reef too.
It doesn't seem to cherry-pick cleanly, so take a look.

@nizamial09 should be backported to Reef too. It doesn't seem to cherry-pick cleanly, so take a look.

alright!

@ljflores #52562, the conflict was because in reef the line number was different.

Perhaps we should consider wrapping pip install with a timeout?

Noting that --timeout 300 is already passed in PIP_OPTS.

Perhaps we should consider wrapping pip install with a timeout?

Noting that --timeout 300 is already passed in PIP_OPTS.

I forgot about this, but thanks for confirming.

Possibly related: pypa/pip#12156
So FWIW pip developers are aware of issues with the legacy resolver in 23.2.

Possibly related: pypa/pip#12156 So FWIW pip developers are aware of issues with the legacy resolver in 23.2.

ah good! atleast its being tracked in the community. thanks for pointing it out. @phlogistonjohn