Skip to content

remove -C option from yum#4790

Closed
bikallem wants to merge 1 commit intoocaml:masterfrom
bikallem:master
Closed

remove -C option from yum#4790
bikallem wants to merge 1 commit intoocaml:masterfrom
bikallem:master

Conversation

@bikallem
Copy link
Copy Markdown

@bikallem bikallem commented Aug 7, 2021

opam 2.1 issuing yum -q -C list in red hat linux 8.4 seems to be problematic. Consider the following

$ opam upgrade -vvv 
...
... 
+ /bin/yum "-q" "-C" "list"
- 2021-08-07 21:51:36,409 [ERROR] yum:21580:MainThread @logutil.py:194 - [Errno 13] Permission denied: '/var/log/rhsm/rhsm.log' - Further logging output will be written to stderr
- 2021-08-07 21:51:36,410 [ERROR] yum:21580:MainThread @identity.py:156 - Reload of consumer identity cert /etc/pki/consumer/cert.pem raised an exception with msg: [Errno 13] Permission denied: '/etc/pki/consumer/key.pem'
- Importing GPG key 0x442DF0F8:
-  Userid     : "PostgreSQL RPM Building Project <pgsql-pkg-yum@postgresql.org>"
-  Fingerprint: 68C9 E2B9 1A37 D136 FE74 D176 1F16 D2E1 442D F0F8
-  From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
- Is this ok [y/N]:

We continue with y and yum -q -C list fails due to some cache issue (- Error: Cache-only enabled but no cache for 'pgdg-common') and opam upgrade fails.

- Error: Cache-only enabled but no cache for 'pgdg-common'
[WARNING] Opam packages conf-diffutils.1, conf-libev.4-12, conf-pkg-config.2 and
          conf-postgresql.1 depend on the following system packages that are no longer installed:
          diffutils libev-devel pkgconf-pkg-config postgresql-devel
  - conf-diffutils.1: depends on diffutils
  - conf-libev.4-12: depends on libev-devel
  - conf-pkg-config.2: depends on pkgconf-pkg-config
  - conf-postgresql.1: depends on postgresql-devel
[WARNING] Upgrade is not possible because of conflicts or packages that are no longer available:
    - Missing dependency:
    - angstrom >= 0.15.0


You may run "opam upgrade --fixup" to let opam fix the current state.
'opam upgrade -vvv' failed.

The UX experience is even worse if one only issues opam upgrade. In the latter case the command never completes because it is waiting for user input.

This PR fixes this use case by removing -C option from yum -q list and this seems to handle this error case correctly.

@kit-ty-kate
Copy link
Copy Markdown
Member

kit-ty-kate commented Aug 7, 2021

This PR is fine on its own, however it comes at the cost of non-predictible performance each time opam is called.
However overall I'm a bit puzzeled as to how no cache exists on your system. Has yum/dnf update never been run before?

The original failure is a main concerned howover as we parse the output

+ /bin/yum "-q" "-C" "list"
- 2021-08-07 21:51:36,409 [ERROR] yum:21580:MainThread @logutil.py:194 - [Errno 13] Permission denied: '/var/log/rhsm/rhsm.log' - Further logging output will be written to stderr
- 2021-08-07 21:51:36,410 [ERROR] yum:21580:MainThread @identity.py:156 - Reload of consumer identity cert /etc/pki/consumer/cert.pem raised an exception with msg: [Errno 13] Permission denied: '/etc/pki/consumer/key.pem'
- Importing GPG key 0x442DF0F8:
-  Userid     : "PostgreSQL RPM Building Project <pgsql-pkg-yum@postgresql.org>"
-  Fingerprint: 68C9 E2B9 1A37 D136 FE74 D176 1F16 D2E1 442D F0F8
-  From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
- Is this ok [y/N]:

Do you have a reproducible case using docker or something similar, so we can check it out? Are you sure that removing -C would avoid this failure too? I'm not sure what should be added here, maybe --assumeno or --assumeyes but it seems weird either way on a list command (which we would have expected to be readonly)

@kit-ty-kate kit-ty-kate added this to the 2.1.1 milestone Aug 8, 2021
@bikallem
Copy link
Copy Markdown
Author

bikallem commented Aug 8, 2021

However overall I'm a bit puzzeled as to how no cache exists on your system. Has yum/dnf update never been run before?
I periodically run sudo yum/df update.

The main issue seems to be that a yum repository 'pgdg-common' for now does not seem to function(work?) correctly when issuing yum -C list, i.e. it always asks to import gpg key and then attempts to build a cache for it. This is evidenced by the error above as Error: Cache-only enabled but no cache for 'pgdg-common'. Due to this the command yum -q -C list that opam attempts to execute fails and then subsequently opam update, opam install etc all fails. Basically opam becomes pretty unusable is such a scenario.

I'm not sure what should be added here, maybe --assumeno or --assumeyes but it seems weird either way on a list command (which we would have expected to be readonly)

I tried with yum -q -C -y list and yum -q -C --assumeyes list but get that same error

PostgreSQL common RPMs for RHEL/CentOS 8 - x86_64         1.6 MB/s | 1.7 kB     00:00    
Importing GPG key 0x442DF0F8:
 Userid     : "PostgreSQL RPM Building Project <pgsql-pkg-yum@postgresql.org>"
 Fingerprint: 68C9 E2B9 1A37 D136 FE74 D176 1F16 D2E1 442D F0F8
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
Error: Cache-only enabled but no cache for 'pgdg-common'

Note that even though the repo seems not to work with yum -C list it works correctly when I do yum install from the same repo. As mentioned earlier, the issue seems to be some yum repo (key?) error, which is an upstream repo issue and not something I as a user of opam and/or yum can resolve easily.

From an opam user experience perspective, I am not sure depending on a cache behaviour of yum is robust as this case shows. I think - but haven't confirmed - yum list does use cache if available and if not then it lists the packages from the remote repo. So not using -C is quite performant and robust.

Are you sure that removing -C would avoid this failure too

Yes, it seems to solve the issue, just executing yum -q list doesn't display key prompt or give cache error for a yum repo. Without this PR opam is pretty useless in rhel/centos/rockylinux/fedora when yum/dnf repo doesn't work correctly. Could this perhaps be a hotfix release of opam?

Also the error messages (opam interactions) are only visible when issueing opam update -vvv. The ux is not very helpful when yum -C list fails, i.e. the command never completes. I didn't do -vvv at first thinking something was wrong with my upgrade to opam 2.1.0. So I wasted a few hours doing a few opam upgrade/downgrade thinking that was the issue. So perhaps, there is an improvement to be made in this regards.

I am not very familiar with docker so haven't tried the docker route.

@kit-ty-kate
Copy link
Copy Markdown
Member

kit-ty-kate commented Aug 8, 2021

I was able to reproduce something similar with the following Dockerfile:

FROM centos:7
RUN yum -y install https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm
#RUN yum -y makecache # can also be yum -y update
RUN yum -q -C list

In this particular case, removing the -C argument is not enough. It will still fail regardless (because the gpg key is missing)
If we uncomment line 3 however, it works fine.

@kit-ty-kate
Copy link
Copy Markdown
Member

kit-ty-kate commented Aug 8, 2021

Given yum list does not seem to be reliable command and there doesn't to be any options to make it so, see e.g.:

Step 4/5 : RUN yum -qy list > /test
 ---> Running in bc9a75b1b27d
Ignored option -q, -v, -d or -e (probably due to merging: -yq != -y -q)
Importing GPG key 0x442DF0F8:
 Userid     : "PostgreSQL RPM Building Project <pgsql-pkg-yum@postgresql.org>"
 Fingerprint: 68c9 e2b9 1a37 d136 fe74 d176 1f16 d2e1 442d f0f8
 Package    : pgdg-redhat-repo-42.0-18.noarch (@/pgdg-redhat-repo-latest.noarch)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
Importing GPG key 0x442DF0F8:
 Userid     : "PostgreSQL RPM Building Project <pgsql-pkg-yum@postgresql.org>"
 Fingerprint: 68c9 e2b9 1a37 d136 fe74 d176 1f16 d2e1 442d f0f8
 Package    : pgdg-redhat-repo-42.0-18.noarch (@/pgdg-redhat-repo-latest.noarch)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
Importing GPG key 0x442DF0F8:
 Userid     : "PostgreSQL RPM Building Project <pgsql-pkg-yum@postgresql.org>"
 Fingerprint: 68c9 e2b9 1a37 d136 fe74 d176 1f16 d2e1 442d f0f8
 Package    : pgdg-redhat-repo-42.0-18.noarch (@/pgdg-redhat-repo-latest.noarch)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
Importing GPG key 0x442DF0F8:
 Userid     : "PostgreSQL RPM Building Project <pgsql-pkg-yum@postgresql.org>"
 Fingerprint: 68c9 e2b9 1a37 d136 fe74 d176 1f16 d2e1 442d f0f8
 Package    : pgdg-redhat-repo-42.0-18.noarch (@/pgdg-redhat-repo-latest.noarch)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
Importing GPG key 0x442DF0F8:
 Userid     : "PostgreSQL RPM Building Project <pgsql-pkg-yum@postgresql.org>"
 Fingerprint: 68c9 e2b9 1a37 d136 fe74 d176 1f16 d2e1 442d f0f8
 Package    : pgdg-redhat-repo-42.0-18.noarch (@/pgdg-redhat-repo-latest.noarch)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
Importing GPG key 0x442DF0F8:
 Userid     : "PostgreSQL RPM Building Project <pgsql-pkg-yum@postgresql.org>"
 Fingerprint: 68c9 e2b9 1a37 d136 fe74 d176 1f16 d2e1 442d f0f8
 Package    : pgdg-redhat-repo-42.0-18.noarch (@/pgdg-redhat-repo-latest.noarch)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
Removing intermediate container bc9a75b1b27d
 ---> 25ff652e95f8
Step 5/5 : RUN head /test
 ---> Running in b829c49f4f4d
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: mirror.crexio.com
 * extras: mirror.crexio.com
 * updates: centos.mirrors.proxad.net
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG

I will propose another solution to the issue by simply removing the detection of available packages and only relying on the list of installed packages using rpm -qa, in a separate PR.

@kit-ty-kate kit-ty-kate mentioned this pull request Aug 8, 2021
Merged
@kit-ty-kate
Copy link
Copy Markdown
Member

kit-ty-kate commented Aug 8, 2021

Alternative fix available in #4791. Feel free to comment there

@rjbou
Copy link
Copy Markdown
Collaborator

rjbou commented Aug 23, 2021

Closing in favor to #4791

@rjbou rjbou closed this Aug 23, 2021
@rjbou
Copy link
Copy Markdown
Collaborator

rjbou commented Aug 23, 2021

Thank you for the PR, and for highlighting the issue!

@rjbou rjbou mentioned this pull request Sep 20, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

KIND: BUG

Projects

None yet

Milestone

2.1.1

Development

Successfully merging this pull request may close these issues.

3 participants

@bikallem @kit-ty-kate @rjbou