tedge-p11-server reports "Failed to find a private key" #3766
Description
Describe the bug
We run tedge-p11-server (1.5.1) on the host and tedge (1.5.1) as podman container).
tedge-p11-server is started and configured with a pin:
[device.cryptoki]
pin = "1111111"
module_path = "/usr/lib/x86_64-linux-gnu/pkcs11/opensc-pkcs11.so"
tedge is running in a podman container with the following start script:
podman run -d \
--userns keep-id \
--name tedge \
--network tedge \
-p "1883:1883" \
-p "8000:8000" \
-p "8001:8001" \
-v "tedge-data:/data/tedge" \
-v "/run/tedge-p11-server:/run/tedge-p11-server" \
-v "/mnt/data/tedge/config:/local-conf" \
-v "/run/user/$UID/podman/podman.sock:/var/run/docker.sock:rw" \
-v "/mnt/data/home/tedge/mosquitto.conf:/etc/mosquitto/mosquitto.conf:rw" \
-e TEDGE_C8Y_OPERATIONS_AUTO_LOG_UPLOAD=always \
-e TEDGE_MQTT_BRIDGE_BUILT_IN=true \
-e TEDGE_DEVICE_CRYPTOKI_MODE=socket \
-e TEDGE_DEVICE_CERT_PATH=/local-conf/tedge-certificate.pem \
-e S6_CMD_WAIT_FOR_SERVICES_MAXTIME=$S6_CMD_WAIT_FOR_SERVICES_MAXTIME \
--env-file /mnt/data/tedge/config/wec-iot.cfg \
<OUR_REGISTRY>/thinedge:stable
Before phase 1 no cert is issued in /mnt/data/tedge/config, so tedge tries to reconnect.
The sequence of the following actions is as follows:
- phase 1:
- tedge-p11-server running, configured with user PIN
- tedge-container running , new certificate arrived, after 30 seconds restarted manually
- phase 2:
- tedge-p11-server unchanged
- tedge-container unchanged after previous restart
- phase 3:
- tedge-p11-server restarted
- tedge-container restarted
In the logs of the tedge-p11-server the following error is reported:
tedge_p11_server::server: Incoming request failed: Failed to find a signing key
Aug 21 11:25:08 tedge-device tedge-p11-server[463]: Caused by:
Aug 21 11:25:08 tedge-device tedge-p11-server[463]: Failed to find a private key
Aug 21 11:25:09 tedge-device tedge-p11-server[463]: 2025-08-21T09:25:09.047462Z INFO tedge_p11_server::server: Incoming request successful
Expected behavior
Is is expected that tedge can use the private key without restarting the tedge-p11-server.
No configuration relevant in tedge-p11-server in all phases was schanged.
Environment (please complete the following information):
- OS: Debian 6.1.147-1
- System-Architecture: Linux 100666-1-123456-nc-hmi 6.1.0-38-amd64
- thin-edge.io version: 1.5.1
- using tedge version 1.5.1
- using device certificates, private key saved in HSM (Nitrokey)
- tedge runs in container (podman) and accesses Nitrokey through tedge-p11-server (1.5.1)
Additional context
************************** PHASE 1 **************************
Aug 21 09:31:01 tedge-device systemd[1]: Started tedge-p11-server.service - tedge-p11-server.
Aug 21 09:31:01 tedge-device tedge-p11-server[463]: 2025-08-21T07:31:01.847979Z INFO tedge_p11_server: Using cryptoki configuration cryptoki_config=CryptokiConfigDirect { module_path: "/usr/lib/x86_64-linux-gnu/pkcs11/opensc-pkcs11.so", pin: "[REDACTED]", uri: None }
Aug 21 09:31:01 tedge-device tedge-p11-server[463]: 2025-08-21T07:31:01.848094Z INFO tedge_p11_server: Server listening listener=Some("/run/tedge-p11-server/tedge-p11-server.sock")
Aug 21 11:14:10 tedge-device tedge-p11-server[463]: 2025-08-21T09:14:10.772863Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:14:11 tedge-device tedge-p11-server[463]: 2025-08-21T09:14:11.212337Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:14:11 tedge-device tedge-p11-server[463]: 2025-08-21T09:14:11.829115Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:14:42 tedge-device tedge-p11-server[463]: 2025-08-21T09:14:42.882564Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:14:43 tedge-device tedge-p11-server[463]: 2025-08-21T09:14:43.498157Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:16:09 tedge-device tedge-p11-server[463]: 2025-08-21T09:16:09.240861Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:16:09 tedge-device tedge-p11-server[463]: 2025-08-21T09:16:09.857875Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:19:07 tedge-device tedge-p11-server[463]: 2025-08-21T09:19:07.555378Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:19:08 tedge-device tedge-p11-server[463]: 2025-08-21T09:19:08.173431Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:21:42 tedge-device tedge-p11-server[463]: 2025-08-21T09:21:42.265267Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:21:42 tedge-device tedge-p11-server[463]: 2025-08-21T09:21:42.884941Z INFO tedge_p11_server::server: Incoming request successful
************************** PHASE 2 **************************
Aug 21 11:25:07 tedge-device tedge-p11-server[463]: 2025-08-21T09:25:07.691890Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:25:08 tedge-device tedge-p11-server[463]: 2025-08-21T09:25:08.008004Z ERROR tedge_p11_server::server: Incoming request failed: Failed to find a signing key
Aug 21 11:25:08 tedge-device tedge-p11-server[463]: Caused by:
Aug 21 11:25:08 tedge-device tedge-p11-server[463]: Failed to find a private key
Aug 21 11:25:08 tedge-device tedge-p11-server[463]: 2025-08-21T09:25:08.363687Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:25:08 tedge-device tedge-p11-server[463]: 2025-08-21T09:25:08.661738Z ERROR tedge_p11_server::server: Incoming request failed: Failed to find a signing key
Aug 21 11:25:08 tedge-device tedge-p11-server[463]: Caused by:
Aug 21 11:25:08 tedge-device tedge-p11-server[463]: Failed to find a private key
Aug 21 11:25:09 tedge-device tedge-p11-server[463]: 2025-08-21T09:25:09.047462Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:25:09 tedge-device tedge-p11-server[463]: 2025-08-21T09:25:09.350037Z ERROR tedge_p11_server::server: Incoming request failed: Failed to find a signing key
Aug 21 11:25:09 tedge-device tedge-p11-server[463]: Caused by:
Aug 21 11:25:09 tedge-device tedge-p11-server[463]: Failed to find a private key
Aug 21 11:25:09 tedge-device tedge-p11-server[463]: 2025-08-21T09:25:09.725829Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:25:10 tedge-device tedge-p11-server[463]: 2025-08-21T09:25:10.019798Z ERROR tedge_p11_server::server: Incoming request failed: Failed to find a signing key
Aug 21 11:25:10 tedge-device tedge-p11-server[463]: Caused by:
Aug 21 11:25:10 tedge-device tedge-p11-server[463]: Failed to find a private key
Aug 21 11:25:10 tedge-device tedge-p11-server[463]: 2025-08-21T09:25:10.350413Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:25:10 tedge-device tedge-p11-server[463]: 2025-08-21T09:25:10.643925Z ERROR tedge_p11_server::server: Incoming request failed: Failed to find a signing key
Aug 21 11:25:10 tedge-device tedge-p11-server[463]: Caused by:
Aug 21 11:25:10 tedge-device tedge-p11-server[463]: Failed to find a private key
Aug 21 11:25:11 tedge-device tedge-p11-server[463]: 2025-08-21T09:25:11.092034Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:25:11 tedge-device tedge-p11-server[463]: 2025-08-21T09:25:11.553501Z ERROR tedge_p11_server::server: Incoming request failed: Failed to find a signing key
Aug 21 11:25:11 tedge-device tedge-p11-server[463]: Caused by:
Aug 21 11:25:11 tedge-device tedge-p11-server[463]: Failed to find a private key
************************** PHASE 3 **************************
Aug 21 11:26:05 tedge-device systemd[1]: Stopping tedge-p11-server.service - tedge-p11-server...
Aug 21 11:26:05 tedge-device systemd[1]: tedge-p11-server.service: Deactivated successfully.
Aug 21 11:26:05 tedge-device systemd[1]: Stopped tedge-p11-server.service - tedge-p11-server.
Aug 21 11:26:05 tedge-device systemd[1]: Started tedge-p11-server.service - tedge-p11-server.
Aug 21 11:26:05 tedge-device tedge-p11-server[94391]: 2025-08-21T09:26:05.820181Z INFO tedge_p11_server: Using cryptoki configuration cryptoki_config=CryptokiConfigDirect { module_path: "/usr/lib/x86_64-linux-gnu/pkcs11/opensc-pkcs11.so", pin: "[REDACTED]", uri: None }
Aug 21 11:26:05 tedge-device tedge-p11-server[94391]: 2025-08-21T09:26:05.820236Z INFO tedge_p11_server: Server listening listener=Some("/run/tedge-p11-server/tedge-p11-server.sock")
Aug 21 11:26:31 tedge-device tedge-p11-server[94391]: 2025-08-21T09:26:31.120645Z WARN tedge_p11_server::pkcs11: Multiple keys were found. If the wrong one was chosen, please use a URI that uniquely identifies a key.
Aug 21 11:30:36 tedge-device tedge-p11-server[94391]: 2025-08-21T09:30:36.035687Z WARN choose_scheme: tedge_p11_server::pkcs11: Multiple keys were found. If the wrong one was chosen, please use a URI that uniquely identifies a key.
Aug 21 11:30:36 tedge-device tedge-p11-server[94391]: 2025-08-21T09:30:36.078132Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:30:50 tedge-device tedge-p11-server[94391]: 2025-08-21T09:30:50.579731Z WARN choose_scheme: tedge_p11_server::pkcs11: Multiple keys were found. If the wrong one was chosen, please use a URI that uniquely identifies a key.
Aug 21 11:30:50 tedge-device tedge-p11-server[94391]: 2025-08-21T09:30:50.622041Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:30:50 tedge-device tedge-p11-server[94391]: 2025-08-21T09:30:50.915202Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:30:51 tedge-device tedge-p11-server[94391]: 2025-08-21T09:30:51.402524Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:30:52 tedge-device tedge-p11-server[94391]: 2025-08-21T09:30:52.081634Z WARN choose_scheme: tedge_p11_server::pkcs11: Multiple keys were found. If the wrong one was chosen, please use a URI that uniquely identifies a key.
Aug 21 11:30:52 tedge-device tedge-p11-server[94391]: 2025-08-21T09:30:52.123425Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:30:52 tedge-device tedge-p11-server[94391]: 2025-08-21T09:30:52.628906Z INFO tedge_p11_server::server: Incoming request successful
Aug 21 11:30:53 tedge-device tedge-p11-server[94391]: 2025-08-21T09:30:53.113939Z INFO tedge_p11_server::server: Incoming request successful