Skip to content

Snakemake output prints sensitive information with storage plugin #3087

@MattMonk

Description

@MattMonk

Snakemake version
8.20.3

Describe the bug
When using a storage plugin e.g. snakemake-storage-plugin-xrootd the user can provide a username/password or encode an authentication token in the file path. When snakemake then prints the DAG or error messages the full path, including the token/username/password whatever, is printed in the terminal exposing potentially sensitive information.

It would be great to have a way to be able to ensure that only ever a censored form of the URL is printed (but still allow for the full, uncensored URL to be passed as input/output to the shell part).

Minimal example

storage:
    provider="xrootd",
    host="eoslhcb.cern.ch",
    username="my_username"

rule test_rule:
    input: storage("root://eos/my_file.root")
    output: "test.flag"
    shell: "touch {output}"

It will correctly fail due to the file not existing and incorrect credentials:

$ snakemake -c1 test.flag -n --storage-xrootd-password="my_password"
A password has been specified -- it will be printed in plaintext when Snakemake displays the inputs/outputs of jobs! Only use this option in trusted environments.
Building DAG of jobs...
Error checking existence of root://my_username:****@eoslhcb.cern.ch:1094//eos/my_file.root on XRootD: [ERROR] Server responded with an error: [3010] Unable to give access - user access restricted - unauthorized identity used ; Permission denied
, attempt 1/3 failed - retrying in 3 seconds...
Unrecoverable error, no more retries
WorkflowError:
Failed to check existence of root://my_username:my_password@eoslhcb.cern.ch:1094//eos/my_file.root
XRootDFatalException: Error checking existence of root://my_username:****@eoslhcb.cern.ch:1094//eos/my_file.root on XRootD: [ERROR] Server responded with an error: [3010] Unable to give access - user access restricted - unauthorized identity used ; Permission denied

where you can see (more than once) the password would be printed in plain text.

Additional context
The snakemake-storage-plugin-xrootd has this postprocess_query method which can be changed to print out a censored URL instead but, as far as I can tell, this then prevents the plugin from working correctly because the censored URL is used as the "file" rather than actual full URL.

Our particular use-case for this is running CI/CD tests where the authentication is handeled by appending a token into the URL parameters and we would prefer not to be exposing that in the CI logs if we can avoid it!

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    Status

    In progress

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions