Skip to content

cli: add filtering support for --list_* options#1367

Merged
jmartin-tech merged 14 commits intoNVIDIA:mainfrom
JosephDavisC:feat/list-filtering-issue-1362
Sep 30, 2025
Merged

cli: add filtering support for --list_* options#1367
jmartin-tech merged 14 commits intoNVIDIA:mainfrom
JosephDavisC:feat/list-filtering-issue-1362

Conversation

@JosephDavisC
Copy link
Copy Markdown
Contributor

Title

cli: add filtering support for --list_probes and --list_detectors (fix #1362)

Description

This PR adds filtering support to the --list_probes and --list_detectors CLI options.

  • Users can now filter by plugin family (e.g., dan) or by a specific plugin (e.g., dan.AntiDAN).
  • If no matches are found, the CLI will display a clear message (e.g., “No detectors match the provided 'detector_spec': misleading.Invalid”) and exit with a non-zero code.

This resolves issue #1362.

Verification

Steps to verify:

  • Run tests: pytest -q tests/test_cli_list_filtering.py (all pass).
  • Check filtering works for probes:
garak --list_probes -p dan
garak --list_probes -p dan,dan.AntiDAN
  • Check filtering works for detectors:
garak --list_detectors -d misleading
garak --list_detectors -d misleading.Invalid
  • Confirm invalid spec exits with error message.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 15, 2025

DCO Assistant Lite bot All contributors have signed the DCO ✍️ ✅

@JosephDavisC
Copy link
Copy Markdown
Contributor Author

I have read the DCO Document and I hereby sign the DCO

@JosephDavisC
Copy link
Copy Markdown
Contributor Author

recheck

github-actions bot added a commit that referenced this pull request Sep 15, 2025
@JosephDavisC JosephDavisC force-pushed the feat/list-filtering-issue-1362 branch from 4f85798 to 74187d5 Compare September 15, 2025 22:43
@leondz leondz requested review from jmartin-tech and leondz and removed request for jmartin-tech September 16, 2025 08:40
Copy link
Copy Markdown
Collaborator

@leondz leondz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. Would prefer to retain print_plugins functionality for probes & detectors. Can we factor this filtering up into that function, perhaps also covering other plugin types?

Copy link
Copy Markdown
Collaborator

@jmartin-tech jmartin-tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not the layer I would expect to find responsible for this work. All _config and cli argument parsing is preformed by cli.py and loaded into the runtime config state.

In fact all the logic required to determine what plugins will be activated by the config exists in cli.py where it calls _config.parse_plugin_spec().

A refactor of print_plugins() could possibly be a place to receive the filtered set. Depending on the how implemented I could see this refactoring signatures for other print_*() methods to improve code reuse.

@leondz leondz added the cli Command-line interface functions label Sep 16, 2025
@JosephDavisC
Copy link
Copy Markdown
Contributor Author

Hey there, just a heads-up, I accidentally clicked re-request review 😅. Still working on the changes, I’ll push updates soon and ping again when it’s ready!

JosephDavisC and others added 2 commits September 16, 2025 12:41
Co-authored-by: Leon Derczynski <leonderczynski@gmail.com>
Signed-off-by: Joseph Davis Chamdani <joseph.chamdani@gmail.com>
- Remove duplicate imports (typing.Iterable, List and sys)
- Remove unused import (typing)
- Organize imports above constants
- Address reviewer feedback from PR NVIDIA#1362
@JosephDavisC JosephDavisC requested a review from leondz September 17, 2025 06:50
@JosephDavisC
Copy link
Copy Markdown
Contributor Author

Just finished making all the requested fixes. Hope everything is good now! Let me know if there’s anything else I should tweak. Thanks for the feedback!

Copy link
Copy Markdown
Collaborator

@jmartin-tech jmartin-tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current revision still attempts to evaluate cli arguments in a location that should not be responsible for accessing them. See comments for further detail.

…y, and apply code formatting

- Move all CLI/config parsing and plugin selection logic to cli.py, keeping command.py focused on display logic only.
- Update print_plugins and related function signatures to use selected_plugins.
@JosephDavisC
Copy link
Copy Markdown
Contributor Author

I just pushed the changes and moved the CLI parsing into cli.py and updated command.py as suggested. Thanks so much for all the feedback and patience throughout this process. This is actually my first open-source PR, so I really appreciate the guidance and the time you’ve taken to help me improve it! 🙏

Copy link
Copy Markdown
Collaborator

@jmartin-tech jmartin-tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing shows this is really close to the desired function. I noticed there was some confusion about the desired output. I believe the comment original mentioning family inclusion was based on ensuring the changes did not impact the default display where all values that can validly be offered in a spec are returned with emoji entires to designate family or inactive plugins.

Co-authored-by: Jeffrey Martin <jmartin@Op3n4M3.dev>
Signed-off-by: Joseph Davis Chamdani <joseph.chamdani@gmail.com>
@JosephDavisC
Copy link
Copy Markdown
Contributor Author

Thanks for the suggestion! I've committed your recommended change as-is, and the code now matches your review.

@jmartin-tech jmartin-tech changed the title cli: add filtering support for --list_probes and --list_detectors (fix #1362) cli: add filtering support for --list_* options Sep 19, 2025
Signed-off-by: Jeffrey Martin <jemartin@nvidia.com>
Signed-off-by: Jeffrey Martin <jemartin@nvidia.com>
@JosephDavisC
Copy link
Copy Markdown
Contributor Author

Hi @jmartin-tech, I saw you made some changes to the code. Is there anything you need me to update or fix now, or should I just wait for review/merge?

@jmartin-tech
Copy link
Copy Markdown
Collaborator

@JosephDavisC just wait for now this will either land or a direct ask will be posted. The acceptance process is still ongoing.

Copy link
Copy Markdown
Collaborator

@leondz leondz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

almost there

Co-authored-by: Leon Derczynski <leonderczynski@gmail.com>
Signed-off-by: Joseph Davis Chamdani <joseph.chamdani@gmail.com>
@JosephDavisC JosephDavisC force-pushed the feat/list-filtering-issue-1362 branch from a6681ff to ef17758 Compare September 29, 2025 16:17
JosephDavisC and others added 2 commits September 29, 2025 09:25
Signed-off-by: Joseph Davis Chamdani <joseph.chamdani@gmail.com>
@JosephDavisC JosephDavisC requested a review from leondz September 29, 2025 16:33
@JosephDavisC
Copy link
Copy Markdown
Contributor Author

Hi there, I’ve added the CLI docs for filtering, so you should see the usage examples in the help output for --list_probes and --list_detectors. Sorry for all the commits, I accidentally pushed some extra files earlier instead of just command.py and cli.py. It should be sorted now! Let me know if anything else needs fixing. Thanks!

Copy link
Copy Markdown
Collaborator

@jmartin-tech jmartin-tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The help comments really only make sense when a --list options is passed, some ideas offered that I think are more clear about the impacts of usage and to focus the examples on when they are expected to have an impact on output.

JosephDavisC and others added 3 commits September 29, 2025 13:04
Co-authored-by: Jeffrey Martin <jmartin@Op3n4M3.dev>
Signed-off-by: Joseph Davis Chamdani <joseph.chamdani@gmail.com>
Co-authored-by: Jeffrey Martin <jmartin@Op3n4M3.dev>
Signed-off-by: Joseph Davis Chamdani <joseph.chamdani@gmail.com>
…etectors help texts

Signed-off-by: Joseph Davis Chamdani <joseph.chamdani@gmail.com>
@JosephDavisC
Copy link
Copy Markdown
Contributor Author

Reverted the help text change for --probes as suggested. Thanks for the feedback! Let me know if anything else needs fixing

@jmartin-tech jmartin-tech merged commit 589ae2e into NVIDIA:main Sep 30, 2025
15 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Sep 30, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

cli Command-line interface functions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

support helper command list filtering

3 participants