Skip to content

introduce dynamic filtering#1387

Merged
ehsandeep merged 6 commits intodevfrom
introduce_dynamic_output_filtering
Mar 26, 2025
Merged

introduce dynamic filtering#1387
ehsandeep merged 6 commits intodevfrom
introduce_dynamic_output_filtering

Conversation

@dogancanbakir
Copy link
Copy Markdown
Member

@dogancanbakir dogancanbakir commented Feb 15, 2025 •
edited by coderabbitai bot
Loading

Closes #1386

Summary by CodeRabbit

  • New Features

    • Introduced new command-line options to list all available output fields and to exclude specified fields. This allows users more control over the displayed JSON and CSV outputs.

  • Chores

    • Updated and added several dependencies to improve overall performance and reliability.

@dogancanbakir dogancanbakir self-assigned this Feb 15, 2025
@ehsandeep ehsandeep mentioned this pull request Feb 27, 2025
Closed
@CodeBeaverAI CodeBeaverAI mentioned this pull request Mar 3, 2025
Open
@erlapso erlapso mentioned this pull request Mar 4, 2025
Closed
ehsandeep
ehsandeep requested changes Mar 4, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@ehsandeep ehsandeep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

merge conflict

@dogancanbakir dogancanbakir marked this pull request as ready for review March 4, 2025 19:23
@dogancanbakir dogancanbakir requested a review from ehsandeep March 4, 2025 19:24
@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Mar 25, 2025 •
edited
Loading

Walkthrough

The pull request updates dependency versions in the project’s go.mod file while introducing new direct and indirect dependencies. It extends the configuration options by adding new fields to manage output formatting in the Options structure, and adjusts both JSON and CSV output methods to support dynamic filtering. The pull request also modifies internal flow in the Runner component to pass the dynamic filtering options during output generation.

Changes

Files Change Summary
go.mod Added new dependency github.com/projectdiscovery/katana v1.1.2; updated github.com/projectdiscovery/utils (v0.4.15 → v0.4.16) and github.com/stretchr/testify (v1.9.0 → v1.10.0); added several new indirect dependencies.
pkg/runner/options.go Added ListOutputFields bool and ExcludeОutputFields goflags.StringSlice to the Options struct; updated ParseOptions to include corresponding flags; added conditional block to list output fields and exit if requested.
pkg/runner/output.go Modified Result and jsonResult types to include additional fields; updated method signatures of JSON, CSVHeaders, and CSVFields to accept an excludedFields []string parameter; adjusted CSV writing functions to propagate the filtering parameter.
pkg/runner/runner.go Updated calls in onReceive and handleOutput to pass r.options.ExcludeОutputFields to JSON and CSV output functions, ensuring the dynamic exclusion of specified fields during output generation.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant Runner
    participant Options
    participant Result
    participant Formatter

    User->>Runner: Execute scan with CLI flags
    Runner->>Options: Parse flags (ListOutputFields, ExcludeOutputFields)
    Options-->>Runner: Provide parsed options
    alt Listing fields requested
        Runner->>Result: Retrieve struct fields via structs.GetStructFields
        Runner->>User: Display available output fields
        Runner-->>User: Exit program
    else Generate output
        Runner->>Result: Call JSON(excludedFields) / CSV(excludedFields)
        Result->>Formatter: Filter fields using excludedFields
        Formatter-->>Runner: Return formatted output
        Runner->>User: Present the output
    end
Loading

Assessment against linked issues

Objective Addressed Explanation
Introduce dynamic filtering (#1386) ✅

Poem

I'm a little rabbit, swift and keen,
Hopping through the code, fresh and clean,
Excluding fields with flair and delight,
Listing outputs that shine so bright,
Carrots and code—everything's just right!
🥕🐰

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d4d8e92 and 27c5905.

⛔ Files ignored due to path filters (1)
  • go.sum is excluded by !**/*.sum
📒 Files selected for processing (4)
  • go.mod (6 hunks)
  • pkg/runner/options.go (5 hunks)
  • pkg/runner/output.go (7 hunks)
  • pkg/runner/runner.go (3 hunks)
🧰 Additional context used
🧬 Code Definitions (2)
pkg/runner/runner.go (1)
pkg/runner/output.go (6)
  • r (46-65)
  • r (72-82)
  • r (84-104)
  • writeCSVHeaders (180-191)
  • writeCSVRow (193-203)
  • WriteCsvOutput (155-178)
pkg/runner/output.go (4)
pkg/protocol/protocol.go (4)
  • Protocol (7-7)
  • p (15-26)
  • p (28-30)
  • p (32-52)
pkg/runner/runner.go (17)
  • r (171-261)
  • r (264-637)
  • r (639-653)
  • r (655-666)
  • r (668-691)
  • r (693-699)
  • r (702-724)
  • r (727-737)
  • r (739-748)
  • r (750-752)
  • r (754-782)
  • r (784-786)
  • r (788-790)
  • r (792-815)
  • r (818-831)
  • r (833-863)
  • err (948-948)
pkg/result/results.go (1)
  • Result (21-26)
pkg/port/port.go (2)
  • Port (9-15)
  • p (17-19)
🔇 Additional comments (18)
pkg/runner/options.go (4)

14-14: Added a new import for struct field inspection.

The structs package is now imported to support the new dynamic field filtering functionality.

46-47: New fields added for dynamic output filtering.

Two new configuration options have been added:

  • ListOutputFields to allow listing available output fields
  • ExcludeОutputFields to exclude specific fields from output

These additions support the new dynamic filtering functionality.

149-150: Added CLI flags for the new output field options.

This adds CLI support for the new functionality:

  • -lof/--list-output-fields flag to list available output fields
  • -eof/--exclude-output-fields flag to exclude specific fields from output

The flags correctly use the appropriate types and provide descriptive help text.

222-231: Added implementation for listing available output fields.

This block handles the --list-output-fields flag by:

  1. Retrieving the fields from the Result struct
  2. Displaying them to the user
  3. Exiting immediately afterward

This provides a useful way for users to discover which fields they can exclude.

pkg/runner/runner.go (4)

232-232: Updated JSON output to support field exclusion.

The JSON method now receives the field exclusion list from the options, enabling dynamic filtering of JSON output.

239-240: Updated CSV output functions to support field exclusion.

Both CSV header and row generation now receive the field exclusion list from options, ensuring consistent field filtering across the entire CSV output.

Also applies to: 242-242

1022-1022: Updated file output for CSV to support field exclusion.

The WriteCsvOutput function call now includes the field exclusion list, ensuring that file output respects the same field exclusion rules as console output.

1084-1084: Consistent field exclusion in another CSV output path.

This ensures that field exclusion is also applied in the IP-only results case, maintaining consistency throughout all CSV output paths.

go.mod (3)

28-28: Updated utils dependency version.

The github.com/projectdiscovery/utils package has been updated from v0.4.15 to v0.4.16, which likely contains the enhanced structs package functionality needed for dynamic field filtering.

30-30:

Details

❓ Verification inconclusive

Updated dependencies and added new indirect dependencies.

Several dependencies have been updated, including:

  • github.com/stretchr/testify to v1.10.0
  • Multiple new indirect dependencies added

This is likely a result of updating the direct dependencies. These changes should be verified to ensure they don't introduce any compatibility issues.

Let's check if there are any major changes in these dependencies:

Also applies to: 39-107

🏁 Script executed:

#!/bin/bash
# Check the changelog for utils to see what changed between v0.4.15 and v0.4.16
curl -s https://api.github.com/repos/projectdiscovery/utils/releases | grep -A 10 "v0.4.16"

# Check if the FilterStruct function exists in the utils/structs package
curl -s https://api.github.com/repos/projectdiscovery/utils/contents/structs?ref=v0.4.16 | grep -A 2 -B 2 "FilterStruct"

Length of output: 1812

Manual Verification Required: Confirm Dependency Compatibility and Function Existence

  • The go.mod update shows that github.com/stretchr/testify is now at v1.10.0 along with several new indirect dependencies.
  • The changelog check for projectdiscovery/utils v0.4.16 confirms its release, but details are minimal.
  • The automated check for the existence of FilterStruct in the utils/structs package returned no output. This may indicate that the function was removed, renamed, or relocated.
  • Action: Please manually verify that the absence of output for FilterStruct is expected and that the dependency update does not introduce compatibility issues.

22-22:

Details

❓ Verification inconclusive

Added new dependency: katana.

Adding github.com/projectdiscovery/katana v1.1.2 as a new dependency.

The PR is introducing dynamic filtering functionality and I don't see direct usage of the katana package in the changed files. Let's verify if this dependency is actually needed:

🏁 Script executed:

#!/bin/bash
# Check if katana is used in the codebase
grep -r "\"github.com/projectdiscovery/katana" --include="*.go" .

Length of output: 66

Double-check the necessity of the added Katana dependency
Your PR introduced github.com/projectdiscovery/katana v1.1.2 in the go.mod file to support dynamic filtering functionality. However, a grep search for "github.com/projectdiscovery/katana" across Go files did not return any direct references. It’s possible this dependency is used indirectly or via side effects that a simple grep might not capture. Could you please manually verify whether the Katana package is indeed required (for example, by checking for any dynamic or reflective usage) or if it can be removed to avoid unnecessary maintenance overhead?

pkg/runner/output.go (7)

20-20: Added structs package import.

The structs package from projectdiscovery/utils is now imported to support the dynamic field filtering functionality.

35-44: Updated jsonResult struct to align with Result struct.

The jsonResult struct has been extended to include fields from the Result struct but with appropriate JSON tags. This ensures proper mapping between the internal representation and JSON output.

46-65: Enhanced JSON method to support field exclusion.

The JSON method now accepts an excludedFields parameter and uses structs.FilterStruct to filter out specified fields before marshaling to JSON. This implements the core dynamic filtering functionality for JSON output.

72-82: Updated CSVHeaders method to support field exclusion.

The CSVHeaders method now accepts an excludedFields parameter and excludes the specified fields from the CSV headers. This ensures consistent field exclusion across different output formats.

84-104: Enhanced CSVFields method to support field exclusion.

The CSVFields method now accepts an excludedFields parameter and uses structs.FilterStruct to filter out specified fields before generating CSV field values. This implements dynamic filtering functionality for CSV output.

155-175: Updated CSV output functions to support field exclusion.

The WriteCsvOutput, writeCSVHeaders, and writeCSVRow functions now accept an excludedFields parameter and pass it to their respective methods. This ensures field exclusion is applied consistently throughout the CSV output generation process.

180-203: Updated CSV helper functions to support field exclusion.

Both writeCSVHeaders and writeCSVRow now pass the excludedFields parameter to their respective methods, completing the implementation of dynamic filtering in CSV output. This allows for consistent exclusion of fields across different parts of the codebase.

✨ Finishing Touches
  • 📝 Generate Docstrings
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai plan to trigger planning for file edits and PR creation.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@ehsandeep ehsandeep merged commit e790a6a into dev Mar 26, 2025
12 checks passed
@ehsandeep ehsandeep deleted the introduce_dynamic_output_filtering branch March 26, 2025 09:19
@BrewTestBot BrewTestBot mentioned this pull request Jul 14, 2025
Merged
@dogancanbakir dogancanbakir mentioned this pull request Jul 24, 2025
Closed
@dogancanbakir dogancanbakir linked an issue Jul 24, 2025 that may be closed by this pull request
Support field filtering in CSV output (-f option for -csv) #1371
Closed
@hanXen hanXen mentioned this pull request Aug 27, 2025
Closed
@dogancanbakir dogancanbakir mentioned this pull request Sep 1, 2025
Closed
@hanXen hanXen mentioned this pull request Sep 3, 2025
Merged
@coderabbitai coderabbitai bot mentioned this pull request Sep 24, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ehsandeep ehsandeep ehsandeep approved these changes

Assignees

@dogancanbakir dogancanbakir

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

introduce dynamic filtering Support field filtering in CSV output (-f option for -csv)

2 participants

@dogancanbakir @ehsandeep