fix(output): ensure -eof completely omit fields from JSON output#1546
fix(output): ensure -eof completely omit fields from JSON output#1546Mzack9999 merged 2 commits intoprojectdiscovery:devfrom
-eof completely omit fields from JSON output#1546Conversation
WalkthroughDependency updates in go.mod (including a replace for projectdiscovery/utils). pkg/runner/output.go switches JSON exclusion handling to a map-based path when exclusions are present and renames internal jsonResult field Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant Runner
participant JSON as jsonResult.JSON()
participant Filter as FilterStructToMap
participant Marshal as json.Marshal
Runner->>JSON: construct jsonResult (data)
alt no excluded fields
JSON->>Marshal: Marshal(struct data)
Marshal-->>Runner: bytes / error
else excluded fields present
JSON->>Filter: FilterStructToMap(struct data, exclusions)
alt filter success
JSON->>Marshal: Marshal(filtered map)
Marshal-->>Runner: bytes / error
else filter error
JSON-->>Runner: error
end
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Assessment against linked issues
Out-of-scope changes
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 💡 Knowledge Base configuration:
You can enable these sources in your CodeRabbit configuration. ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (2)
go.mod(4 hunks)pkg/runner/output.go(3 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
pkg/runner/output.go (1)
pkg/port/port.go (1)
Port(10-17)
🔇 Additional comments (5)
go.mod (2)
23-25: Dependency Updates Approved: Patch-Level Bumps OnlyThe following ProjectDiscovery modules have been bumped in go.mod:
Direct dependencies
- github.com/projectdiscovery/networkpolicy → v0.1.20
- github.com/projectdiscovery/retryablehttp-go → v1.0.120
Indirect dependencies
- github.com/projectdiscovery/fastdialer → v0.4.6
- github.com/projectdiscovery/hmap → v0.0.92
- github.com/projectdiscovery/retryabledns → v1.0.105
These are all low-risk, patch-level updates. No breaking changes are expected. Please ensure you’ve skimmed the corresponding release notes and run the full integration test suite to catch any edge-case regressions before merging.
27-27: The script will check that the upstreamstructs.FilterStructToMapexists onmainbut that nov0.4.24release is yet published (latest isv0.4.23), indicating we must hold the temporary replace until the officialv0.4.24tag is cut. Once confirmed, update the review to:
Note that
v0.4.24is not yet released (latestv0.4.23).Keep the
replacein place until the release is cut.After the
utilsrepo publishesv0.4.24withFilterStructToMap, updatego.mod:- github.com/projectdiscovery/utils v0.4.22 + github.com/projectdiscovery/utils v0.4.24and drop the
replace.This ensures consumers aren’t pinned to a non-canonical fork and can smoothly upgrade once the official module is available.
pkg/runner/output.go (3)
36-44: jsonResult shape aligns with external schema; port rename reflected.Tags are correct; keeping port/protocol/tls non-omitempty ensures presence unless explicitly excluded.
59-61: Map-based filtering and test coverage verificationThe switch to a map for excluding JSON keys correctly omits unwanted fields, but marshaling a map may emit keys in a nondeterministic order. Additionally, there’s no evidence of existing tests covering the
JSON()method with variousExcludeOutputFieldscombinations (only a single call atpkg/runner/runner.go:232was found, and no corresponding tests were located).• Potential nondeterministic key order when using
json.Marshalon a map.
• No tests detected fordata.JSON(r.options.ExcludeOutputFields)edge cases—coverage should be confirmed or added.
• Optional refactor if downstream tooling/tests require stable ordering:
– Construct and marshal an ordered struct with fields in a fixed sequence, or
– Manually encode JSON by iterating over a sorted slice of keys.
• Otherwise, update or add tests to accept any key order (order-agnostic assertions).Please verify test coverage for excluded-fields scenarios and adjust accordingly.
55-55: Migration complete – no stalePortNumberreferences remain.
- Searched for
PortNumber(exact match) across the codebase; no occurrences found.- Checked for lowercase
portNumberusages; the only hits are local variables inpkg/runner/ports.go(lines 180–184), which are unrelated to the struct field migration.- Verified there are no lingering JSON or YAML tags referencing
portNumber.All references to the old field name have been removed.
| } | ||
| for _, p := range ports { | ||
| data.PortNumber = p.Port | ||
| data.Port = p.Port |
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
WriteJSONOutput still ignores -eof; confirm call-site filtering or plumb exclusions.
If no upstream layer filters before Encode, JSON streaming output will not honor -eof.
If needed, add a variant that accepts excludedFields and reuses the map-filter path to ensure consistency between r.JSON() and streaming.
🏁 Script executed:
#!/bin/bash
# Find call sites and see if excluded fields are applied on the streaming path
rg -nP '\bWriteJSONOutput\s*\(' -C3
rg -nP 'excludedFields|eof' -n -C2Length of output: 5991
Ensure JSON streaming honors -eof exclude-output-fields
- The
WriteJSONOutputfunction (pkg/runner/output.go:134–137) does not accept anexcludedFieldsslice, so it never applies the-eoffilter on its output. - All call-sites in pkg/runner/runner.go (lines 1027 and 1089) invoke
WriteJSONOutputwithout filtering, unlikeWriteCsvOutputwhich correctly takes and appliesr.options.ExcludeОutputFields.
Recommended change:
- Extend
WriteJSONOutput(and its helper types) to acceptexcludedFields []stringand, before callingjson.NewEncoder, filter the data (e.g. viastructs.FilterStructToMap) as is done in(*Result).JSON(). This will ensure parity between JSON and CSV output formats.
🤖 Prompt for AI Agents
In pkg/runner/output.go around lines 134–147, WriteJSONOutput currently doesn't
accept excludedFields so the -eof exclude-output-fields filter is never applied;
change WriteJSONOutput signature to accept excludedFields []string (and update
any helper types/functions in this file) and, before creating json.NewEncoder,
convert or filter the value to a map using the same filtering used by
(*Result).JSON() (e.g. call structs.FilterStructToMap(value, excludedFields) or
equivalent) so excluded fields are removed, then pass the filtered map to the
encoder; finally update the two callers in pkg/runner/runner.go (around lines
1027 and 1089) to pass r.options.ExcludeOutputFields to the updated
WriteJSONOutput.
Resolves #1545
Depends on projectdiscovery/utils#674
Summary
This PR fixes a bug where the
-eofflag set fields to their zero-value (e.g.,"timestamp": "0001-01-01...") instead of omitting them from the JSON output.This is resolved by using a new
structs.FilterStructToMaputility function, which is being introduced in the companionprojectdiscovery/utilsPR.Solution
To ensure excluded fields are truly omitted,
JSON()inpkg/runner/output.gonow callsstructs.FilterStructToMap. This helper converts the result struct into amap[string]anyand removes entries for excludedjsonkeys (respectingomitempty) beforejson.Marshalis called, ensuring fields specified by-eofare absent from the final JSON output.Additionally, this fix includes a necessary internal change to ensure
-eof portworks as expected. The internal DTO fieldPortNumberwas renamed toPortto align with the user-facing field list from-lof. This is a safe modification that does not change the final JSON output key, which remains"port".Development & Testing Notes
To test this PR before the dependency in
projectdiscovery/utilsis merged, thego.modfile was temporarily modified with areplacedirective. This directive should be removed once the dependency PR is merged.Summary by CodeRabbit
Refactor
Chores