fix(output): apply -eof filter to JSON file output#1548
fix(output): apply -eof filter to JSON file output#1548Mzack9999 merged 5 commits intoprojectdiscovery:devfrom
-eof filter to JSON file output#1548Conversation
WalkthroughAdds support for excluding specified output fields for file outputs: fixes a typo in the Options struct name, extends WriteJSONOutput/WriteCsvOutput signatures to accept excludedFields, updates runner call sites and tests to pass the new parameter, and removes a go.mod replace directive. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor CLI
participant Runner
participant Output as WriteJSONOutput
participant Result
participant File as Writer
CLI->>Runner: start with -j -eof <fields> -o <file>
Runner->>Output: WriteJSONOutput(host, ip, ports, ..., excludedFields, writer)
Note right of Output: create Result{Host,IP,TimeStamp,(+CDN fields if set)}
loop per port
Output->>Result: set Port, Protocol, TLS
Output->>Result: call Result.JSON(excludedFields)
Result-->>Output: JSON bytes (filtered)
Output->>File: write JSON line\n
end
File-->>Runner: flush/close or error
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
pkg/runner/output.go (1)
133-161: Handle alive-hosts (no ports) and buffer writes for performance.Currently nothing is written when ports is nil/empty. Also, buffering can reduce syscall overhead for large outputs.
Apply this diff:
-func WriteJSONOutput(host, ip string, ports []*port.Port, outputCDN bool, isCdn bool, cdnName string, excludedFields []string, writer io.Writer) error { - result := &Result{ +func WriteJSONOutput(host, ip string, ports []*port.Port, outputCDN bool, isCdn bool, cdnName string, excludedFields []string, writer io.Writer) error { + bw := bufio.NewWriter(writer) + result := &Result{ Host: host, IP: ip, TimeStamp: time.Now().UTC(), } if outputCDN { result.IsCDNIP = isCdn result.CDNName = cdnName } - - for _, p := range ports { + // Alive-host output (no ports) + if len(ports) == 0 { + b, err := result.JSON(excludedFields) + if err != nil { + return err + } + if _, err := bw.Write(append(b, '\n')); err != nil { + _ = bw.Flush() + return err + } + return bw.Flush() + } + + for _, p := range ports { result.Port = p.Port result.Protocol = p.Protocol.String() //nolint result.TLS = p.TLS - - b, err := result.JSON(excludedFields) + b, err := result.JSON(excludedFields) if err != nil { - return err + _ = bw.Flush() + return err } - - if _, err := writer.Write(append(b, '\n')); err != nil { - return err + if _, err := bw.Write(append(b, '\n')); err != nil { + _ = bw.Flush() + return err } } - return nil + return bw.Flush() }
🧹 Nitpick comments (3)
pkg/runner/output_test.go (2)
35-35: Add a test that asserts -eof filtering is applied to file JSON output.Current test only checks line count. Add coverage to ensure excluded fields are actually omitted in file output as per the PR goal.
Apply this diff to extend the test:
func TestWriteJSONOutput(t *testing.T) { host := "localhost" ip := "127.0.0.1" ports := []*port.Port{ {Port: 80, Protocol: protocol.TCP}, {Port: 8080, Protocol: protocol.TCP}, } var s string buf := bytes.NewBufferString(s) - assert.Nil(t, WriteJSONOutput(host, ip, ports, true, false, "", nil, buf)) + assert.Nil(t, WriteJSONOutput(host, ip, ports, true, false, "", nil, buf)) assert.Equal(t, 3, len(strings.Split(buf.String(), "\n"))) + + // Verify excluded fields are omitted in file output + buf2 := bytes.NewBufferString("") + excluded := []string{"timestamp", "protocol", "tls"} + assert.Nil(t, WriteJSONOutput(host, ip, ports, true, false, "", excluded, buf2)) + out := buf2.String() + assert.NotContains(t, out, `"timestamp":`) + assert.NotContains(t, out, `"protocol":`) + assert.NotContains(t, out, `"tls":`) }
35-35: Clarify behavior for alive-hosts (ports == nil) JSON file output and test it.WriteJSONOutput currently emits nothing when ports is nil. If alive-only JSON lines are expected, add a focused test and handle the case in WriteJSONOutput.
Proposed additional test:
+func TestWriteJSONOutput_NoPorts_AliveHost(t *testing.T) { + host := "alive.local" + ip := "127.0.0.2" + var ports []*port.Port // nil + buf := bytes.NewBuffer(nil) + err := WriteJSONOutput(host, ip, ports, false, false, "", []string{"port","protocol","tls"}, buf) + assert.NoError(t, err) + lines := strings.Split(strings.TrimSpace(buf.String()), "\n") + // Expect exactly one JSON line for the alive host + assert.Equal(t, 1, len(lines)) + assert.Contains(t, lines[0], `"ip":"127.0.0.2"`) + assert.NotContains(t, lines[0], `"port"`) + assert.NotContains(t, lines[0], `"protocol"`) + assert.NotContains(t, lines[0], `"tls"`) +}pkg/runner/output.go (1)
35-44: Consider omitting port/protocol/tls when absent.If you adopt alive-host JSON lines, make these fields omittable to avoid emitting port:0, empty protocol, tls:false for host-only records.
Apply this diff:
type jsonResult struct { Host string `json:"host,omitempty" csv:"host"` IP string `json:"ip,omitempty" csv:"ip"` IsCDNIP bool `json:"cdn,omitempty" csv:"cdn"` CDNName string `json:"cdn-name,omitempty" csv:"cdn-name"` TimeStamp time.Time `json:"timestamp,omitempty" csv:"timestamp"` - Port int `json:"port"` - Protocol string `json:"protocol"` - TLS bool `json:"tls"` + Port int `json:"port,omitempty"` + Protocol string `json:"protocol,omitempty"` + TLS bool `json:"tls,omitempty"` }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (4)
go.mod(1 hunks)pkg/runner/output.go(1 hunks)pkg/runner/output_test.go(1 hunks)pkg/runner/runner.go(2 hunks)
🧰 Additional context used
🧬 Code graph analysis (3)
pkg/runner/output.go (2)
pkg/port/port.go (1)
Port(10-17)pkg/result/results.go (1)
Result(21-26)
pkg/runner/runner.go (1)
pkg/runner/output.go (1)
WriteJSONOutput(134-161)
pkg/runner/output_test.go (1)
pkg/runner/output.go (1)
WriteJSONOutput(134-161)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (6)
- GitHub Check: release-test-mac
- GitHub Check: build-mac
- GitHub Check: build-windows
- GitHub Check: build-linux
- GitHub Check: release-test-windows
- GitHub Check: Functional Test (ubuntu-latest)
🔇 Additional comments (3)
go.mod (1)
27-27: Update to utils pseudo-version looks correct; ensure go.sum and CI are green.Pinning to a commit to use FilterStructToMap makes sense. Please confirm go.sum is updated and that the new utils API is available across all platforms in CI.
pkg/runner/runner.go (1)
1089-1089: Passing nil ports produces no JSON lines; confirm intended behavior.With the new WriteJSONOutput loop, nil/empty ports yield no output for alive-only hosts in the HasIPS branch. If a host-level JSON line is desired, handle this case in WriteJSONOutput (see suggested change in pkg/runner/output.go).
pkg/runner/output.go (1)
46-69: JSON filtering path LGTM; verifies -eof behavior.Using structs.FilterStructToMap with excludedFields is the right way to align file output with console filtering.
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (2)
go.mod(0 hunks)pkg/runner/runner.go(3 hunks)
💤 Files with no reviewable changes (1)
- go.mod
🧰 Additional context used
🧬 Code graph analysis (1)
pkg/runner/runner.go (1)
pkg/runner/output.go (2)
WriteJSONOutput(134-161)WriteCsvOutput(164-187)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
- GitHub Check: build-windows
- GitHub Check: build-linux
- GitHub Check: release-test-mac
- GitHub Check: build-mac
- GitHub Check: Functional Test (ubuntu-latest)
- GitHub Check: release-test-linux
- GitHub Check: Lint Test
| if file != nil { | ||
| if r.options.JSON { | ||
| err = WriteJSONOutput(host, hostIP, nil, r.options.OutputCDN, isCDNIP, cdnName, file) | ||
| err = WriteJSONOutput(host, hostIP, nil, r.options.OutputCDN, isCDNIP, cdnName, r.options.ExcludeOutputFields, file) | ||
| } else if r.options.CSV { | ||
| err = WriteCsvOutput(host, hostIP, nil, r.options.OutputCDN, isCDNIP, cdnName, csvFileHeaderEnabled, r.options.ExcludeОutputFields, file) | ||
| err = WriteCsvOutput(host, hostIP, nil, r.options.OutputCDN, isCDNIP, cdnName, csvFileHeaderEnabled, r.options.ExcludeOutputFields, file) | ||
| } else { |
There was a problem hiding this comment.
Restore JSON output for host-only results
In the HasIPS() branch we now pass ports == nil into WriteJSONOutput. The refactored implementation in pkg/runner/output.go immediately returns after looping the (empty) slice, so nothing ever hits the writer. Result: running host-discovery flows (e.g. -sn/-verify with -j -o) yields an empty JSON file even though the console still shows the alive hosts. Please reinstate the host-only emission by guarding for an empty slice inside WriteJSONOutput (or otherwise emitting one record when no ports are present).
diff --git a/pkg/runner/output.go b/pkg/runner/output.go
@@
func WriteJSONOutput(host, ip string, ports []*port.Port, outputCDN bool, isCdn bool, cdnName string, excludedFields []string, writer io.Writer) error {
result := &Result{
Host: host,
IP: ip,
TimeStamp: time.Now().UTC(),
}
@@
- for _, p := range ports {
+ if len(ports) == 0 {
+ b, err := result.JSON(excludedFields)
+ if err != nil {
+ return err
+ }
+ _, err = writer.Write(append(b, '\n'))
+ return err
+ }
+
+ for _, p := range ports {📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if file != nil { | |
| if r.options.JSON { | |
| err = WriteJSONOutput(host, hostIP, nil, r.options.OutputCDN, isCDNIP, cdnName, file) | |
| err = WriteJSONOutput(host, hostIP, nil, r.options.OutputCDN, isCDNIP, cdnName, r.options.ExcludeOutputFields, file) | |
| } else if r.options.CSV { | |
| err = WriteCsvOutput(host, hostIP, nil, r.options.OutputCDN, isCDNIP, cdnName, csvFileHeaderEnabled, r.options.ExcludeОutputFields, file) | |
| err = WriteCsvOutput(host, hostIP, nil, r.options.OutputCDN, isCDNIP, cdnName, csvFileHeaderEnabled, r.options.ExcludeOutputFields, file) | |
| } else { | |
| func WriteJSONOutput(host, ip string, ports []*port.Port, outputCDN bool, isCdn bool, cdnName string, excludedFields []string, writer io.Writer) error { | |
| result := &Result{ | |
| Host: host, | |
| IP: ip, | |
| TimeStamp: time.Now().UTC(), | |
| } | |
| // Emit a record even when there are no ports | |
| if len(ports) == 0 { | |
| b, err := result.JSON(excludedFields) | |
| if err != nil { | |
| return err | |
| } | |
| _, err = writer.Write(append(b, '\n')) | |
| return err | |
| } | |
| for _, p := range ports { | |
| // existing per-port emission logic… | |
| result.Ports = append(result.Ports, p) | |
| if outputCDN && isCdn { | |
| result.CDNName = cdnName | |
| } | |
| b, err := result.JSON(excludedFields) | |
| if err != nil { | |
| return err | |
| } | |
| if _, err := writer.Write(append(b, '\n')); err != nil { | |
| return err | |
| } | |
| } | |
| return nil | |
| } |
🤖 Prompt for AI Agents
In pkg/runner/runner.go around lines 1087-1092, host-only results currently pass
ports == nil into WriteJSONOutput which in the refactored pkg/runner/output.go
returns early and emits nothing; update WriteJSONOutput to detect a nil or empty
ports slice and emit a single JSON record for the host (with ports set to null
or an empty array as appropriate, and preserving other metadata like hostIP,
isCDNIP, cdnName, and output options) instead of returning immediately; ensure
the function still iterates and emits normal per-port records when ports are
present and that excluded fields and OutputCDN behavior are respected.
Resolves #1547
Related to projectdiscovery/utils#674
Summary
Fixes an issue where the
-eofflag was ignored when saving JSON output with-o.Ensures consistent field filtering between console and file outputs.
This fix relies on the new
FilterStructToMapfunction, so theprojectdiscovery/utilsdependency has been updated to the latest commit on themainbranch.Solution
pkg/runner/runner.goUpdated the
handleOutputfunction to passr.options.ExcludeOutputFieldstoWriteJSONOutput.pkg/runner/output.goWriteJSONOutputto acceptexcludedFields []string.*Resultand reuse(r *Result) JSON()for consistent filtering.Test Plan
Run the following command and confirm that excluded fields (
timestamp,protocol,tls) are not present in the JSON file:Summary by CodeRabbit
New Features
Tests
Chores