Skip to content

Add optional API key support for sources#1700

Merged
dogancanbakir merged 1 commit intodevfrom
feature/optional-api-key-support
Jan 5, 2026
Merged

Add optional API key support for sources#1700
dogancanbakir merged 1 commit intodevfrom
feature/optional-api-key-support

Conversation

@dogancanbakir
Copy link
Copy Markdown
Member

@dogancanbakir dogancanbakir commented Jan 5, 2026 •
edited by coderabbitai bot
Loading

Summary

  • Adds KeyRequirement enum with three states: NoKey, OptionalKey, RequiredKey
  • Adds KeyRequirement() method to Source interface
  • Updates config generation to include optional sources in provider-config.yaml
  • Updates env var loading to work for optional sources
  • Updates source listing (-ls) with ~ marker for optional sources
  • Marks hackertarget, leakix, reconeer as OptionalKey
  • Fixes hackertarget and reconeer to work without API key

Source Listing Output

Sources marked with an * require key(s) or token(s) to work.
Sources marked with a ~ optionally support key(s) for better results.

Test Plan

  • Run subfinder -ls and verify markers display correctly
  • Test hackertarget without API key configured
  • Test hackertarget with HACKERTARGET_API_KEY env var
  • Verify provider-config.yaml includes optional sources

Closes #1695

Summary by CodeRabbit

  • New Features
    • Implemented a granular key requirement system that distinguishes between required and optional API keys, enabling sources to function with or without authentication.
    • Updated help text to clearly mark sources that require keys (*) and those that optionally support keys (~) for improved results.

✏️ Tip: You can customize this high-level summary in your review settings.

@dogancanbakir
add optional API key support for sources
1ca2a67 
- add KeyRequirement enum with three states: NoKey, OptionalKey, RequiredKey
- add KeyRequirement() method to Source interface
- update config generation to include optional sources
- update env var loading for optional sources
- update source listing with ~ marker for optional sources
- mark hackertarget, leakix, reconeer as OptionalKey
- fix hackertarget and reconeer to work without API key

closes #1695
@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Jan 5, 2026 •
edited
Loading

Walkthrough

This PR introduces a KeyRequirement type to distinguish API key requirements across all sources. Instead of a binary NeedsKey() check, sources now declare whether keys are required, optional, or not needed. Core logic updated to dispatch based on this new requirement semantics.

Changes

Cohort / File(s) Summary
Core infrastructure
pkg/subscraping/types.go, pkg/runner/config.go, pkg/runner/options.go, pkg/passive/sources.go		 New KeyRequirement type (int) with constants NoKey, OptionalKey, RequiredKey. Source interface extended with KeyRequirement() method; NeedsKey() marked deprecated. Runner and passive source logic updated to dispatch on KeyRequirement() values instead of boolean checks. User-facing text clarified to distinguish required vs. optional keys.
Sources with required keys
pkg/subscraping/sources/alienvault/alienvault.go, bevigil/bevigil.go, bufferover/bufferover.go, builtwith/builtwith.go, c99/c99.go, censys/censys.go, certspotter/certspotter.go, chaos/chaos.go, chinaz/chinaz.go, dnsdb/dnsdb.go, dnsdumpster/dnsdumpster.go, dnsrepo/dnsrepo.go, domainsproject/domainsproject.go, driftnet/driftnet.go, facebook/ctlogs.go, fofa/fofa.go, fullhunt/fullhunt.go, github/github.go, gitlab/gitlab.go, intelx/intelx.go, merklemap/merklemap.go, netlas/netlas.go, onyphe/onyphe.go, profundis/profundis.go, pugrecon/pugrecon.go, quake/quake.go, redhuntlabs/redhuntlabs.go, robtex/robtext.go, rsecloud/rsecloud.go, securitytrails/securitytrails.go, shodan/shodan.go, threatbook/threatbook.go, virustotal/virustotal.go, whoisxmlapi/whoisxmlapi.go, windvane/windvane.go, zoomeyeapi/zoomeyeapi.go		 Each source adds KeyRequirement() method returning subscraping.RequiredKey. NeedsKey() updated to delegate to KeyRequirement(), returning true only when key is required.
Sources with optional keys
pkg/subscraping/sources/hackertarget/hackertarget.go, leakix/leakix.go, reconeer/reconeer.go		 Added KeyRequirement() returning OptionalKey. Modified control flow in Run() methods to conditionally apply API keys only when present, allowing requests to proceed without keys. NeedsKey() updated accordingly.
Sources without keys
pkg/subscraping/sources/anubis/anubis.go, commoncrawl/commoncrawl.go, crtsh/crtsh.go, digitorus/digitorus.go, hudsonrock/hudsonrock.go, rapiddns/rapiddns.go, reconcloud/reconcloud.go, riddler/riddler.go, sitedossier/sitedossier.go, thc/thc.go, threatcrowd/threatcrowd.go, threatminer/threatminer.go, waybackarchive/waybackarchive.go		 Each source adds KeyRequirement() method returning subscraping.NoKey. NeedsKey() updated to return false when key is not required.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

Poem

🐰 A new requirement hops through the code,
Keys now have nuance—required, suggested, or unloaded.
Sixty sources unified by semantic grace,
OptionalKey lets systems breathe in the search space!
Better results await those with credentials to share. 🔑✨

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Add optional API key support for sources' is fully related to the main changeset, which introduces KeyRequirement enum with OptionalKey state and updates multiple sources and config files accordingly.
Linked Issues check ✅ Passed The PR addresses issue #1695 by introducing KeyRequirement enum with OptionalKey support, updating sources to mark optional key scenarios, and modifying config and environment variable loading accordingly.
Out of Scope Changes check ✅ Passed All changes directly support the objectives: KeyRequirement type definition, source interface updates, source-specific implementations marking key requirements, configuration generation, and CLI output enhancements for key requirement visibility.
✨ Finishing touches
  • 📝 Generate docstrings

Comment @coderabbitai help to get the list of available commands and usage tips.

coderabbitai[bot]
coderabbitai bot reviewed Jan 5, 2026
View reviewed changes
Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (2)
pkg/subscraping/sources/thc/thc.go (1)

117-117: Consider consolidating the NeedsKey() implementation.

The NeedsKey() method is implemented identically across all sources: return s.KeyRequirement() == subscraping.RequiredKey. While this works correctly, you might consider providing a default implementation (e.g., via an embedded helper struct or interface default method if Go version supports it) to reduce duplication across dozens of source files.

pkg/subscraping/sources/riddler/riddler.go (1)

85-87: Consider reducing code duplication with a helper or embedded struct.

The NeedsKey() implementation is identical across all sources reviewed. You could reduce duplication by:

  • Providing a default implementation in a base struct that sources can embed, or
  • Adding a helper function in the subscraping package

Since this affects multiple files across the codebase, it may be worth addressing in a follow-up refactoring.

Example approach using an embedded struct

In pkg/subscraping/types.go, you could define:

type BaseSource struct {
    keyRequirement KeyRequirement
}

func (b *BaseSource) NeedsKey() bool {
    return b.keyRequirement == RequiredKey
}

func (b *BaseSource) KeyRequirement() KeyRequirement {
    return b.keyRequirement
}

Then sources would embed BaseSource and set the requirement, eliminating the need to implement these methods individually.

📜 Review details

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c1977c0 and 1ca2a67.

📒 Files selected for processing (57)
  • pkg/passive/sources.go
  • pkg/runner/config.go
  • pkg/runner/options.go
  • pkg/subscraping/sources/alienvault/alienvault.go
  • pkg/subscraping/sources/anubis/anubis.go
  • pkg/subscraping/sources/bevigil/bevigil.go
  • pkg/subscraping/sources/bufferover/bufferover.go
  • pkg/subscraping/sources/builtwith/builtwith.go
  • pkg/subscraping/sources/c99/c99.go
  • pkg/subscraping/sources/censys/censys.go
  • pkg/subscraping/sources/certspotter/certspotter.go
  • pkg/subscraping/sources/chaos/chaos.go
  • pkg/subscraping/sources/chinaz/chinaz.go
  • pkg/subscraping/sources/commoncrawl/commoncrawl.go
  • pkg/subscraping/sources/crtsh/crtsh.go
  • pkg/subscraping/sources/digitalyama/digitalyama.go
  • pkg/subscraping/sources/digitorus/digitorus.go
  • pkg/subscraping/sources/dnsdb/dnsdb.go
  • pkg/subscraping/sources/dnsdumpster/dnsdumpster.go
  • pkg/subscraping/sources/dnsrepo/dnsrepo.go
  • pkg/subscraping/sources/domainsproject/domainsproject.go
  • pkg/subscraping/sources/driftnet/driftnet.go
  • pkg/subscraping/sources/facebook/ctlogs.go
  • pkg/subscraping/sources/fofa/fofa.go
  • pkg/subscraping/sources/fullhunt/fullhunt.go
  • pkg/subscraping/sources/github/github.go
  • pkg/subscraping/sources/gitlab/gitlab.go
  • pkg/subscraping/sources/hackertarget/hackertarget.go
  • pkg/subscraping/sources/hudsonrock/hudsonrock.go
  • pkg/subscraping/sources/intelx/intelx.go
  • pkg/subscraping/sources/leakix/leakix.go
  • pkg/subscraping/sources/merklemap/merklemap.go
  • pkg/subscraping/sources/netlas/netlas.go
  • pkg/subscraping/sources/onyphe/onyphe.go
  • pkg/subscraping/sources/profundis/profundis.go
  • pkg/subscraping/sources/pugrecon/pugrecon.go
  • pkg/subscraping/sources/quake/quake.go
  • pkg/subscraping/sources/rapiddns/rapiddns.go
  • pkg/subscraping/sources/reconcloud/reconcloud.go
  • pkg/subscraping/sources/reconeer/reconeer.go
  • pkg/subscraping/sources/redhuntlabs/redhuntlabs.go
  • pkg/subscraping/sources/riddler/riddler.go
  • pkg/subscraping/sources/robtex/robtext.go
  • pkg/subscraping/sources/rsecloud/rsecloud.go
  • pkg/subscraping/sources/securitytrails/securitytrails.go
  • pkg/subscraping/sources/shodan/shodan.go
  • pkg/subscraping/sources/sitedossier/sitedossier.go
  • pkg/subscraping/sources/thc/thc.go
  • pkg/subscraping/sources/threatbook/threatbook.go
  • pkg/subscraping/sources/threatcrowd/threatcrowd.go
  • pkg/subscraping/sources/threatminer/threatminer.go
  • pkg/subscraping/sources/virustotal/virustotal.go
  • pkg/subscraping/sources/waybackarchive/waybackarchive.go
  • pkg/subscraping/sources/whoisxmlapi/whoisxmlapi.go
  • pkg/subscraping/sources/windvane/windvane.go
  • pkg/subscraping/sources/zoomeyeapi/zoomeyeapi.go
  • pkg/subscraping/types.go
🧰 Additional context used
🧬 Code graph analysis (43)
pkg/subscraping/sources/digitalyama/digitalyama.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/threatcrowd/threatcrowd.go (1)
pkg/subscraping/types.go (3)
  • KeyRequirement (37-37)
  • NoKey (40-40)
  • RequiredKey (42-42)
pkg/subscraping/sources/leakix/leakix.go (1)
pkg/subscraping/types.go (3)
  • KeyRequirement (37-37)
  • OptionalKey (41-41)
  • RequiredKey (42-42)
pkg/passive/sources.go (1)
pkg/subscraping/types.go (3)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
  • OptionalKey (41-41)
pkg/subscraping/sources/bevigil/bevigil.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/hudsonrock/hudsonrock.go (1)
pkg/subscraping/types.go (3)
  • KeyRequirement (37-37)
  • NoKey (40-40)
  • RequiredKey (42-42)
pkg/subscraping/sources/securitytrails/securitytrails.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/pugrecon/pugrecon.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/reconcloud/reconcloud.go (1)
pkg/subscraping/types.go (3)
  • KeyRequirement (37-37)
  • NoKey (40-40)
  • RequiredKey (42-42)
pkg/subscraping/sources/chaos/chaos.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/netlas/netlas.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/alienvault/alienvault.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/rapiddns/rapiddns.go (1)
pkg/subscraping/types.go (3)
  • KeyRequirement (37-37)
  • NoKey (40-40)
  • RequiredKey (42-42)
pkg/subscraping/sources/threatminer/threatminer.go (1)
pkg/subscraping/types.go (3)
  • KeyRequirement (37-37)
  • NoKey (40-40)
  • RequiredKey (42-42)
pkg/subscraping/sources/commoncrawl/commoncrawl.go (1)
pkg/subscraping/types.go (3)
  • KeyRequirement (37-37)
  • NoKey (40-40)
  • RequiredKey (42-42)
pkg/subscraping/sources/waybackarchive/waybackarchive.go (1)
pkg/subscraping/types.go (3)
  • KeyRequirement (37-37)
  • NoKey (40-40)
  • RequiredKey (42-42)
pkg/subscraping/sources/riddler/riddler.go (1)
pkg/subscraping/types.go (3)
  • KeyRequirement (37-37)
  • NoKey (40-40)
  • RequiredKey (42-42)
pkg/subscraping/sources/dnsrepo/dnsrepo.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/whoisxmlapi/whoisxmlapi.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/zoomeyeapi/zoomeyeapi.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/facebook/ctlogs.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/merklemap/merklemap.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/gitlab/gitlab.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/domainsproject/domainsproject.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/onyphe/onyphe.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/fullhunt/fullhunt.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/runner/config.go (1)
pkg/subscraping/types.go (3)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
  • OptionalKey (41-41)
pkg/subscraping/sources/rsecloud/rsecloud.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/censys/censys.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/shodan/shodan.go (1)
pkg/subscraping/types.go (3)
  • Source (46-75)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/fofa/fofa.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/sitedossier/sitedossier.go (1)
pkg/subscraping/types.go (3)
  • KeyRequirement (37-37)
  • NoKey (40-40)
  • RequiredKey (42-42)
pkg/subscraping/sources/threatbook/threatbook.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/reconeer/reconeer.go (2)
pkg/subscraping/utils.go (1)
  • PickRandom (12-20)
pkg/subscraping/types.go (3)
  • KeyRequirement (37-37)
  • OptionalKey (41-41)
  • RequiredKey (42-42)
pkg/subscraping/sources/redhuntlabs/redhuntlabs.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/profundis/profundis.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/certspotter/certspotter.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/dnsdb/dnsdb.go (1)
pkg/subscraping/types.go (3)
  • Source (46-75)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/virustotal/virustotal.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/builtwith/builtwith.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/subscraping/sources/driftnet/driftnet.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
pkg/runner/options.go (2)
pkg/passive/sources.go (1)
  • AllSources (64-117)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • OptionalKey (41-41)
pkg/subscraping/sources/dnsdumpster/dnsdumpster.go (1)
pkg/subscraping/types.go (2)
  • KeyRequirement (37-37)
  • RequiredKey (42-42)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (5)
  • GitHub Check: Test Builds (macOS-latest)
  • GitHub Check: Test Builds (ubuntu-latest)
  • GitHub Check: Test Builds (windows-latest)
  • GitHub Check: Analyze (go)
  • GitHub Check: release-test
🔇 Additional comments (70)
pkg/subscraping/sources/threatcrowd/threatcrowd.go (2)

111-114: LGTM! Correct KeyRequirement implementation.

The KeyRequirement() method correctly returns NoKey, which aligns with ThreatCrowd's implementation that doesn't use API keys.

118-118: LGTM! Correct delegation to KeyRequirement().

The NeedsKey() method correctly derives its value from KeyRequirement(), returning false for sources that don't require keys.

pkg/subscraping/sources/leakix/leakix.go (2)

91-93: LGTM! Correct OptionalKey implementation.

The KeyRequirement() method correctly returns OptionalKey, which aligns with the LeakIX implementation that can work both with and without API keys (see lines 38-41 where the api-key header is only set if a key is available).

96-96: LGTM! Correct delegation for optional key source.

The NeedsKey() method correctly returns false for this OptionalKey source, allowing it to be used without an API key while still supporting keys when available.

pkg/subscraping/sources/crtsh/crtsh.go (2)

203-205: LGTM! Correct KeyRequirement implementation.

The KeyRequirement() method correctly returns NoKey, which aligns with crt.sh's implementation that uses direct SQL queries and unauthenticated HTTP requests.

208-208: LGTM! Consistent implementation.

The NeedsKey() method correctly derives its value from KeyRequirement().

pkg/subscraping/sources/rapiddns/rapiddns.go (2)

105-107: LGTM! Correct KeyRequirement implementation.

The KeyRequirement() method correctly returns NoKey, which aligns with RapidDNS's implementation that scrapes data without authentication.

110-110: LGTM! Consistent implementation.

The NeedsKey() method correctly derives its value from KeyRequirement().

pkg/subscraping/sources/thc/thc.go (1)

112-114: LGTM! Correct KeyRequirement implementation.

The KeyRequirement() method correctly returns NoKey, which aligns with THC's implementation that uses an unauthenticated API.

pkg/subscraping/sources/threatbook/threatbook.go (1)

116-122: LGTM! Clean implementation of the key requirement interface.

The KeyRequirement() method correctly returns RequiredKey, and NeedsKey() properly delegates to it for a single source of truth. This refactoring aligns with the PR's goal of introducing granular key requirements.

pkg/subscraping/sources/chaos/chaos.go (1)

77-83: LGTM! Consistent implementation.

The key requirement pattern is correctly implemented, matching the refactoring across other sources.

pkg/subscraping/sources/gitlab/gitlab.go (1)

163-169: LGTM! Properly integrated.

The key requirement interface is correctly implemented following the established pattern.

pkg/subscraping/sources/driftnet/driftnet.go (1)

107-115: LGTM! Well-documented implementation.

The key requirement interface is correctly implemented with clear documentation on KeyRequirement(). The delegation pattern in NeedsKey() is appropriate.

pkg/subscraping/sources/rsecloud/rsecloud.go (1)

111-117: LGTM! Correct implementation.

The key requirement interface is properly implemented, consistent with the refactoring pattern across all sources.

pkg/subscraping/sources/pugrecon/pugrecon.go (1)

136-144: LGTM! Clean implementation of the KeyRequirement interface.

The KeyRequirement() method correctly returns RequiredKey, and NeedsKey() properly derives its value from it. This aligns with the source's behavior in Run() where it skips execution if no API key is available.

pkg/subscraping/sources/onyphe/onyphe.go (1)

150-156: LGTM! Correct implementation of the KeyRequirement interface.

The implementation correctly identifies this as a required-key source and derives NeedsKey() from KeyRequirement(). This is consistent with the source's behavior that skips execution when no API key is provided.

pkg/subscraping/sources/censys/censys.go (1)

200-206: LGTM! Proper implementation of the KeyRequirement interface.

The source correctly declares itself as requiring an API key, and NeedsKey() appropriately derives its return value from KeyRequirement(). This matches the implementation in Run() where the source skips if no PAT is available.

pkg/subscraping/sources/bevigil/bevigil.go (1)

97-103: LGTM! Consistent implementation of the KeyRequirement interface.

The implementation correctly returns RequiredKey and updates NeedsKey() to derive from it. This aligns perfectly with the source's API key requirement shown in the Run() method.

pkg/subscraping/sources/sitedossier/sitedossier.go (1)

100-106: LGTM! Correct implementation for a no-key source.

The source properly returns NoKey from KeyRequirement(), and NeedsKey() correctly evaluates to false. This aligns with the source's implementation which doesn't require or use API keys.

pkg/subscraping/sources/chinaz/chinaz.go (1)

94-100: LGTM! Implementation correctly reflects API key requirement.

The KeyRequirement() method correctly returns RequiredKey, which aligns with the source's behavior (lines 35-39 where it skips if no API key is available). The NeedsKey() delegation is consistent with the new pattern.

pkg/subscraping/sources/threatminer/threatminer.go (1)

83-89: LGTM! Implementation correctly reflects no API key requirement.

The KeyRequirement() method correctly returns NoKey, which aligns with the source's behavior (lines 91-93 where AddApiKeys is a no-op with explicit comment "no key needed"). The NeedsKey() delegation is consistent with the new pattern.

pkg/subscraping/sources/facebook/ctlogs.go (1)

167-175: LGTM! Implementation correctly reflects API key requirement.

The KeyRequirement() method correctly returns RequiredKey, which aligns with the source's behavior (lines 90-94 where it skips if no API keys are present). The doc comments provide helpful context for these methods.

pkg/subscraping/sources/shodan/shodan.go (1)

118-124: LGTM! Implementation correctly reflects API key requirement.

The KeyRequirement() method correctly returns RequiredKey, which aligns with the source's behavior (lines 43-47 where it skips if no API key is available). The NeedsKey() delegation is consistent with the new pattern.

pkg/subscraping/sources/securitytrails/securitytrails.go (1)

145-151: LGTM! Implementation correctly reflects API key requirement.

The KeyRequirement() method correctly returns RequiredKey, which aligns with the source's behavior (lines 49-53 where it skips if no API key is available). The NeedsKey() delegation is consistent with the new pattern.

pkg/subscraping/sources/whoisxmlapi/whoisxmlapi.go (1)

102-108: LGTM! Clean refactoring to support the new KeyRequirement contract.

The implementation correctly returns RequiredKey and delegates NeedsKey() to the new method, maintaining backward compatibility.

pkg/subscraping/sources/profundis/profundis.go (1)

112-118: LGTM! Consistent implementation of the KeyRequirement contract.

The changes correctly identify this source as requiring an API key and maintain the expected behavior.

pkg/subscraping/sources/merklemap/merklemap.go (1)

142-148: LGTM! Proper implementation of the KeyRequirement contract.

The source correctly declares its key requirement and delegates the boolean check appropriately.

pkg/subscraping/sources/windvane/windvane.go (1)

139-145: LGTM! Consistent with the KeyRequirement pattern.

The implementation correctly marks this source as requiring an API key.

pkg/subscraping/sources/riddler/riddler.go (1)

81-87: LGTM! Correctly implements NoKey requirement.

The source properly declares it doesn't need an API key, and NeedsKey() will correctly return false.

pkg/passive/sources.go (1)

187-194: LGTM! Correct implementation of KeyRequirement-based key loading.

The logic correctly loads API keys for sources that require them (RequiredKey) or optionally use them (OptionalKey), replacing the previous boolean NeedsKey() check with more granular semantics.

pkg/subscraping/sources/digitorus/digitorus.go (1)

90-96: LGTM! Correct KeyRequirement implementation for key-free source.

The source correctly returns NoKey since digitorus doesn't require or use API keys, and NeedsKey() properly delegates to the new KeyRequirement() method.

pkg/subscraping/sources/builtwith/builtwith.go (1)

103-109: LGTM! Correct KeyRequirement implementation.

The source correctly returns RequiredKey (confirmed by the early return at lines 54-56 when no API key is available), and NeedsKey() properly delegates to the new pattern.

pkg/subscraping/sources/redhuntlabs/redhuntlabs.go (1)

139-145: LGTM! Correct KeyRequirement implementation.

The source correctly returns RequiredKey (confirmed by the skip logic at lines 46-55 when no valid API key is available), and NeedsKey() properly delegates to the new KeyRequirement() method.

pkg/subscraping/sources/dnsdb/dnsdb.go (1)

186-192: LGTM! Correct KeyRequirement implementation.

The source correctly returns RequiredKey (confirmed by the early return at lines 63-66 when no API key is available), and NeedsKey() properly delegates to the new KeyRequirement() method.

pkg/subscraping/sources/dnsrepo/dnsrepo.go (1)

103-109: LGTM! Clean refactoring to support the new key requirement contract.

The implementation correctly declares this source as requiring an API key and derives NeedsKey() from KeyRequirement(). This aligns with the existing behavior where the source skips execution when no API key is available (lines 38-42).

pkg/subscraping/sources/dnsdumpster/dnsdumpster.go (1)

94-100: LGTM! Consistent implementation of the key requirement refactoring.

The changes correctly implement the new KeyRequirement() contract and update NeedsKey() to derive from it. This is consistent with the source's existing behavior of requiring an API key (lines 43-47).

pkg/subscraping/sources/virustotal/virustotal.go (1)

116-122: LGTM! Proper implementation of the key requirement interface.

The addition of KeyRequirement() and the update to NeedsKey() correctly reflect that this source requires an API key. The implementation is consistent with the broader refactoring effort across all sources.

pkg/subscraping/sources/github/github.go (1)

268-274: LGTM! Clean integration of the key requirement contract.

The implementation correctly declares GitHub as a source that requires an API key, which aligns with the explicit key check at lines 61-65. The refactoring properly delegates NeedsKey() to the new KeyRequirement() method.

pkg/subscraping/sources/fullhunt/fullhunt.go (1)

89-95: LGTM! Consistent refactoring implementation.

The changes correctly implement the new key requirement interface. The source properly declares itself as requiring an API key, which matches its existing behavior of skipping when no key is available (lines 40-44).

pkg/subscraping/sources/reconcloud/reconcloud.go (1)

92-98: LGTM! KeyRequirement implementation is correct.

The source correctly declares NoKey since it doesn't use API keys, and NeedsKey() properly derives from KeyRequirement(). The pattern is consistent with the broader refactoring.

pkg/subscraping/sources/robtex/robtext.go (1)

134-140: LGTM! KeyRequirement implementation is correct.

The source correctly declares RequiredKey since it requires API keys to function (Line 50-53 skips execution if no key is available). The NeedsKey() method properly derives from KeyRequirement().

pkg/subscraping/sources/waybackarchive/waybackarchive.go (1)

87-93: LGTM! KeyRequirement implementation is correct.

The source correctly declares NoKey since it doesn't require API keys, and NeedsKey() properly derives from KeyRequirement(). The implementation is consistent with other no-key sources.

pkg/subscraping/sources/netlas/netlas.go (1)

187-193: LGTM! KeyRequirement implementation is correct.

The source correctly declares RequiredKey since it uses API keys for authentication (Lines 68, 125). The NeedsKey() method properly derives from KeyRequirement(), maintaining consistency with other required-key sources.

pkg/subscraping/sources/quake/quake.go (2)

142-148: LGTM! KeyRequirement implementation is correct.

The source correctly declares RequiredKey since it requires API keys to function (Lines 54-58 skip execution if no key is available). The NeedsKey() method properly derives from KeyRequirement().

142-144: OptionalKey is properly defined in the KeyRequirement enum.

The enum in pkg/subscraping/types.go correctly includes all three expected constants: NoKey, OptionalKey, and RequiredKey. No action needed.

pkg/subscraping/sources/domainsproject/domainsproject.go (1)

120-126: LGTM! Clean implementation of KeyRequirement pattern.

The KeyRequirement() method correctly returns RequiredKey, and NeedsKey() properly delegates to it. This aligns with the source's behavior at lines 47-50 where it skips execution when API keys are missing.

pkg/subscraping/sources/c99/c99.go (1)

110-116: LGTM! Consistent implementation.

The KeyRequirement pattern is correctly implemented, matching the source's requirement for API keys (verified at lines 46-50).

pkg/subscraping/sources/fofa/fofa.go (1)

120-126: LGTM! Proper KeyRequirement implementation.

The implementation correctly declares this source as requiring keys, consistent with the skip logic at lines 50-54.

pkg/subscraping/sources/bufferover/bufferover.go (1)

122-128: LGTM! Implementation aligns with source requirements.

The KeyRequirement pattern is correctly implemented. The source properly returns RequiredKey given its skip behavior at lines 45-49.

pkg/runner/config.go (2)

11-11: Necessary import for KeyRequirement types.

The subscraping package import is required for accessing the KeyRequirement type and constants used in the updated logic below.

29-33: LGTM! Correctly implements optional key support.

The updated logic now includes both RequiredKey and OptionalKey sources in the generated configuration file, enabling the optional API key feature described in the PR objectives.

pkg/subscraping/sources/digitalyama/digitalyama.go (1)

122-128: LGTM! Clean implementation of the new key requirement pattern.

The KeyRequirement() method correctly returns RequiredKey for this source, and NeedsKey() properly delegates to it. This aligns with the existing behavior where the source skips execution if no API key is available (line 45-47).

pkg/subscraping/sources/zoomeyeapi/zoomeyeapi.go (1)

121-127: LGTM! Consistent implementation.

The KeyRequirement() method correctly returns RequiredKey, consistent with the source's requirement for an API key in host:apiKey format (lines 51-57).

pkg/subscraping/sources/certspotter/certspotter.go (1)

143-149: LGTM! Correct implementation.

The source properly implements the new key requirement pattern. The RequiredKey classification matches the existing behavior where an API key is mandatory (line 40-44).

pkg/subscraping/sources/alienvault/alienvault.go (1)

101-107: LGTM! Properly implemented.

The key requirement implementation is correct. This source requires an API key for authentication (line 49), so RequiredKey is the appropriate classification.

pkg/subscraping/sources/hudsonrock/hudsonrock.go (1)

90-96: LGTM! Correct NoKey classification.

This source correctly returns NoKey since it doesn't require authentication. The implementation aligns with the no-op AddApiKeys() method (lines 98-100) and the unauthenticated API call (line 43).

pkg/runner/options.go (3)

18-18: LGTM: Import addition is correct.

The subscraping import is necessary to access the KeyRequirement constants used in the updated listSources function.

227-228: LGTM: Clear documentation of key requirement markers.

The updated messages clearly distinguish between required (*) and optional (~) API key sources, improving user understanding of the -ls output.

233-240: LGTM: Switch statement correctly implements key requirement display logic.

The switch on KeyRequirement() properly handles all three states (RequiredKey, OptionalKey, NoKey) and displays the appropriate markers as documented in the help text.

pkg/subscraping/sources/intelx/intelx.go (1)

168-174: LGTM: Correct KeyRequirement implementation for intelx.

The implementation correctly identifies intelx as requiring an API key (line 67-69 show it skips execution without a key), and the NeedsKey() delegation maintains backward compatibility while supporting the new key requirement system.

pkg/subscraping/types.go (2)

36-43: LGTM: Well-designed KeyRequirement enum.

The three-state enum (NoKey, OptionalKey, RequiredKey) provides the necessary granularity to distinguish API key requirements across sources, addressing the PR objectives effectively.

64-69: LGTM: Interface extension maintains backward compatibility.

The addition of KeyRequirement() to the Source interface provides the new functionality while keeping the deprecated NeedsKey() method ensures backward compatibility. The deprecation notice appropriately directs users to the more granular method.

pkg/subscraping/sources/anubis/anubis.go (1)

85-91: LGTM: Correct KeyRequirement implementation for anubis.

The implementation correctly identifies anubis as not requiring an API key, consistent with the unauthenticated API call in the Run method (line 34) and the no-op AddApiKeys implementation (line 93-95).

pkg/subscraping/sources/commoncrawl/commoncrawl.go (1)

108-114: LGTM: Correct KeyRequirement implementation for commoncrawl.

The implementation correctly identifies commoncrawl as not requiring an API key, consistent with its use of the public Common Crawl Index API and the no-op AddApiKeys implementation (line 116-118).

pkg/subscraping/sources/hackertarget/hackertarget.go (3)

36-38: LGTM! Conditional API key usage correctly implemented.

The implementation properly checks for a non-empty API key before appending it to the URL, allowing HackerTarget to function without authentication while still supporting API keys for enhanced limits when available.

94-96: LGTM! Correctly declares optional key requirement.

The source properly implements the new KeyRequirement() interface method, returning OptionalKey to indicate the API key is optional for this source.

98-100: LGTM! Correctly derives key necessity from KeyRequirement.

The NeedsKey() method properly delegates to KeyRequirement(), ensuring consistent behavior across the new key requirement model. Since this source returns OptionalKey, it will correctly report that a key is not strictly required.

pkg/subscraping/sources/reconeer/reconeer.go (3)

39-45: LGTM! Conditional header construction correctly implemented.

The implementation properly initializes headers with the required Accept field and conditionally adds the X-API-KEY header only when a non-empty API key is available. This allows Reconeer to function without authentication while supporting API keys for enhanced access when provided.

93-95: LGTM! Correctly declares optional key requirement.

The source properly implements the new KeyRequirement() interface method, returning OptionalKey to indicate the API key is optional for this source.

97-99: LGTM! Correctly derives key necessity from KeyRequirement.

The NeedsKey() method properly delegates to KeyRequirement(), ensuring consistent behavior across the new key requirement model. Since this source returns OptionalKey, it will correctly report that a key is not strictly required.

@dogancanbakir dogancanbakir merged commit a2615fa into dev Jan 5, 2026
7 of 10 checks passed
@dogancanbakir dogancanbakir deleted the feature/optional-api-key-support branch January 5, 2026 18:01
@BrewTestBot BrewTestBot mentioned this pull request Jan 12, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments

@Mzack9999 Mzack9999 Mzack9999 approved these changes

@knakul853 knakul853 Awaiting requested review from knakul853

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Make API key optional

2 participants

@dogancanbakir @Mzack9999