add apikey support for hackertarget#1622
Conversation
WalkthroughThe changes add optional API key handling for the HackerTarget subscraping source (random key selection, skip when none), and simplify configuration logic to always forward configured API keys to sources when any keys are present. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant Config as Config Loader
participant User as Caller (session)
participant Source as HackerTarget Source
participant API as HackerTarget API
Config->>Source: AddApiKeys(apiKeys)
User->>Source: Run(domain, session)
alt apiKeys present
Source->>Source: PickRandom(apiKey)
Source->>API: HTTP GET /<endpoint>?q=<domain>&apikey=<key>
API-->>Source: response
Source-->>User: results
else no apiKeys
Source->>Source: skipped = true
Source-->>User: return (no results)
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
Pre-merge checks and finishing touches✅ Passed checks (5 passed)
✨ Finishing touches
📜 Recent review detailsConfiguration used: Organization UI Review profile: CHILL Plan: Pro 📒 Files selected for processing (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (5)
🔇 Additional comments (3)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
pkg/runner/config.go(1 hunks)pkg/subscraping/sources/hackertarget/hackertarget.go(3 hunks)
🧠 Learnings (2)
pkg/runner/config.go (1)
Learnt from: 0x4500
PR: #1612
File: pkg/subscraping/agent.go:0-0
Timestamp: 2025-07-17T12:07:51.521Z
Learning: In subfinder subscraping sources, when a source needs to handle specific HTTP status codes differently (like treating 204 as success), the check should be implemented within the individual source's code rather than modifying the global httpRequestWrapper in agent.go. This keeps the special handling localized and avoids affecting other sources.
pkg/subscraping/sources/hackertarget/hackertarget.go (3)
Learnt from: 0x4500
PR: #1612
File: pkg/subscraping/agent.go:0-0
Timestamp: 2025-07-17T12:07:51.521Z
Learning: In subfinder subscraping sources, when a source needs to handle specific HTTP status codes differently (like treating 204 as success), the check should be implemented within the individual source's code rather than modifying the global httpRequestWrapper in agent.go. This keeps the special handling localized and avoids affecting other sources.
Learnt from: x-stp
PR: projectdiscovery/subfinder#0
File: :0-0
Timestamp: 2025-06-24T16:03:30.467Z
Learning: When fixing HTTP response handling bugs in subfinder sources, the correct pattern is to use defer session.DiscardHTTPResponse(resp) after successful requests to ensure the response body remains open for reading, and call session.DiscardHTTPResponse(resp) immediately in error cases.
Learnt from: x-stp
PR: #1608
File: v2/pkg/subscraping/sources/shrewdeye/shrewdeye.go:32-38
Timestamp: 2025-06-20T19:02:59.053Z
Learning: The DiscardHTTPResponse method in subfinder's Session already includes a built-in nil check for the response parameter, so it's safe to call with a potentially nil http.Response without additional nil checking.
🧰 Additional context used
🧠 Learnings (2)
pkg/runner/config.go (1)
Learnt from: 0x4500
PR: #1612
File: pkg/subscraping/agent.go:0-0
Timestamp: 2025-07-17T12:07:51.521Z
Learning: In subfinder subscraping sources, when a source needs to handle specific HTTP status codes differently (like treating 204 as success), the check should be implemented within the individual source's code rather than modifying the global httpRequestWrapper in agent.go. This keeps the special handling localized and avoids affecting other sources.
pkg/subscraping/sources/hackertarget/hackertarget.go (3)
Learnt from: 0x4500
PR: #1612
File: pkg/subscraping/agent.go:0-0
Timestamp: 2025-07-17T12:07:51.521Z
Learning: In subfinder subscraping sources, when a source needs to handle specific HTTP status codes differently (like treating 204 as success), the check should be implemented within the individual source's code rather than modifying the global httpRequestWrapper in agent.go. This keeps the special handling localized and avoids affecting other sources.
Learnt from: x-stp
PR: projectdiscovery/subfinder#0
File: :0-0
Timestamp: 2025-06-24T16:03:30.467Z
Learning: When fixing HTTP response handling bugs in subfinder sources, the correct pattern is to use defer session.DiscardHTTPResponse(resp) after successful requests to ensure the response body remains open for reading, and call session.DiscardHTTPResponse(resp) immediately in error cases.
Learnt from: x-stp
PR: #1608
File: v2/pkg/subscraping/sources/shrewdeye/shrewdeye.go:32-38
Timestamp: 2025-06-20T19:02:59.053Z
Learning: The DiscardHTTPResponse method in subfinder's Session already includes a built-in nil check for the response parameter, so it's safe to call with a potentially nil http.Response without additional nil checking.
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (5)
- GitHub Check: Test Builds (windows-latest)
- GitHub Check: Test Builds (ubuntu-latest)
- GitHub Check: Test Builds (macOS-13)
- GitHub Check: release-test
- GitHub Check: Analyze (go)
🔇 Additional comments (5)
pkg/runner/config.go (1)
49-49: LGTM! Simplified logic aligns with optional API key pattern.The removal of the
source.NeedsKey()check enables sources to receive API keys even when they don't strictly require them, allowing for optional API key usage. The nil check removal is safe sincelen()on a nil slice returns 0.pkg/subscraping/sources/hackertarget/hackertarget.go (4)
15-15: LGTM! Struct fields properly support optional API key functionality.The addition of
apiKeysandskippedfields enables tracking of API key state and execution status as intended.Also applies to: 19-19
43-49: LGTM! HTTP request handling follows established patterns.The HTTP request implementation correctly uses
session.SimpleGet()and proper error handling withsession.DiscardHTTPResponse()as per subfinder conventions.
92-95: LGTM! API key storage implementation is correct.The
AddApiKeysmethod now properly stores the provided keys, and the TODO comment appropriately highlights the broader refactoring needs for API key management.
102-102: LGTM! Statistics correctly report skip status.The addition of the
Skippedfield to statistics provides proper visibility into when the source was skipped due to missing API keys.
| htSearchUrl := fmt.Sprintf("https://api.hackertarget.com/hostsearch/?q=%s", domain) | ||
| randomApiKey := subscraping.PickRandom(s.apiKeys, s.Name()) | ||
| if randomApiKey == "" { | ||
| s.skipped = true | ||
| return | ||
| } | ||
|
|
||
| htSearchUrl = fmt.Sprintf("%s&apikey=%s", htSearchUrl, randomApiKey) | ||
|
|
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Verify that HackerTarget API requires authentication.
The implementation skips execution entirely when no API key is available, but this may be overly restrictive if HackerTarget's API supports unauthenticated requests with rate limits.
Please verify if HackerTarget's API works without authentication:
🌐 Web query:
Does HackerTarget's hostsearch API work without an API key, and what are the rate limits for authenticated vs unauthenticated requests?
💡 Result:
HackerTarget's hostsearch API is accessible without an API key, allowing users to perform queries without authentication. However, unauthenticated users are subject to certain rate limits to ensure fair usage and maintain service stability.
Rate Limits for Unauthenticated Users:
-
Daily Limit: Unauthenticated users are permitted up to 50 API calls per day from a single IP address. (hackertarget.com)
-
Per-Second Limit: The API enforces a rate limit of 2 requests per second from a single IP address. Exceeding this limit will result in an HTTP 429 response code. (hackertarget.com)
Rate Limits for Authenticated Users:
Authenticated users, who have a HackerTarget membership and possess an API key, enjoy enhanced access:
-
Unlimited Results: Authenticated users can retrieve unlimited results, whereas free queries are limited to 500 results. (hackertarget.com)
-
Generous Daily Quota: The daily quota for authenticated users is based on their membership plan, offering a higher allowance compared to unauthenticated users. (hackertarget.com)
-
Additional Features: Membership provides access to advanced tools like Attack Surface Mapping with Domain Profiler and full access to 28 vulnerability scanners and tools. (hackertarget.com)
To access the API as an authenticated user, you need to sign up for a HackerTarget membership. Once registered, you can find your API key in the user dashboard. This key can be included in your API requests either as a query parameter (&apikey=) or as a custom header (X-API-Key). (hackertarget.com)
By adhering to these rate limits and utilizing the appropriate authentication, you can effectively use HackerTarget's hostsearch API for your network security assessments.
Citations:
- 1: https://hackertarget.com/ip-tools/?utm_source=openai
- 2: https://hackertarget.com/ip-tools/?utm_source=openai
- 3: https://hackertarget.com/ip-tools/?utm_source=openai
- 4: https://hackertarget.com/ip-tools/?utm_source=openai
- 5: https://hackertarget.com/ip-tools/?utm_source=openai
- 6: https://hackertarget.com/ip-tools/?utm_source=openai
Allow Unauthenticated HackerTarget API Requests
HackerTarget’s hostsearch endpoint supports unauthenticated queries (up to 50 calls/day and 2 calls/sec per IP). Skipping entirely when no API key is present is overly restrictive—let’s fall back to an unauthenticated request instead.
Locations to update:
- pkg/subscraping/sources/hackertarget/hackertarget.go: lines 34–42
Suggested diff:
- randomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())
- if randomApiKey == "" {
- s.skipped = true
- return
- }
-
- htSearchUrl = fmt.Sprintf("%s&apikey=%s", htSearchUrl, randomApiKey)
+ randomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())
+ if randomApiKey != "" {
+ // Authenticated request: unlimited results per plan
+ htSearchUrl = fmt.Sprintf("%s&apikey=%s", htSearchUrl, randomApiKey)
+ } else {
+ // Unauthenticated fallback: limited to 50 calls/day and 2 calls/sec
+ // Proceed without &apikey parameter
+ }• Remove s.skipped = true and return to allow unauthenticated queries
• Optionally, add logging to warn when falling back to unauthenticated mode
• Be mindful of rate limits (50/day, 2/sec) to avoid HTTP 429 errors
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| htSearchUrl := fmt.Sprintf("https://api.hackertarget.com/hostsearch/?q=%s", domain) | |
| randomApiKey := subscraping.PickRandom(s.apiKeys, s.Name()) | |
| if randomApiKey == "" { | |
| s.skipped = true | |
| return | |
| } | |
| htSearchUrl = fmt.Sprintf("%s&apikey=%s", htSearchUrl, randomApiKey) | |
| htSearchUrl := fmt.Sprintf("https://api.hackertarget.com/hostsearch/?q=%s", domain) | |
| randomApiKey := subscraping.PickRandom(s.apiKeys, s.Name()) | |
| if randomApiKey != "" { | |
| // Authenticated request: unlimited results per plan | |
| htSearchUrl = fmt.Sprintf("%s&apikey=%s", htSearchUrl, randomApiKey) | |
| } else { | |
| // Unauthenticated fallback: limited to 50 calls/day and 2 calls/sec | |
| // Proceed without &apikey parameter | |
| } |
🤖 Prompt for AI Agents
In pkg/subscraping/sources/hackertarget/hackertarget.go around lines 34 to 42,
the current code skips the request entirely if no API key is available, which is
too restrictive since HackerTarget allows unauthenticated queries with limits.
Remove the lines setting s.skipped to true and the return statement to allow the
request to proceed without an API key. Optionally, add a log warning indicating
fallback to unauthenticated mode. Ensure the URL is constructed without the
apikey parameter when no key is present, and consider implementing rate limiting
to respect the 50 calls/day and 2 calls/sec limits.
There was a problem hiding this comment.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
pkg/subscraping/sources/hackertarget/hackertarget.go (1)
97-99: Inconsistency: NeedsKey() returns false but execution requires a key.
NeedsKey()returningfalsesignals that this source functions without API keys, yet lines 36-39 skip execution entirely when no key is present. This creates a contradictory API contract.Either:
- Change
NeedsKey()to returntrueto match the new behavior, or- Allow unauthenticated requests (recommended, as per previous review)
♻️ Duplicate comments (1)
pkg/subscraping/sources/hackertarget/hackertarget.go (1)
34-42: Regression: Breaks unauthenticated HackerTarget access.This change requires an API key to function, which breaks existing behavior. HackerTarget's API explicitly supports unauthenticated requests (50 calls/day, 2 calls/sec) as documented in the previous review. The PR objective was to add API key support for paid users, not to require keys and break free tier access.
🔎 Recommended approach from previous review
- randomApiKey := subscraping.PickRandom(s.apiKeys, s.Name()) - if randomApiKey == "" { - s.skipped = true - return - } - - htSearchUrl = fmt.Sprintf("%s&apikey=%s", htSearchUrl, randomApiKey) + randomApiKey := subscraping.PickRandom(s.apiKeys, s.Name()) + if randomApiKey != "" { + // Authenticated request: unlimited results per plan + htSearchUrl = fmt.Sprintf("%s&apikey=%s", htSearchUrl, randomApiKey) + } + // else: proceed with unauthenticated request (50 calls/day, 2/sec limit)Remove the early return to allow unauthenticated fallback.
Based on previous review findings that documented HackerTarget's support for unauthenticated requests.
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
pkg/subscraping/sources/hackertarget/hackertarget.go
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (5)
- GitHub Check: Test Builds (ubuntu-latest)
- GitHub Check: Test Builds (macOS-latest)
- GitHub Check: Test Builds (windows-latest)
- GitHub Check: Analyze (go)
- GitHub Check: release-test
🔇 Additional comments (3)
pkg/subscraping/sources/hackertarget/hackertarget.go (3)
15-15: LGTM on struct field additions.The
apiKeysandskippedfields appropriately support API key management and skip tracking.Also applies to: 19-19
101-104: TODO acknowledges architectural debt but doesn't address it.The TODO correctly identifies that "entire api key management needs to be refactored," but this technical debt shouldn't block the current PR. However, the immediate inconsistency between
NeedsKey() = falseand the required-key behavior should be resolved first (already flagged above).
106-113: LGTM on statistics tracking.Including the
Skippedfield provides useful observability into source execution. The tracking mechanism itself is sound; the issue lies in the skip logic (already flagged above).
closes #1576
Summary by CodeRabbit
New Features
Bug Fixes
✏️ Tip: You can customize this high-level summary in your review settings.