Update Censys PAT integration and docs#1654
Conversation
WalkthroughThe Censys source integration is migrated from a GET-based search API to Censys API v3 using POST requests. The change introduces new request/response structures, API key management with organization ID support, cursor-based pagination via NextPageToken, and adds a public AddApiKeys method for key parsing. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Comment |
|
Hello. I have been informed by a distributor that Censys API v2 is scheduled for EOL (End of Life) on December 15th. Since many users rely on the Censys integration in subfinder, failing to address this before the deadline will likely cause issues for a significant number of people. Do you have a rough timeline for when this could be reviewed or merged? I understand the team is busy, but we would greatly appreciate it if you could handle this before the 15th. Thank you! |
333eaec to
b7193b4
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (3)
pkg/subscraping/sources/censys/censys.go (2)
149-163: Consider validating HTTP status code before decoding.The code decodes the response body directly without checking
resp.StatusCode. If the API returns a 4xx (e.g., 401 Unauthorized, 429 Rate Limited) or 5xx error, the error response body will be decoded into the successresponsestruct, likely resulting in silently empty results rather than a meaningful error.if err != nil { results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err} s.errors++ session.DiscardHTTPResponse(resp) return } + if resp.StatusCode != http.StatusOK { + _ = resp.Body.Close() + results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf("unexpected status code: %d", resp.StatusCode)} + s.errors++ + return + } + var censysResponse responseNote: You'll need to import
"fmt"if not already imported.
204-217: Clarify docstring to document both supported formats.The docstring mentions
PAT:ORG_IDformat but the implementation also supports PAT-only keys for free users (lines 211-216). Consider updating the docstring to document both formats explicitly.-// AddApiKeys parses and adds API keys. -// Format: "PAT:ORG_ID" where ORG_ID is required for paid accounts. -// Example: "censys_xxx_token:12345678-91011-1213" +// AddApiKeys parses and adds API keys. +// Supported formats: +// - "PAT:ORG_ID" for paid accounts (e.g., "censys_xxx_token:12345678-91011-1213") +// - "PAT" for free users (e.g., "censys_xxx_token") func (s *Source) AddApiKeys(keys []string) {pkg/subscraping/sources/censys/censys_test.go (1)
17-25: Consider handling the error from NewMultiLimiter.Ignoring the error could mask test setup issues. While unlikely to fail with these options, handling it improves test reliability.
-func createTestMultiRateLimiter(ctx context.Context) *ratelimit.MultiLimiter { - mrl, _ := ratelimit.NewMultiLimiter(ctx, &ratelimit.Options{ +func createTestMultiRateLimiter(t *testing.T, ctx context.Context) *ratelimit.MultiLimiter { + t.Helper() + mrl, err := ratelimit.NewMultiLimiter(ctx, &ratelimit.Options{ Key: "censys", IsUnlimited: false, MaxCount: math.MaxInt32, Duration: time.Millisecond, }) + require.NoError(t, err) return mrl }Note: Update all call sites to pass
tas the first argument.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
pkg/subscraping/sources/censys/censys.go(5 hunks)pkg/subscraping/sources/censys/censys_test.go(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
pkg/subscraping/sources/censys/censys.go (1)
pkg/subscraping/utils.go (2)
PickRandom(12-20)CreateApiKeys(22-30)
🔇 Additional comments (9)
pkg/subscraping/sources/censys/censys.go (4)
16-33: LGTM!The constants are well-documented and clearly define the API endpoint, pagination limits, and header values for the Censys Platform API v3 integration.
35-70: LGTM!The request/response structures are well-defined and appropriately map to the Censys Platform API v3 schema. The
searchRequestsupports cursor-based pagination, and the response hierarchy correctly targetscertificate_v1.resource.namesfor subdomain extraction.
93-101: LGTM!Good use of the
PickRandomutility for load distribution across multiple API keys, with clear documentation explaining the rationale. The empty PAT check correctly handles the skip case.
165-181: LGTM!The result extraction correctly iterates through hits and their certificate names, with proper context cancellation handling. The pagination logic correctly uses
NextPageTokenand respects the max page limit.pkg/subscraping/sources/censys/censys_test.go (5)
27-51: LGTM!This test correctly validates that the source is skipped when no API keys are configured. Since the skip happens before any HTTP request is made, the test is deterministic and doesn't require network access.
91-98: LGTM!Simple and effective metadata verification for the source interface methods.
100-122: LGTM!Thorough testing of both API key formats. The subtests clearly validate that:
PAT:ORG_IDformat correctly populates both fieldsPATformat (for free users) populates only the PAT field with empty orgIDGood use of
require.Lenbefore accessing slice elements to prevent panics.
124-137: LGTM!Correctly validates that the
Statistics()method properly maps all internal fields to the returnedsubscraping.Statisticsstruct.
53-89: Context cancellation is properly checked before HTTP requests are made.The
Run()method checksctx.Done()at the start of its main loop (before building any request body or making HTTP calls), so whencancel()is called immediately afterRun()returns, the context will be cancelled before any HTTP request is sent. The theoretical race window between goroutine startup and the context check is negligible in practice. Additionally,HTTPRequest()useshttp.NewRequestWithContext()which respects context cancellation. The test design is sound and follows the project's patterns.Likely an incorrect or invalid review comment.
|
closes #1614
deprecate censys v2 api
Summary by CodeRabbit
New Features
Refactor
Tests
✏️ Tip: You can customize this high-level summary in your review settings.