fix:The rl,rls options are not supported #1434#1731
fix:The rl,rls options are not supported #1434#1731falcolnic wants to merge 3 commits intoprojectdiscovery:devfrom
Conversation
feat:tests lint fix:edge-case
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Tip Issue Planner is now in beta. Read the docs and try it out! Share your feedback on Discord. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@pkg/passive/passive.go`:
- Around line 85-105: The buildMultiRateLimiter function currently dereferences
rateLimit.Custom and can panic when rateLimit is nil; update
buildMultiRateLimiter (method on Agent) to first check for nil rateLimit and nil
rateLimit.Custom (e.g., create a local safe reference or treat it as empty)
before calling rateLimit.Custom.Get(strings.ToLower(source.Name())); if nil,
always fall back to the globalRateLimit path so per-source lookup is skipped;
ensure EnumerateSubdomainsWithCtx's nil customRateLimiter case is handled safely
by this guard.
🧹 Nitpick comments (1)
pkg/passive/passive_test.go (1)
34-96: Add a subtest for negative global rate limits.
This exercises the new clamp logic and guards against regression.✅ Suggested subtest
t.Run("GlobalRateLimitAppliedWhenSourceNotInCustom", func(t *testing.T) { agent := New([]string{"crtsh", "hackertarget"}, []string{}, false, false) customRL := &subscraping.CustomRateLimit{ Custom: mapsutil.SyncLockMap[string, subscraping.RateLimitSpec]{Map: make(map[string]subscraping.RateLimitSpec)}, } @@ require.NoError(t, err) require.NotNil(t, multiRateLimiter) }) + + t.Run("NegativeGlobalRateLimitClampsToZero", func(t *testing.T) { + agent := New([]string{"crtsh"}, []string{}, false, false) + customRL := &subscraping.CustomRateLimit{ + Custom: mapsutil.SyncLockMap[string, subscraping.RateLimitSpec]{Map: make(map[string]subscraping.RateLimitSpec)}, + } + + multiRateLimiter, err := agent.buildMultiRateLimiter(ctx, -5, customRL) + + require.NoError(t, err) + require.NotNil(t, multiRateLimiter) + })
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@pkg/passive/passive.go`:
- Around line 97-110: The indentation in the if-block that handles rateLimit
(the block checking "if rateLimit != nil" where sourceRateLimit is retrieved and
rl/duration/globalRateLimit are assigned) uses spaces instead of tabs; run gofmt
or goimports on pkg/passive/passive.go (or manually re-indent that block) to
convert those leading spaces to tabs so the file matches Go formatting
conventions and linter expectations.
|
@dogancanbakir Hey, any news? |
|
Closing this PR as #1764 has been merged to address this issue. Thanks for your contribution — your PR correctly identified and fixed all three bugs (global |
Proposed changes
/claim #1434
Fixed a bug in the global rate limit (
-rl) implementation where it was not applied to sources without explicit per-source rate limits (-rls).Proof
Before fix (v2.6.7) — all domains throttled to ~30s:
After fix (v2.12.0) — fast domains complete quickly, rate limit respected:
Also added
passive_test.goChecklist
Summary by CodeRabbit