fix(ratelimit): honour -rls duration and apply -rl to all sources (#1434)#1762
fix(ratelimit): honour -rls duration and apply -rl to all sources (#1434)#1762marchantdev wants to merge 3 commits intoprojectdiscovery:devfrom
Conversation
) Two bugs in the rate-limit pipeline: 1. **Duration dropped (runner.go):** `CustomRateLimit` only stored `MaxCount`; the `Duration` from `-rls key=N/d` was silently discarded. A new `CustomDuration` map is added to `CustomRateLimit` and populated alongside `Custom` in `runner.go`. 2. **Global `-rl` ignored for unregistered sources (passive.go):** `buildMultiRateLimiter` only applied a rate limit when a source was present in the custom map. Sources absent from `-rls` received `math.MaxUint32` (unlimited) even when `-rl` was set. The fix applies `globalRateLimit` as a fallback for those sources. Fixes #1434.
Neo - PR Security ReviewNo security issues found Highlights
Comment |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
WalkthroughAdds per-source duration support to passive rate limiting: introduces Changes
Sequence DiagramsequenceDiagram
participant Runner
participant CustomRateLimit
participant Builder as buildMultiRateLimiter
participant RateLimiter
Runner->>CustomRateLimit: populate `Custom` (counts) and `CustomDuration` (durations) with lowercased keys
Builder->>CustomRateLimit: read per-source count & duration (lowercased)
Builder->>Builder: determine effective duration (per-source -> global -> default)
Builder->>RateLimiter: add per-source limiter with count & duration (or unlimited fallback)
RateLimiter->>RateLimiter: enforce request pacing per source
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
pkg/passive/passive.go (1)
85-114:⚠️ Potential issue | 🔴 CriticalNil pointer dereference when
rateLimitis nil.If
EnumerateSubdomainsWithCtxis called without theWithCustomRateLimitoption,enumerateOptions.customRateLimiterwill be nil, causing a panic at line 93 when callingrateLimit.Custom.Get(sourceName).🐛 Proposed fix: Add nil guard at the start of the function
func (a *Agent) buildMultiRateLimiter(ctx context.Context, globalRateLimit int, rateLimit *subscraping.CustomRateLimit) (*ratelimit.MultiLimiter, error) { var multiRateLimiter *ratelimit.MultiLimiter var err error + + if rateLimit == nil { + rateLimit = &subscraping.CustomRateLimit{} + } + for _, source := range a.sources { var rl uint duration := time.Second // default duration when a rate limit is activeAlternatively, you could add nil checks before each
Getcall, but initializing an empty struct is cleaner.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/passive/passive.go` around lines 85 - 114, The function buildMultiRateLimiter can nil-deref rateLimit when enumerateOptions.customRateLimiter is not set; add a nil-guard at the top of buildMultiRateLimiter (or initialize enumerateOptions.customRateLimiter earlier) so rateLimit is non-nil (e.g., if rateLimit == nil { rateLimit = &subscraping.CustomRateLimit{} }) before calling rateLimit.Custom.Get or rateLimit.CustomDuration.Get; this prevents the panic when EnumerateSubdomainsWithCtx is called without WithCustomRateLimit while keeping the current per-source/global logic intact.
🧹 Nitpick comments (1)
pkg/passive/ratelimit_test.go (1)
34-126: Good test coverage for the rate-limiting scenarios.The four tests effectively cover:
- Per-source limits with custom duration
- Global rate limit fallback for unlisted sources
- Count=0 fallback to global rate limit
- Unlimited behavior when no flags are set
Consider adding a test for when
rateLimitis nil to verify the fix for the nil pointer issue flagged inpassive.go:🧪 Optional: Add test for nil rateLimit handling
// TestBuildMultiRateLimiter_NilRateLimit verifies that a nil CustomRateLimit // does not cause a panic and correctly applies the global rate limit. func TestBuildMultiRateLimiter_NilRateLimit(t *testing.T) { agent := New([]string{"hackertarget"}, []string{}, false, false) globalRateLimit := 5 mrl, err := agent.buildMultiRateLimiter(context.Background(), globalRateLimit, nil) require.NoError(t, err) require.NotNil(t, mrl) limit, err := mrl.GetLimit("hackertarget") require.NoError(t, err) assert.Equal(t, uint(globalRateLimit), limit, "source should use global rate limit when CustomRateLimit is nil") }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/passive/ratelimit_test.go` around lines 34 - 126, Add a new test named TestBuildMultiRateLimiter_NilRateLimit that calls agent.buildMultiRateLimiter with a non-zero globalRateLimit and a nil CustomRateLimit, then assert no error and that mrl.GetLimit("hackertarget") returns the global rate (uint(globalRateLimit)); this verifies buildMultiRateLimiter and its handling of a nil *CustomRateLimit (reference buildMultiRateLimiter, CustomRateLimit, GetLimit, and the new TestBuildMultiRateLimiter_NilRateLimit).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Outside diff comments:
In `@pkg/passive/passive.go`:
- Around line 85-114: The function buildMultiRateLimiter can nil-deref rateLimit
when enumerateOptions.customRateLimiter is not set; add a nil-guard at the top
of buildMultiRateLimiter (or initialize enumerateOptions.customRateLimiter
earlier) so rateLimit is non-nil (e.g., if rateLimit == nil { rateLimit =
&subscraping.CustomRateLimit{} }) before calling rateLimit.Custom.Get or
rateLimit.CustomDuration.Get; this prevents the panic when
EnumerateSubdomainsWithCtx is called without WithCustomRateLimit while keeping
the current per-source/global logic intact.
---
Nitpick comments:
In `@pkg/passive/ratelimit_test.go`:
- Around line 34-126: Add a new test named
TestBuildMultiRateLimiter_NilRateLimit that calls agent.buildMultiRateLimiter
with a non-zero globalRateLimit and a nil CustomRateLimit, then assert no error
and that mrl.GetLimit("hackertarget") returns the global rate
(uint(globalRateLimit)); this verifies buildMultiRateLimiter and its handling of
a nil *CustomRateLimit (reference buildMultiRateLimiter, CustomRateLimit,
GetLimit, and the new TestBuildMultiRateLimiter_NilRateLimit).
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: cb7587eb-71d8-4236-9db1-1260878a74f5
📒 Files selected for processing (4)
pkg/passive/passive.gopkg/passive/ratelimit_test.gopkg/runner/runner.gopkg/subscraping/types.go
…miter Prevent a nil pointer dereference when EnumerateSubdomainsWithCtx is called without the WithCustomRateLimit option (rateLimit would be nil, causing a panic on rateLimit.Custom.Get). Also add TestBuildMultiRateLimiter_NilRateLimit to cover this path.
|
Addressed CodeRabbit review:
|
There was a problem hiding this comment.
🧹 Nitpick comments (1)
pkg/passive/ratelimit_test.go (1)
34-57: Consider adding a test for case-insensitive source name lookup.All tests use lowercase keys (e.g.,
"hackertarget"), which won't catch the case mismatch issue between storage and lookup. Adding a test with mixed-case keys would validate that the lowercasing behavior works end-to-end.🧪 Suggested test for case-insensitive lookup
// TestBuildMultiRateLimiter_CaseInsensitiveLookup verifies that per-source // limits work regardless of source name casing in the custom map. func TestBuildMultiRateLimiter_CaseInsensitiveLookup(t *testing.T) { agent := New([]string{"hackertarget"}, []string{}, false, false) // Simulate user providing mixed-case source name via -rls crl := newCustomRateLimit( map[string]uint{"HackerTarget": 8}, map[string]time.Duration{"HackerTarget": time.Minute}, ) mrl, err := agent.buildMultiRateLimiter(context.Background(), 0, crl) require.NoError(t, err) require.NotNil(t, mrl) // Should still match despite case difference limit, err := mrl.GetLimit("hackertarget") require.NoError(t, err) assert.Equal(t, uint(8), limit, "should find per-source limit regardless of case in custom map") }Note: This test would currently fail due to the case mismatch bug identified in
passive.go.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/passive/ratelimit_test.go` around lines 34 - 57, Add a new unit test (e.g., TestBuildMultiRateLimiter_CaseInsensitiveLookup) alongside TestBuildMultiRateLimiter_PerSourceLimit that constructs an agent with a lowercase source list, then creates a custom rate limit via newCustomRateLimit using a mixed-case key (e.g., "HackerTarget") and duration, calls agent.buildMultiRateLimiter(context.Background(), 0, crl), and asserts mrl.GetLimit("hackertarget") returns the configured uint value; this verifies case-insensitive lookup end-to-end and will expose the current casing bug.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@pkg/passive/ratelimit_test.go`:
- Around line 34-57: Add a new unit test (e.g.,
TestBuildMultiRateLimiter_CaseInsensitiveLookup) alongside
TestBuildMultiRateLimiter_PerSourceLimit that constructs an agent with a
lowercase source list, then creates a custom rate limit via newCustomRateLimit
using a mixed-case key (e.g., "HackerTarget") and duration, calls
agent.buildMultiRateLimiter(context.Background(), 0, crl), and asserts
mrl.GetLimit("hackertarget") returns the configured uint value; this verifies
case-insensitive lookup end-to-end and will expose the current casing bug.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: b979cbd2-dd74-4910-9bef-5fa6ad2e0a5a
📒 Files selected for processing (2)
pkg/passive/passive.gopkg/passive/ratelimit_test.go
passive.go looks up custom rate limits using strings.ToLower(source.Name()), but runner.go stored them with the original casing from the CLI flag. A user passing -rls HackerTarget=5/m would never match the lowercase lookup key, silently falling back to unlimited instead of the configured limit. Fix: lowercase the source key when storing in runner.go so both storage and lookup are consistent. Also add TestBuildMultiRateLimiter_CaseInsensitiveLookup to verify that a mixed-case key in the custom map is found correctly end-to-end. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Addressed CodeRabbit's nitpick and found the underlying bug: Root cause: Fix in New test All 6 |
|
Hi @Ice3man543 / @ehsandeep — could you take a look when you get a chance? The automated reviews (Neo security + CodeRabbit) both came back clean, and the nil pointer guard has been addressed. Happy to make any changes you need. 🙏 |
|
Closing this PR as #1764 has been merged to address this issue. Your PR was very close to being selected — it fixed all three bugs (global |
What
Fixes #1434 —
-rland-rlsrate-limit flags were silently ignored.Two independent bugs:
Bug 1 — Duration discarded in
runner.goCustomRateLimitonly storedMaxCount; theDurationfrom-rls key=N/d(e.g.hackertarget=2/m) was dropped.Fix: Added
CustomDuration mapsutil.SyncLockMap[string, time.Duration]toCustomRateLimitand populated it alongsideCustom.Bug 2 — Global
-rlnot applied to unlisted sources inpassive.gobuildMultiRateLimiteronly entered the rate-limit branch when a source was present inrateLimit.Custom. Sources not listed in-rlsreceivedmath.MaxUint32(unlimited) even when-rlwas set.Fix: Added
else if globalRateLimit > 0branch so all sources without a per-source override receive the global limit.Affected files
pkg/subscraping/types.goCustomDurationfield toCustomRateLimitpkg/runner/runner.goCustomDuration; addtimeimportpkg/passive/passive.gopkg/passive/ratelimit_test.goTests
Summary by CodeRabbit
New Features
Bug Fixes
Tests