Skip to content

fix(ratelimit): honour -rls duration and apply -rl to all sources (#1434)#1762

Closed
marchantdev wants to merge 3 commits intoprojectdiscovery:devfrom
marchantdev:fix/rate-limit-duration-and-global
Closed

fix(ratelimit): honour -rls duration and apply -rl to all sources (#1434)#1762
marchantdev wants to merge 3 commits intoprojectdiscovery:devfrom
marchantdev:fix/rate-limit-duration-and-global

Conversation

@marchantdev
Copy link
Copy Markdown

@marchantdev marchantdev commented Mar 9, 2026

What

Fixes #1434-rl and -rls rate-limit flags were silently ignored.

Two independent bugs:

Bug 1 — Duration discarded in runner.go

CustomRateLimit only stored MaxCount; the Duration from -rls key=N/d (e.g. hackertarget=2/m) was dropped.

// Before — Duration from RateLimits.AsMap() silently discarded
_ = runner.rateLimit.Custom.Set(source, sourceRateLimit.MaxCount)

Fix: Added CustomDuration mapsutil.SyncLockMap[string, time.Duration] to CustomRateLimit and populated it alongside Custom.

Bug 2 — Global -rl not applied to unlisted sources in passive.go

buildMultiRateLimiter only entered the rate-limit branch when a source was present in rateLimit.Custom. Sources not listed in -rls received math.MaxUint32 (unlimited) even when -rl was set.

// Before — global rate limit path never reached for unlisted sources
if sourceRateLimit, ok := rateLimit.Custom.Get(sourceName); ok {
    rl = sourceRateLimitOrDefault(uint(globalRateLimit), sourceRateLimit)
}
// rl stays 0 if source not in map → assigned MaxUint32 (unlimited)

Fix: Added else if globalRateLimit > 0 branch so all sources without a per-source override receive the global limit.

Affected files

File Change
pkg/subscraping/types.go Add CustomDuration field to CustomRateLimit
pkg/runner/runner.go Populate CustomDuration; add time import
pkg/passive/passive.go Use per-source duration; apply global rate limit as fallback
pkg/passive/ratelimit_test.go 4 unit tests covering all fix branches

Tests

=== RUN   TestBuildMultiRateLimiter_PerSourceLimit
--- PASS (0.00s)
=== RUN   TestBuildMultiRateLimiter_GlobalRateLimit
--- PASS (0.00s)
=== RUN   TestBuildMultiRateLimiter_GlobalFallbackInCustomMap
--- PASS (0.00s)
=== RUN   TestBuildMultiRateLimiter_NoLimits
--- PASS (0.00s)
ok  github.com/projectdiscovery/subfinder/v2/pkg/passive   0.051s

Summary by CodeRabbit

  • New Features

    • Per-source duration-based rate limits with global fallback and case-insensitive source names.
  • Bug Fixes

    • Safeguards when rate-limit config is missing to avoid panics and ensure correct global behavior.
  • Tests

    • Added tests for per-source limits, global fallback, nil/missing configs, and unlimited scenarios.

)

Two bugs in the rate-limit pipeline:

1. **Duration dropped (runner.go):** `CustomRateLimit` only stored
   `MaxCount`; the `Duration` from `-rls key=N/d` was silently discarded.
   A new `CustomDuration` map is added to `CustomRateLimit` and populated
   alongside `Custom` in `runner.go`.

2. **Global `-rl` ignored for unregistered sources (passive.go):**
   `buildMultiRateLimiter` only applied a rate limit when a source was
   present in the custom map.  Sources absent from `-rls` received
   `math.MaxUint32` (unlimited) even when `-rl` was set.
   The fix applies `globalRateLimit` as a fallback for those sources.

Fixes #1434.
@neo-by-projectdiscovery-dev
Copy link
Copy Markdown

neo-by-projectdiscovery-dev bot commented Mar 9, 2026

Neo - PR Security Review

No security issues found

Highlights

  • Adds case normalization (strings.ToLower) for source names in runner.go to ensure consistent rate-limit map lookup
  • Includes corresponding test case to verify case-insensitive lookup behavior works correctly
  • Changes improve robustness of rate-limiting configuration without introducing attack vectors

Comment @pdneo help for available commands. · Open in Neo

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Mar 9, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 9112fb64-5a20-4918-a9d8-a2a39f53199a

📥 Commits

Reviewing files that changed from the base of the PR and between b0f724c and 317bd60.

📒 Files selected for processing (2)
  • pkg/passive/ratelimit_test.go
  • pkg/runner/runner.go
🚧 Files skipped from review as they are similar to previous changes (2)
  • pkg/runner/runner.go
  • pkg/passive/ratelimit_test.go

Walkthrough

Adds per-source duration support to passive rate limiting: introduces CustomDuration mapping, populates it during Runner initialization, and updates the multi-rate-limiter builder to use lowercased source keys and apply per-source durations (with fallbacks) when creating rate limiters.

Changes

Cohort / File(s) Summary
Rate limiter logic
pkg/passive/passive.go
Adds nil-check for rate limit, lowercases source keys, computes per-source duration (defaults to second), applies per-source or global duration when creating rate limiters, preserves unlimited fallback behavior.
Tests
pkg/passive/ratelimit_test.go
Adds tests and helper newCustomRateLimit to validate per-source limits, global fallback, nil handling, and no-limit behavior.
Runner configuration
pkg/runner/runner.go
Populates CustomDuration for sources with configured durations and stores keys lowercased alongside existing per-source count limits.
Type definition
pkg/subscraping/types.go
Adds exported CustomDuration mapsutil.SyncLockMap[string, time.Duration] to CustomRateLimit to store per-source durations.

Sequence Diagram

sequenceDiagram
    participant Runner
    participant CustomRateLimit
    participant Builder as buildMultiRateLimiter
    participant RateLimiter

    Runner->>CustomRateLimit: populate `Custom` (counts) and `CustomDuration` (durations) with lowercased keys
    Builder->>CustomRateLimit: read per-source count & duration (lowercased)
    Builder->>Builder: determine effective duration (per-source -> global -> default)
    Builder->>RateLimiter: add per-source limiter with count & duration (or unlimited fallback)
    RateLimiter->>RateLimiter: enforce request pacing per source
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 I tuned the clocks with nimble paws,
Per-source beats now heed the laws,
Lowercase names, each pause well-timed,
Hop—requests hum in rhythm, primed! 🥕

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main changes: fixing rate limit duration honor and applying global rate limit to all sources, directly matching the PR objectives.
Linked Issues check ✅ Passed The PR fully addresses issue #1434: it implements per-source duration handling from -rls, applies global -rl to unlisted sources, adds nil-safety, and includes comprehensive test coverage for all scenarios.
Out of Scope Changes check ✅ Passed All changes are directly in scope: CustomDuration field addition, per-source duration handling, global fallback logic, nil-check safety, and comprehensive test coverage all directly support fixing #1434 rate-limiting bugs.
Docstring Coverage ✅ Passed Docstring coverage is 87.50% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
pkg/passive/passive.go (1)

85-114: ⚠️ Potential issue | 🔴 Critical

Nil pointer dereference when rateLimit is nil.

If EnumerateSubdomainsWithCtx is called without the WithCustomRateLimit option, enumerateOptions.customRateLimiter will be nil, causing a panic at line 93 when calling rateLimit.Custom.Get(sourceName).

🐛 Proposed fix: Add nil guard at the start of the function
 func (a *Agent) buildMultiRateLimiter(ctx context.Context, globalRateLimit int, rateLimit *subscraping.CustomRateLimit) (*ratelimit.MultiLimiter, error) {
 	var multiRateLimiter *ratelimit.MultiLimiter
 	var err error
+	
+	if rateLimit == nil {
+		rateLimit = &subscraping.CustomRateLimit{}
+	}
+	
 	for _, source := range a.sources {
 		var rl uint
 		duration := time.Second // default duration when a rate limit is active

Alternatively, you could add nil checks before each Get call, but initializing an empty struct is cleaner.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/passive/passive.go` around lines 85 - 114, The function
buildMultiRateLimiter can nil-deref rateLimit when
enumerateOptions.customRateLimiter is not set; add a nil-guard at the top of
buildMultiRateLimiter (or initialize enumerateOptions.customRateLimiter earlier)
so rateLimit is non-nil (e.g., if rateLimit == nil { rateLimit =
&subscraping.CustomRateLimit{} }) before calling rateLimit.Custom.Get or
rateLimit.CustomDuration.Get; this prevents the panic when
EnumerateSubdomainsWithCtx is called without WithCustomRateLimit while keeping
the current per-source/global logic intact.
🧹 Nitpick comments (1)
pkg/passive/ratelimit_test.go (1)

34-126: Good test coverage for the rate-limiting scenarios.

The four tests effectively cover:

  • Per-source limits with custom duration
  • Global rate limit fallback for unlisted sources
  • Count=0 fallback to global rate limit
  • Unlimited behavior when no flags are set

Consider adding a test for when rateLimit is nil to verify the fix for the nil pointer issue flagged in passive.go:

🧪 Optional: Add test for nil rateLimit handling
// TestBuildMultiRateLimiter_NilRateLimit verifies that a nil CustomRateLimit
// does not cause a panic and correctly applies the global rate limit.
func TestBuildMultiRateLimiter_NilRateLimit(t *testing.T) {
	agent := New([]string{"hackertarget"}, []string{}, false, false)

	globalRateLimit := 5
	mrl, err := agent.buildMultiRateLimiter(context.Background(), globalRateLimit, nil)
	require.NoError(t, err)
	require.NotNil(t, mrl)

	limit, err := mrl.GetLimit("hackertarget")
	require.NoError(t, err)
	assert.Equal(t, uint(globalRateLimit), limit, "source should use global rate limit when CustomRateLimit is nil")
}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/passive/ratelimit_test.go` around lines 34 - 126, Add a new test named
TestBuildMultiRateLimiter_NilRateLimit that calls agent.buildMultiRateLimiter
with a non-zero globalRateLimit and a nil CustomRateLimit, then assert no error
and that mrl.GetLimit("hackertarget") returns the global rate
(uint(globalRateLimit)); this verifies buildMultiRateLimiter and its handling of
a nil *CustomRateLimit (reference buildMultiRateLimiter, CustomRateLimit,
GetLimit, and the new TestBuildMultiRateLimiter_NilRateLimit).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@pkg/passive/passive.go`:
- Around line 85-114: The function buildMultiRateLimiter can nil-deref rateLimit
when enumerateOptions.customRateLimiter is not set; add a nil-guard at the top
of buildMultiRateLimiter (or initialize enumerateOptions.customRateLimiter
earlier) so rateLimit is non-nil (e.g., if rateLimit == nil { rateLimit =
&subscraping.CustomRateLimit{} }) before calling rateLimit.Custom.Get or
rateLimit.CustomDuration.Get; this prevents the panic when
EnumerateSubdomainsWithCtx is called without WithCustomRateLimit while keeping
the current per-source/global logic intact.

---

Nitpick comments:
In `@pkg/passive/ratelimit_test.go`:
- Around line 34-126: Add a new test named
TestBuildMultiRateLimiter_NilRateLimit that calls agent.buildMultiRateLimiter
with a non-zero globalRateLimit and a nil CustomRateLimit, then assert no error
and that mrl.GetLimit("hackertarget") returns the global rate
(uint(globalRateLimit)); this verifies buildMultiRateLimiter and its handling of
a nil *CustomRateLimit (reference buildMultiRateLimiter, CustomRateLimit,
GetLimit, and the new TestBuildMultiRateLimiter_NilRateLimit).

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: cb7587eb-71d8-4236-9db1-1260878a74f5

📥 Commits

Reviewing files that changed from the base of the PR and between 13e5db7 and 902b08a.

📒 Files selected for processing (4)
  • pkg/passive/passive.go
  • pkg/passive/ratelimit_test.go
  • pkg/runner/runner.go
  • pkg/subscraping/types.go

…miter

Prevent a nil pointer dereference when EnumerateSubdomainsWithCtx is
called without the WithCustomRateLimit option (rateLimit would be nil,
causing a panic on rateLimit.Custom.Get).

Also add TestBuildMultiRateLimiter_NilRateLimit to cover this path.
@marchantdev
Copy link
Copy Markdown
Author

Addressed CodeRabbit review:

  • Critical nil pointer dereference fixed: Added nil guard at start of buildMultiRateLimiter — if rateLimit is nil, it's initialized to an empty &subscraping.CustomRateLimit{} before any field access. This prevents panic when EnumerateSubdomainsWithCtx is called without the WithCustomRateLimit option.
  • Added TestBuildMultiRateLimiter_NilRateLimit test to cover this path — all 5 rate limiter tests pass.

Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
pkg/passive/ratelimit_test.go (1)

34-57: Consider adding a test for case-insensitive source name lookup.

All tests use lowercase keys (e.g., "hackertarget"), which won't catch the case mismatch issue between storage and lookup. Adding a test with mixed-case keys would validate that the lowercasing behavior works end-to-end.

🧪 Suggested test for case-insensitive lookup
// TestBuildMultiRateLimiter_CaseInsensitiveLookup verifies that per-source
// limits work regardless of source name casing in the custom map.
func TestBuildMultiRateLimiter_CaseInsensitiveLookup(t *testing.T) {
	agent := New([]string{"hackertarget"}, []string{}, false, false)

	// Simulate user providing mixed-case source name via -rls
	crl := newCustomRateLimit(
		map[string]uint{"HackerTarget": 8},
		map[string]time.Duration{"HackerTarget": time.Minute},
	)

	mrl, err := agent.buildMultiRateLimiter(context.Background(), 0, crl)
	require.NoError(t, err)
	require.NotNil(t, mrl)

	// Should still match despite case difference
	limit, err := mrl.GetLimit("hackertarget")
	require.NoError(t, err)
	assert.Equal(t, uint(8), limit, "should find per-source limit regardless of case in custom map")
}

Note: This test would currently fail due to the case mismatch bug identified in passive.go.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/passive/ratelimit_test.go` around lines 34 - 57, Add a new unit test
(e.g., TestBuildMultiRateLimiter_CaseInsensitiveLookup) alongside
TestBuildMultiRateLimiter_PerSourceLimit that constructs an agent with a
lowercase source list, then creates a custom rate limit via newCustomRateLimit
using a mixed-case key (e.g., "HackerTarget") and duration, calls
agent.buildMultiRateLimiter(context.Background(), 0, crl), and asserts
mrl.GetLimit("hackertarget") returns the configured uint value; this verifies
case-insensitive lookup end-to-end and will expose the current casing bug.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@pkg/passive/ratelimit_test.go`:
- Around line 34-57: Add a new unit test (e.g.,
TestBuildMultiRateLimiter_CaseInsensitiveLookup) alongside
TestBuildMultiRateLimiter_PerSourceLimit that constructs an agent with a
lowercase source list, then creates a custom rate limit via newCustomRateLimit
using a mixed-case key (e.g., "HackerTarget") and duration, calls
agent.buildMultiRateLimiter(context.Background(), 0, crl), and asserts
mrl.GetLimit("hackertarget") returns the configured uint value; this verifies
case-insensitive lookup end-to-end and will expose the current casing bug.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: b979cbd2-dd74-4910-9bef-5fa6ad2e0a5a

📥 Commits

Reviewing files that changed from the base of the PR and between 902b08a and b0f724c.

📒 Files selected for processing (2)
  • pkg/passive/passive.go
  • pkg/passive/ratelimit_test.go

passive.go looks up custom rate limits using strings.ToLower(source.Name()),
but runner.go stored them with the original casing from the CLI flag. A user
passing -rls HackerTarget=5/m would never match the lowercase lookup key,
silently falling back to unlimited instead of the configured limit.

Fix: lowercase the source key when storing in runner.go so both storage and
lookup are consistent.

Also add TestBuildMultiRateLimiter_CaseInsensitiveLookup to verify that a
mixed-case key in the custom map is found correctly end-to-end.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@marchantdev
Copy link
Copy Markdown
Author

Addressed CodeRabbit's nitpick and found the underlying bug:

Root cause: runner.go stored -rls keys with original user casing (e.g., HackerTarget), but passive.go looks them up with strings.ToLower(source.Name()) — always lowercase. Mixed-case keys would silently fall back to unlimited rate.

Fix in runner/runner.go: Normalise -rls source keys to lowercase when storing so storage and lookup are consistent.

New test TestBuildMultiRateLimiter_CaseInsensitiveLookup: Verifies the end-to-end match works correctly.

All 6 TestBuildMultiRateLimiter_* tests pass.

@marchantdev
Copy link
Copy Markdown
Author

Hi @Ice3man543 / @ehsandeep — could you take a look when you get a chance? The automated reviews (Neo security + CodeRabbit) both came back clean, and the nil pointer guard has been addressed. Happy to make any changes you need. 🙏

@Mzack9999
Copy link
Copy Markdown
Member

Closing this PR as #1764 has been merged to address this issue.

Your PR was very close to being selected — it fixed all three bugs (global -rl fallback, -rls duration, nil safety) and had strong tests with actual value assertions via mrl.GetLimit(). The key normalization to lowercase in runner.go was a nice defensive touch that the merged PR didn't include. Ultimately #1764 was chosen for its slightly cleaner resolveSourceRateLimit helper and broader test coverage (11 test functions). Great work — thank you for the thorough contribution!

@Mzack9999 Mzack9999 closed this Mar 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Issue] The rl,rls options are not supported

2 participants