Skip to content

Add rate limiting to feeds endpoints.#927

Merged
regulartim merged 3 commits intoGreedyBear-Project:developfrom
manik3160:feat/add-feeds-rate-limiting
Mar 4, 2026
Merged

Add rate limiting to feeds endpoints.#927
regulartim merged 3 commits intoGreedyBear-Project:developfrom
manik3160:feat/add-feeds-rate-limiting

Conversation

@manik3160
Copy link
Copy Markdown
Contributor

@manik3160 manik3160 commented Mar 3, 2026 •
edited
Loading

Description

Adds rate limiting to all feeds API endpoints using DRF's ScopedRateThrottle to prevent abuse and reduce unnecessary load on the database and Elasticsearch.

Per @regulartim's feedback on #923, throttling is applied only to feeds endpoints, not to enrichment, command_sequence, or cowrie_session (since researchers may need to use these heavily, and they require authentication).

Changes

  • api/views/feeds.py: Added ScopedRateThrottle to all 4 feeds views with scoped rates:
    • feeds + feeds_pagination (public) → 30 requests/minute per IP
    • feeds_advanced + feeds_asn (authenticated) → 100 requests/minute per user
  • greedybear/settings.py: Added DEFAULT_THROTTLE_RATES configuration
  • tests/api/views/test_feeds_throttle.py: Added 6 tests verifying throttle behavior

Related issues

Closes #923

Type of change

  • Bug fix (non-breaking change which fixes an issue).
  • New feature (non-breaking change which adds functionality).
  • Breaking change (fix or feature that would cause existing functionality to not work as expected).
  • Chore (refactoring, dependency updates, CI/CD changes, code cleanup, docs-only changes).

Checklist

Formalities

  • I have read and understood the rules about how to Contribute to this project.
  • I chose an appropriate title for the pull request in the form: <feature name>. Closes #999
  • My branch is based on develop.
  • The pull request is for the branch develop.
  • I have reviewed and verified any LLM-generated code included in this PR.

Docs and tests

  • I documented my code changes with docstrings and/or comments.
  • I have checked if my changes affect user-facing behavior that is described in the docs. If so, I also created a pull request in the docs repository.
  • Linter (Ruff) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.
  • I have added tests for the feature/bug I solved.
  • All the tests gave 0 errors.

@manik3160 manik3160 marked this pull request as draft March 3, 2026 07:20
@manik3160 manik3160 force-pushed the feat/add-feeds-rate-limiting branch from 7508c97 to 6abffeb Compare March 3, 2026 07:36
@manik3160 manik3160 marked this pull request as ready for review March 3, 2026 09:33
regulartim
regulartim requested changes Mar 3, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@regulartim regulartim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please make sure CI is happy.

Comment thread api/views/feeds.py Outdated
return feeds_response(iocs_queryset, feed_params, valid_feed_types)


feeds.throttle_scope = "feeds"
Copy link
Copy Markdown
Member

@regulartim regulartim Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The feeds.throttle_scope assignment sitting between function definitions is weird. According to the DRF docs there are cleaner ways to do that.

@manik3160
Copy link
Copy Markdown
Contributor Author

manik3160 commented Mar 3, 2026

Thanks for the feedback!
I’ll refactor the throttle configuration according to DRF best practices and make sure CI passes. I’ll push an update shortly.

@manik3160 manik3160 force-pushed the feat/add-feeds-rate-limiting branch from 6abffeb to c5658b7 Compare March 3, 2026 10:24
@manik3160 manik3160 requested a review from regulartim March 3, 2026 13:18
@manik3160 manik3160 marked this pull request as draft March 3, 2026 13:51
@manik3160 manik3160 force-pushed the feat/add-feeds-rate-limiting branch from c5658b7 to ff91817 Compare March 3, 2026 13:57
@manik3160 manik3160 marked this pull request as ready for review March 3, 2026 13:58
@manik3160 manik3160 force-pushed the feat/add-feeds-rate-limiting branch from ff91817 to 48e0cbb Compare March 3, 2026 14:01
fix: use SimpleRateThrottle instead of ScopedRateThrottle
9bc91ac 
ScopedRateThrottle.allow_request() always overwrites self.scope from
view.throttle_scope, which our function-based views don't set. This
caused scope to be None and all requests to bypass throttling entirely.

Switch to SimpleRateThrottle (like DRF's built-in AnonRateThrottle and
UserRateThrottle), which properly reads scope from the class attribute
and applies throttling in __init__.
@manik3160
Copy link
Copy Markdown
Contributor Author

manik3160 commented Mar 3, 2026

Hope CI is good, switched to SimpleRateThrottle for correct rate limiting, and added cache.clear() between tests to prevent throttle limits from leaking across the test suite.

regulartim
regulartim requested changes Mar 4, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@regulartim regulartim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @manik3160 , core implementation looks good. A few things:

  • cache.clear() runs on every of our ~500 test cases although it is only needed on the throttle tests
  • throttle rates should be configurable via environment variables in the env_file_templte
  • I can't find any test for unauthenticated access to public endpoints

@manik3160 manik3160 requested a review from regulartim March 4, 2026 10:27
@manik3160
Copy link
Copy Markdown
Contributor Author

manik3160 commented Mar 4, 2026

@regulartim Thanks for the review and feedback. I’ve updated the implementation by limiting cache.clear() to the throttle tests, making the throttle rate configurable via environment variables in the env_file_template, and adding a test for unauthenticated access to the public endpoints. Please let me know if anything else should be adjusted.

regulartim
regulartim approved these changes Mar 4, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@regulartim regulartim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good now, thanks! :)

@regulartim regulartim merged commit acd0688 into GreedyBear-Project:develop Mar 4, 2026
4 checks passed
@regulartim regulartim mentioned this pull request Mar 5, 2026
Merged
8 tasks
cclts pushed a commit to cclts/GreedyBear that referenced this pull request Mar 11, 2026
@manik3160 @cclts
Add rate limiting to feeds endpoints. Closes GreedyBear-Project#923 (G…
67fc7fe 
…reedyBear-Project#927)

* Add rate limiting to feeds endpoints.

* fix: use SimpleRateThrottle instead of ScopedRateThrottle

ScopedRateThrottle.allow_request() always overwrites self.scope from
view.throttle_scope, which our function-based views don't set. This
caused scope to be None and all requests to bypass throttling entirely.

Switch to SimpleRateThrottle (like DRF's built-in AnonRateThrottle and
UserRateThrottle), which properly reads scope from the class attribute
and applies throttling in __init__.

* refactor: address review — scope cache.clear, env-configurable rates, unauth tests

---------

Co-authored-by: Manik <maniksingh3606@gmaile.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@regulartim regulartim regulartim approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@manik3160 @regulartim