refactor:honeypot extraction using DB-driven exclusion. closes #631 by drona-gyawali · Pull Request #670 · GreedyBear-Project/GreedyBear

drona-gyawali · 2026-01-02T17:18:23Z

Description

This PR refactors the is_ready_for_extraction method to respect the GeneralHoneypot.active flag, making honeypot exclusion fully database-driven. No honeypots are hardcoded, and no migrations were added as discussed previously.

Changes

Normalizes honeypot names for cache lookup while preserving original casing in the DB.
Uses name__iexact for case-insensitive DB lookup.
Dynamically creates new honeypots with active=True.
Updates _honeypot_cache to reflect the DB active state.
Extraction pipeline now automatically skips honeypots where active=False.

Notes / Observations

The code assumes _honeypot_cache always uses normalized keys. If elsewhere the cache is populated with original casing, it could cause a cache miss and fallback to the DB. This was already the case before this change.
Currently, the GeneralHoneypot.name field is not unique at the DB level. I didn’t change this, but we might consider adding a unique constraint. Am I missing anything here?

Related issues

closes #631

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue).
New feature (non-breaking change which adds functionality).
Breaking change (fix or feature that would cause existing functionality to not work as expected).

Checklist

I have read and understood the rules about how to Contribute to this project.
The pull request is for the branch develop.
I have added documentation of the new features.
Linters (Black, Flake, Isort) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.
I have added tests for the feature/bug I solved. All the tests (new and old ones) gave 0 errors.
If changes were made to an existing model/serializer/view, the docs were updated and regenerated (check CONTRIBUTE.md).
If the GUI has been modified:
- I have a provided a screenshot of the result in the PR.
- I have created new frontend tests for the new component or updated existing ones.

Important Rules

If you miss to compile the Checklist properly, your PR won't be reviewed by the maintainers.
If your changes decrease the overall tests coverage (you will know after the Codecov CI job is done), you should add the required tests to fix the problem
Everytime you make changes to the PR and you think the work is done, you should explicitly ask for a review. After being reviewed and received a "change request", you should explicitly ask for a review again once you have made the requested changes.

drona-gyawali · 2026-01-02T17:27:08Z

Hi @regulartim,
I’ve refactored is_ready_for_extraction to fully respect GeneralHoneypot.active, and all tests are passing.

Could you clarify the plan for setting up the initial honeypot exclusion list? Will it be handled via a migration file, raw SQL, the admin interface, or some other method? I want to make sure our implementation aligns with the intended workflow.

regulartim

I think your approach makes matters more complicated than they were before: it has a cache that contains the not-normalized honeypot names as keys (they usually start with an upper case character) so your first cache lookup always fails and, in the next step, you write a cache entry for the normalized name. Am I right?
What do you think about only writing normalized keys to the cache? That would make things easier, right?

greedybear/cronjobs/repositories/ioc.py

regulartim · 2026-01-02T19:24:07Z

Could you clarify the plan for setting up the initial honeypot exclusion list? Will it be handled via a migration file, raw SQL, the admin interface, or some other method? I want to make sure our implementation aligns with the intended workflow.

Yep, will write that into the issue. 👍

regulartim · 2026-01-04T21:56:13Z

Is this ready to get reviewed again?

drona-gyawali · 2026-01-05T05:19:26Z

Is this ready to get reviewed again?

Hi @regulartim,

Yes, it’s ready for review again.

I added a migration file for the initial honeypot setup. In the migration, I’m using a try/except (get → create) pattern intentionally, since it’s limited strictly to the migration and only used for the one-time initial setup.

Runtime extraction now relies solely on the normalized cache, with no DB creation or fallback logic involved.

Please let me know if you’d like any adjustments.

regulartim · 2026-01-05T19:37:48Z

Ah, I didn't notice it was ready. You can click "re-request review" next time, then that I get notified. I'll take a look now.

regulartim

I seems like we are having some communication issues. If your not sure what to do, please thoroughly read the issue and the comments in the PR again. If you still have questions or things are unclear, please ask (in the issue or the PR, as you like).

greedybear/cronjobs/repositories/ioc.py

greedybear/migrations/0026_disabled_unwanted_honeypots.py

…alize cache keys

regulartim

Looks good. Thanks for your work. One last thing that I am concerned about:
Say we have a honeypot named "Cowrie" in our database. Now for some odd reason we extract an event from T-Pot where the name of the honeypot is lower case "cowrie". What do you think would happen then? Can we cover this with one or more tests?

tests/test_repositories.py

drona-gyawali · 2026-01-06T11:47:19Z

Looks good. Thanks for your work. One last thing that I am concerned about:
Say we have a honeypot named "Cowrie" in our database. Now for some odd reason we extract an event from T-Pot where the name of the honeypot is lower case "cowrie". What do you think would happen then? Can we cover this with one or more tests?

Thanks for the review!

I’ve added three tests that together cover the case-insensitive handling of honeypot names:

Ensures an enabled honeypot like "Cowrie" works even when called as lowercase "cowrie".
Ensures a disabled honeypot returns False even with a lowercase lookup.
Confirms that special honeypots (Cowrie, Log4Pot) are always enabled and normal honeypots respect their active status, all in a case-insensitive manner.

I hope this tests satisfy the concern about case-insensitive extraction from T-Pot events.

…code-val

regulartim

We're getting close! 👍

greedybear/migrations/0027_disable_unwanted_honeypots.py

tests/test_repositories.py

greedybear/cronjobs/repositories/ioc.py

regulartim · 2026-01-08T10:41:24Z

Thanks for your work! :) As a follow up, would you like to open a new issue describing the problem of the not-normalized honeypot names in the DB and how to handle this?

drona-gyawali · 2026-01-08T11:16:18Z

Thanks for your work! :) As a follow up, would you like to open a new issue describing the problem of the not-normalized honeypot names in the DB and how to handle this?

Thanks for the review and merge!
I’ll open a new issue describing the problem and the approach to handle this.

refactor:honeypot extraction using DB-driven exclusion

e33996c

regulartim marked this pull request as ready for review January 2, 2026 19:02

regulartim requested changes Jan 2, 2026

View reviewed changes

greedybear/cronjobs/repositories/ioc.py Outdated Show resolved Hide resolved

greedybear/cronjobs/repositories/ioc.py Outdated Show resolved Hide resolved

drona-gyawali added 3 commits January 3, 2026 05:46

feat/refactor: added migration file and changes in extraction

fd52854

Merge branch 'develop' into refactor/no-hardcode-val

ff4e446

refactor: DB creation behavior is deferred

6844810

regulartim requested changes Jan 5, 2026

View reviewed changes

greedybear/cronjobs/repositories/ioc.py Outdated Show resolved Hide resolved

greedybear/migrations/0026_disabled_unwanted_honeypots.py Outdated Show resolved Hide resolved

drona-gyawali added 2 commits January 6, 2026 07:23

fix(ioc): restore create_honeypot in is_ready_for_extraction and norm…

abe0571

…alize cache keys

Merge branch 'develop' into refactor/no-hardcode-val

39219ed

drona-gyawali requested a review from regulartim January 6, 2026 07:37

regulartim reviewed Jan 6, 2026

View reviewed changes

tests/test_repositories.py Outdated Show resolved Hide resolved

test(repo): Add case-insensitive tests for honeypot extraction

a248c10

drona-gyawali added 2 commits January 6, 2026 12:04

Merge remote-tracking branch 'upstream/develop' into refactor/no-hard…

6512ed3

…code-val

resolve: conflict

e635d48

drona-gyawali requested a review from regulartim January 6, 2026 12:14

add test for insesitive honeypot retrieval

1d0dbac

regulartim requested changes Jan 8, 2026

View reviewed changes

greedybear/migrations/0027_disable_unwanted_honeypots.py Show resolved Hide resolved

tests/test_repositories.py Show resolved Hide resolved

refactor(repo): implement insensitive lookup in get_hp_by_name

da5cc08

drona-gyawali requested a review from regulartim January 8, 2026 09:58

regulartim reviewed Jan 8, 2026

View reviewed changes

greedybear/cronjobs/repositories/ioc.py Show resolved Hide resolved

regulartim approved these changes Jan 8, 2026

View reviewed changes

regulartim merged commit fc5b5f1 into GreedyBear-Project:develop Jan 8, 2026
5 checks passed

drona-gyawali mentioned this pull request Jan 8, 2026

Normalize GeneralHoneypot names and enforce uniqueness #689

Closed

Uh oh!

Conversation

drona-gyawali commented Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Notes / Observations

Related issues

Type of change

Checklist

Important Rules

Uh oh!

drona-gyawali commented Jan 2, 2026

Uh oh!

regulartim left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

regulartim commented Jan 2, 2026

Uh oh!

regulartim commented Jan 4, 2026

Uh oh!

drona-gyawali commented Jan 5, 2026

Uh oh!

regulartim commented Jan 5, 2026

Uh oh!

regulartim left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

regulartim left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

drona-gyawali commented Jan 6, 2026

Uh oh!

regulartim left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

regulartim commented Jan 8, 2026

Uh oh!

drona-gyawali commented Jan 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

drona-gyawali commented Jan 2, 2026 •

edited

Loading