Adding a plain btree index on Tag.value is unlikely to help the new tag_value substring filter, since PostgreSQL typically cannot use a btree index for ILIKE/__icontains queries. This adds write/storage overhead without improving the main query.

If tag_value filtering needs to be performant, consider a trigram GIN/GiST index (pg_trgm) or changing the filter semantics to equality/prefix where an index can be used.

Hi @regulartim
When I was integrating Phase 2, my PR had previously created the 0038_add_tag_model migration file. Due to copilot review feedback, i needed to optimize the database queries by adding db_index=True directly into the Tag model table. To avoid polluting the DB migrations with two sequential model modifications back-to-back, I deleted 0038 and completely regenerated it as a clean new migration called 0038_tag.py.

In the meantime, develop was merged into my branch, bringing in a new migration 0039 which had 0038_add_tag_model hardcoded as its dependency. I manually updated dependencies = [('greedybear', '0038_tag')] inside 0039_ioc_attacker_country_sensor_country_and_more.py to fix this clash and restore the DB schema graph cleanly!

is it fine or would it create some complications?

Hey @opbot-xd ! :) I think it's better to just create a new migration. Changing old migration files after they landed in a branch that other people could use, might break their instances in unpredictable ways.

Why do you fetch the tag here and not in get_queryset? There you could just join them like the general honeypots:

    if not is_aggregated:
        iocs = iocs.filter(general_honeypot__active=True)
        iocs = iocs.annotate(honeypots=ArrayAgg("general_honeypot__name"))
        iocs = iocs.annotate(tags_json=ArrayAgg(...)
        iocs = iocs.order_by(feed_params.ordering)
        iocs = iocs[: int(feed_params.feed_size)]

This would be much simpler as we don't need to cover the case where iocs is no queryset.

Nice!
This approach is much cleaner! Though I had to also add the tags_json annotation to the repository methods (get_scanners_by_pks and get_recent_scanners) since they bypass get_queryset but still call feeds_response for ML scoring workflows.

The benefit is worth it though.

Changes made:

• Moved tags_json annotation to get_queryset alongside honeypots
• Removed tags_lookup dictionary logic from feeds_response
• Added tags_json annotation to IocRepository.get_scanners_by_pks() and IocRepository.get_recent_scanners()

If I understand your code correctly, the changes to this file should server the tags to the ML jobs, right? But there, they are thrown away again, because get_features does not know of them.

Also, I am not a 100% sure if it is a good idea to use the Tags for training. This might introduce weird biases. I had the similar doubts when I added the ip_reputation as a feature, but decided to include it after testing the inclusion thoroughly.

So, bottom line: I would prefer not to add the Tags to scoring for now. This would be a separate issue / PR requiring a lot of research and testing.

Thanks for pointing this out!
Updated the logic to handle both cases correctly:

if isinstance(iocs, list):
    has_tags_annotation = bool(iocs) and hasattr(iocs[0], "tags_json")
else:
    has_tags_annotation = "tags_json" in getattr(
        iocs, "query", type("", (), {"annotations": {}})()"
    ).annotations