Skip to content

1.4.0#433

Merged
mlodic merged 20 commits intomainfrom
develop
Feb 10, 2025
Merged

1.4.0#433
mlodic merged 20 commits intomainfrom
develop

Conversation

@mlodic
Copy link
Copy Markdown
Collaborator

@mlodic mlodic commented Feb 7, 2025

Checklist for creating a new release

  • Change version number in docker/.version
  • Verify CI Tests
  • Verify that the PR is named with a correct version number like x.x.x
  • Merge the PR to the main branch. The release will be done automatically by the CI

Note: Only use "Merge and commit" as the merge strategy and not "Squash and merge". Using "Squash and merge" makes history between branches misaligned.

dependabot bot and others added 15 commits December 5, 2024 19:20
Bumps [django-ses](https://github.com/django-ses/django-ses) from 4.1.1 to 4.3.0.
- [Release notes](https://github.com/django-ses/django-ses/releases)
- [Changelog](https://github.com/django-ses/django-ses/blob/main/CHANGES.md)
- [Commits](django-ses/django-ses@v4.1.1...v4.3.0)

---
updated-dependencies:
- dependency-name: django-ses
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [elasticsearch-dsl](https://github.com/elasticsearch/elasticsearch-dsl-py) from 8.15.0 to 8.16.0.
- [Release notes](https://github.com/elasticsearch/elasticsearch-dsl-py/releases)
- [Changelog](https://github.com/elastic/elasticsearch-dsl-py/blob/main/Changelog.rst)
- [Commits](elastic/elasticsearch-dsl-py@v8.15.0...v8.16.0)

---
updated-dependencies:
- dependency-name: elasticsearch-dsl
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [slack-sdk](https://github.com/slackapi/python-slack-sdk) from 3.31.0 to 3.33.4.
- [Release notes](https://github.com/slackapi/python-slack-sdk/releases)
- [Commits](slackapi/python-slack-sdk@v3.31.0...v3.33.4)

---
updated-dependencies:
- dependency-name: slack-sdk
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [uwsgi](https://uwsgi-docs.readthedocs.io/en/latest/) from 2.0.26 to 2.0.28.

---
updated-dependencies:
- dependency-name: uwsgi
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [psycopg2-binary](https://github.com/psycopg/psycopg2) from 2.9.9 to 2.9.10.
- [Changelog](https://github.com/psycopg/psycopg2/blob/master/NEWS)
- [Commits](https://github.com/psycopg/psycopg2/commits)

---
updated-dependencies:
- dependency-name: psycopg2-binary
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* making pendng migration

* add container class for general honeypots such that every honeypot gets its own instance

* adapt test to new class name

* Revert "making pendng migration", as it has nothing to do with the fix. This commit will be included in a future PR.

This reverts commit ca95b9a.

* fix method calls having wrong arguments

* add test case for #395 (strongly depended on the individual setup, therefore in manual_tests)
* making pendng migration

* Revert "making pendng migration", as it has nothing to do with the fix. This commit will be included in a future PR.

This reverts commit ca95b9a.

* improve API performance by prefetching data from general honeypot model

* improve API performance by replacing IOC serialization via Serializer class with custom function

* add basic validation functions

* adapt tests to new validation methods

* slightly modify feed_type_validation to pass test cases

* revert Django Serializers removal

* retrieve valid feed types from GeneralHoneypots at the start of the API views

* adapt test to new feed type validation method

* add constant to have the option to skip feed validation for better performance

* add comment to clarify why we do not use .first()

* move SKIP_FEED_VALIDATION option to env_file

* use the object-level validation with extra-context for feed type validation

* remove caching as the cache does not get cleared between requests which may lead to ignoring admin changes

* Revert "adapt test to new feed type validation method"

This reverts commit 3fbbcc0.

* add extra-context to serializers in tests

---------

Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com>
* introduce options for using the old extraction process and for modifying the extraction interval in ev_file

* add options in settings

* add custom extraction interval to cronjobs

* rework extraction process
* making pendng migration

* add cowrie session model and extend IOC model

* apply psf black formatting

* change ioc model default for more simple handling in _add_ioc function

* fix error in _add_ioc: new IOC instances not able to access ManyToMany relation with GeneralHoneypot

* minor model modifications

* remove unnecessary exception class

* add method to extract more information about attackers from TPot

* rewrite data extraction process for general honeypot class to extract more data

* rewrite data extraction process for cowrie class to extract more data and add cowrie session data extraction

* revert already made migration

* rename times_seen to attack_count

* minor model tweaks

* add model migration

* add data migration

* fill attack and interaction count correctly

* Rename header in frontend code

* base first_seen and last_seen on TPot timestamps instead of extraction time

* add model tests

* change default value of login_attempts to 0

* minor improvements

* increment IOCs login attempt counter on detection in cowrie session extraction

* bump alpine from 3.18 to 3.21 in frontend build
* create index on name field of IOC model to speed up _add_ioc function

* use QuerySet.exist() for better performance

* hand over previously added IOC record to _get_sessions method to reduce number of DB queries

* fix returning wrong IOC object

* add more error-resistant time window calculation

* document additional_lookback argument

* minor improvements to get_time_window function

* add test cases for get_time_window function

* fix error in docstring

* remove argument from function that is already a configuration setting and adapt tests accordingly
* add new feeds API

* adapt tests to changed serializer

* switch to Token Authentication to match other views

* remove unused import

* add destination_port_count to serializer

* update documentation

* add validation for feed requests

* add api tests

* add serializer tests

* fix serializer

* store feed request parameters as strings for easier validation and adapt tests accordingly

* store request parameters in FeedRequestParams object in old feeds APIs and remove unused code

* correct age conversion

* pass FeedRequestParams instance to feeds_response function

* rename new API endpoint to "advanced"

* split views in separate files

* force json format for paginated responses

* add option to exclude mass scanners to old APIs

* fix ordering_validation function and add docstrings to both helper functions for field-level validation

* add docstring to get_valid_feed_types()

* add support for sorting by feed_type in FeedRequestParams class

* multiple performance optimizations
- remove logging the number of returned IOCs in get_queryset() as this forces early QuerySet evaluation
- only pass filter for number_of_days_seen if it is > 1
- build list of general honeypots, that have seen an IOC, directly on the database
- rewrite feeds_response() function

* remove SKIP_FEED_VALIDATION setting as this can now be done by a query parameter

* change the way data is excluded from inactive honeypots (see #429)

* adjusted log

---------

Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com>
@mlodic
Copy link
Copy Markdown
Collaborator Author

mlodic commented Feb 7, 2025

@regulartim if you think we are ready to test it, I can try to upload it in the Honeynet servers.

@mlodic
Copy link
Copy Markdown
Collaborator Author

mlodic commented Feb 7, 2025

@drosetti FYI this deploy could impact internal Certego's integration.

@regulartim
Copy link
Copy Markdown
Collaborator

It is ready for testing in my opinion. I am running this changes for a month now on my system without any problems.

dependabot bot and others added 5 commits February 10, 2025 09:54
Bumps [slack-sdk](https://github.com/slackapi/python-slack-sdk) from 3.33.4 to 3.34.0.
- [Release notes](https://github.com/slackapi/python-slack-sdk/releases)
- [Commits](slackapi/python-slack-sdk@v3.33.4...v3.34.0)

---
updated-dependencies:
- dependency-name: slack-sdk
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps library/nginx from 1.27.0-alpine to 1.27.3-alpine.

---
updated-dependencies:
- dependency-name: library/nginx
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…424)

* Bump axios from 0.27.2 to 0.28.0 in /frontend (#270)

Bumps [axios](https://github.com/axios/axios) from 0.27.2 to 0.28.0.
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/v0.28.0/CHANGELOG.md)
- [Commits](axios/axios@v0.27.2...v0.28.0)

---
updated-dependencies:
- dependency-name: axios
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Revert "Bump axios from 0.27.2 to 0.28.0 in /frontend (#270)" (#286)

This reverts commit 080321e.

* Bump django from 4.2.15 to 4.2.17 in /requirements in the pip group

Bumps the pip group in /requirements with 1 update: [django](https://github.com/django/django).


Updates `django` from 4.2.15 to 4.2.17
- [Commits](django/django@4.2.15...4.2.17)

---
updated-dependencies:
- dependency-name: django
  dependency-type: direct:production
  dependency-group: pip
...

Signed-off-by: dependabot[bot] <support@github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Martina Carella <m.carella@external.certego.net>
Co-authored-by: Daniele Rosetti <d.rosetti@certego.net>
Co-authored-by: 0ssigeno <s.berni@certego.net>
Co-authored-by: Martina Carella <m.carella@certego.net>
Co-authored-by: Daniele Rosetti <55402684+drosetti@users.noreply.github.com>
Bumps [django-ses](https://github.com/django-ses/django-ses) from 4.3.0 to 4.4.0.
- [Release notes](https://github.com/django-ses/django-ses/releases)
- [Changelog](https://github.com/django-ses/django-ses/blob/main/CHANGES.md)
- [Commits](django-ses/django-ses@v4.3.0...v4.4.0)

---
updated-dependencies:
- dependency-name: django-ses
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@mlodic mlodic merged commit 19b4a3a into main Feb 10, 2025
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants