Skip to content

Add Recorded Future support to threatintel module#26481

Merged
adriansr merged 3 commits intoelastic:masterfrom
adriansr:recorded_future
Jun 28, 2021
Merged

Add Recorded Future support to threatintel module#26481
adriansr merged 3 commits intoelastic:masterfrom
adriansr:recorded_future

Conversation

@adriansr
Copy link
Copy Markdown
Contributor

@adriansr adriansr commented Jun 24, 2021

This adds a new fileset, recordedfuture, to the treatintel module. It ingests indicators via the Recorded Future Connect API.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Screenshots

image

@adriansr
Add Recorded Future support to threatintel module
0625638 
This adds a new fileset, `recordedfuture`, to the treatintel module. It
ingests indicators via the Recorded Future Connect API.
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 24, 2021

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 24, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 24, 2021 •
edited by jenkins-beats-ci bot
Loading

❕ Build Aborted

Either there was a build timeout or someone aborted the build.'}

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: Pull request #26481 updated

  • Start Time: 2021-06-27T16:26:58.662+0000

  • Duration: 183 min 34 sec

  • Commit: 3e2d0e0

Test stats 🧪

Test Results
Failed 0
Passed 12505
Skipped 1972
Total 14477

Trends 🧪

Image of Build Times

Image of Tests

Log output

Expand to view the last 100 lines of log output 

[2021-06-27T17:55:27.542Z] 3.50s call     filebeat/tests/system/test_registrar.py::Test::test_state_after_rotation
[2021-06-27T17:55:27.542Z] 3.47s call     filebeat/tests/system/test_reload_inputs.py::Test::test_start_stop
[2021-06-27T17:55:27.542Z] 3.46s call     filebeat/tests/system/test_reload_modules.py::Test::test_start_stop
[2021-06-27T17:55:27.542Z] 3.28s call     filebeat/tests/system/test_shutdown.py::Test::test_shutdown
[2021-06-27T17:55:27.542Z] 2.91s call     filebeat/tests/system/test_multiline.py::Test::test_timeout
[2021-06-27T17:55:27.542Z] 2.49s call     filebeat/tests/system/test_registrar.py::Test::test_registrar_files_with_input_level_processors
[2021-06-27T17:55:27.542Z] ================ 166 passed, 183 skipped in 206.02s (0:03:26) =================
[2021-06-27T17:55:27.542Z] >> python test: Unit Testing Complete
[2021-06-27T17:55:27.885Z] 
[2021-06-27T17:55:27.885Z] C:\Users\jenkins\workspace\PR-26481-3-654d075a-3954-4bc5-8333-fcdaed68f56e\src\github.com\elastic\beats>FOR / %d IN ("ve") DO @IF EXIST "%d" rmdir /s /q "%d" 
[2021-06-27T17:55:30.732Z] 
[2021-06-27T17:55:30.732Z] C:\Users\jenkins\workspace\PR-26481-3-654d075a-3954-4bc5-8333-fcdaed68f56e\src\github.com\elastic\beats>python .ci/scripts/pre_archive_test.py 
[2021-06-27T17:55:36.000Z] Copy .\filebeat\build into build\filebeat\build
[2021-06-27T17:55:36.000Z] Copy .\filebeat\null\build into build\filebeat\null\build
[2021-06-27T17:55:36.012Z] Running in C:\Users\jenkins\workspace\PR-26481-3-654d075a-3954-4bc5-8333-fcdaed68f56e\src\github.com\elastic\beats\build
[2021-06-27T17:55:36.027Z] Recording test results
[2021-06-27T17:55:37.193Z] [Checks API] No suitable checks publisher found.
[2021-06-27T17:55:37.537Z] 
[2021-06-27T17:55:37.537Z] C:\Users\jenkins\workspace\PR-26481-3-654d075a-3954-4bc5-8333-fcdaed68f56e\src\github.com\elastic\beats>go clean -modcache 
[2021-06-27T17:55:51.523Z] ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'C:\\Users\\jenkins\\AppData\\Local\\Temp\\pip-uninstall-rs3aawsg\\pip.exe'
[2021-06-27T17:55:51.523Z] Consider using the `--user` option or check the permissions.
[2021-06-27T17:55:51.523Z] 
[2021-06-27T17:56:11.847Z] + gsutil --version
[2021-06-27T17:56:13.810Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-06-27T17:56:14.122Z] + gcloud auth activate-service-account --key-file ****
[2021-06-27T17:56:15.075Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-06-27T17:56:15.401Z] + gsutil -m -q cp -a public-read eC1wYWNrL2ZpbGViZWF0LXdpbmRvd3MtNy13aW5kb3dzLTczZTJkMGUwZDM5ODE4N2E0YjczYTAwN2FmMTg1ZTRlOWU4MThkZWQy gs://beats-ci-temp/ci/cache/
[2021-06-27T17:56:46.637Z] + gsutil --version
[2021-06-27T17:56:48.609Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-06-27T17:56:48.928Z] + gcloud auth activate-service-account --key-file ****
[2021-06-27T17:56:49.881Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-06-27T17:56:50.202Z] + gsutil -m -q cp -a public-read ZmlsZWJlYXQtd2luZG93cy0xMC13aW5kb3dzLTEwM2UyZDBlMGQzOTgxODdhNGI3M2EwMDdhZjE4NWU0ZTllODE4ZGVkMg gs://beats-ci-temp/ci/cache/
[2021-06-27T17:57:13.299Z] warn: failed to upgrade pip (ignoring): running "null\build\ve\windows\Scripts\pip install -U pip" failed with exit code 1============================= test session starts =============================
[2021-06-27T17:57:13.299Z] platform win32 -- Python 3.8.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
[2021-06-27T17:57:13.299Z] rootdir: C:\Users\jenkins\workspace\PR-26481-3-8b885fb8-5e8f-4472-baa3-95aff92ca9bc\src\github.com\elastic\beats, configfile: pytest.ini
[2021-06-27T17:57:13.299Z] plugins: rerunfailures-9.1.1, timeout-1.4.2
[2021-06-27T17:57:13.299Z] timeout: 90.0s
[2021-06-27T17:57:13.299Z] timeout method: thread
[2021-06-27T17:57:13.299Z] timeout func_only: True
[2021-06-27T17:57:13.299Z] collected 167 items
[2021-06-27T17:57:13.299Z] 
[2021-06-27T17:57:20.830Z] tests\system\test_filebeat_xpack.py .....                                [  2%]
[2021-06-27T17:57:56.204Z] tests\system\test_http_endpoint.py ...................                   [ 14%]
[2021-06-27T17:57:56.204Z] tests\system\test_xpack_modules.py sssssssssssssssssssssssssssssssssssss [ 36%]
[2021-06-27T17:57:56.480Z] ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 79%]
[2021-06-27T17:57:56.480Z] ssssssssssssssssssssssssssssssssss                                       [100%]
[2021-06-27T17:57:56.480Z] 
[2021-06-27T17:57:56.480Z] - generated xml file: C:\Users\jenkins\workspace\PR-26481-3-8b885fb8-5e8f-4472-baa3-95aff92ca9bc\src\github.com\elastic\beats\x-pack\filebeat\build\TEST-python-unit.xml -
[2021-06-27T17:57:56.480Z] ============================ slowest 20 durations =============================
[2021-06-27T17:57:56.480Z] 4.08s call     x-pack/filebeat/tests/system/test_filebeat_xpack.py::FilebeatXPackTest::test_export_index_pattern
[2021-06-27T17:57:56.480Z] 4.01s call     x-pack/filebeat/tests/system/test_filebeat_xpack.py::FilebeatXPackTest::test_export_index_pattern_migration
[2021-06-27T17:57:56.480Z] 3.43s call     x-pack/filebeat/tests/system/test_filebeat_xpack.py::FilebeatXPackTest::test_export_template
[2021-06-27T17:57:56.480Z] 2.41s call     x-pack/filebeat/tests/system/test_filebeat_xpack.py::FilebeatXPackTest::test_export_config
[2021-06-27T17:57:56.480Z] 2.14s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_request
[2021-06-27T17:57:56.480Z] 2.11s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_request_ndjson
[2021-06-27T17:57:56.480Z] 2.06s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_request_multiple_documents
[2021-06-27T17:57:56.480Z] 2.01s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_valid_hmac
[2021-06-27T17:57:56.480Z] 2.01s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_get_request
[2021-06-27T17:57:56.480Z] 2.01s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_include_headers_without_header
[2021-06-27T17:57:56.480Z] 1.98s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_include_headers_not_canonical_config
[2021-06-27T17:57:56.480Z] 1.98s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_correct_auth_header
[2021-06-27T17:57:56.480Z] 1.96s call     x-pack/filebeat/tests/system/test_filebeat_xpack.py::FilebeatXPackTest::test_export_ilm_policy
[2021-06-27T17:57:56.480Z] 1.96s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_preserve_original_event
[2021-06-27T17:57:56.480Z] 1.95s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_wrong_auth_value
[2021-06-27T17:57:56.480Z] 1.95s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_empty_body
[2021-06-27T17:57:56.480Z] 1.95s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_wrong_auth_header
[2021-06-27T17:57:56.480Z] 1.94s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_include_headers_empty_value
[2021-06-27T17:57:56.480Z] 1.94s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_include_headers_single_value
[2021-06-27T17:57:56.480Z] 1.94s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_invalid_hmac
[2021-06-27T17:57:56.480Z] ====================== 24 passed, 143 skipped in 53.38s =======================
[2021-06-27T17:57:56.741Z] >> python test: Unit Testing Complete
[2021-06-27T17:57:57.084Z] 
[2021-06-27T17:57:57.085Z] C:\Users\jenkins\workspace\PR-26481-3-8b885fb8-5e8f-4472-baa3-95aff92ca9bc\src\github.com\elastic\beats>FOR / %d IN ("ve") DO @IF EXIST "%d" rmdir /s /q "%d" 
[2021-06-27T17:57:59.316Z] 
[2021-06-27T17:57:59.316Z] C:\Users\jenkins\workspace\PR-26481-3-8b885fb8-5e8f-4472-baa3-95aff92ca9bc\src\github.com\elastic\beats>python .ci/scripts/pre_archive_test.py 
[2021-06-27T17:58:00.270Z] Copy .\x-pack\filebeat\build into build\x-pack\filebeat\build
[2021-06-27T17:58:00.270Z] Copy .\x-pack\filebeat\null\build into build\x-pack\filebeat\null\build
[2021-06-27T17:58:00.283Z] Running in C:\Users\jenkins\workspace\PR-26481-3-8b885fb8-5e8f-4472-baa3-95aff92ca9bc\src\github.com\elastic\beats\build
[2021-06-27T17:58:00.298Z] Recording test results
[2021-06-27T17:58:01.665Z] [Checks API] No suitable checks publisher found.
[2021-06-27T17:58:02.015Z] 
[2021-06-27T17:58:02.015Z] C:\Users\jenkins\workspace\PR-26481-3-8b885fb8-5e8f-4472-baa3-95aff92ca9bc\src\github.com\elastic\beats>go clean -modcache 
[2021-06-27T17:59:11.646Z] + gsutil --version
[2021-06-27T17:59:13.608Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-06-27T17:59:13.922Z] + gcloud auth activate-service-account --key-file ****
[2021-06-27T17:59:14.502Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-06-27T17:59:15.082Z] + gsutil -m -q cp -a public-read eC1wYWNrL2ZpbGViZWF0LXdpbmRvd3MtMTAtd2luZG93cy0xMDNlMmQwZTBkMzk4MTg3YTRiNzNhMDA3YWYxODVlNGU5ZTgxOGRlZDI gs://beats-ci-temp/ci/cache/
[2021-06-27T19:29:31.747Z] Cancelling nested steps due to timeout
[2021-06-27T19:29:31.780Z] Failed in branch filebeat-windows-8-windows-8
[2021-06-27T19:29:31.803Z] Failed in branch x-pack/filebeat-windows-8-windows-8
[2021-06-27T19:29:31.867Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-06-27T19:29:31.895Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-06-27T19:29:31.944Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-26481/src/github.com/elastic/beats
[2021-06-27T19:29:32.218Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-26481
[2021-06-27T19:29:32.263Z] [INFO] getVaultSecret: Getting secrets
[2021-06-27T19:29:32.300Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-06-27T19:29:32.943Z] + chmod 755 generate-build-data.sh
[2021-06-27T19:29:32.943Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26481/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26481/runs/3 ABORTED 10954019
[2021-06-27T19:29:32.943Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26481/runs/3/steps/?limit=10000 -o steps-info.json
[2021-06-27T19:29:34.286Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26481/runs/3/tests/?status=FAILED -o tests-errors.json

@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Jun 24, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b recorded_future upstream/recorded_future
git merge upstream/master
git push upstream recorded_future

andrewkroh
andrewkroh approved these changes Jun 25, 2021
View reviewed changes
Copy link
Copy Markdown
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@adriansr
Copy link
Copy Markdown
Contributor Author

adriansr commented Jun 28, 2021

CI failure is due to Windows workers being offline. Merging.

@adriansr adriansr merged commit 6d89566 into elastic:master Jun 28, 2021
mergify bot pushed a commit that referenced this pull request Jun 28, 2021
@adriansr
Add Recorded Future support to threatintel module (#26481)
cb591a9 
This adds a new fileset, `recordedfuture`, to the treatintel module. It
ingests indicators via the Recorded Future Connect API.

(cherry picked from commit 6d89566)
@mergify mergify bot mentioned this pull request Jun 28, 2021
Merged
adriansr added a commit that referenced this pull request Jun 28, 2021
@mergify @adriansr
Add Recorded Future support to threatintel module (#26481) (#26516)
dbe17e4 
This adds a new fileset, `recordedfuture`, to the treatintel module. It
ingests indicators via the Recorded Future Connect API.

(cherry picked from commit 6d89566)

Co-authored-by: Adrian Serrano <adrisr83@gmail.com>
mdelapenya added a commit to mdelapenya/beats that referenced this pull request Jun 29, 2021
@mdelapenya
Merge branch 'master' into archive-system-tests
d4a87cb 
* master:
  Osquerybeat: set the raw index name to supress the timestamp suffix (elastic#26545)
  [Heartbeat] add screenshots config to synthetics (elastic#26455)
  [Elastic Agent] Use http2 to connect to Fleet Server. (elastic#26474)
  Remove all docs about  Beats central management (elastic#26399)
  update data.json for gcp billing (elastic#26506)
  Skip x-pack metricbeat tests (elastic#26537)
  [Elastic Agent] Fix issue with FLEET_CA not being used with Fleet Server in container (elastic#26529)
  Add changelog entry for  elastic#26224 (elastic#26531)
  Add inttests for the x-pack/metricbeat on a PR/branches basis (elastic#26526)
  Suppress too many errors (elastic#26224)
  Fix master's linting issue (elastic#26517)
  [libbeat] Fix encoding and file offset issues in the disk queue (elastic#26484)
  Add log_group_name_prefix config option for aws-cloudwatch input (elastic#26187)
  Update shared-deduplication.asciidoc (elastic#26492)
  Add Recorded Future support to threatintel module (elastic#26481)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrewkroh andrewkroh andrewkroh approved these changes

Assignees

No one assigned

Labels

backport-v7.14.0 Automated backport with mergify enhancement review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@adriansr @elasticmachine @andrewkroh