[aws] [s3] Introduce ignore_older & start_timestamp for S3 input allowing better registry cleanups by Kavindu-Dodan · Pull Request #41817 · elastic/beats

Kavindu-Dodan · 2024-11-27T21:05:27Z

Proposed commit message

Introduce ignore_older and start_timestamp properties to AWS S3 input. This is a follow-up for #41694.

The configurations introduced here act as input object filters. If the object fails to match derived filters, the entries will be cleaned up from the registry, reducing filebeat memory consumption.

Introduced configurations are,

ignore_older : Accepts a time duration in which entries are accepted for processing
start_timestamp: A timestamp from which objects are accepted for processing

For both inputs, the object's last modified timestamp is taken into comparison. See Use cases section for further explanation

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

None as defaults are disabled. However, when configurations introduced here are used, the following can have an impact on the user,

Whenstart_timestamp is defined, then objects with the last modified timestamps prior to the timestamp are ignored from processing (documented ¹)
When ignore_older is defined, then objects that do not fall within the look-back period when processing starts (polling run) are ignored (documented ¹)
When both start_timestamp & ignore_older are defined, the initial run will process all entries up to start_timestamp. The subsequent runs will not include entries that do not fall within ignore_older even if processing failed for an object. (documented ¹)

How to test this PR locally

Build filebeat from the changest included in the PR
Source S3 bucket with objects (you may use this tool ² to create entries)
Try configuring filebeat with alternative values for ignore_older & start_timestamp to see how data ingestion change with their values. See Use cases section for further explanation

Related issues

Use cases

Consider below diagrams where there're 3 objects Object A, Object B and Object C with their last modified timestamps of t1, t2 and t3.

And consider how filebeat processes and tracks registry entries based on the following scenarios

Default behavior

If none of the configurations are used, then filebeat will process and the internal registry will track all objects continuously unless they are removed from the bucket.

Use start_timestamp

If start_timestamp is used, objects newer than the timestamp are accepted for processing. The registry will grow unless objects are removed from the bucket by other means (ex:- lifecycle policy).

Use ignore_older

If ignore_older is defined, input will process objects within the provided duration, calculated from the current time. The registry will track objects within the current timeframe and others will get cleaned up eventually by subsequent runs.

Use both ignore_older & start_timestamp

If both properties are defined,

The initial run will include entries within the start_timestamp (ignoring ignore_older duration).
Subsequent runs will only consider entries within the ignore_older duration.

CHANGELOG.asciidoc

leehinman

LGTM, but I'd like @faec to take a look.

elasticmachine · 2025-01-06T15:11:36Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

…them Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # x-pack/filebeat/input/awss3/s3_test.go

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

…tation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

…wing better registry cleanups (#41817) * add changelog entry Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * sort config entries Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * introduce ignore old and start timestamp configurations and document them Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add filtering logic Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * filter tests Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add component test for filtering and fix lint issues Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # x-pack/filebeat/input/awss3/s3_test.go * add changelog entry Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * improve documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review changes - improve naming, change signature and improve documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> (cherry picked from commit 4ba7d1c)

…wing better registry cleanups (#41817) (#42246) * add changelog entry Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * sort config entries Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * introduce ignore old and start timestamp configurations and document them Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add filtering logic Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * filter tests Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add component test for filtering and fix lint issues Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # x-pack/filebeat/input/awss3/s3_test.go * add changelog entry Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * improve documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review changes - improve naming, change signature and improve documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> (cherry picked from commit 4ba7d1c) Co-authored-by: Kavindu Dodanduwa <Kavindu-Dodan@users.noreply.github.com>

Kavindu-Dodan · 2025-02-14T14:58:14Z

After considering with the team and with the background from this conversation when upgrading the Integration ¹, I have decided to backport the change to 8.16.x and 8.17.x release tracks.

https://github.com/elastic/integrations/pull/12645#discussion_r1950711236 ↩

…wing better registry cleanups (#41817) * add changelog entry Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * sort config entries Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * introduce ignore old and start timestamp configurations and document them Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add filtering logic Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * filter tests Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add component test for filtering and fix lint issues Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # x-pack/filebeat/input/awss3/s3_test.go * add changelog entry Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * improve documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review changes - improve naming, change signature and improve documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> (cherry picked from commit 4ba7d1c) # Conflicts: # x-pack/filebeat/input/awss3/s3_test.go

…wing better registry cleanups (#41817) * add changelog entry Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * sort config entries Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * introduce ignore old and start timestamp configurations and document them Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add filtering logic Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * filter tests Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add component test for filtering and fix lint issues Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # x-pack/filebeat/input/awss3/s3_test.go * add changelog entry Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * improve documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review changes - improve naming, change signature and improve documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> (cherry picked from commit 4ba7d1c) # Conflicts: # x-pack/filebeat/docs/inputs/input-aws-s3.asciidoc # x-pack/filebeat/input/awss3/s3_test.go

…wing better registry cleanups (#41817) * add changelog entry Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * sort config entries Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * introduce ignore old and start timestamp configurations and document them Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add filtering logic Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * filter tests Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add component test for filtering and fix lint issues Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add changelog entry Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * improve documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review changes - improve naming, change signature and improve documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

…estamp for S3 input allowing better registry cleanups (#42716) * [aws] [s3] Introduce ignore_older & start_timestamp for S3 input allowing better registry cleanups (#41817) * add changelog entry Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * sort config entries Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * introduce ignore old and start timestamp configurations and document them Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add filtering logic Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * filter tests Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add component test for filtering and fix lint issues Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # x-pack/filebeat/input/awss3/s3_test.go * add changelog entry Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * improve documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review changes - improve naming, change signature and improve documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> (cherry picked from commit 4ba7d1c) # Conflicts: # x-pack/filebeat/input/awss3/s3_test.go * [aws] [s3] Introduce ignore_older & start_timestamp for S3 input allowing better registry cleanups (#41817) * add changelog entry Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * sort config entries Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * introduce ignore old and start timestamp configurations and document them Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add filtering logic Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * filter tests Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add component test for filtering and fix lint issues Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add changelog entry Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * improve documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review changes - improve naming, change signature and improve documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * fix backport conflicts Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> Co-authored-by: Kavindu Dodanduwa <Kavindu-Dodan@users.noreply.github.com> Co-authored-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

…estamp for S3 input allowing better registry cleanups (#42717) * [aws] [s3] Introduce ignore_older & start_timestamp for S3 input allowing better registry cleanups (#41817) * add changelog entry Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * sort config entries Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * introduce ignore old and start timestamp configurations and document them Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add filtering logic Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * filter tests Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add component test for filtering and fix lint issues Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # x-pack/filebeat/input/awss3/s3_test.go * add changelog entry Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * improve documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review changes - improve naming, change signature and improve documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> (cherry picked from commit 4ba7d1c) # Conflicts: # x-pack/filebeat/docs/inputs/input-aws-s3.asciidoc # x-pack/filebeat/input/awss3/s3_test.go * fix backport conflicts Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> Co-authored-by: Kavindu Dodanduwa <Kavindu-Dodan@users.noreply.github.com> Co-authored-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Kavindu-Dodan added enhancement Team:obs-ds-hosted-services Label for the Observability Hosted Services team backport-8.x Automated backport to the 8.x branch with mergify labels Nov 27, 2024

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Nov 27, 2024

mergify bot assigned Kavindu-Dodan Nov 27, 2024

Kavindu-Dodan mentioned this pull request Nov 27, 2024

[aws] feat: aws-s3 input registry cleanup for untracked s3 objects #41694

Merged

6 tasks

Kavindu-Dodan force-pushed the feat/s3-input-start-time-and-ignore-old branch 2 times, most recently from 4924d70 to 79ae2c1 Compare November 27, 2024 22:32

Kavindu-Dodan commented Nov 27, 2024

View reviewed changes

CHANGELOG.asciidoc Outdated Show resolved Hide resolved

Kavindu-Dodan mentioned this pull request Nov 28, 2024

[AWS] Introduce ignore_older & start_timestamp elastic/integrations#11919

Closed

Kavindu-Dodan requested a review from a team November 29, 2024 16:06

elastic deleted a comment from mergify bot Dec 3, 2024

Kavindu-Dodan force-pushed the feat/s3-input-start-time-and-ignore-old branch from 52fad61 to 6f5472c Compare December 3, 2024 23:06

elastic deleted a comment from mergify bot Dec 3, 2024

Kavindu-Dodan force-pushed the feat/s3-input-start-time-and-ignore-old branch from 6f5472c to ec00024 Compare December 6, 2024 16:58

Kavindu-Dodan marked this pull request as ready for review December 6, 2024 17:12

Kavindu-Dodan requested review from a team as code owners December 6, 2024 17:12

Kavindu-Dodan requested review from leehinman and rdner December 6, 2024 17:12

Kavindu-Dodan mentioned this pull request Dec 6, 2024

[AWS-S3] Add a timestamp filter for s3 polling mode #41232

Closed

Kavindu-Dodan force-pushed the feat/s3-input-start-time-and-ignore-old branch 2 times, most recently from 85f883e to fb4990b Compare December 6, 2024 19:53

elastic deleted a comment from mergify bot Dec 6, 2024

elastic deleted a comment from elasticmachine Dec 6, 2024

Kavindu-Dodan force-pushed the feat/s3-input-start-time-and-ignore-old branch from fb4990b to dab88c6 Compare December 6, 2024 20:48

elastic deleted a comment from mergify bot Dec 6, 2024

leehinman requested a review from faec December 9, 2024 16:29

leehinman approved these changes Dec 9, 2024

View reviewed changes

Kavindu-Dodan added 9 commits January 7, 2025 07:12

add changelog entry

2b30cb8

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

sort config entries

41e6d44

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

introduce ignore old and start timestamp configurations and document …

d948836

…them Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

add filtering logic

1609d54

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

filter tests

4f1c94d

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

add component test for filtering and fix lint issues

40fff60

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # x-pack/filebeat/input/awss3/s3_test.go

add changelog entry

af056c1

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

improve documentation

28ed690

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

review changes - improve naming, change signature and improve documen…

82db670

…tation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Kavindu-Dodan force-pushed the feat/s3-input-start-time-and-ignore-old branch from 3a4c0bd to 82db670 Compare January 7, 2025 15:13

rdner removed their request for review January 7, 2025 15:42

Kavindu-Dodan merged commit 4ba7d1c into elastic:main Jan 7, 2025

mergify bot mentioned this pull request Jan 7, 2025

[8.x](backport #41817) [aws] [s3] Introduce ignore_older & start_timestamp for S3 input allowing better registry cleanups #42246

Merged

6 tasks

Kavindu-Dodan mentioned this pull request Jan 7, 2025

aws-s3 input's bucket polling accumulates state in the registry #39116

Closed

Kavindu-Dodan deleted the feat/s3-input-start-time-and-ignore-old branch January 7, 2025 19:24

Kavindu-Dodan mentioned this pull request Feb 6, 2025

[AWS S3] Introduce start timestamp and ignore older timespan to AWS S3 based integrations elastic/integrations#12645

Merged

5 tasks

Kavindu-Dodan added backport-8.17 Automated backport with mergify backport-8.16 Automated backport with mergify labels Feb 14, 2025

lucabelluccini mentioned this pull request Feb 21, 2025

Filebeat ignores ignore_older option and processes all files in S3 folder #42824

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[aws] [s3] Introduce ignore_older & start_timestamp for S3 input allowing better registry cleanups#41817

[aws] [s3] Introduce ignore_older & start_timestamp for S3 input allowing better registry cleanups#41817
Kavindu-Dodan merged 9 commits intoelastic:mainfrom
Kavindu-Dodan:feat/s3-input-start-time-and-ignore-old

Kavindu-Dodan commented Nov 27, 2024 •

edited

Loading

Uh oh!

Uh oh!

leehinman left a comment

Uh oh!

elasticmachine commented Jan 6, 2025

Uh oh!

Kavindu-Dodan commented Feb 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

Kavindu-Dodan commented Nov 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Disruptive User Impact

How to test this PR locally

Related issues

Use cases

Default behavior

Use start_timestamp

Use ignore_older

Use both ignore_older & start_timestamp

Footnotes

Uh oh!

Uh oh!

leehinman left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jan 6, 2025

Uh oh!

Kavindu-Dodan commented Feb 14, 2025

Footnotes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Kavindu-Dodan commented Nov 27, 2024 •

edited

Loading