Skip to content

[8.19](backport #43251) feat: Introduce lastSync start position to AWS CloudWatch input backed by state registry#44736

Merged
Kavindu-Dodan merged 3 commits into8.19from
mergify/bp/8.19/pr-43251
Jun 11, 2025
Merged

[8.19](backport #43251) feat: Introduce lastSync start position to AWS CloudWatch input backed by state registry#44736
Kavindu-Dodan merged 3 commits into8.19from
mergify/bp/8.19/pr-43251

Conversation

@mergify
Copy link
Copy Markdown
Contributor

@mergify mergify bot commented Jun 10, 2025

Proposed commit message

Introduce lastSync option to AWS CloudWatch input's start_position configuration. This feature utilizes the state registry, which with this update stores the last successful sync timestamp under the last_sync_epoch key.

start_position configuration by default uses beginning option. This can cause data duplications when restarting filebeats or migrating it to a newer version. Users can avoid such data duplocates by utilizing lastSync option for the start_position configuration while targeting beats to use the same registry location.

Acknowledgement backed state updates

After the initial review, the need for acknowledgments prior to the state registry update was identified thanks to this remark - #43251 (comment)

Based on this feedback, I have implemented the state registry update with following component interactions,

image

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

None - start_position by default use beginning. So this new option is an opt-in feature.

Screenshots

Example filebeat.yml configuration,

image

Example state entries with last sync timestamp updates,

image

This is an automatic backport of pull request #43251 done by [Mergify](https://mergify.com).

@Kavindu-Dodan @mergify
feat: Introduce lastSync start position to AWS CloudWatch input backe…
b9ccd31 
…d by state registry (#43251)

* add state registry support for cloudwatch input

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

finalize AWS CW lastSync start position usage

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

update changelog next

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

review changes

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

review changes - improve state store id, fix timestamp and return nil for error

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

documentation update with md migration

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

# Conflicts:
#	x-pack/filebeat/input/awscloudwatch/config.go

* review change: use ackers for confirmed delivery prior to saving state

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

* review: documentation updates

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

* review changes : ctx handling, shutdown improvement and race condition handling

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

* review : fix ctx condition

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

---------

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>
(cherry picked from commit 42e50cf)

# Conflicts:
#	docs/reference/filebeat/filebeat-input-aws-cloudwatch.md
@mergify mergify bot added backport conflicts There is a conflict in the backported pull request labels Jun 10, 2025
@mergify mergify bot requested review from a team as code owners June 10, 2025 17:55
@mergify mergify bot added backport conflicts There is a conflict in the backported pull request labels Jun 10, 2025
@mergify mergify bot requested review from AndersonQ and faec and removed request for a team June 10, 2025 17:55
@mergify
Copy link
Copy Markdown
Contributor Author

mergify bot commented Jun 10, 2025

Cherry-pick of 42e50cf has failed:

On branch mergify/bp/8.19/pr-43251
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 42e50cf7c.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   CHANGELOG.next.asciidoc
	modified:   NOTICE.txt
	modified:   go.mod
	modified:   go.sum
	modified:   x-pack/filebeat/input/awscloudwatch/cloudwatch.go
	modified:   x-pack/filebeat/input/awscloudwatch/cloudwatch_test.go
	new file:   x-pack/filebeat/input/awscloudwatch/cloudwatch_worker.go
	new file:   x-pack/filebeat/input/awscloudwatch/cloudwatch_worker_test.go
	modified:   x-pack/filebeat/input/awscloudwatch/config.go
	modified:   x-pack/filebeat/input/awscloudwatch/input.go
	modified:   x-pack/filebeat/input/awscloudwatch/input_integration_test.go
	modified:   x-pack/filebeat/input/awscloudwatch/processor.go
	new file:   x-pack/filebeat/input/awscloudwatch/state_handler.go
	new file:   x-pack/filebeat/input/awscloudwatch/state_handler_test.go
	modified:   x-pack/filebeat/input/default-inputs/inputs_darwin.go
	modified:   x-pack/filebeat/input/default-inputs/inputs_other.go
	modified:   x-pack/filebeat/input/default-inputs/inputs_windows.go

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   docs/reference/filebeat/filebeat-input-aws-cloudwatch.md

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 10, 2025
@github-actions github-actions bot added Team:obs-ds-hosted-services Label for the Observability Hosted Services team Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team labels Jun 10, 2025
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 10, 2025

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 10, 2025
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 10, 2025

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

Kavindu-Dodan and others added 2 commits June 10, 2025 11:04
@Kavindu-Dodan
Update CHANGELOG.next.asciidoc
d2cf6ea
@Kavindu-Dodan
update CloudWatch documentation for backport
8ed39cb 
Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>
@Kavindu-Dodan Kavindu-Dodan force-pushed the mergify/bp/8.19/pr-43251 branch from 9af93ff to 8ed39cb Compare June 10, 2025 18:16
Kavindu-Dodan
Kavindu-Dodan approved these changes Jun 10, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@Kavindu-Dodan Kavindu-Dodan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed Change Log conflicts and documentation upgrades.

@Kavindu-Dodan Kavindu-Dodan merged commit 441da35 into 8.19 Jun 11, 2025
201 checks passed
@Kavindu-Dodan Kavindu-Dodan deleted the mergify/bp/8.19/pr-43251 branch June 11, 2025 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Kavindu-Dodan Kavindu-Dodan Kavindu-Dodan approved these changes

@AndersonQ AndersonQ Awaiting requested review from AndersonQ AndersonQ is a code owner automatically assigned from elastic/elastic-agent-data-plane

@faec faec Awaiting requested review from faec faec is a code owner automatically assigned from elastic/elastic-agent-data-plane

Assignees

@Kavindu-Dodan Kavindu-Dodan

Labels

backport conflicts There is a conflict in the backported pull request Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team Team:obs-ds-hosted-services Label for the Observability Hosted Services team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@elasticmachine @Kavindu-Dodan