Skip to content

[Entity Store] Use last_seen for automated resolution watermark#258574

Merged
maxcold merged 10 commits intomainfrom
maxcold/watermark-last-seen
Mar 24, 2026
Merged

[Entity Store] Use last_seen for automated resolution watermark#258574
maxcold merged 10 commits intomainfrom
maxcold/watermark-last-seen

Conversation

@maxcold
Copy link
Copy Markdown
Contributor

@maxcold maxcold commented Mar 19, 2026

Summary

Switches the automated email resolution incremental scan watermark from @timestamp to entity.lifecycle.last_seen.

Problem: The history snapshot operation resets @timestamp on all entity documents to the current time. This causes automated resolution to do a full re-scan of all entities after every snapshot, instead of only processing entities with genuinely new data.

Fix: Use entity.lifecycle.last_seen instead, which reflects the newest @timestamp from source logs and is not affected by the history snapshot operation.

Depends on #258495 (merged).

Checklist

  • Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
  • Documentation was added for features that require explanation or tutorials
  • Unit or functional tests were updated or added to match the most common scenarios
  • If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
  • This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
  • Flaky Test Runner was used on any tests changed
  • The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
  • Review the backport guidelines and apply applicable backport:* labels.

Identify risks

  • Low risk: This change only affects the field used for incremental scan filtering in automated resolution. If entity.lifecycle.last_seen is not populated on an entity (e.g., before the first log extraction after init), the first run is a full scan anyway (lastProcessedTimestamp is null → no range filter).

@maxcold maxcold changed the title [WIP] switch to last_seen for automated email resolution [Entity Store] Use last_seen for automated resolution watermark Mar 23, 2026
@maxcold maxcold added release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting Team:Cloud Security Cloud Security team related labels Mar 23, 2026
@maxcold maxcold marked this pull request as ready for review March 23, 2026 15:38
@maxcold maxcold requested a review from a team as a code owner March 23, 2026 15:38
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/contextual-security-apps (Team:Cloud Security)

@maxcold maxcold requested a review from kubasobon March 24, 2026 08:54
@elasticmachine
Copy link
Copy Markdown
Contributor

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #38 / console app console autocomplete feature Autocomplete shouldnt trigger within a multiline block comment
  • [job] [logs] Jest Tests #2 / ServicesTable Performance renders large datasets efficiently

Metrics [docs]

✅ unchanged

Copy link
Copy Markdown
Member

@kubasobon kubasobon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! 🚀

@maxcold maxcold merged commit 76243f3 into main Mar 24, 2026
18 checks passed
@maxcold maxcold deleted the maxcold/watermark-last-seen branch March 24, 2026 14:37
mbondyra added a commit to mbondyra/kibana that referenced this pull request Mar 24, 2026
…ra/kibana into dashboard_align_attachment_to_api

* 'dashboard_align_attachment_to_api' of github.com:mbondyra/kibana: (45 commits)
  [OTel Tracing] HTTP instrumentation (elastic#258663)
  Replace deprecated EUI icons in files owned by @elastic/ml-ui (elastic#255624)
  [Codeowners] add missing codeowners for security_solution_api_integration tests (elastic#259223)
  [CI] fix bad imports that came from a merge-race (elastic#259383)
  Add `.claude/worktrees/` to `.gitignore` (elastic#259192)
  Improve unknown-key validation error message in @kbn/config-schema (elastic#258633)
  [ML] Update Security ML jobs to use entity analytics fields for host and user fields (elastic#255339)
  [Table sweep] Update table columns responsiveness in Index Management and Dashboards (elastic#259340)
  skip failing test suite (elastic#258790)
  skip failing test suite (elastic#259261)
  chore: util to clean cached images (elastic#259335)
  [Entity Store] Use last_seen for automated resolution watermark (elastic#258574)
  [One Workflow] Fix flaky alert trigger Scout test by removing order-dependent assertions (elastic#259299)
  Skip serverless Discover request counts tests for MKI (elastic#259333)
  [Security Solution] render header title in new document flyout in Security Solution and Discover (elastic#258166)
  [Agent Builder] register inference endpoint feature (elastic#259259)
  [Agent Builder] Skills Command Menu - Add descriptions and scope options to agent (elastic#258964)
  [Streams][Streamlang][API] Fully use meta({id}) to reuse schema partials in OAS output (elastic#259275)
  fix(files_example): add tableCaption to EuiInMemoryTable for a11y (elastic#258289)
  [Entity Store] Adding list endpoint with query filter (elastic#258320)
  ...
jeramysoucy pushed a commit to jeramysoucy/kibana that referenced this pull request Mar 26, 2026
…tic#258574)

## Summary

Switches the automated email resolution incremental scan watermark from
`@timestamp` to `entity.lifecycle.last_seen`.

**Problem:** The history snapshot operation resets `@timestamp` on all
entity documents to the current time. This causes automated resolution
to do a full re-scan of all entities after every snapshot, instead of
only processing entities with genuinely new data.

**Fix:** Use `entity.lifecycle.last_seen` instead, which reflects the
newest `@timestamp` from source logs and is not affected by the history
snapshot operation.

Depends on elastic#258495 (merged).

### Checklist

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

### Identify risks

- Low risk: This change only affects the field used for incremental scan
filtering in automated resolution. If `entity.lifecycle.last_seen` is
not populated on an entity (e.g., before the first log extraction after
init), the first run is a full scan anyway (`lastProcessedTimestamp` is
null → no range filter).

---------

Co-authored-by: kubasobon <kuba.sobon@elastic.co>
Co-authored-by: Kuba Soboń <wtty.fool@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting release_note:skip Skip the PR/issue when compiling release notes Team:Cloud Security Cloud Security team related v9.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants