Skip to content

[Entity Store] Extract first_seen and last_seen from @timestamp#258495

Merged
kubasobon merged 8 commits intomainfrom
entity-store-lifecycle
Mar 20, 2026
Merged

[Entity Store] Extract first_seen and last_seen from @timestamp#258495
kubasobon merged 8 commits intomainfrom
entity-store-lifecycle

Conversation

@kubasobon
Copy link
Copy Markdown
Member

@kubasobon kubasobon commented Mar 19, 2026

Summary

Modifies entity.lifecycle.first_seen and entity.lifecycle.last_seen to represent true state by holding onto first and latest timestamps respectively.
EDIT: entity.lifecycle.last_activity should stay the same, last_seen added.

Closes #255084

Checklist

@kubasobon kubasobon requested a review from a team as a code owner March 19, 2026 09:47
@kubasobon kubasobon self-assigned this Mar 19, 2026
@kubasobon kubasobon added release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting labels Mar 19, 2026
@kubasobon kubasobon changed the title [Entity Store] Extract first_seen and last_seen from @timestamp [Entity Store] Extract first_seen and last_activity from @timestamp Mar 19, 2026
@kubasobon kubasobon changed the title [Entity Store] Extract first_seen and last_activity from @timestamp [Entity Store] Extract first_seen and last_seen from @timestamp Mar 19, 2026
@kubasobon kubasobon requested a review from uri-weisman March 19, 2026 11:45
@kubasobon kubasobon enabled auto-merge (squash) March 19, 2026 11:50
@kubasobon kubasobon merged commit 6dbf951 into main Mar 20, 2026
19 checks passed
@kubasobon kubasobon deleted the entity-store-lifecycle branch March 20, 2026 12:07
@elasticmachine
Copy link
Copy Markdown
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

cc @kubasobon

kubasobon added a commit that referenced this pull request Mar 24, 2026
Fixes a string templating bug introduced by
#258495
- Use the validateQuery from @kbn/esql-language to va;idate the ESQL
queries we generate.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
kubasobon added a commit that referenced this pull request Mar 24, 2026
Fixes a string templating bug introduced by
#258495
- Use the validateQuery from @kbn/esql-language to va;idate the ESQL
queries we generate.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
maxcold added a commit that referenced this pull request Mar 24, 2026
)

## Summary

Switches the automated email resolution incremental scan watermark from
`@timestamp` to `entity.lifecycle.last_seen`.

**Problem:** The history snapshot operation resets `@timestamp` on all
entity documents to the current time. This causes automated resolution
to do a full re-scan of all entities after every snapshot, instead of
only processing entities with genuinely new data.

**Fix:** Use `entity.lifecycle.last_seen` instead, which reflects the
newest `@timestamp` from source logs and is not affected by the history
snapshot operation.

Depends on #258495 (merged).

### Checklist

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

### Identify risks

- Low risk: This change only affects the field used for incremental scan
filtering in automated resolution. If `entity.lifecycle.last_seen` is
not populated on an entity (e.g., before the first log extraction after
init), the first run is a full scan anyway (`lastProcessedTimestamp` is
null → no range filter).

---------

Co-authored-by: kubasobon <kuba.sobon@elastic.co>
Co-authored-by: Kuba Soboń <wtty.fool@gmail.com>
jeramysoucy pushed a commit to jeramysoucy/kibana that referenced this pull request Mar 26, 2026
…tic#258495)

## Summary

Modifies `entity.lifecycle.first_seen` and `entity.lifecycle.last_seen`
to represent true state by holding onto first and latest timestamps
respectively.
**EDIT:** `entity.lifecycle.last_activity` should stay the same,
`last_seen` added.

Closes elastic#255084

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
jeramysoucy pushed a commit to jeramysoucy/kibana that referenced this pull request Mar 26, 2026
…59057)

Fixes a string templating bug introduced by
elastic#258495
- Use the validateQuery from @kbn/esql-language to va;idate the ESQL
queries we generate.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
jeramysoucy pushed a commit to jeramysoucy/kibana that referenced this pull request Mar 26, 2026
…tic#258574)

## Summary

Switches the automated email resolution incremental scan watermark from
`@timestamp` to `entity.lifecycle.last_seen`.

**Problem:** The history snapshot operation resets `@timestamp` on all
entity documents to the current time. This causes automated resolution
to do a full re-scan of all entities after every snapshot, instead of
only processing entities with genuinely new data.

**Fix:** Use `entity.lifecycle.last_seen` instead, which reflects the
newest `@timestamp` from source logs and is not affected by the history
snapshot operation.

Depends on elastic#258495 (merged).

### Checklist

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

### Identify risks

- Low risk: This change only affects the field used for incremental scan
filtering in automated resolution. If `entity.lifecycle.last_seen` is
not populated on an entity (e.g., before the first log extraction after
init), the first run is a full scan anyway (`lastProcessedTimestamp` is
null → no range filter).

---------

Co-authored-by: kubasobon <kuba.sobon@elastic.co>
Co-authored-by: Kuba Soboń <wtty.fool@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting release_note:skip Skip the PR/issue when compiling release notes v9.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Entity Store] Implement first_seen and last_seen from logs extraction

4 participants