Skip to content

experiment: store ignored source as binary doc values.#143396

Draft
martijnvg wants to merge 8 commits intoelastic:mainfrom
martijnvg:store_synthetic_source_as_binary_dv
Draft

experiment: store ignored source as binary doc values.#143396
martijnvg wants to merge 8 commits intoelastic:mainfrom
martijnvg:store_synthetic_source_as_binary_dv

Conversation

@martijnvg
Copy link
Copy Markdown
Member

@martijnvg martijnvg commented Mar 2, 2026

Following changes:

  • Introduced another IgnoredSourceFormat that stores ignored source using binary doc values.
  • Re-used the legacy ignored source encoding, given that binary doc values api just loads all ignored source values as one binary blob. And legacy encoding is best suited for this.
  • Used binary doc values with integrated count. Storing the counts separately has no additional advantage.
  • Increased binary doc values block size from 128kb to 1mb.

@martijnvg martijnvg force-pushed the store_synthetic_source_as_binary_dv branch from e037047 to 0024eea Compare March 2, 2026 19:18
@martijnvg
Copy link
Copy Markdown
Member Author

Buildkite benchmark this with elastic-logs-streams please

1 similar comment
@martijnvg
Copy link
Copy Markdown
Member Author

Buildkite benchmark this with elastic-logs-streams please

@elasticmachine
Copy link
Copy Markdown
Collaborator

elasticmachine commented Mar 5, 2026

💚 Build Succeeded

This build ran two elastic-logs-streams benchmarks to evaluate performance impact of this PR.

History

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants