[Metrics][Discover] Build Metrics Grid chart from METRICS_INFO response#257206
[Metrics][Discover] Build Metrics Grid chart from METRICS_INFO response#257206kpatticha merged 29 commits intoelastic:mainfrom
Conversation
- Introduced `fetchMetricsInfo` and `parseMetricsInfoResponse` functions to handle fetching and parsing of metrics information. - Updated existing components to utilize the new metrics info structure, including adjustments to metric fields and dimensions. - Added utility functions for normalizing and appending metrics info queries. - Implemented tests for new functionalities to ensure reliability and correctness. This update improves the overall metrics experience by providing a more structured approach to handling metrics data.
| @@ -1,67 +0,0 @@ | |||
| /* | |||
There was a problem hiding this comment.
Note: I just moved this file
| @@ -41,14 +41,17 @@ export const OverviewTab = ({ metric, description }: OverviewTabProps) => { | |||
| const [itemsPerPage, setItemsPerPage] = useState(DEFAULT_PAGINATION_SIZE); | |||
| const [containerRef, setContainerRef] = useState<HTMLDivElement | null>(null); | |||
|
|
|||
| const unitLabel = useMemo(() => getUnitLabel({ unit: metric.unit }), [metric.unit]); | |||
| const unitLabel = useMemo( | |||
There was a problem hiding this comment.
The multi value for all field in the flyout it will be handled in a follow up ticket: #253718
|
/oblt-deploy |
lucaslopezf
left a comment
There was a problem hiding this comment.
thanks for sharing! Overall, I agree with the approach. I’ll let you mark the PR as ready for review before adding the comments, and I’ll keep an eye on it so I can provide quick feedback.
justinkambic
left a comment
There was a problem hiding this comment.
Great work! I did an initial review. I commented on a couple type errors but tried to stick more toward things I think would be useful as initial feedback.
One thing I noticed in practical testing is if I click a breakdown field from Discover's fields list, basically nothing happens. Are we considering how this will interact with fields that don't display in the Dimensions dropdown list, but do display in Discover's fields list?
…na into metrics_info_metrics_expr
lucaslopezf
left a comment
There was a problem hiding this comment.
Nice work! I left some comments. I haven’t finished the review yet, so I’ll continue, but I wanted to give you some initial feedback.
|
/flaky scoutConfig:src/platform/plugins/shared/discover/test/scout/ui/parallel.playwright.config.ts:1 |
Flaky Test Runner✅ Build triggered - kibana-flaky-test-suite-runner#11087
|
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#11087[✅] src/platform/plugins/shared/discover/test/scout/ui/parallel.playwright.config.ts: 1/1 tests passed. |
lucaslopezf
left a comment
There was a problem hiding this comment.
Nice!! We’re really close! Regardless of the comments (nits), while looking at the E2E tests I realized that tests related to queries like TS metrics-* | SORT @timestamp DESC are not working. I tested it locally and the same thing happens
Yes, these test are gold. It turns out that we cannot use METRICS_INFO after limit and sort. https://github.com/elastic/elasticsearch/blob/44207cb2e522baa392ba84d18ba0ac83ca1da24b/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/VerifierTests.java#L3762-L3768 . I'm fixing the limit but for sort I will remove it for now and create a seperate ticket cause I need product input on this |
|
/flaky scoutConfig:src/platform/plugins/shared/discover/test/scout/ui/parallel.playwright.config.ts:30 |
|
/oblt-deploy |
Flaky Test Runner✅ Build triggered - kibana-flaky-test-suite-runner#11103
|
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#11103[❌] src/platform/plugins/shared/discover/test/scout/ui/parallel.playwright.config.ts: 25/30 tests passed. |
💚 Build Succeeded
Metrics [docs]Module Count
Public APIs missing comments
Async chunks
Public APIs missing exports
History
|
|
Pinging @elastic/obs-exploration-team (Team:obs-exploration) |
|
/flaky scoutConfig:src/platform/plugins/shared/discover/test/scout/ui/parallel.playwright.config.ts:30 |
Flaky Test Runner✅ Build triggered - kibana-flaky-test-suite-runner#11111
|
lucaslopezf
left a comment
There was a problem hiding this comment.
Amazing work 🚀 !! Everything looks good to me! Let’s just wait for the flaky tool before merging, and “fix” any flaky tests if we find any
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#11111[❌] src/platform/plugins/shared/discover/test/scout/ui/parallel.playwright.config.ts: 26/30 tests passed. |
|
The flaky tests failed again, 26/30 but I'm going to merge this PR and fix this in a follow up |
…metrics experience tests (#258274) Closes: #258227 ## Summary The test used a relative time range (`Last 30 days`) that doesn't overlap with the test data (Jan 2025), so the `METRICS_INFO` query returns empty results and the grid disappears. Replaced with an absolute range (`Jan 1 – Jun 30, 2025`) that covers the test data while still validating grid persistence across time range changes. Surfaced by #257206, which switched to time-range-aware `METRICS_INFO` queries. The test had a latent bug (relative date with no data) that was previously masked by the old field-metadata-based approach. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ...
…metrics experience tests (elastic#258274) Closes: elastic#258227 ## Summary The test used a relative time range (`Last 30 days`) that doesn't overlap with the test data (Jan 2025), so the `METRICS_INFO` query returns empty results and the grid disappears. Replaced with an absolute range (`Jan 1 – Jun 30, 2025`) that covers the test data while still validating grid persistence across time range changes. Surfaced by elastic#257206, which switched to time-range-aware `METRICS_INFO` queries. The test had a latent bug (relative date with no data) that was previously masked by the old field-metadata-based approach. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ...
…se (elastic#257206) close elastic#253717 close elastic#253435 ## Summary Changes how the Metrics experience grid is built and rendered: we no longer use Field Caps and now rely on the ES|QL `METRICS_INFO` pipeline as the single source of metris grid. Data is fetched, parsed, and typed in `kbn-unified-chart-section-viewer`. ## Major changes - **Data source**: Grid is driven by METRICS_INFO instead of Field Caps. Fetch via `useFetchMetricsData` (ES|QL + `| METRICS_INFO`), parse with `parseMetricsResponse`; grid and toolbar consume `metricItems` and `allDimensions` from this hook. - **Filtering**: When multiple dimension names are selected, a WHERE dimension filter (e.g. `field IS NOT NULL`) is applied in `buildMetricsInfoQuery` before `METRICS_INFO`. - **Types**: `ParsedMetricItem` and `ParsedMetricsResult` describe the flow; grid, chart layers, flyout, and content use this shape (`metricName`, `dataStream`, `fieldTypes`, `metricTypes`, `units`, `dimensionFields`). Public API exports `ParsedMetricItem`, `ParsedMetricsResult`, `MetricsInfoResponse`, `MetricsESQLResponseObject`, `Dimension`, `MetricUnit`, `UnifiedMetricsGridProps`. - **Naming**: Parser returns `metricItems` / `allDimensions`; hook returns `metricItems`; - **Search/filter**: Client-side only via `useMetricFieldsFilter` and `filteredMetricItems` (no Field Caps). - **Tests**: Updated and added mocks for `useFetchMetricsData`, `useMetricFieldsFilter`, and related hooks. ## Behavior - **SORT only:** `FROM x | SORT timestamp DESC` → `FROM x | METRICS_INFO` (SORT removed). - **LIMIT only:** `FROM x | LIMIT 10` → `FROM x | METRICS_INFO | LIMIT 10`. - **SORT + LIMIT:** `FROM x | SORT ts | LIMIT 100` → `FROM x | METRICS_INFO | LIMIT 100`. - **WHERE + SORT + LIMIT:** Existing WHERE is kept; SORT is removed; LIMIT is re-appended after METRICS_INFO. ## Not addressed - **Multi-value in flyout**: Handling/display of multi-value fields in the metric insights flyout will be done in a separate ticket. - ~**Dimension/Naming refactor**: Simplification of dimension handling (e.g. `Dimension[]` vs `string[]`, selector, state) is out of scope here and will be done in a separate change.~ Keep the Dimension[] type https://github.com/user-attachments/assets/0d865735-ac21-418e-9fb0-16078dfc52f2 ## How to test 1. use https://github.com/simianhacker/simian-forge 2. run `npm run build` 3. `./forge --dataset histograms --count 3 --interval 10s` 4. go to discover --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Lucas Francisco López <lucaslopezf@gmail.com>
…metrics experience tests (elastic#258274) Closes: elastic#258227 ## Summary The test used a relative time range (`Last 30 days`) that doesn't overlap with the test data (Jan 2025), so the `METRICS_INFO` query returns empty results and the grid disappears. Replaced with an absolute range (`Jan 1 – Jun 30, 2025`) that covers the test data while still validating grid persistence across time range changes. Surfaced by elastic#257206, which switched to time-range-aware `METRICS_INFO` queries. The test had a latent bug (relative date with no data) that was previously masked by the old field-metadata-based approach. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ...
…se (elastic#257206) close elastic#253717 close elastic#253435 ## Summary Changes how the Metrics experience grid is built and rendered: we no longer use Field Caps and now rely on the ES|QL `METRICS_INFO` pipeline as the single source of metris grid. Data is fetched, parsed, and typed in `kbn-unified-chart-section-viewer`. ## Major changes - **Data source**: Grid is driven by METRICS_INFO instead of Field Caps. Fetch via `useFetchMetricsData` (ES|QL + `| METRICS_INFO`), parse with `parseMetricsResponse`; grid and toolbar consume `metricItems` and `allDimensions` from this hook. - **Filtering**: When multiple dimension names are selected, a WHERE dimension filter (e.g. `field IS NOT NULL`) is applied in `buildMetricsInfoQuery` before `METRICS_INFO`. - **Types**: `ParsedMetricItem` and `ParsedMetricsResult` describe the flow; grid, chart layers, flyout, and content use this shape (`metricName`, `dataStream`, `fieldTypes`, `metricTypes`, `units`, `dimensionFields`). Public API exports `ParsedMetricItem`, `ParsedMetricsResult`, `MetricsInfoResponse`, `MetricsESQLResponseObject`, `Dimension`, `MetricUnit`, `UnifiedMetricsGridProps`. - **Naming**: Parser returns `metricItems` / `allDimensions`; hook returns `metricItems`; - **Search/filter**: Client-side only via `useMetricFieldsFilter` and `filteredMetricItems` (no Field Caps). - **Tests**: Updated and added mocks for `useFetchMetricsData`, `useMetricFieldsFilter`, and related hooks. ## Behavior - **SORT only:** `FROM x | SORT timestamp DESC` → `FROM x | METRICS_INFO` (SORT removed). - **LIMIT only:** `FROM x | LIMIT 10` → `FROM x | METRICS_INFO | LIMIT 10`. - **SORT + LIMIT:** `FROM x | SORT ts | LIMIT 100` → `FROM x | METRICS_INFO | LIMIT 100`. - **WHERE + SORT + LIMIT:** Existing WHERE is kept; SORT is removed; LIMIT is re-appended after METRICS_INFO. ## Not addressed - **Multi-value in flyout**: Handling/display of multi-value fields in the metric insights flyout will be done in a separate ticket. - ~**Dimension/Naming refactor**: Simplification of dimension handling (e.g. `Dimension[]` vs `string[]`, selector, state) is out of scope here and will be done in a separate change.~ Keep the Dimension[] type https://github.com/user-attachments/assets/0d865735-ac21-418e-9fb0-16078dfc52f2 ## How to test 1. use https://github.com/simianhacker/simian-forge 2. run `npm run build` 3. `./forge --dataset histograms --count 3 --interval 10s` 4. go to discover --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Lucas Francisco López <lucaslopezf@gmail.com>
…metrics experience tests (elastic#258274) Closes: elastic#258227 ## Summary The test used a relative time range (`Last 30 days`) that doesn't overlap with the test data (Jan 2025), so the `METRICS_INFO` query returns empty results and the grid disappears. Replaced with an absolute range (`Jan 1 – Jun 30, 2025`) that covers the test data while still validating grid persistence across time range changes. Surfaced by elastic#257206, which switched to time-range-aware `METRICS_INFO` queries. The test had a latent bug (relative date with no data) that was previously masked by the old field-metadata-based approach. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ...
close #253717
close #253435
Summary
Changes how the Metrics experience grid is built and rendered: we no longer use Field Caps and now rely on the ES|QL
METRICS_INFOpipeline as the single source of metris grid. Data is fetched, parsed, and typed inkbn-unified-chart-section-viewer.Major changes
useFetchMetricsData(ES|QL +| METRICS_INFO), parse withparseMetricsResponse; grid and toolbar consumemetricItemsandallDimensionsfrom this hook.field IS NOT NULL) is applied inbuildMetricsInfoQuerybeforeMETRICS_INFO.ParsedMetricItemandParsedMetricsResultdescribe the flow; grid, chart layers, flyout, and content use this shape (metricName,dataStream,fieldTypes,metricTypes,units,dimensionFields). Public API exportsParsedMetricItem,ParsedMetricsResult,MetricsInfoResponse,MetricsESQLResponseObject,Dimension,MetricUnit,UnifiedMetricsGridProps.metricItems/allDimensions; hook returnsmetricItems;useMetricFieldsFilterandfilteredMetricItems(no Field Caps).useFetchMetricsData,useMetricFieldsFilter, and related hooks.Behavior
FROM x | SORT timestamp DESC→FROM x | METRICS_INFO(SORT removed).FROM x | LIMIT 10→FROM x | METRICS_INFO | LIMIT 10.FROM x | SORT ts | LIMIT 100→FROM x | METRICS_INFO | LIMIT 100.Not addressed
Dimension/Naming refactor: Simplification of dimension handling (e.g.Keep the Dimension[] typeDimension[]vsstring[], selector, state) is out of scope here and will be done in a separate change.Screen.Recording.2026-03-13.at.4.32.14.PM.mov
How to test
npm run build./forge --dataset histograms --count 3 --interval 10s