[Chargeback] Fix chargeback visualizations and add automated config lookup by JohannesMahne · Pull Request #16936 · elastic/integrations

JohannesMahne · 2026-01-13T05:28:09Z

Summary

Fixes visualization loading issues and automates chargeback_conf_lookup index creation.

Changes

Bug Fixes

Fix visualizations not loading by adding TO_DOUBLE type conversion in ESQL queries to prevent integer division returning zero (fixes #69)

Enhancements

Add automated chargeback_conf_lookup index creation via transform (fixes #70)
- Bootstrap transform creates index with default config values
- Default ECU rate: 0.85 EUR, weights: 20/20/40
- Date range: 2010-01-01 (Elasticsearch birthdate) to 2046-12-31
- Eliminates need for manual index creation

Maintenance

Bump transform pipeline versions to 0.2.10
Update package version to 0.2.10

Testing

Tested on local docker stack
Verified transforms create lookup index automatically
Confirmed dashboard visualizations display correctly

Closes

- Add TO_DOUBLE() wrapper to all division operations in ESQL queries - Prevents integer division from returning zero - Fixes tier_sum_indexing_time / deployment_sum_indexing_time - Fixes tier_sum_query_time / deployment_sum_query_time - Fixes tier_sum_data_set_store_size / deployment_sum_data_set_store_size - Fixes tier_sum_store_size / deployment_sum_store_size - Bump version to 0.2.10 Fixes: elastic/elasticsearch-chargeback#69

- Add bootstrap transform that creates chargeback_conf_lookup index with default config - Uses cluster_deployment_contribution_lookup as source - Sets default values: ECU rate 0.85 EUR, weights 20/20/40 - Date range: 2010-01-01 (ES birthdate) to 2046-12-31 - Eliminates need for manual index creation

@timestamp

- Update pipeline references from 0.2.9 to 0.2.10 - Revert billing_cluster_cost sync field to event.ingested (was temporarily @timestamp)

Copilot

Pull request overview

This PR addresses visualization display issues and automates chargeback configuration index creation. It fixes integer division returning zero in ESQL queries by adding TO_DOUBLE type conversions, and introduces a transform-based approach to automatically create and populate the chargeback_conf_lookup index with default configuration values.

Changes:

Fixed integer division in ESQL queries causing visualizations to display zero values
Automated chargeback_conf_lookup index creation through transform configuration
Updated pipeline versions from 0.2.9 to 0.2.10 across all transforms

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
packages/chargeback/pre-setup.sh	Removed manual index creation script (replaced by automated transform)
packages/chargeback/manifest.yml	Bumped package version to 0.2.10
packages/chargeback/kibana/dashboard/chargeback-39a39857-746c-4a29-adca-3c2fcb6bcfb6.json	Added TO_DOUBLE type conversions in ESQL queries to fix visualization calculations
packages/chargeback/elasticsearch/transform/cluster_tier_contribution/transform.yml	Updated pipeline version to 0.2.10
packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/transform.yml	Updated pipeline version to 0.2.10
packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/transform.yml	Updated pipeline version to 0.2.10
packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/transform.yml	Updated pipeline version to 0.2.10
packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/transform.yml	Added transform configuration for automated lookup index creation
packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/manifest.yml	Added manifest for chargeback_conf_lookup transform
packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/fields/ecs.yml	Added empty ECS fields definition
packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/fields/base-fields.yml	Added field definitions for configuration lookup index
packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml	Updated pipeline version to 0.2.10
packages/chargeback/elasticsearch/ingest_pipeline/chargeback_conf_lookup.yml	Added pipeline to extract currency unit from terms aggregation
packages/chargeback/changelog.yml	Added changelog entry for version 0.2.10

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/fields/base-fields.yml

JohannesMahne · 2026-01-13T05:34:40Z

@sholzhauer-es - would you please check this PR?
I built the assets and prepared PR on elasticsearch-chargeback: elastic/elasticsearch-chargeback#71

…okup/fields/base-fields.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…on fixes - Removed manual setup_commands.json (no longer needed) - Updated Instructions.md: removed manual config setup step, added Configuration section at end - Updated README.md with version 0.2.10 release notes - Documented automated chargeback_conf_lookup creation via bootstrap transform - Simplified installation steps - configuration index now auto-created - Default config: ECU rate 0.85 EUR, weights 20/20/40, date range 2010-2046 Based on PR: elastic/integrations#16936

JohannesMahne · 2026-01-29T15:33:06Z

@elastic-abhi - can you please review the PR here too? You tested the integration on the elasticsearch-chargeback repo, but it relates to this PR.

elastic-abhi

the chargeback_conf_lookup transform didn't load in my test setup due to missing data view "cluster_deployment_contribution_lookup"

rest is aligned with 0.2.10 changes review earlier

@elastic-abhi

Change source from cluster_deployment_contribution_lookup to metrics-ess_billing.billing-* to fix transform loading in fresh setups. The transform uses runtime_mappings to generate all config values, so the source index content doesn't matter - it only needs any document to trigger. Billing data is guaranteed to exist in chargeback deployments. Fixes feedback from @elastic-abhi review.

JohannesMahne · 2026-01-30T08:03:11Z

@elastic-abhi Thanks for catching this! I've pushed a fix.

Fix Applied

Changed the chargeback_conf_lookup transform source from cluster_deployment_contribution_lookup to metrics-ess_billing.billing-*.

Why this fixes the issue

The original source index cluster_deployment_contribution_lookup doesn't exist in fresh setups until the cluster_deployment_contribution transform runs with monitoring data - creating a dependency problem.

The transform uses runtime_mappings to generate all config values (ECU rate, weights, dates), so the actual source data content doesn't matter. It only needs any document to trigger execution.

Using metrics-ess_billing.billing-* is better because:

Billing data is guaranteed to exist in any chargeback deployment (it's the core requirement)
Eliminates the dependency on the monitoring integration being configured first
The billing_cluster_cost transform already uses this source successfully

Please re-test when you get a chance!

elasticmachine · 2026-01-30T08:24:41Z

💚 Build Succeeded

Buildkite Build
Commit: 5dd0e7e

History

💚 Build #36347 succeeded 3b6d518
💚 Build #36346 succeeded bf477bb

elastic-abhi

look perfect

@timestamp

…, deployment groups (#17170) * WIP: early chargeback code for review * Working config integration - 0.0.2 * Version 0.0.3: working from Stack monitoring data * Fixed query for one visualisation * Update instructions * Working with the correct alias * Changes to transforms * Bug fix: Fix sorting on visualisation. * Update setup instructions * 0.1.0: Adding ECU value (normalised cost). * Bug: Aligned fields returned to field names used in visualisation * Fixing bug: aligning esql returned field names with field names used in lens * move to packages * not starting transforms on integration installation * Update version number * Made sure the colour palette is predictable by using the eui_amsterdam_color_blind palate. Add ECU rate to the dashboard. * Update sequence and comments on pre-setup to promote ES integration * Consistent naming of datastream. Add LIMIT 5000 to ESQL top query to cater for large organisations. * Add correct code owner * Delete wrong test files * Updated the directory structure to remove superfluous directory * Rem reference to sample logs and logos * Switch off dynamic mappings for the results of the transforms - we know exactly what the output be. * Removed agent folders in data stream, as it is not used. * Updated the readme file to refer to integration, rather than module. Also added explanation about the rest of the config. * Re-add image * Formatting * NOT WORKING: settings index.mode: lookup is not supported * Fixing the control error in the dashboard by adding a data view. * Updated to push back usage data transform to ES Integration * Updated readme * Update transfrom version numbers * Swap the use of deployment_id or deployment name to a concatenation of both, to make it easier to identify the deployment in the dashboard. * Make use of the new elastic-package version, which will create the lookup index automatically when installing the package. * Update version number * Updated pre-setup, and version number * Adding casting to double for division to avoid null instead of very small numbers. elastic/elasticsearch-chargeback#50 * Update version * Allowing for setting converion rate per time window * fixing pipeline versions * adding pipeline stuff * correcting version * [Chargeback] Dashboard control and Dataview (#16153) * dashboard control * updating version + DV * SKU based chargeback (#16182) * adding sku and cost_type to billing data for node granularity * working on sku with pipeline to do parsing * downplaying version * transform * Chargeback Integration: Extract deployment group from Billing tags (#16185) * Add deployment_group extraction from ESS Billing tags - Extract chargeback_group tag value to deployment_group field in billing pipeline - Add deployment_group to billing_cluster_cost transform group_by - Add deployment_group field definition - Fix transforms to use elasticsearch.cluster.name without .keyword - Update changelog for v0.2.4 * Add deployment_group extraction using runtime mappings from ESS Billing tags * Update dashboard with deployment_group filter and definitions * Bump version to 0.2.5 for deployment_group feature after merging SKU/cost_type changes * known [bug](elastic/elasticsearch-chargeback#60) from 0.2.4 * Fixing bug introduced in 0.2.4 (#16192) * adding sku and cost_type to billing data for node granularity * working on sku with pipeline to do parsing * downplaying version * transform * merge * dashboard * setting version * Add observability alerts for chargeback integration (#16205) * Add observability alerts for chargeback integration - Add two ES|QL alerting rules: detect new chargeback groups and detect deployments missing usage data - Add comprehensive documentation for alert setup and configuration - Update Elasticsearch version requirement to 9.2.0+ for smart lookup join support - Add transform startup and monitoring instructions to README * Update changelog with PR #16205 * Remove wrong information * Update chargeback README documentation * Improve observability alert action message formatting * Clarify configuration update vs add new period documentation * Fix mustache template escaping in alert actions documentation * [Chargeback] Alerting rule (#16229) * Add alerting rule templates and enable auto-start for all transforms - Add 3 Kibana alerting rule templates: - Transform health monitoring for all Chargeback transforms - New chargeback group detection - Deployment with missing usage data detection - Enable auto-start for all transforms (start: true in manifests) - Update transform pipeline references to version 0.2.8 - Add performance warning about initial transform execution - Update README with alerting documentation - Bump package version to 0.2.8 * Fix: Revert transform frequencies back to 60m * Update PR number in changelog * Chargeback css (#16326) * WIP: early chargeback code for review * Working config integration - 0.0.2 * Version 0.0.3: working from Stack monitoring data * Fixed query for one visualisation * Update instructions * Working with the correct alias * Changes to transforms * Bug fix: Fix sorting on visualisation. * Update setup instructions * 0.1.0: Adding ECU value (normalised cost). * Bug: Aligned fields returned to field names used in visualisation * Fixing bug: aligning esql returned field names with field names used in lens * move to packages * not starting transforms on integration installation * Update version number * Made sure the colour palette is predictable by using the eui_amsterdam_color_blind palate. Add ECU rate to the dashboard. * Update sequence and comments on pre-setup to promote ES integration * Consistent naming of datastream. Add LIMIT 5000 to ESQL top query to cater for large organisations. * Add correct code owner * Delete wrong test files * Updated the directory structure to remove superfluous directory * Rem reference to sample logs and logos * Switch off dynamic mappings for the results of the transforms - we know exactly what the output be. * Removed agent folders in data stream, as it is not used. * Updated the readme file to refer to integration, rather than module. Also added explanation about the rest of the config. * Re-add image * Formatting * NOT WORKING: settings index.mode: lookup is not supported * Fixing the control error in the dashboard by adding a data view. * Updated to push back usage data transform to ES Integration * Updated readme * Update transfrom version numbers * Swap the use of deployment_id or deployment name to a concatenation of both, to make it easier to identify the deployment in the dashboard. * Make use of the new elastic-package version, which will create the lookup index automatically when installing the package. * Update version number * Updated pre-setup, and version number * Adding casting to double for division to avoid null instead of very small numbers. elastic/elasticsearch-chargeback#50 * Update version * Allowing for setting converion rate per time window * fixing pipeline versions * adding pipeline stuff * correcting version * [Chargeback] Dashboard control and Dataview (#16153) * dashboard control * updating version + DV * SKU based chargeback (#16182) * adding sku and cost_type to billing data for node granularity * working on sku with pipeline to do parsing * downplaying version * transform * Chargeback Integration: Extract deployment group from Billing tags (#16185) * Add deployment_group extraction from ESS Billing tags - Extract chargeback_group tag value to deployment_group field in billing pipeline - Add deployment_group to billing_cluster_cost transform group_by - Add deployment_group field definition - Fix transforms to use elasticsearch.cluster.name without .keyword - Update changelog for v0.2.4 * Add deployment_group extraction using runtime mappings from ESS Billing tags * Update dashboard with deployment_group filter and definitions * Bump version to 0.2.5 for deployment_group feature after merging SKU/cost_type changes * known [bug](elastic/elasticsearch-chargeback#60) from 0.2.4 * wip on css * adding "local" cluster for ones without remote clusters --------- Co-authored-by: Johannes Mahne <johannes.mahne@elastic.co> * Fix: Correct PR number for CSS changes in changelog (0.2.9) * [Chargeback] Fix chargeback visualizations and add automated config lookup (#16936) * Fix visualizations not loading by adding TO_DOUBLE type conversion - Add TO_DOUBLE() wrapper to all division operations in ESQL queries - Prevents integer division from returning zero - Fixes tier_sum_indexing_time / deployment_sum_indexing_time - Fixes tier_sum_query_time / deployment_sum_query_time - Fixes tier_sum_data_set_store_size / deployment_sum_data_set_store_size - Fixes tier_sum_store_size / deployment_sum_store_size - Bump version to 0.2.10 Fixes: elastic/elasticsearch-chargeback#69 * Add automated chargeback_conf_lookup index creation via transform - Add bootstrap transform that creates chargeback_conf_lookup index with default config - Uses cluster_deployment_contribution_lookup as source - Sets default values: ECU rate 0.85 EUR, weights 20/20/40 - Date range: 2010-01-01 (ES birthdate) to 2046-12-31 - Eliminates need for manual index creation * Bump transform pipeline versions to 0.2.10 - Update pipeline references from 0.2.9 to 0.2.10 - Revert billing_cluster_cost sync field to event.ingested (was temporarily @timestamp) * Removed now redundant pre-setup. * Update PR number in changelog, and recover billing cost sync time field. * Update packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/fields/base-fields.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix chargeback_conf_lookup transform source index dependency Change source from cluster_deployment_contribution_lookup to metrics-ess_billing.billing-* to fix transform loading in fresh setups. The transform uses runtime_mappings to generate all config values, so the source index content doesn't matter - it only needs any document to trigger. Billing data is guaranteed to exist in chargeback deployments. Fixes feedback from @elastic-abhi review. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Chargeback dashboard: ECU/ERU wording in Configuration Information section (#17168) * Chargeback 0.3.0: chargeable units (ECU/ERU), bump manifest and transform pipeline/versions * Apply suggestion from @Copilot Spell correction Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Spell correction Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Spell correction Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Spell correction Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Stijn Holzhauer <stijn.holzhauer@elastic.co> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Abhi <252589011+elastic-abhi@users.noreply.github.com>

JohannesMahne added 4 commits January 12, 2026 14:33

Bump transform pipeline versions to 0.2.10

ac560c7

- Update pipeline references from 0.2.9 to 0.2.10 - Revert billing_cluster_cost sync field to event.ingested (was temporarily @timestamp)

Removed now redundant pre-setup.

063c2a6

JohannesMahne requested a review from a team as a code owner January 13, 2026 05:28

JohannesMahne changed the title ~~Fix chargeback visualizations and add automated config lookup~~ [Chargeback] Fix chargeback visualizations and add automated config lookup Jan 13, 2026

Update PR number in changelog, and recover billing cost sync time field.

bf477bb

JohannesMahne requested review from Copilot and sholzhauer-es January 13, 2026 05:33

Copilot AI reviewed Jan 13, 2026

View reviewed changes

packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/fields/base-fields.yml Outdated Show resolved Hide resolved

Update packages/chargeback/elasticsearch/transform/chargeback_conf_lo…

3b6d518

…okup/fields/base-fields.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

This was referenced Jan 13, 2026

Update to version 0.2.10 with automated config lookup and visualization fix elastic/elasticsearch-chargeback#71

Merged

integration package elastic/elasticsearch-chargeback#14

Open

andrewkroh added the dashboard Relates to a Kibana dashboard bug, enhancement, or modification. label Jan 13, 2026

JohannesMahne requested review from elastic-abhi and removed request for a team and sholzhauer-es January 29, 2026 14:59

elastic-abhi reviewed Jan 30, 2026

View reviewed changes

JohannesMahne requested a review from elastic-abhi January 30, 2026 08:04

elastic-abhi reviewed Jan 30, 2026

View reviewed changes

JohannesMahne merged commit 8113829 into johannes-chargeback-wip Jan 30, 2026
9 checks passed

JohannesMahne deleted the fix-issue-69-visualization-to-double branch January 30, 2026 08:47

JohannesMahne mentioned this pull request Jan 30, 2026

Chargeback: chargeable units (ECU/ERU), visualizations, config lookup, deployment groups #17169

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Chargeback] Fix chargeback visualizations and add automated config lookup#16936

[Chargeback] Fix chargeback visualizations and add automated config lookup#16936
JohannesMahne merged 7 commits intojohannes-chargeback-wipfrom
fix-issue-69-visualization-to-double

JohannesMahne commented Jan 13, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

JohannesMahne commented Jan 13, 2026 •

edited

Loading

Uh oh!

JohannesMahne commented Jan 29, 2026

Uh oh!

elastic-abhi left a comment •

edited

Loading

Uh oh!

JohannesMahne commented Jan 30, 2026

Uh oh!

elasticmachine commented Jan 30, 2026

Uh oh!

elastic-abhi left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

JohannesMahne commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Bug Fixes

Enhancements

Maintenance

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

JohannesMahne commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JohannesMahne commented Jan 29, 2026

Uh oh!

elastic-abhi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JohannesMahne commented Jan 30, 2026

Fix Applied

Why this fixes the issue

Uh oh!

elasticmachine commented Jan 30, 2026

💚 Build Succeeded

History

Uh oh!

elastic-abhi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

JohannesMahne commented Jan 13, 2026 •

edited

Loading

JohannesMahne commented Jan 13, 2026 •

edited

Loading

elastic-abhi left a comment •

edited

Loading