On-Prem Billing: mERU cost model, 0.2.0 ERU config and deployment_tags by JohannesMahne · Pull Request #17167 · elastic/integrations

JohannesMahne · 2026-01-30T10:35:20Z

Summary

On-Prem Billing integration: mERU (milli-ERU) cost model and 0.2.0 alignment
ERU-based config (organization + deployment config), deployment_tags in config_bootstrap
Docs and transform updates for ERU/deployment_tags

Target

Merges into wip-johannes-chargeback-chargable-units (Chargeback WIP branch).

Testing

E2E tested with elasticsearch-chargeback run_e2e_tests.sh (On-Prem Billing + Chargeback)

…in lens

…m_color_blind palate. Add ECU rate to the dashboard.

…cater for large organisations.

…ow exactly what the output be.

…Also added explanation about the rest of the config.

- Add TO_DOUBLE() wrapper to all division operations in ESQL queries - Prevents integer division from returning zero - Fixes tier_sum_indexing_time / deployment_sum_indexing_time - Fixes tier_sum_query_time / deployment_sum_query_time - Fixes tier_sum_data_set_store_size / deployment_sum_data_set_store_size - Fixes tier_sum_store_size / deployment_sum_store_size - Bump version to 0.2.10 Fixes: elastic/elasticsearch-chargeback#69

- Add bootstrap transform that creates chargeback_conf_lookup index with default config - Uses cluster_deployment_contribution_lookup as source - Sets default values: ECU rate 0.85 EUR, weights 20/20/40 - Date range: 2010-01-01 (ES birthdate) to 2046-12-31 - Eliminates need for manual index creation

@timestamp

- Update pipeline references from 0.2.9 to 0.2.10 - Revert billing_cluster_cost sync field to event.ingested (was temporarily @timestamp)

Initial release of the On-Premises Billing integration that generates ESS Billing-compatible metrics for on-premises, ECE, and ECK deployments, enabling the Chargeback integration to work in non-cloud environments. Features: - Fixed daily ECU model per deployment - Bootstrap transform for auto-discovery of deployments - Configurable deployment_tags for Chargeback grouping - Manual enrich policy setup for deployment configuration - Output to metrics-ess_billing.billing-onprem index Requires manual post-installation setup (enrich policy, pipeline, transform start).

Introduces milli-ERU as the internal cost unit (1 ERU = 1000 mERU) to handle fractional ERU allocations cleanly. This allows deployments ranging from 0.25 ERU to 10+ ERU without awkward decimals or large rate multipliers. Changes: - Add organization-level config (license cost, total ERUs, ERU-to-RAM ratio) - Support per-deployment config via direct ERU input OR RAM-based calculation - Update config_bootstrap transform for mERU defaults - Comprehensive documentation on gathering config data and calculations WIP: Preliminary implementation - needs testing

…m alignment

Copilot

Pull request overview

This PR integrates on-premises billing with the Chargeback integration by introducing a milli-ERU (mERU) cost model and aligning with version 0.2.0. It adds support for ERU-based and RAM-based deployment configuration, deployment tags for grouping, and comprehensive documentation for setup and usage.

Changes:

Introduces new onprem_billing integration package (v0.2.0) with mERU cost model
Adds chargeback_conf_lookup transform for automated configuration bootstrap
Updates Chargeback dashboard ESQL queries to fix integer division issues with TO_DOUBLE conversion
Removes manual setup script in favor of automated transform-based configuration

Reviewed changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
packages/onprem_billing/manifest.yml	Defines new integration metadata for on-premises billing with version 0.2.0
packages/onprem_billing/elasticsearch/transform/config_bootstrap/transform.yml	Bootstrap transform that discovers deployments and creates mERU-based config records
packages/onprem_billing/elasticsearch/transform/billing/transform.yml	Aggregates daily billing data from Stack Monitoring to ESS-compatible format
packages/onprem_billing/docs/README.md	Comprehensive documentation for mERU model, configuration, and troubleshooting
packages/chargeback/manifest.yml	Version bump to 0.2.10
packages/chargeback/pre-setup.sh	Deleted manual setup script replaced by automated transform
packages/chargeback/kibana/dashboard/chargeback-39a39857-746c-4a29-adca-3c2fcb6bcfb6.json	Fixes ESQL queries with TO_DOUBLE to prevent integer division
packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/transform.yml	New transform for automated chargeback configuration bootstrap

Comments suppressed due to low confidence (1)

packages/onprem_billing/docs/README.md:1

Corrected spelling of 'birthdate' to 'birth date' for better readability.

# On-Premises Billing Integration

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/fields/base-fields.yml

packages/chargeback/elasticsearch/transform/cluster_tier_contribution/transform.yml

packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/transform.yml

packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/transform.yml

packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/transform.yml

…okup/fields/base-fields.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…_contribution/transform.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…ibution/transform.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…s_contribution/transform.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…_contribution/transform.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

elasticmachine · 2026-01-30T10:47:13Z

💔 Build Failed

Buildkite Build
Commit: be6c183

Failed CI Steps

:pipeline:⬆️ Upload Pipeline: .buildkite/pipeline.yml

History

💔 Build #37195 failed 71a472a

elastic-abhi

Looks perfect.

JohannesMahne and others added 30 commits December 2, 2025 12:16

WIP: early chargeback code for review

de717b8

Working config integration - 0.0.2

14941da

Version 0.0.3: working from Stack monitoring data

5343625

Fixed query for one visualisation

fb96f41

Update instructions

c776186

Working with the correct alias

5fa139c

Changes to transforms

4a643d7

Bug fix: Fix sorting on visualisation.

fac07bc

Update setup instructions

19a902b

0.1.0: Adding ECU value (normalised cost).

668a7cf

Bug: Aligned fields returned to field names used in visualisation

0482b9a

Fixing bug: aligning esql returned field names with field names used …

5b0607e

…in lens

move to packages

aace7b6

not starting transforms on integration installation

aad3921

Update version number

224b6ca

Made sure the colour palette is predictable by using the eui_amsterda…

9656a03

…m_color_blind palate. Add ECU rate to the dashboard.

Update sequence and comments on pre-setup to promote ES integration

c271271

Consistent naming of datastream. Add LIMIT 5000 to ESQL top query to …

5155220

…cater for large organisations.

Add correct code owner

1ca257f

Delete wrong test files

5b0e668

Updated the directory structure to remove superfluous directory

d3048ef

Rem reference to sample logs and logos

79508a5

Switch off dynamic mappings for the results of the transforms - we kn…

4e471f0

…ow exactly what the output be.

Removed agent folders in data stream, as it is not used.

e5efd8f

Updated the readme file to refer to integration, rather than module. …

18b0b0c

…Also added explanation about the rest of the config.

Re-add image

3101319

Formatting

4dfb302

NOT WORKING: settings index.mode: lookup is not supported

2a6ecdb

Fixing the control error in the dashboard by adding a data view.

ae4263d

Updated to push back usage data transform to ES Integration

55d66f2

JohannesMahne added 8 commits January 12, 2026 14:33

Bump transform pipeline versions to 0.2.10

ac560c7

- Update pipeline references from 0.2.9 to 0.2.10 - Revert billing_cluster_cost sync field to event.ingested (was temporarily @timestamp)

Removed now redundant pre-setup.

063c2a6

Update PR number in changelog, and recover billing cost sync time field.

bf477bb

On-Prem Billing: 0.2.0 ERU config, deployment_tags, docs and transfor…

71a472a

…m alignment

JohannesMahne requested review from a team as code owners January 30, 2026 10:35

mergify bot mentioned this pull request Jan 30, 2026

On-Prem Billing: mERU cost model, 0.2.0 ERU config and deployment_tags #17166

Closed

JohannesMahne requested review from Copilot and elastic-abhi and removed request for a team January 30, 2026 10:37

Copilot AI reviewed Jan 30, 2026

View reviewed changes

JohannesMahne and others added 5 commits January 30, 2026 12:44

Update packages/chargeback/elasticsearch/transform/chargeback_conf_lo…

73d314e

…okup/fields/base-fields.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update packages/chargeback/elasticsearch/transform/cluster_deployment…

a722bd2

…_contribution/transform.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update packages/chargeback/elasticsearch/transform/cluster_tier_contr…

470afcb

…ibution/transform.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update packages/chargeback/elasticsearch/transform/cluster_tier_and_d…

389372f

…s_contribution/transform.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update packages/chargeback/elasticsearch/transform/cluster_datastream…

be6c183

…_contribution/transform.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

JohannesMahne mentioned this pull request Jan 30, 2026

Chargeback: chargeable units (ECU/ERU), visualizations, config lookup, deployment groups #17169

Closed

JohannesMahne changed the base branch from wip-johannes-chargeback-chargable-units to wip-johannes-chargeback January 30, 2026 12:04

andrewkroh added New Integration Issue or pull request for creating a new integration package. dashboard Relates to a Kibana dashboard bug, enhancement, or modification. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. labels Jan 30, 2026

elastic-abhi approved these changes Feb 6, 2026

View reviewed changes

JohannesMahne merged commit 3d24e27 into wip-johannes-chargeback Feb 9, 2026
11 of 12 checks passed

JohannesMahne deleted the wip/onprem-billing-integration branch February 9, 2026 07:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

On-Prem Billing: mERU cost model, 0.2.0 ERU config and deployment_tags#17167

On-Prem Billing: mERU cost model, 0.2.0 ERU config and deployment_tags#17167
JohannesMahne merged 64 commits intowip-johannes-chargebackfrom
wip/onprem-billing-integration

JohannesMahne commented Jan 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Jan 30, 2026

Uh oh!

elastic-abhi left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

JohannesMahne commented Jan 30, 2026

Summary

Target

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Jan 30, 2026

💔 Build Failed

Failed CI Steps

History

Uh oh!

elastic-abhi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants