Fix/billing transform runtime fields deployment tags#17409
Open
JohannesMahne wants to merge 60 commits intowip-johannes-chargebackfrom
Open
Fix/billing transform runtime fields deployment tags#17409JohannesMahne wants to merge 60 commits intowip-johannes-chargebackfrom
JohannesMahne wants to merge 60 commits intowip-johannes-chargebackfrom
Conversation
…m_color_blind palate. Add ECU rate to the dashboard.
…cater for large organisations.
…ow exactly what the output be.
…Also added explanation about the rest of the config.
…16185) * Add deployment_group extraction from ESS Billing tags - Extract chargeback_group tag value to deployment_group field in billing pipeline - Add deployment_group to billing_cluster_cost transform group_by - Add deployment_group field definition - Fix transforms to use elasticsearch.cluster.name without .keyword - Update changelog for v0.2.4 * Add deployment_group extraction using runtime mappings from ESS Billing tags * Update dashboard with deployment_group filter and definitions * Bump version to 0.2.5 for deployment_group feature after merging SKU/cost_type changes * known [bug](elastic/elasticsearch-chargeback#60) from 0.2.4
* adding sku and cost_type to billing data for node granularity * working on sku with pipeline to do parsing * downplaying version * transform * merge * dashboard * setting version
* Add observability alerts for chargeback integration - Add two ES|QL alerting rules: detect new chargeback groups and detect deployments missing usage data - Add comprehensive documentation for alert setup and configuration - Update Elasticsearch version requirement to 9.2.0+ for smart lookup join support - Add transform startup and monitoring instructions to README * Update changelog with PR #16205 * Remove wrong information * Update chargeback README documentation * Improve observability alert action message formatting * Clarify configuration update vs add new period documentation
* Add alerting rule templates and enable auto-start for all transforms - Add 3 Kibana alerting rule templates: - Transform health monitoring for all Chargeback transforms - New chargeback group detection - Deployment with missing usage data detection - Enable auto-start for all transforms (start: true in manifests) - Update transform pipeline references to version 0.2.8 - Add performance warning about initial transform execution - Update README with alerting documentation - Bump package version to 0.2.8 * Fix: Revert transform frequencies back to 60m * Update PR number in changelog
* WIP: early chargeback code for review * Working config integration - 0.0.2 * Version 0.0.3: working from Stack monitoring data * Fixed query for one visualisation * Update instructions * Working with the correct alias * Changes to transforms * Bug fix: Fix sorting on visualisation. * Update setup instructions * 0.1.0: Adding ECU value (normalised cost). * Bug: Aligned fields returned to field names used in visualisation * Fixing bug: aligning esql returned field names with field names used in lens * move to packages * not starting transforms on integration installation * Update version number * Made sure the colour palette is predictable by using the eui_amsterdam_color_blind palate. Add ECU rate to the dashboard. * Update sequence and comments on pre-setup to promote ES integration * Consistent naming of datastream. Add LIMIT 5000 to ESQL top query to cater for large organisations. * Add correct code owner * Delete wrong test files * Updated the directory structure to remove superfluous directory * Rem reference to sample logs and logos * Switch off dynamic mappings for the results of the transforms - we know exactly what the output be. * Removed agent folders in data stream, as it is not used. * Updated the readme file to refer to integration, rather than module. Also added explanation about the rest of the config. * Re-add image * Formatting * NOT WORKING: settings index.mode: lookup is not supported * Fixing the control error in the dashboard by adding a data view. * Updated to push back usage data transform to ES Integration * Updated readme * Update transfrom version numbers * Swap the use of deployment_id or deployment name to a concatenation of both, to make it easier to identify the deployment in the dashboard. * Make use of the new elastic-package version, which will create the lookup index automatically when installing the package. * Update version number * Updated pre-setup, and version number * Adding casting to double for division to avoid null instead of very small numbers. elastic/elasticsearch-chargeback#50 * Update version * Allowing for setting converion rate per time window * fixing pipeline versions * adding pipeline stuff * correcting version * [Chargeback] Dashboard control and Dataview (#16153) * dashboard control * updating version + DV * SKU based chargeback (#16182) * adding sku and cost_type to billing data for node granularity * working on sku with pipeline to do parsing * downplaying version * transform * Chargeback Integration: Extract deployment group from Billing tags (#16185) * Add deployment_group extraction from ESS Billing tags - Extract chargeback_group tag value to deployment_group field in billing pipeline - Add deployment_group to billing_cluster_cost transform group_by - Add deployment_group field definition - Fix transforms to use elasticsearch.cluster.name without .keyword - Update changelog for v0.2.4 * Add deployment_group extraction using runtime mappings from ESS Billing tags * Update dashboard with deployment_group filter and definitions * Bump version to 0.2.5 for deployment_group feature after merging SKU/cost_type changes * known [bug](elastic/elasticsearch-chargeback#60) from 0.2.4 * wip on css * adding "local" cluster for ones without remote clusters --------- Co-authored-by: Johannes Mahne <johannes.mahne@elastic.co>
- Add TO_DOUBLE() wrapper to all division operations in ESQL queries - Prevents integer division from returning zero - Fixes tier_sum_indexing_time / deployment_sum_indexing_time - Fixes tier_sum_query_time / deployment_sum_query_time - Fixes tier_sum_data_set_store_size / deployment_sum_data_set_store_size - Fixes tier_sum_store_size / deployment_sum_store_size - Bump version to 0.2.10 Fixes: elastic/elasticsearch-chargeback#69
- Add bootstrap transform that creates chargeback_conf_lookup index with default config - Uses cluster_deployment_contribution_lookup as source - Sets default values: ECU rate 0.85 EUR, weights 20/20/40 - Date range: 2010-01-01 (ES birthdate) to 2046-12-31 - Eliminates need for manual index creation
- Update pipeline references from 0.2.9 to 0.2.10 - Revert billing_cluster_cost sync field to event.ingested (was temporarily @timestamp)
Initial release of the On-Premises Billing integration that generates ESS Billing-compatible metrics for on-premises, ECE, and ECK deployments, enabling the Chargeback integration to work in non-cloud environments. Features: - Fixed daily ECU model per deployment - Bootstrap transform for auto-discovery of deployments - Configurable deployment_tags for Chargeback grouping - Manual enrich policy setup for deployment configuration - Output to metrics-ess_billing.billing-onprem index Requires manual post-installation setup (enrich policy, pipeline, transform start).
Introduces milli-ERU as the internal cost unit (1 ERU = 1000 mERU) to handle fractional ERU allocations cleanly. This allows deployments ranging from 0.25 ERU to 10+ ERU without awkward decimals or large rate multipliers. Changes: - Add organization-level config (license cost, total ERUs, ERU-to-RAM ratio) - Support per-deployment config via direct ERU input OR RAM-based calculation - Update config_bootstrap transform for mERU defaults - Comprehensive documentation on gathering config data and calculations WIP: Preliminary implementation - needs testing
💔 Build Failed
Failed CI Steps |
|
Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as |
| - set: | ||
| field: cost_type | ||
| value: 'snapshots' | ||
| if: ctx.sku =~ /^(aws|gcp|azure).snapshot-storage&/ |
There was a problem hiding this comment.
🟠 High ingest_pipeline/billing.yml:19
The regex on line 19 ends with & instead of .*, so the condition ctx.sku =~ /^(aws|gcp|azure).snapshot-storage&/ never matches real snapshot-storage SKUs (which use . separators like aws.snapshot-storage.xxx). This causes all snapshot storage documents to skip the 'snapshots' cost_type assignment and fall through to the grok processor, resulting in incorrect cost categorization.
- if: ctx.sku =~ /^(aws|gcp|azure).snapshot-storage&/
+ if: ctx.sku =~ /^(aws|gcp|azure).snapshot-storage.*/🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file packages/chargeback/elasticsearch/ingest_pipeline/billing.yml around line 19:
The regex on line 19 ends with `&` instead of `.*`, so the condition `ctx.sku =~ /^(aws|gcp|azure).snapshot-storage&/` never matches real snapshot-storage SKUs (which use `.` separators like `aws.snapshot-storage.xxx`). This causes all snapshot storage documents to skip the 'snapshots' `cost_type` assignment and fall through to the grok processor, resulting in incorrect cost categorization.
There was a problem hiding this comment.
I'm working on the fix — it will be delivered via a new PR shortly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug fix: elastic/elasticsearch-chargeback#91
Proposed commit message
Fix billing_cluster_cost transform when deployment_tags is a string
The deployment_group runtime mapping in the billing_cluster_cost transform
assumed ess.billing.deployment_tags is always a list and used
for (tag in params._source.ess.billing.deployment_tags). When the fieldis indexed as a single string (e.g. one tag), Painless throws
"Cannot iterate over [java.lang.String]" and the transform fails with
ValidationException / BAD_REQUEST.
when it is a List, iterate as before; when it is a String, check that
single value for the chargeback_group: prefix. This keeps behavior
unchanged for array values and fixes the string case.
deployment_tags as either an array or a single keyword, so the
runtime mapping must support both to avoid transform failures.
Bumped fleet_transform_version (0.2.5 → 0.2.6) so the updated transform
is reinstalled when the package is deployed.
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
elastic-package buildand install in a test stack).ess.billing.deployment_tagsis a single string (e.g."chargeback_group:my-team") into the billing source index.billing_cluster_costtransform run. It should complete without "Cannot iterate over [java.lang.String]" or BAD_REQUEST.deployment_tagsas an array; confirm deployment_group extraction still works.Related issues
Screenshots
N/A – runtime script change only; no UI changes.