Chargeback 0.3.1: chargeable units (ECU/ERU rename), E2E testing scripts, On-Prem Billing pipeline fix#89
Open
JohannesMahne wants to merge 9 commits intomainfrom
Open
Chargeback 0.3.1: chargeable units (ECU/ERU rename), E2E testing scripts, On-Prem Billing pipeline fix#89JohannesMahne wants to merge 9 commits intomainfrom
JohannesMahne wants to merge 9 commits intomainfrom
Conversation
…ign with integrations PR #17168)
…deployment_group patch (8h), enrich wait, jq prereq
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Chargeback integration to version 0.3.0 with a breaking change from ECU (Elastic Consumption Units) terminology to "chargeable units", and adds comprehensive local testing infrastructure for end-to-end validation of Chargeback and On-Prem Billing integrations.
Changes:
- Renamed ECU-related fields to chargeable units across documentation and instructions (
total_ecu→total_chargeable_units,conf_ecu_rate→conf_chargeable_unit_rate, etc.) - Added automated E2E test script (
run_e2e_tests.sh) that installs Elasticsearch, On-Prem Billing, and Chargeback integrations with a 12 ERU test scenario - Added cleanup script and comprehensive testing documentation
Reviewed changes
Copilot reviewed 9 out of 12 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/run_e2e_tests.sh | New automated E2E test script for installing and verifying integrations with 12 ERU test setup |
| scripts/onprem_billing_calculate_cost_pipeline.json | New ingest pipeline for On-Prem Billing cost calculations (mERU/ERU/RAM to ESS Billing schema) |
| scripts/cleanup_testing_env.sh | New cleanup script for uninstalling test environment |
| scripts/README.md | New comprehensive testing documentation |
| integration/README.md | Updated version to 0.3.0 and documented breaking changes from ECU to chargeable units |
| integration/Instructions.md | Updated instructions with new chargeable unit field names |
| integration/.gitattributes | Added beads JSONL merge strategy |
| README.md | Updated ECU terminology to chargeable units |
| CHANGELOG.md | Added 0.3.0 release notes with breaking changes |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…orm sync works Add a set processor for event.ingested (= _ingest.timestamp) to onprem_billing_calculate_cost_pipeline.json. The billing_cluster_cost transform is configured with sync.time.field: event.ingested; without this, newly ingested On-Prem billing docs are invisible to the transform and the lookup index stays empty. Also update chargeback-0.3.0.zip to include related integration changes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… cleanup improvements run_e2e_tests.sh: - Seed monitoring-indices with 30 days × 3 deployments × 3 datastreams × 3 tiers (810 docs) instead of 3 single docs; vary metrics by tier/datastream realistically - Seed billing index (step 8d) with 30 days × 3 deployments (90 docs) via bulk API, each doc has event.ingested set (required for billing_cluster_cost transform sync); removes jq dependency for this step - Fix race condition in step 8g: stop transform with wait_for_completion, sleep 2s between stop/reset/start, call _schedule_now, bump SLEEP_BILLING_AFTER_PATCH to 30s, verify lookup doc count after transform runs - Add stack version check (fails fast if running stack != STACK_VERSION=9.2.2) - Prefer installed elastic-package binary to avoid go run version warnings - Named sleep constants (SLEEP_CONFIG_BOOT, SLEEP_TRANSFORMS, etc.) for easy tuning - Cross-verify against total_chargeable_units (Chargeback 0.3.0 field rename from total_ecu) cleanup_testing_env.sh: - Delete On-Prem Billing package policies before package removal (required by Fleet API) - Use ?force=true on package DELETE; prefer installed elastic-package binary Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…erences - Add integration/assets/0.3.1/chargeback-0.3.1.zip (built from integrations PR elastic/integrations#18102) - Update Instructions.md asset link to chargeback-0.3.1.zip - Update README.md version header, compatibility table, and add 0.3.1 release notes - Bump CHARGEBACK_VERSION default to 0.3.1 in run_e2e_tests.sh and cleanup_testing_env.sh Key changes in 0.3.1 vs 0.3.0: deployment_tags string/array fix in billing transform, explicit lookup mappings removed from manifest, pipeline versions bumped to 0.3.1-billing / 0.3.1-usage, dashboard @timestamp exists filter added. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Collaborator
Author
|
Hey @sholzhauer-es — requesting your review here. This PR ships the Chargeback 0.3.1 zip (built from the companion integrations#18102), updates the E2E test scripts, and fixes the On-Prem Billing ingest pipeline to set |
Collaborator
Author
|
See elastic/integrations#18102 (comment) as evidence of testing |
…peline Rebuilt from integrations wip-johannes-chargeback-chargable-units after fixing the billing.yml ingest pipeline regex (& -> .*) so snapshot-storage SKUs are correctly classified as cost_type 'snapshots'. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sholzhauer-es
requested changes
Mar 30, 2026
Collaborator
sholzhauer-es
left a comment
There was a problem hiding this comment.
Clarify the "upgrade documentation" as in-place upgrade 2.10 -> failed
Built with elastic-package from integrations branch wip-johannes-chargeback-chargable-units @ 3a87a3c98b (PR elastic/integrations#18102). Includes dashboard ES|QL COALESCE for 0.3.0 upgrade compatibility, README sync via _dev/build/docs, format_version 3.5.0, and removal of validation.yml SVR00002 override. Refreshes asset for #89. Made-with: Cursor
Built with elastic-package from branch wip-johannes-chargeback-chargable-units @ 24c5f464e9 (elastic/integrations#18102). Includes onprem_billing changelog link lint fix for CI; chargeback package unchanged in zip content vs prior rebuild if only sibling package differed— rebuilt from current integrations tree for PR #89. Made-with: Cursor
This was referenced Apr 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces Chargeback 0.3.1 with a breaking field rename (ECU → chargeable units), ships comprehensive E2E testing scripts, and fixes On-Prem Billing pipeline/transform sync.
Breaking change: ECU → chargeable units
total_ecu→total_chargeable_unitsconf_ecu_rate→conf_chargeable_unit_rateconf_ecu_rate_unit→conf_chargeable_unit_rate_unitReflects that the unit is deployment/ERU-based, not strictly ECU.
On-Prem Billing pipeline fix (
scripts/onprem_billing_calculate_cost_pipeline.json)setprocessor forevent.ingested(=_ingest.timestamp)billing_cluster_costtransform sync (sync.time.field: event.ingested) to detect newly ingested docs — without it the lookup index stays emptyE2E test scripts (
scripts/run_e2e_tests.sh,scripts/cleanup_testing_env.sh)STACK_VERSION=9.2.2)elastic-packagebinary overgo runmonitoring-indiceswith 30 days × 3 deployments × 3 datastreams × 3 tiers (810 docs) with realistic per-tier metricsevent.ingestedset — eliminates jq dependency for this step_stop?wait_for_completion=true, explicit gaps between stop/reset/start,_schedule_now,SLEEP_BILLING_AFTER_PATCH=30s, post-run doc count verificationtotal_chargeable_units(0.3.1 field rename)?force=trueon Fleet DELETEChargeback integration zip (
integration/assets/0.3.1/chargeback-0.3.1.zip)deployment_tagsstring/array fix in billing transform0.3.1-billing/0.3.1-usage@timestampexists filter addedRelated PRs (must merge together)
elastic/integrationsTesting
Tested end-to-end with
./scripts/run_e2e_tests.shagainst Elasticsearch/Kibana 9.2.2:event.ingested✓billing_cluster_cost_lookuppopulated (dev/prod/monitoring) ✓deployment_group✓