Billing integration add tags by 3kt · Pull Request #15950 · elastic/integrations

3kt · 2025-11-12T13:56:38Z

Proposed commit message

This adds a toggle button on the billing integration allowing the collection of deployment tags, which are then added to each billing object related to the corresponding deployment. In case the toggle isn't activated, the old behavior is kept identical

Considerations

There are some caveats:

As the commercial cloud URL is hardcoded, this is only functional for commercial cloud and not gov cloud. This also prevents the override to admin.
The tags are stored as a list of key:value strings, rather than nested objects to prevent mapping explosion.
This doesn't consider historical tags. In other words, the deployment receive the tags that they have at the moment of the data pull, rather than at the time the billing occurred.
This doesn't work for serverless projects.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

Validate that the caveats are accepted
-> moved to configurable parameter for deployment search
Evaluate whether we want to extend dashboards
Evaluate whether we want to make the tag pull behavior default, and remove the control of the user over the setting altogether.
-> Kept default behavior

How to test this PR locally

Spin a local environment with elastic-package stack up -v -d
Set some relevant tags on the deployment of your choice
Collect the organization ID and billing API key for your organization.
Deploy the integration while activating the Add deployment tags toggle.

Related issues

None

Screenshots

… deployment.id

elastic-vault-github-plugin-prod · 2025-11-12T14:16:33Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

asazallesmilner · 2025-11-21T18:18:20Z

We definitely have people using this for gov cloud. Can we expose that as a configurable parameter (separate from the main base URL if needed ) ?

JohannesMahne · 2025-11-24T14:09:40Z

The functionality works on ECH deployments, but, for some reason we cannot add tags on Serverless projects:

Something went wrong

patch request validation failed: field is not allowed in the current configuration: project tagging is not enabled

asazallesmilner · 2025-11-24T15:18:16Z

I tried patching the code to be configurable, but doesn't seem to work with govcloud still.

packages/ess_billing/data_stream/billing/agent/stream/cel.yml.hbs

JohannesMahne

I tested the positive flows, and the tags are coming through nicely.
Please just check the 2 questions I had on the edge cases.

packages/ess_billing/data_stream/billing/agent/stream/cel.yml.hbs

botelastic · 2026-01-02T05:48:43Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

elasticmachine · 2026-01-07T21:39:11Z

💚 Build Succeeded

Buildkite Build
Commit: 5d16bab

History

💚 Build #34627 succeeded ea5159a
💚 Build #34612 succeeded d93e9b1
💚 Build #34038 succeeded 58992e8
💚 Build #34037 succeeded 53919f8

JohannesMahne

All good then.

JohannesMahne · 2026-01-15T14:45:46Z

@asazallesmilner - would you mind reviewing this PR again?

agithomas · 2026-01-16T06:52:36Z

packages/ess_billing/data_stream/billing/fields/fields.yml

+    - name: deployment_tags
+      type: keyword
+      description: Tags associated with the Elasticsearch Service deployment.


Suggested change

- name: deployment_tags

type: keyword

description: Tags associated with the Elasticsearch Service deployment.

- name: deployment_tags

type: keyword

description: >-

Tags associated with the Elasticsearch Service deployment, stored as

"key:value" strings. Reflects current tags at collection time.

agithomas

Left a minor suggestion. The change looks good to me.

@gpop63 , kindly check if this can be approved from your end as well.

agithomas · 2026-01-16T07:37:27Z

packages/ess_billing/data_stream/billing/agent/stream/cel.yml.hbs

+      bytes(deployments_resp.Body).decode_json().as(deployments_body,
+        zip(
+          deployments_body.deployments.map(d,
+            d.resources.elasticsearch[0].id


Could you double-check if this must be replaced as follows

zip( deployments_body.deployments.map(d, d.id), // ← Use deployment ID deployments_body.deployments.map(d, d.metadata.collate("tags")) )

asazallesmilner

LGTM. Work for both cloud.elastic.co and console.us-gov-east-1.aws.elastic-cloud.com.
Should we add docs for the urls to use for govcloud?
Billing endpoint: https://billing.us-gov-east-1.aws.elastic-cloud.com and Tags endpoint: https://console.us-gov-east-1.aws.elastic-cloud.com/api/v1/deployments/_search

elastic-vault-github-plugin-prod · 2026-01-27T07:45:13Z

Package ess_billing - 1.7.0 containing this change is available at https://epr.elastic.co/package/ess_billing/1.7.0/

* Working tag pull * Fixed deployment identification: elasticsearch_id is used rather than deployment.id * Added changelog entry * Added dep_search_url parameter to control tag fetch

3kt added 2 commits November 12, 2025 11:47

Working tag pull

5ad485c

Fixed deployment identification: elasticsearch_id is used rather than…

53919f8

… deployment.id

3kt added enhancement New feature or request Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] Integration:ess_billing Elasticsearch Service Billing labels Nov 12, 2025

Added changelog entry

58992e8

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Nov 12, 2025

3kt added 2 commits November 25, 2025 09:41

Merge branch 'main' into billing_integration_add_tags

d93e9b1

Added dep_search_url parameter to control tag fetch

ea5159a

3kt marked this pull request as ready for review November 25, 2025 14:27

3kt requested review from a team as code owners November 25, 2025 14:27

3kt requested a review from asazallesmilner November 25, 2025 14:27

gpop63 reviewed Nov 26, 2025

View reviewed changes

packages/ess_billing/data_stream/billing/agent/stream/cel.yml.hbs Show resolved Hide resolved

JohannesMahne requested changes Dec 3, 2025

View reviewed changes

packages/ess_billing/data_stream/billing/agent/stream/cel.yml.hbs Show resolved Hide resolved

packages/ess_billing/data_stream/billing/agent/stream/cel.yml.hbs Show resolved Hide resolved

packages/ess_billing/data_stream/billing/agent/stream/cel.yml.hbs Show resolved Hide resolved

botelastic bot added Stalled and removed Stalled labels Jan 2, 2026

Merge branch 'main' into billing_integration_add_tags

5d16bab

JohannesMahne approved these changes Jan 12, 2026

View reviewed changes

agithomas reviewed Jan 16, 2026

View reviewed changes

agithomas approved these changes Jan 16, 2026

View reviewed changes

agithomas reviewed Jan 16, 2026

View reviewed changes

gpop63 approved these changes Jan 16, 2026

View reviewed changes

asazallesmilner approved these changes Jan 23, 2026

View reviewed changes

JohannesMahne mentioned this pull request Jan 26, 2026

Release 0.2.10 - Automated config lookup and visualization fixes elastic/elasticsearch-chargeback#83

Merged

3 tasks

JohannesMahne merged commit df5b77c into elastic:main Jan 27, 2026
8 checks passed

Conversation

3kt commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Considerations

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elastic-vault-github-plugin-prod bot commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Uh oh!

asazallesmilner commented Nov 21, 2025

Uh oh!

JohannesMahne commented Nov 24, 2025

Uh oh!

asazallesmilner commented Nov 24, 2025

Uh oh!

Uh oh!

JohannesMahne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

botelastic bot commented Jan 2, 2026

Uh oh!

elasticmachine commented Jan 7, 2026

💚 Build Succeeded

History

Uh oh!

JohannesMahne left a comment

Choose a reason for hiding this comment

Uh oh!

JohannesMahne commented Jan 15, 2026

Uh oh!

agithomas Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

agithomas left a comment

Choose a reason for hiding this comment

Uh oh!

agithomas Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

asazallesmilner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Jan 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

3kt commented Nov 12, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented Nov 12, 2025 •

edited

Loading