Add environment variable to support backend storage tag enrichment#1006
Merged
ViBiOh merged 5 commits intoaws-forwarder-v5-preparationfrom Oct 16, 2025
Merged
Add environment variable to support backend storage tag enrichment#1006ViBiOh merged 5 commits intoaws-forwarder-v5-preparationfrom
ViBiOh merged 5 commits intoaws-forwarder-v5-preparationfrom
Conversation
aa79f3a to
f7fb1e1
Compare
3994c02 to
e7004da
Compare
Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
e7004da to
c9ec9ae
Compare
Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
aws/logs_monitoring/README.md
Outdated
| ### Advanced (optional) | ||
|
|
||
| `DD_ENRICH_S3_TAGS` | ||
| : Instruct Datadog backend to enrich a log coming from a S3 bucket with the tag attached to this bucket. It's the equivalent behavior of `DD_FETCH_S3_TAG` but done after ingestion. This require Resource Collection to be enabled. Flag is enabled by default. |
Member
There was a problem hiding this comment.
I'd rephrase with something like
True by default. Enables S3 tag enrichment at intake time for logs coming from S3 buckets. Equivalent to DD_FETCH_S3_TAG` once the logs are ingested in Datadog. This parameter requires Resource Collection to be enabled.
Contributor
Author
There was a problem hiding this comment.
"at intake time" is a bit ambiguous, before or after intake? because it impacts billing.
"S3 tag" is also ambiguous, "S3 Bucket tag" is precise.
Member
There was a problem hiding this comment.
I think I feel strongly about having a link to our documentation for resource collection.
For the other parts I don't mind keeping them as is, the key part that we want the reader to understand is that the tags will appear in the logs after they're ingested by Datadog, meaning they don't pay for the volume of the log the tag contribute to
Contributor
Author
There was a problem hiding this comment.
Revamped it with Claude
RaphaelAllier
approved these changes
Oct 16, 2025
|
|
||
| storage_tag = get_dd_storage_tag_header() | ||
| if storage_tag != "": | ||
| _HEADERS["DD-STORAGE-TAG"] = storage_tag |
Member
There was a problem hiding this comment.
maybe set the header key as a constant (now that I comment that I'd argue the other should be too)
Contributor
Author
There was a problem hiding this comment.
It's only used in one place. I don't think it worth it. We'll do some back and forth in code for something that is used at only one place.
… metrics Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
ViBiOh
added a commit
that referenced
this pull request
Oct 23, 2025
* refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only log message (#996) * refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only on log Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * fixup! refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only on log Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> --------- Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * feat(aws)!: Drop deprecated TCP support (#1000) Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * feat(aws)!: Remove deprecated PrivateLink env variable (#1002) Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * docs(aws): Removing JSON explanation in the include at match documentation (#1007) Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * Add environment variable to support backend storage tag enrichment (#1006) * feat(aws): AWSX-1566 Adding storage tag HTTP header Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * feat(aws): AWSX-1566 Adding new variable in the cloudformation template Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * feat: AWSX-1566 Restoring fetch in the integration test to see metrics Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * fixup! feat: AWSX-1566 Restoring fetch in the integration test to see metrics Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * docs(aws): AWSX-1566 Refine documentation around the new variable Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> --------- Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * fix(aws): Fixing discrepancies of the fetch configuration between cloudformation and code Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * feat: Add a changelog for the v5 (#1009) * feat: Add a changelog for the v5 Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * fixup! feat: Add a changelog for the v5 Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * Update aws/logs_monitoring/CHANGELOG.md Co-authored-by: Raphaël Allier <118757729+RaphaelAllier@users.noreply.github.com> * fixup! Update aws/logs_monitoring/CHANGELOG.md Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> --------- Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> Co-authored-by: Raphaël Allier <118757729+RaphaelAllier@users.noreply.github.com> * Update aws/logs_monitoring/CHANGELOG.md Co-authored-by: Georgi <georgi.ajaeiya@datadoghq.com> * Update aws/logs_monitoring/CHANGELOG.md Co-authored-by: Georgi <georgi.ajaeiya@datadoghq.com> * Update aws/logs_monitoring/CHANGELOG.md Co-authored-by: Georgi <georgi.ajaeiya@datadoghq.com> * Update aws/logs_monitoring/CHANGELOG.md Co-authored-by: Georgi <georgi.ajaeiya@datadoghq.com> * Update aws/logs_monitoring/CHANGELOG.md Co-authored-by: Georgi <georgi.ajaeiya@datadoghq.com> * docs: Updating changelog from comments Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * fixup! docs: Updating changelog from comments Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * Add reference to the changelog in README (for public doc) (#1011) * feat: Add link to the changelog in the read toe reflect the documentation The readme is feeding this page https://docs.datadoghq.com/logs/guide/forwarder/?tab=cloudformation Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * fixup! feat: Add link to the changelog in the read toe reflect the documentation Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> --------- Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * refactor: Improve error handling for exception (#1012) * refactor: Improve error handling for exception Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * fixup! refactor: Improve error handling for exception Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * fixup! refactor: Improve error handling for exception Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> --------- Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * Update aws/logs_monitoring/lambda_function.py Co-authored-by: Georgi <georgi.ajaeiya@datadoghq.com> * fixup! Update aws/logs_monitoring/lambda_function.py Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> --------- Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> Co-authored-by: Raphaël Allier <118757729+RaphaelAllier@users.noreply.github.com> Co-authored-by: Georgi <georgi.ajaeiya@datadoghq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This pull-request switch the tag enrichment done by the forwarder to logs-backend by default for both s3 and cloudwatch.
Motivation
Reduce ingestion payload size, s3 reduce cache burden and cost.
Testing Guidelines
Additional Notes
Types of changes
Check all that apply