feat(kinesisfirehose): support dynamic partitioning by Tietew · Pull Request #35903 · aws/aws-cdk

Tietew · 2025-10-31T04:00:45Z

Issue # (if applicable)

Reason for this change

Enables dynamic partitioning with inline parsing or Lambda function.
https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning.html

Description of changes

Added dynamicPartitioning prop to configure dynamic partitioning
Added following data processors:
- RecordDeAggregationProcessor - multi record deaggregation
- MetadataExtractionProcessor - configure inline parsing

Usage

const s3Destination = new firehose.S3Bucket(bucket, {
  // Enables dynamic partitioning
  dynamicPartitioning: { enabled: true },
  processors: [
    // Configures multi record deaggregation
    firehose.RecordDeAggregationProcessor.json(),
    // Configures inline parsing with JQ-1.6
    firehose.MetadataExtractionProcessor.jq16({
      customer_id: '.customer_id',
      device: '.type.device',
      year: '.event_timestamp|strftime("%Y")',
    }),
    // Partition keys can also be constructed by lambda function processor
    new firehose.LambdaFunctionProcessor(lambdaFunction),
  ],
  // Specify partition keys in the S3 output prefix
  dataOutputPrefix: '!{partitionKeyFromQuery:year}/!{partitionKeyFromQuery:device}/!{partitionKeyFromQuery:customer_id}/!{partitionKeyFromLambda:lambda_key}/',
  // Required due to expressions in dataOutputPrefix
  errorOutputPrefix: '!{firehose:error-output-type}/!{timestamp:yyyy/MM}/',
});

Describe any new or updated permissions being added

N/A

Description of how you validated changes

Unit tests and integ tests.
The integ tests also assert that records are correctly partitioned by JQ-1.6 or lambda function.

Checklist

My code adheres to the CONTRIBUTING GUIDE and DESIGN GUIDELINES

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

github-actions · 2025-12-19T09:57:05Z

⚠️ Experimental Feature: This security report is currently in experimental phase. Results may include false positives and the rules are being actively refined.
Please try merge from main to avoid findings unrelated to the PR.

	Tests	Passed ✅	Skipped	Failed
Security Guardian Results	96 ran	96 passed

Test	Result
No test annotations available

View Security Guardian Results

github-actions · 2025-12-19T09:57:08Z

⚠️ Experimental Feature: This security report is currently in experimental phase. Results may include false positives and the rules are being actively refined.
Please try merge from main to avoid findings unrelated to the PR.

	Tests	Passed ✅	Skipped	Failed
Security Guardian Results with resolved templates	96 ran	96 passed

Test	Result
No test annotations available

View Security Guardian Results with resolved templates

kumvprat

@Tietew Thank you for the contribution and extensive testing of this(based on you finding undocumented defaults)

I have added a few inline comments.

If you still see the same undocumented defaults being enforced do inform me and I can check if there is documentation update needed(or maybe these defaults are not being enforced anymore)

kumvprat · 2026-02-13T12:55:35Z

I will try to dissect this a bit : #35903 (comment)

Thanks a lot for digging deep to find the behaviour for all these edge cases.
I have added inline question where it was confusing and highlighted the items that are pretty clear to me.
I will verify if all the edge cases you mentioned are tested and have corresponding validation error in the PR.

My opinion : Just the formatting could be a bit better to put the message across

My questions :

Does the cloudformation section represent using sdk or cloudformation sdk or api for getting the information ?

Specify RecordDeAggregation processor without enabling dynamic partitioning

CloudFormation

class com.amazonaws.services.firehose.internal.model.RecordDeAggregationProcessor can only be pres
ent when Dynamic Partitioning is enabled.

Console

Cannot enable Inline parsing for JSON (the field is hidden)

The cloudformation seems to informing about deaggregation processor but console seems to be talking about something else ?

Specify MetadataExtraction processor without enabling dynamic partitioning

CloudFormation

class com.amazonaws.services.firehose.internal.model.MetadataExtractionProcessor can only be prese
nt when Dynamic Partitioning is enabled.

Console

Cannot enable Multi record deaggregation (the field is hidden)

Console part is about de-aggregation here ?

Dynamic partitioning is enabled without ProcessingConfiguration

CloudFormation

Processing Configuration is not enabled when DataPartitioning is enabled.

Console

You must enable data transformation and/or inline parsing when dynamic partitioning is enabled.
When dynamic partitioning is enabled, you must specify the S3 bucket prefix to which Amazon Data Firehose is to deliver partitioned data.

Is "DataPartitioning" = "Dynamic Partitioning" in cloudformation part ? And it seems like data transformation is a hard requirement when enabling dynamic partitioning ? But only from console ? Cloudformation response seems either for something else or unclear? (is this response observed when you use the cloudformation api directly ?)

Dynamic partitioning is enabled with neither "partitionKeyFromQuery" nor "partitionKeyFromLambda"

CloudFormation

S3 Prefix should contain Dynamic Partitioning namespaces when Dynamic Partitioning is enabled

Console

When dynamic partitioning is enabled, you must specify the S3 bucket prefix to which Data Firehose is to deliver partitioned data. With inline parsing, in order for dynamic partitioning to be configured correctly, the specified S3 bucket prefix must include all of the specified partitioning keys. You must specify them using the following format: "!{partitionKeyFromQuery:keyID}".

This seems pretty clear. We need one of the 2 expression when enabling dynamic partitioning.

Prefix has "partitionKeyFromQuery" without MetadataExpression processor

CloudFormation

MetadataExtraction processor should be present when S3 Prefix has partitionKeyFromQuery namespace.

Console

Inline parsing is not enabled, therefore, the prefix can't include !{partitionKeyFromQuery:keyID}.

Again pretty clear from cloudformation response that MetadataExtraction processor is needed with dynamic partition

Prefix has "partitionKeyFromQuery" without ErrorOutputPrefix

CloudFormation

The supplied prefix(es) do not satisfy the following constraint: ErrorOutputPrefix cannot be null
or empty when Prefix contains expressions

Console

Specify the S3 bucket error output prefix in order to use expressions in the S3 bucket prefix.

Again pretty clear from cloudformation response that error prefix is needed when the output prefix is defined is needed with dynamic partition

JQ query has a key where Prefix does not contain matched "partitionKeyFromQuery"

CloudFormation

MetaDataExtraction JQ Query can't contain keys that are not present in the S3 Prefix expression

Console

Once dynamic partitioning is enabled, you must use all specified dynamic partitioning key values for partitioning your data source.

Again pretty clear from cloudformation response that jq query keys should be similar to ones defined in S3 prefix with dynamic partition

Prefix has a "partitionKeyFromQuery" key where JQ query does not contain

CloudFormation

No errors. But every record will emit metadata-extraction-failed.

Console

When dynamic partitioning via inline parsing is enabled, you must specify the S3 bucket prefix using only the specified keys, matching them identically. You cannot use not specified partitioning keys in your S3 bucket prefix. You must specify them using the following format: "!{partitionKeyFromQuery:keyID}".

Makes sense to have this validation in cdk synth to avoid deployment for some known failures that can be caught earlier.

Prefix has "partitionKeyFromLambda" without Lambda processor

CloudFormation

Lambda should be present when S3 Prefix contains keys from partitionKeyFromLambda namespace.

Console

Dynamic partitioning with AWS Lambda is not enabled, therefore, the prefix can't include !{partitionKeyFromLambda:keyID}.

Again pretty clear from cloudformation response that lambda processor is needed when the output prefix has partitionKeyFromLambda with dynamic partition

kumvprat

Added a couple of comments for the new changes

kumvprat · 2026-02-13T13:23:49Z

 * @see https://docs.aws.amazon.com/firehose/latest/dev/s3-prefixes.html#prefix-rules
 */
 function validateOutputPrefix(prefix?: string, errorOutputPrefix?: string) {
+  const ERROR_OUTPUT_TYPE = '!{firehose:error-output-type}';


Should we define it outside this method so that users can use this when generating their prefix strings ?

kumvprat · 2026-02-13T13:24:42Z

 import type { DataProcessorBindOptions, IDataProcessor } from '../processor';
 import type { DynamicPartitioningProps } from '../s3-bucket';

+export const PARTKEY_QUERY = 'partitionKeyFromQuery';


Any reason to not go for a verbose name like PARTITION_KEY_QUERY and PARTITION_KEY_LAMBDA for variable names ?

Tietew · 2026-02-16T12:02:57Z

Does the cloudformation section represent using sdk or cloudformation sdk or api for getting the information ?

I tried a simple CDK app which defines a firehose.DeliveryStream with an escape hatch on ExtendedS3DestinationConfiguration property.
The deployment failed (CREATE_FAILED status) with each error message.

The cloudformation seems to informing about deaggregation processor but console seems to be talking about something else ? Console part is about de-aggregation here ?

Sorry, my bad. I'll correct it.

Is "DataPartitioning" = "Dynamic Partitioning" in cloudformation part ?

Seems to be yes.

And it seems like data transformation is a hard requirement when enabling dynamic partitioning ?

"Processing Configuration is not enabled" means ProcessingConfiguration: { Enabled: true } resource property is not specified. Data transformation (Transform source records with AWS Lambda) is just one of processors.

ProcessingConfiguration: {
  Enabled: true,
  Processors: [
    { Type: 'Lambda', Parameters: [...] },             // Data transformation
    { Type: 'MetadataExtraction', Parameters: [...] }, // Inline parsing
  ]
}

But only from console ? Cloudformation response seems either for something else or unclear? (is this response observed when you use the cloudformation api directly ?)

In console, the error message will be shown when:

Transform source records with AWS Lambda - Unchecked "Turn on data transformation"
Dynamic partitioning - "Enabled"
Inline parsing for JSON - "Not enabled"

kumvprat

Thanks for all the work here @Tietew

mergify · 2026-02-16T13:39:31Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

mergify · 2026-02-16T13:39:35Z

Merge Queue Status

Rule: default-squash

✅ Entered queue — 2026-02-16 13:39 UTC
🚫 Left the queue — 2026-02-16 13:39 UTC · at 92a7feeaa589437db569625967ca69d30904e1ce

This pull request spent 4 seconds in the queue, with no time running CI.

Reason

The pull request can't be updated

For security reasons, Mergify can't update this pull request. Try updating locally.
GitHub response: refusing to allow a GitHub App to create or update workflow .github/workflows/integration-test-deployment.yml without workflows permission

Hint

You should update or rebase your pull request manually. If you do, this pull request will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue, you can requeue the pull request, without updating it, by posting a @mergifyio requeue comment.

Pull request has been modified.

kumvprat · 2026-02-16T16:06:55Z

@Mergifyio queue

mergify · 2026-02-16T16:07:06Z

Merge Queue Status

🛑 Queue command has been cancelled

mergify · 2026-02-16T16:09:20Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

kumvprat · 2026-02-16T16:09:55Z

@Mergifyio requeue

mergify · 2026-02-16T16:10:00Z

requeue

☑️ This pull request is already queued

mergify · 2026-02-16T16:13:33Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

mergify · 2026-02-16T17:02:35Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

mergify · 2026-02-16T17:45:49Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

mergify · 2026-02-16T17:45:58Z

github-actions · 2026-02-16T17:46:04Z

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

dynamic partitioning

1a19fad

github-actions Bot added effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2 labels Oct 31, 2025

aws-cdk-automation requested a review from a team October 31, 2025 04:00

github-actions Bot added the distinguished-contributor [Pilot] contributed 50+ PRs to the CDK label Oct 31, 2025

rosetta fix

a892113

Tietew force-pushed the firehose-dynamic-partitioning branch from 362de35 to a892113 Compare October 31, 2025 07:27

github-actions Bot mentioned this pull request Nov 1, 2025

Monthly PR metrics report #35917

Open

update README

b2e5a8c

Tietew changed the title ~~feat(kinesisfirehose): support dynamic partitioning~~ feat(kinesisfirehose): support dynamic partitioning Nov 5, 2025

aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Nov 5, 2025

kumvprat self-assigned this Dec 19, 2025

Merge branch 'main' into firehose-dynamic-partitioning

b395317

kumvprat reviewed Jan 20, 2026

View reviewed changes

kumvprat added the pr/needs-integration-tests-deployment Requires the PR to deploy the integration test snapshots. label Jan 20, 2026

kumvprat had a problem deploying to deployment-integ-test January 20, 2026 21:07 — with GitHub Actions Failure

Merge branch 'main' into firehose-dynamic-partitioning

8c68ac4

kumvprat temporarily deployed to deployment-integ-test January 20, 2026 21:16 — with GitHub Actions Inactive

Tietew added 6 commits January 22, 2026 19:10

update doc

f88fde2

withResolved -> isUnresolved

e35b833

validate the query object

fbdccae

assertion for errorOutputPrefix

8103fc0

integ test for dynamic partitioning with lambda processor

ec60f85

Merge branch 'main' into firehose-dynamic-partitioning

73f956a

Tietew had a problem deploying to deployment-integ-test January 26, 2026 09:49 — with GitHub Actions Error

Tietew added 2 commits January 29, 2026 19:10

validation on dataOutputPrefix & errorOutputPrefix

af990b2

Merge branch 'main' into firehose-dynamic-partitioning

ea47707

kumvprat reviewed Feb 13, 2026

View reviewed changes

update constants

92a7fee

Tietew temporarily deployed to deployment-integ-test February 16, 2026 10:48 — with GitHub Actions Inactive

kumvprat previously approved these changes Feb 16, 2026

View reviewed changes

kumvprat removed the pr/needs-integration-tests-deployment Requires the PR to deploy the integration test snapshots. label Feb 16, 2026

aws-cdk-automation removed the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Feb 16, 2026

Merge branch 'main' into firehose-dynamic-partitioning

29f0345

aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Feb 16, 2026

kumvprat approved these changes Feb 16, 2026

View reviewed changes

aws-cdk-automation removed the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Feb 16, 2026

Merge branch 'main' into firehose-dynamic-partitioning

849e284

Merge branch 'main' into firehose-dynamic-partitioning

294e3df

mergify Bot merged commit 4f86a05 into aws:main Feb 16, 2026
23 checks passed

github-actions Bot locked as resolved and limited conversation to collaborators Feb 16, 2026

Tietew deleted the firehose-dynamic-partitioning branch February 17, 2026 02:24

Conversation

Tietew commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue # (if applicable)

Reason for this change

Description of changes

Usage

Describe any new or updated permissions being added

Description of how you validated changes

Checklist

Uh oh!

github-actions Bot commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kumvprat left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kumvprat commented Feb 13, 2026

Specify RecordDeAggregation processor without enabling dynamic partitioning

CloudFormation

Console

Specify MetadataExtraction processor without enabling dynamic partitioning

CloudFormation

Console

Dynamic partitioning is enabled without ProcessingConfiguration

CloudFormation

Console

Dynamic partitioning is enabled with neither "partitionKeyFromQuery" nor "partitionKeyFromLambda"

CloudFormation

Console

Prefix has "partitionKeyFromQuery" without MetadataExpression processor

CloudFormation

Console

Prefix has "partitionKeyFromQuery" without ErrorOutputPrefix

CloudFormation

Console

JQ query has a key where Prefix does not contain matched "partitionKeyFromQuery"

CloudFormation

Console

Prefix has a "partitionKeyFromQuery" key where JQ query does not contain

CloudFormation

Console

Prefix has "partitionKeyFromLambda" without Lambda processor

CloudFormation

Console

Uh oh!

kumvprat left a comment

Choose a reason for hiding this comment

Uh oh!

kumvprat Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

kumvprat Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

Tietew commented Feb 16, 2026

Does the cloudformation section represent using sdk or cloudformation sdk or api for getting the information ?

The cloudformation seems to informing about deaggregation processor but console seems to be talking about something else ? Console part is about de-aggregation here ?

Is "DataPartitioning" = "Dynamic Partitioning" in cloudformation part ?

And it seems like data transformation is a hard requirement when enabling dynamic partitioning ?

But only from console ? Cloudformation response seems either for something else or unclear? (is this response observed when you use the cloudformation api directly ?)

Uh oh!

kumvprat left a comment

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Feb 16, 2026

Uh oh!

Tietew commented Oct 31, 2025 •

edited

Loading

github-actions Bot commented Dec 19, 2025 •

edited

Loading

github-actions Bot commented Dec 19, 2025 •

edited

Loading

mergify Bot commented Feb 16, 2026 •

edited

Loading