Skip to content

feat(kinesisfirehose): support HTTP endpoint destination#35969

Open
Tietew wants to merge 16 commits intoaws:mainfrom
Tietew:firehose-http-destination
Open

feat(kinesisfirehose): support HTTP endpoint destination#35969
Tietew wants to merge 16 commits intoaws:mainfrom
Tietew:firehose-http-destination

Conversation

@Tietew
Copy link
Copy Markdown
Contributor

@Tietew Tietew commented Nov 6, 2025 •
edited
Loading

Issue # (if applicable)

Closes #15502.
Closes #33585.

Reason for this change

Amazon Data Firehose delivery stream can deliver records to a custom HTTP endpoint destination.
Some destinations, e.g. Colalogix, Datadog, New Relic, etc., are based on HTTP endpoint destination.

See also:

Description of changes

Added the HttpEndpoint destination class.

Usage:

const httpDestination = new firehose.HttpEndpoint({
  // Endpoint configuration
  url: 'https://example.com/',
  name: 'MyEndpointName', // The endpoint name - optional
  // Authentication - optional
  authentication: firehose.HttpEndpointAuthentication.accessKey('my-access-key'), // Access key
  authentication: firehose.HttpEndpointAuthentication.secretsManager({ secret }), // Secret from AWS Secrets Manager
  // Request configuration - optional
  parameters: {
    'deployment-context': 'pre-prod-gamma',
  },
  contentEncoding: firehose.ContentEncoding.GZIP,
  retryDuration: Duration.minutes(60),
  bufferingInterval: Duration.seconds(300),
  bufferingSize: Size.mebibytes(5),
  // Data Processor - optional
  processors: [lambdaProcessor], // In management console, only one lambda processor is allowed
  // Logging configuration - optional
  loggingConfig: new firehose.LoggingEnabled(),
  // S3 backup configuration - optional
  s3Backup: {
    mode: firehose.BackupMode.FAILED, // default - backup failed records
    mode: firehose.BackupMode.ALL, // backup all records
    bucket: backupBucket, // S3 backup bucket is automatically created by default
  },
});

Describe any new or updated permissions being added

The HTTP endpoint destination will grant following accesses to the destination role:

  • Read access to the secret if present using secret.grantRead()
  • Read/Write access to the backup bucket using bucket.grantReadWrite()
  • Write access to the log group using logGroup.grantWrite()

Description of how you validated changes

Added unit tests and an Integ test.
The integ test also asserts the http endpoint (with access key) is invoked correctly.

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@aws-cdk-automation aws-cdk-automation requested a review from a team November 6, 2025 08:35
@github-actions github-actions bot added effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2 distinguished-contributor [Pilot] contributed 50+ PRs to the CDK labels Nov 6, 2025
@aws-cdk-automation aws-cdk-automation added pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. pr/needs-further-review PR requires additional review from our team specialists due to the scope or complexity of changes. labels Nov 6, 2025
@github-actions github-actions bot mentioned this pull request Dec 1, 2025
Open
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Dec 8, 2025 •
edited
Loading

⚠️ Experimental Feature: This security report is currently in experimental phase. Results may include false positives and the rules are being actively refined.
This security report is NOT a review blocker. Please try merge from main to avoid findings unrelated to the PR.

TestsPassed ✅SkippedFailed
Security Guardian Results48 ran48 passed
TestResult
No test annotations available

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Dec 8, 2025 •
edited
Loading

⚠️ Experimental Feature: This security report is currently in experimental phase. Results may include false positives and the rules are being actively refined.
This security report is NOT a review blocker. Please try merge from main to avoid findings unrelated to the PR.

TestsPassed ✅SkippedFailed
Security Guardian Results with resolved templates48 ran48 passed
TestResult
No test annotations available

@pahud pahud mentioned this pull request Feb 2, 2026
Open
badmintoncryer
badmintoncryer suggested changes Mar 25, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@badmintoncryer badmintoncryer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution. I've added some minor comments.

packages/aws-cdk-lib/aws-kinesisfirehose/lib/http-endpoint.ts Outdated
abstract bind(scope: Construct): HttpEndpointAuthenticationOptions;
}

class HttpEndpointAccesKeyAuthentication extends HttpEndpointAuthentication {
Copy link
Copy Markdown
Contributor

@badmintoncryer badmintoncryer Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: typo

Suggested change
class HttpEndpointAccesKeyAuthentication extends HttpEndpointAuthentication {
class HttpEndpointAccessKeyAuthentication extends HttpEndpointAuthentication {

packages/aws-cdk-lib/aws-kinesisfirehose/lib/http-endpoint.ts Outdated
* @see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding
*/
export enum ContentEncoding {
/** No content-encoing */
Copy link
Copy Markdown
Contributor

@badmintoncryer badmintoncryer Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: typo

Suggested change
/** No content-encoing */
/** No content-encoding */

packages/aws-cdk-lib/aws-kinesisfirehose/lib/http-endpoint.ts Outdated
}) ?? {};

if (this.props.retryDuration && !this.props.retryDuration.isUnresolved() && this.props.retryDuration.toSeconds() > 7200) {
throw new cdk.ValidationError(`Retry duration must be less than or equal to 7200 seconds, got ${this.props.retryDuration.toSeconds()}.`, scope);
Copy link
Copy Markdown
Contributor

@badmintoncryer badmintoncryer Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: It may be slightly better to show a unit.

Suggested change
throw new cdk.ValidationError(`Retry duration must be less than or equal to 7200 seconds, got ${this.props.retryDuration.toSeconds()}.`, scope);
throw new cdk.ValidationError(`Retry duration must be less than or equal to 7200 seconds, got ${this.props.retryDuration.toSeconds()} seconds.`, scope);

packages/aws-cdk-lib/aws-kinesisfirehose/lib/http-endpoint.ts
/**
* The URL of the HTTP endpoint selected as the destination.
*/
readonly url: string;
Copy link
Copy Markdown
Contributor

@badmintoncryer badmintoncryer Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please add a validation to check the url prop begins with https://?

https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html#create-destination-http

packages/aws-cdk-lib/aws-kinesisfirehose/lib/http-endpoint.ts Outdated

bind(scope: Construct): HttpEndpointAuthenticationOptions {
if (!cdk.Token.isUnresolved(this.accessKey) && Buffer.from(this.accessKey).byteLength > 4096) {
throw new cdk.ValidationError('The maximum length of the access key is 4096 bytes.', scope);
Copy link
Copy Markdown
Contributor

@badmintoncryer badmintoncryer Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Error class now needs a error code.

#36934

packages/@aws-cdk-testing/framework-integ/test/aws-kinesisfirehose/test/integ.http-endpoint.ts
Comment on lines +13 to +15
postCliContext: {
'@aws-cdk/aws-lambda:useCdkManagedLogGroup': false,
},
Copy link
Copy Markdown
Contributor

@badmintoncryer badmintoncryer Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this feature flag condition needed?

packages/aws-cdk-lib/aws-kinesisfirehose/lib/common.ts
*
* @default - true if `secret` is specified, false otherwise
*/
readonly enabled?: boolean;
Copy link
Copy Markdown
Contributor

@badmintoncryer badmintoncryer Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we actually need this parameter? One alternative worth considering would be to make secret a required field, and derive the enabled state from it, treating a provided secret as enabled = true. If there are any concrete use cases where explicitly setting enabled = false is necessary, it would be helpful to understand what those look like.

packages/aws-cdk-lib/aws-kinesisfirehose/lib/common.ts Outdated
*
* @default - the destination specific role will be used
*/
readonly role?: iam.IRole;
Copy link
Copy Markdown
Contributor

@badmintoncryer badmintoncryer Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please confirm to use the reference interface instead of L2 interface?
This applies equally to all other arguments that accept an Interface.

https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md#consuming-construct-interfaces-what-interface-type-to-choose

Suggested change
readonly role?: iam.IRole;
readonly role?: iam.IRoleRef;

@aws-cdk-automation aws-cdk-automation removed the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Mar 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@badmintoncryer badmintoncryer badmintoncryer requested changes

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

distinguished-contributor [Pilot] contributed 50+ PRs to the CDK effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2 pr/needs-further-review PR requires additional review from our team specialists due to the scope or complexity of changes.

Projects

None yet

Milestone

No milestone

3 participants

@Tietew @badmintoncryer @aws-cdk-automation