Skip to content

feat(kinesisfirehose-destinations): add support for prefixes in the S3 destination#15552

Merged
mergify[bot] merged 121 commits intomasterfrom
madeline-k/feat/kinesisfirehose-prefixes
Jul 28, 2021
Merged

feat(kinesisfirehose-destinations): add support for prefixes in the S3 destination#15552
mergify[bot] merged 121 commits intomasterfrom
madeline-k/feat/kinesisfirehose-prefixes

Conversation

@madeline-k
Copy link
Copy Markdown
Contributor

Closes #15551


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@gitpod-io
Copy link
Copy Markdown

gitpod-io bot commented Jul 14, 2021

@mergify mergify bot added the contribution/core This is a PR that came from AWS. label Jul 14, 2021
@BenChaimberg BenChaimberg requested a review from skinny85 July 28, 2021 03:23
BenChaimberg
BenChaimberg previously approved these changes Jul 28, 2021
skinny85
skinny85 previously approved these changes Jul 28, 2021
@skinny85 skinny85 changed the title feat(aws-kinesisfirehose-destinations): support prefixes for S3 destination feat(kinesisfirehose-destinations): support prefixes for S3 destination Jul 28, 2021
Base automatically changed from madeline-k/feat/kinesisfirehose-compression to master July 28, 2021 21:58
@mergify mergify bot dismissed stale reviews from skinny85 and BenChaimberg via e0f7901 July 28, 2021 21:58
@madeline-k madeline-k removed the pr/do-not-merge This PR should not be merged at this time. label Jul 28, 2021
@skinny85 skinny85 changed the title feat(kinesisfirehose-destinations): support prefixes for S3 destination feat(kinesisfirehose-destinations): add support for prefixes in the S3 destination Jul 28, 2021
@aws-cdk-automation
Copy link
Copy Markdown
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
  • Commit ID: 929f9bb
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify mergify bot merged commit d227e48 into master Jul 28, 2021
@mergify mergify bot deleted the madeline-k/feat/kinesisfirehose-prefixes branch July 28, 2021 22:45
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Jul 28, 2021

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

TikiTDO pushed a commit to TikiTDO/aws-cdk that referenced this pull request Aug 3, 2021
…3 destination (aws#15552)

Closes aws#15551 

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
hollanddd pushed a commit to hollanddd/aws-cdk that referenced this pull request Aug 26, 2021
…3 destination (aws#15552)

Closes aws#15551 

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@jakejscott
Copy link
Copy Markdown

@madeline-k

I needed to add property overrides to the ExtendedS3DestinationConfiguration because the dataOutputPrefix and errorOutputPrefix props contained partitionKeyFromQuery fragments.

const s3Destination = new destinations.S3Bucket(s3DestinationBucket, {
  logging: true,
  logGroup: s3DestinationLogGroup,
  bufferingInterval: cdk.Duration.seconds(60),
  bufferingSize: cdk.Size.mebibytes(64),
  compression: undefined,
  dataOutputPrefix:
    "user/!{partitionKeyFromQuery:userId}/order/!{partitionKeyFromQuery:orderId}/!{timestamp:yyyy}/!{timestamp:MM}/!{timestamp:dd}/!{timestamp:HH}/",
  errorOutputPrefix:
    "failed/!{firehose:error-output-type}/!{timestamp:yyyy}/!{timestamp:MM}/!{timestamp:dd}/!{timestamp:HH}/",
});

const s3DeliveryStream = new kinesisfirehose.DeliveryStream(
  this,
  "S3DeliveryStream",
  {
    destinations: [s3Destination],
  }
);

s3DeliveryStreamEscapeHatch.addPropertyOverride(
  "ExtendedS3DestinationConfiguration.DynamicPartitioningConfiguration",
  {
    Enabled: true,
    RetryOptions: {
      DurationInSeconds: 300,
    },
  }
);

s3DeliveryStreamEscapeHatch.addPropertyOverride(
  "ExtendedS3DestinationConfiguration.ProcessingConfiguration",
  {
    Enabled: true,
    Processors: [
      {
        Type: "MetadataExtraction",
        Parameters: [
          {
            ParameterName: "MetadataExtractionQuery",
            ParameterValue: "{userId:.userId,orderId:.orderId}",
          },
          {
            ParameterName: "JsonParsingEngine",
            ParameterValue: "JQ-1.6",
          },
        ],
      },
    ],
  }
);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contribution/core This is a PR that came from AWS.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

(aws-kinesisfirehose): prefixes for delivery stream S3 destination

5 participants