feat(kinesisfirehose-destinations): add support for compression on S3 delivery stream destinations by madeline-k · Pull Request #15550 · aws/aws-cdk

madeline-k · 2021-07-14T21:05:53Z

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

…DeliveryStream

…eam destinations

gitpod-io · 2021-07-14T21:05:55Z

…ream

…roperty type and use local var instead

…eam (#15545) Closes #15543 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

otaviomacedo · 2021-07-16T12:52:12Z

packages/@aws-cdk/aws-kinesisfirehose/README.md


+## Compression
+
+Your data can automatically be compressed when it is delivered to S3 as either a final or


If the user specifies a backup option, does the compression apply to both the backup and the final destination? If so, the "either" in the documentation may be confusing.

The backup option will have it's own field for compression. They are not coupled to each other. This should be clear once I have the PR with the backup implementation ready. Will ping you with that, when it's ready.

Take a look at the backup PR, and let me know if you still find this unclear: https://github.com/aws/aws-cdk/pull/15725/files

…t.ts Use `toHaveResourceLike` instead of `toHaveResource` Co-authored-by: Ben Chaimberg <chaimber@amazon.com>

…t.ts Co-authored-by: Ben Chaimberg <chaimber@amazon.com>

skinny85

Looks good! 3 minor suggestions.

packages/@aws-cdk/aws-kinesisfirehose-destinations/lib/common.ts

…metrics' into madeline-k/feat/kinesisfirehose-sse

…at/kinesisfirehose-compression

… private, tests

…k/feat/kinesisfirehose-sse

…at/kinesisfirehose-compression

aws-cdk-automation · 2021-07-28T21:25:50Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
Commit ID: 9f9c7f8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

skinny85

Pure awesomeness.

skinny85 · 2021-07-28T21:57:21Z

packages/@aws-cdk/aws-kinesisfirehose-destinations/lib/s3-bucket.ts

+   * The compression formats SNAPPY or ZIP cannot be specified for Amazon Redshift
+   * destinations because they are not supported by the Amazon Redshift COPY operation
+   * that reads from the S3 bucket.


I fee like this comment is not needed, since this is the S3Destination...?

Good point, this comment will be needed once this prop is inevitably hoisted up to another interface so it can be shared with multiple destinations. But we can delete for now.

mergify · 2021-07-28T21:58:47Z

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

madeline-k · 2021-07-28T22:00:47Z

packages/@aws-cdk/aws-kinesisfirehose-destinations/lib/s3-bucket.ts

+   * The compression formats SNAPPY or ZIP cannot be specified for Amazon Redshift
+   * destinations because they are not supported by the Amazon Redshift COPY operation
+   * that reads from the S3 bucket.
+   *


Suggested change

* The compression formats SNAPPY or ZIP cannot be specified for Amazon Redshift

* destinations because they are not supported by the Amazon Redshift COPY operation

* that reads from the S3 bucket.

*

… delivery stream destinations (aws#15550) Closes aws#15548 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

madeline-k added 6 commits July 14, 2021 10:04

feat(aws-kinesisfirehose): DeliveryStream API and basic S3 destination

8401fb4

fix typo in aws-kinesisfirehose-destinations README

dca12f2

remove unnecessary deps

6fceebf

feat(aws-kinesisfirehose): specific metrics functions for DeliveryStream

58b3fe1

feat(aws-kinesisfirehose): Add support for server-side encryption on …

55cb8d6

…DeliveryStream

feat(aws-kinesisfirehose): support for compression on S3 delivery str…

cfc1427

…eam destinations

madeline-k added this to the [DevPreview] Kinesis Data Firehose milestone Jul 14, 2021

mergify bot added the contribution/core This is a PR that came from AWS. label Jul 14, 2021

madeline-k linked an issue Jul 14, 2021 that may be closed by this pull request

(aws-kinesisfirehose): support for data compression on DeliveryStream destinations #15548

Closed

2 tasks

madeline-k changed the title ~~feat(aws-kinesisfirehose): support for compression on S3 delivery stream destinations~~ feat(aws-kinesisfirehose-destinations): support for compression on S3 delivery stream destinations Jul 14, 2021

madeline-k and others added 17 commits July 14, 2021 14:58

Merge branch 'master' into madeline-k/feat/kinesisfirehose-deliveryst…

7892f58

…ream

remove awslint exception for IConnectable.connections

78e6a9c

Change name of S3 destination to S3Bucket

5f7a15c

add Role to DestinationBindOptions, update S3 to S3Bucket in tests

efe2fe0

remove helper function to enforce ExtendedS3DestinationConfigurationP…

4308f44

…roperty type and use local var instead

remove lambda dependency

172ab43

update README

c0d85f5

remove tagsand ITaggable interface

3d34c46

use splitArn instead of parseArn

5ceff5c

update error message for bad input ARN

ce4dde1

add ArnFormat to formatArn() call

0ed658f

update error message in tests for bad input ARN

1925560

pass deliveryStreamName as physicalName through to super constructor

e46c595

move validation to beginning of constructor

d1463a2

simplify createLoggingOptions

1623199

remove awslint exclude

16a54e5

feat(aws-kinesisfirehose): specific metrics functions for DeliveryStr…

b011392

…eam (#15545) Closes #15543 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

otaviomacedo reviewed Jul 16, 2021

View reviewed changes

change name of s3 destination props to S3BucketProps

56aa16a

madeline-k and others added 2 commits July 26, 2021 09:58

Update packages/@aws-cdk/aws-kinesisfirehose/test/delivery-stream.tes…

43f2ecf

…t.ts Use `toHaveResourceLike` instead of `toHaveResource` Co-authored-by: Ben Chaimberg <chaimber@amazon.com>

Update packages/@aws-cdk/aws-kinesisfirehose/test/delivery-stream.tes…

3325990

…t.ts Co-authored-by: Ben Chaimberg <chaimber@amazon.com>

madeline-k requested a review from otaviomacedo July 26, 2021 17:01

Merge branch 'master' into madeline-k/feat/kinesisfirehose-metrics

8ac8ea3

skinny85 previously approved these changes Jul 27, 2021

View reviewed changes

packages/@aws-cdk/aws-kinesisfirehose-destinations/lib/common.ts Outdated Show resolved Hide resolved

packages/@aws-cdk/aws-kinesisfirehose-destinations/lib/common.ts Outdated Show resolved Hide resolved

BenChaimberg and others added 11 commits July 27, 2021 19:21

README formatting

8bc6c88

deduplicate delivery stream creation in metrics tests

1aa653e

Merge remote-tracking branch 'origin/madeline-k/feat/kinesisfirehose-…

9ee3651

…metrics' into madeline-k/feat/kinesisfirehose-sse

README formatting

44d3805

docstring clarification of defaults

6207614

add comment detailing why explicit dependency on key grant is not added

5c790a6

Merge branch 'madeline-k/feat/kinesisfirehose-sse' into madeline-k/fe…

61391a4

…at/kinesisfirehose-compression

remove Compression.UNCOMPRESSED, add Compression.of, make constructor…

c30312d

… private, tests

Merge branch 'master' into madeline-k/feat/kinesisfirehose-metrics

5f4577a

Merge branch 'madeline-k/feat/kinesisfirehose-metrics' into madeline-…

6ac183e

…k/feat/kinesisfirehose-sse

Merge branch 'madeline-k/feat/kinesisfirehose-sse' into madeline-k/fe…

387fbef

…at/kinesisfirehose-compression

Base automatically changed from madeline-k/feat/kinesisfirehose-sse to master July 28, 2021 20:42

mergify bot dismissed skinny85’s stale review via 387fbef July 28, 2021 20:42

Merge branch 'master' into madeline-k/feat/kinesisfirehose-compression

9f9c7f8

madeline-k removed the pr/do-not-merge This PR should not be merged at this time. label Jul 28, 2021

skinny85 changed the title ~~feat(aws-kinesisfirehose-destinations): support for compression on S3 delivery stream destinations~~ feat(kinesisfirehose-destinations): add support for compression on S3 delivery stream destinations Jul 28, 2021

skinny85 approved these changes Jul 28, 2021

View reviewed changes

mergify bot merged commit 1eb56a0 into master Jul 28, 2021

mergify bot deleted the madeline-k/feat/kinesisfirehose-compression branch July 28, 2021 21:58

madeline-k commented Jul 28, 2021

View reviewed changes

BenChaimberg removed this from the [DevPreview] Kinesis Data Firehose milestone Jul 30, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(kinesisfirehose-destinations): add support for compression on S3 delivery stream destinations#15550

feat(kinesisfirehose-destinations): add support for compression on S3 delivery stream destinations#15550
mergify[bot] merged 105 commits intomasterfrom
madeline-k/feat/kinesisfirehose-compression

madeline-k commented Jul 14, 2021

Uh oh!

gitpod-io bot commented Jul 14, 2021

Uh oh!

otaviomacedo Jul 16, 2021

Uh oh!

madeline-k Jul 20, 2021

Uh oh!

madeline-k Jul 23, 2021

Uh oh!

skinny85 left a comment

Uh oh!

Uh oh!

Uh oh!

aws-cdk-automation commented Jul 28, 2021

Uh oh!

skinny85 left a comment

Uh oh!

skinny85 Jul 28, 2021

Uh oh!

madeline-k Jul 28, 2021

Uh oh!

mergify bot commented Jul 28, 2021

Uh oh!

madeline-k Jul 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants


		## Compression

		Your data can automatically be compressed when it is delivered to S3 as either a final or

Conversation

madeline-k commented Jul 14, 2021

Uh oh!

gitpod-io bot commented Jul 14, 2021

Uh oh!

otaviomacedo Jul 16, 2021

Choose a reason for hiding this comment

Uh oh!

madeline-k Jul 20, 2021

Choose a reason for hiding this comment

Uh oh!

madeline-k Jul 23, 2021

Choose a reason for hiding this comment

Uh oh!

skinny85 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aws-cdk-automation commented Jul 28, 2021

AWS CodeBuild CI Report

Uh oh!

skinny85 left a comment

Choose a reason for hiding this comment

Uh oh!

skinny85 Jul 28, 2021

Choose a reason for hiding this comment

Uh oh!

madeline-k Jul 28, 2021

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Jul 28, 2021

Uh oh!

madeline-k Jul 28, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants