[improve][io] Add support for the complete KinesisProducerConfiguration in KinesisSinkConfig #24489

sandeep-mst · 2025-07-07T19:47:33Z

Motivation

The Apache Pulsar KinesisSink utilises the KinesisProducer from the KPL. The KPL provides a comprehensive set of configuration parameters through its KinesisProducerConfiguration class.

In the current KinesisSink implementation, most of these parameters such as collectionMaxCount, collectionMaxSize, connectTimeout, maxConnections, and minConnections—are not configurable. This restricts users' ability to optimise the sink's performance, or meet network and operational requirements potentially affecting throughput, latency, and resource utilisation.

Modifications

Added the following KPL configuration parameters directly into the KinesisSink configuration.

Parameter	Description
collectionMaxCount	Maximum number of items to pack into an PutRecords request.
collectionMaxSize	Maximum amount of data to send with a PutRecords request.
connectTimeout	Timeout (milliseconds) for establishing TLS connections.
credentialsRefreshDelay	How often to refresh credentials (in milliseconds).
maxConnections	Maximum number of connections to open to the backend.
minConnections	Minimum number of connections to keep open to the backend
rateLimit	Limits the maximum allowed put rate for a shard, as a percentage of the backend limits.
recordTtl	Set a time-to-live on records (milliseconds).
requestTimeout	The maximum total time (milliseconds) allowed for a HTTP request

Introduce an Optional Hashmap for Additional Configuration (extraKinesisProducerConfiguration), which accepts a hashmap of key-value pairs corresponding to any other KPL configuration parameters not directly exposed. This ensures users retain full control over the KinesisProducer's behavior without overloading the primary sink configuration interface.

Verifying this change

Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change added tests and can be verified as follows:

Will be updated

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

Documentation

doc
doc-required
doc-not-needed
doc-complete

Matching PR in forked repository

PR in forked repository:
cognitree#25

dlg99

LGTM

BewareMyPower · 2025-07-11T04:05:03Z

Could you rebase to master so that the checkstyle check for tests will be applied?

sandeep-mst · 2025-07-11T15:43:41Z

Could you rebase to master so that the checkstyle check for tests will be applied?

@BewareMyPower
Done.

sandeep-mst · 2025-07-21T14:57:01Z

/pulsarbot rerun-failure-checks

BewareMyPower · 2025-07-21T15:10:10Z

It's weird for this failure:

Restoring tar from name pulsar-maven-repository-binaries.tar.zst in GitHub Actions Artifacts to /home/runner
  /home/runner/work/_temp/_github_home/.local/bin/gh-actions-artifact-client.js:79024
      throw new ArtifactNotFoundError(`No artifacts found with name: ${name}`);

@sandeep-mst maybe you have to try merging to master again to re-trigger the build.

BTW, @lhotari do you have any idea about the failure above?

…ducerConfiguration hashmap.

…erConfig

sandeep-mst · 2025-07-21T15:24:17Z

@BewareMyPower
Rebased to master again.

codecov-commenter · 2025-07-21T16:52:40Z

Codecov Report

Attention: Patch coverage is 93.33333% with 1 line in your changes missing coverage. Please review.

Project coverage is 74.33%. Comparing base (bbc6224) to head (f609773).
Report is 1208 commits behind head on master.

Files with missing lines	Patch %	Lines
...java/org/apache/pulsar/io/kinesis/KinesisSink.java	92.30%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #24489      +/-   ##
============================================
+ Coverage     73.57%   74.33%   +0.76%     
- Complexity    32624    32933     +309     
============================================
  Files          1877     1869       -8     
  Lines        139502   146105    +6603     
  Branches      15299    16762    +1463     
============================================
+ Hits         102638   108610    +5972     
+ Misses        28908    28889      -19     
- Partials       7956     8606     +650

Flag	Coverage Δ
inttests	`26.88% <ø> (+2.30%)`	⬆️
systests	`23.27% <0.00%> (-1.05%)`	⬇️
unittests	`73.84% <93.33%> (+0.99%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...rg/apache/pulsar/io/kinesis/KinesisSinkConfig.java	`90.90% <100.00%> (+10.13%)`	⬆️
...java/org/apache/pulsar/io/kinesis/KinesisSink.java	`57.53% <92.30%> (+2.91%)`	⬆️

... and 1100 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

sandeep-mst · 2025-07-21T17:45:28Z

@BewareMyPower, @dlg99, @lhotari

The OWASP dependency check is failing on master as well and I have verified that pulsar-io/kinesis has no new vulnerabilities.

Let me know if anything else is required on this PR.

lhotari · 2025-07-21T17:53:45Z

It's weird for this failure:
Restoring tar from name pulsar-maven-repository-binaries.tar.zst in GitHub Actions Artifacts to /home/runner
  /home/runner/work/_temp/_github_home/.local/bin/gh-actions-artifact-client.js:79024
      throw new ArtifactNotFoundError(`No artifacts found with name: ${name}`);
@sandeep-mst maybe you have to try merging to master again to re-trigger the build.

BTW, @lhotari do you have any idea about the failure above?

@BewareMyPower yes. Retrying a failed build after 3 days won't work since the GitHub workflow artifacts retention is set to 3 days so that we don't overflow the size of storage. It's necessary to trigger a completely new build in that case. Usually it's recommended to merge master to the PR branch to trigger that.

…on in KinesisSinkConfig (apache#24489) (cherry picked from commit 80902f8)

…on in KinesisSinkConfig (#24489)

…on in KinesisSinkConfig (apache#24489)

github-actions bot added the doc-required Your PR changes impact docs and you will update later. label Jul 7, 2025

dlg99 approved these changes Jul 8, 2025

View reviewed changes

dlg99 added the ready-to-test label Jul 8, 2025

sandeep-mst force-pushed the kinesis-24422 branch from 29b330a to e6f01d7 Compare July 11, 2025 14:01

BewareMyPower assigned sandeep-mst Jul 21, 2025

sandeep-mst added 6 commits July 21, 2025 20:51

Added new KinesisSinkConfig Parameters and loading of extraKinesisPro…

9295533

…ducerConfiguration hashmap.

updated KinesisSinkConfigTests to check parsing of extraKinesisProduc…

7482217

…erConfig

Added utils to fix jackson parsing issue

6f4a5c6

improved testing of parsing of KinesisProducerConfiguration

c77eca7

updated help to point to the current version parameter list.

ef87eb6

checkstyle formatting fix

f609773

sandeep-mst force-pushed the kinesis-24422 branch from 4d76453 to f609773 Compare July 21, 2025 15:22

dlg99 merged commit 80902f8 into apache:master Jul 22, 2025
96 of 98 checks passed

priyanshu-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 24, 2025

[improve][io] Add support for the complete KinesisProducerConfigurati…

02c54a9

…on in KinesisSinkConfig (apache#24489) (cherry picked from commit 80902f8)

priyanshu-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 24, 2025

[improve][io] Add support for the complete KinesisProducerConfigurati…

b0718a2

…on in KinesisSinkConfig (apache#24489) (cherry picked from commit 80902f8)

priyanshu-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 24, 2025

[improve][io] Add support for the complete KinesisProducerConfigurati…

fb78055

…on in KinesisSinkConfig (apache#24489) (cherry picked from commit 80902f8)

srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 24, 2025

[improve][io] Add support for the complete KinesisProducerConfigurati…

f59b6e9

…on in KinesisSinkConfig (apache#24489) (cherry picked from commit 80902f8)

srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 24, 2025

[improve][io] Add support for the complete KinesisProducerConfigurati…

456ee77

…on in KinesisSinkConfig (apache#24489) (cherry picked from commit 80902f8)

lhotari added this to the 4.1.0 milestone Jul 25, 2025

lhotari added the release/4.0.7 label Jul 25, 2025

sandeep-mst mentioned this pull request Jul 30, 2025

[improve][io] Add support for the complete KinesisProducerConfiguration in KinesisSinkConfig cognitree/pulsar#25

Closed

15 tasks

Technoboy- pushed a commit that referenced this pull request Jul 31, 2025

[improve][io] Add support for the complete KinesisProducerConfigurati…

28b5df0

…on in KinesisSinkConfig (#24489)

Technoboy- added the cherry-picked/branch-4.0 label Jul 31, 2025

KannarFr pushed a commit to CleverCloud/pulsar that referenced this pull request Sep 22, 2025

[improve][io] Add support for the complete KinesisProducerConfigurati…

984a9c3

…on in KinesisSinkConfig (apache#24489)

walkinggo pushed a commit to walkinggo/pulsar that referenced this pull request Oct 8, 2025

[improve][io] Add support for the complete KinesisProducerConfigurati…

4854c6b

…on in KinesisSinkConfig (apache#24489)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[improve][io] Add support for the complete KinesisProducerConfiguration in KinesisSinkConfig #24489

[improve][io] Add support for the complete KinesisProducerConfiguration in KinesisSinkConfig #24489

Uh oh!

sandeep-mst commented Jul 7, 2025

Uh oh!

dlg99 left a comment

Uh oh!

BewareMyPower commented Jul 11, 2025

Uh oh!

sandeep-mst commented Jul 11, 2025 •

edited

Loading

Uh oh!

sandeep-mst commented Jul 21, 2025

Uh oh!

BewareMyPower commented Jul 21, 2025

Uh oh!

sandeep-mst commented Jul 21, 2025

Uh oh!

codecov-commenter commented Jul 21, 2025

Uh oh!

sandeep-mst commented Jul 21, 2025 •

edited

Loading

Uh oh!

lhotari commented Jul 21, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

[improve][io] Add support for the complete KinesisProducerConfiguration in KinesisSinkConfig #24489

[improve][io] Add support for the complete KinesisProducerConfiguration in KinesisSinkConfig #24489

Uh oh!

Conversation

sandeep-mst commented Jul 7, 2025

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Matching PR in forked repository

Uh oh!

dlg99 left a comment

Choose a reason for hiding this comment

Uh oh!

BewareMyPower commented Jul 11, 2025

Uh oh!

sandeep-mst commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sandeep-mst commented Jul 21, 2025

Uh oh!

BewareMyPower commented Jul 21, 2025

Uh oh!

sandeep-mst commented Jul 21, 2025

Uh oh!

codecov-commenter commented Jul 21, 2025

Codecov Report

Uh oh!

sandeep-mst commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lhotari commented Jul 21, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

sandeep-mst commented Jul 11, 2025 •

edited

Loading

sandeep-mst commented Jul 21, 2025 •

edited

Loading