[Parent issue] Support for Apache Pulsar #31078
Description
Since there is a continious demand for it,
just an overview for current state of Support for Apache Pulsar:
-
when looking into beams i/o connector matrix pulsar is not listed yet:
https://beam.apache.org/documentation/io/connectors/ -
but there is a sdk package
Package org.apache.beam.sdk.io.pulsar
https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/pulsar/package-summary.html -
in v2.39 of beam a relatively basic pulsar io was introduced:
Implemented Apache PulsarIO https://beam.apache.org/blog/beam-2.39.0/
for details see:
https://issues.apache.org/jira/browse/BEAM-8218 -
there are some open issues in beam for pulsar:
https://github.com/apache/beam/issues?q=is%3Aissue+pulsar+is%3Aopen- [Bug]: PulsarIO.read() is failing with java.lang.RuntimeException: Could not find a way to create AutoValue class class com.idfy.beam.pulsar.PulsarSourceDescriptor #30688
- [Bug]: PulsarIO write() complains of missing ClientUrl even when provided with withClientUrl #30152
- [Feature Request]: Set option for acknowledge the messages in PulsarIO reader #22054
- [Feature Request]: Pulsar Connector should accept auth parameters to connect #22027
- [Feature Request]: Make Pulsar IO connector publish messages asynchronously #22025
- PulsarIOTest.testReadFromSimpleTopic is very flaky #21714
- org.apache.beam.sdk.io.pulsar.PulsarIOTest.testReadFromSimpleTopic flaky #21560
- [Doc Request]: current state of Pulsar support not in doc #31079
-
there were also some non merged pull request marked as stale and automatically closed, which may be worth to have a look on
https://github.com/apache/beam/pulls?q=is%3Apr+pulsar+is%3Aclosed+label%3Astale- Bump Jackson dependency due to CVE-2022-1471 #26410
- fix pulsar reader calls #22375
- pulsar connector accepts auth params #22028
- pulsar connector produces message asynchronously #22026
- [BEAM-14269] PulsarIOTest.testReadFromSimpleTopic flaky test #17473
- Validate timestamps when creating windowed or timestamped values #16749
-
in general it would be a good longterm target, to reach the same quality of integration in Beam as there is for Kafka