Skip to content

Conversation

@poorbarcode
Copy link
Contributor

@poorbarcode poorbarcode commented Jul 3, 2025

Motivation & Modifications

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: x

@poorbarcode poorbarcode added this to the 4.1.0 milestone Jul 3, 2025
@poorbarcode poorbarcode self-assigned this Jul 3, 2025
@github-actions github-actions bot added PIP doc-required Your PR changes impact docs and you will update later. labels Jul 3, 2025
@poorbarcode
Copy link
Contributor Author

The PIP also solves the issue #24417

@poorbarcode poorbarcode requested a review from codelipenghui July 7, 2025 01:56
@poorbarcode poorbarcode requested a review from nodece July 7, 2025 14:42
Copy link
Member

@nodece nodece left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-1

This PIP attempts to work around user misconfigurations rather than enforce proper system boundaries. As a result, it introduces hidden complexity and encourages fragile, non-transparent behavior. The main issues are:

  1. Replicator relies on implicit topic auto-creation on the remote cluster, assuming the auto-creation policies (like number of partitions and topic type) match. But this does not solve the root problem. Remote cluster policies may change at any time, and relying on them does not eliminate the risk of topic-type mismatches.

  2. Bypassing schema auto-update restrictions for replication is a workaround, not a proper solution.
    If a remote cluster disables schema auto-updates, it reflects an explicit user decision to reject schema changes. Replicator should respect this setting. Injecting schemas automatically breaks the data contract and risks corrupting consumers.
    A safer and more transparent approach would be to allow users to manually reset the cursor to a compatible schema point if they wish to resume replication.

  3. Automatically modifying remote cluster policies is a dangerous design.
    It violates cluster isolation and can introduce unintended side effects. In large-scale, multi-tenant deployments, such implicit adjustments are risky and potentially disruptive.

  4. Instead of using automatic adjustments or bypass logic, replication should require explicit provisioning of topics and schemas on both clusters.
    This keeps behavior predictable, under user control, and consistent with system boundaries.

In short, this PIP prioritizes tolerating misconfigurations over enforcing correct and safe behavior. That may offer short-term convenience, but it makes the system harder to reason about and less reliable over time.

A better direction would be to improve observability of the replicator (e.g., metrics, failure visibility), and help users fix configuration issues explicitly and safely.

@poorbarcode poorbarcode requested a review from nodece July 10, 2025 02:26
@poorbarcode poorbarcode requested a review from Technoboy- July 10, 2025 10:00
Co-authored-by: Penghui Li <penghui@apache.org>
@BewareMyPower
Copy link
Contributor

Could you rebase to master so the checkstyle for tests will be applied?

@poorbarcode
Copy link
Contributor Author

poorbarcode commented Jul 13, 2025

@BewareMyPower

Could you rebase to master so the checkstyle for tests will be applied?

Seems this comment is in order to leave under other PRs?

@poorbarcode poorbarcode requested a review from nodece July 18, 2025 09:07
@poorbarcode poorbarcode merged commit bda1ec4 into apache:master Jul 24, 2025
20 checks passed
KannarFr pushed a commit to CleverCloud/pulsar that referenced this pull request Sep 22, 2025
… automatic creation mechanisms, including the automatic creation of topics and schemas (apache#24485)

Co-authored-by: Penghui Li <penghui@apache.org>
walkinggo pushed a commit to walkinggo/pulsar that referenced this pull request Oct 8, 2025
… automatic creation mechanisms, including the automatic creation of topics and schemas (apache#24485)

Co-authored-by: Penghui Li <penghui@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-required Your PR changes impact docs and you will update later. PIP

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants