Skip to content

Conversation

@Denovo1998
Copy link
Contributor

@Denovo1998 Denovo1998 commented May 27, 2023

Fixes #20414

Master Issue: ##20414

Motivation

#17221 describes an environment when multiple bookie copies are corrupted, or a Ledger has been deleted. The loss of schema ledger results in new producers and consumers not even being created and working properly.

So we need a solution that does not just skip the schema with the missing ledger, but actually supplements the broken schema ledger.

Modifications

Add a new method tryCompleteTheLostSchema() in SchemaStorage and SchemaRegistry

CompletableFuture<Long> tryCompleteTheLostSchemaLedger(String key, SchemaVersion version, SchemaData schema);
  1. get schemalocator from metastore
  2. Create a new ledger. And write SchemaStorageFormat.SchemaEntry built with schemaData and schemaVersion.
  3. update schemalocator to metastore(new ledger id)

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: Denovo1998#4

@Denovo1998
Copy link
Contributor Author

@poorbarcode @congbobo184 @codelipenghui
SchemaData and SchemaVersion has moved to the org.apache.pulsar.broker.service.AbstractTopic, rather than save in each producer and consumer. Check out the solution in issue #20414. Is this way okay now?

@github-actions
Copy link

The pr had no activity for 30 days, mark with Stale label.

@github-actions github-actions bot added the Stale label Jul 30, 2023
@Technoboy- Technoboy- modified the milestones: 3.1.0, 3.2.0 Jul 31, 2023
@Denovo1998
Copy link
Contributor Author

Waiting to discuss whether this plan is feasible. I will send an email to discuss it later.

@github-actions github-actions bot removed the Stale label Aug 30, 2023
@github-actions
Copy link

The pr had no activity for 30 days, mark with Stale label.

@Denovo1998
Copy link
Contributor Author

In the alternative, the implementation is updated. Needs to be discussed.

# Conflicts:
#	pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
#	pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java
#	pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
#	pulsar-broker/src/test/java/org/apache/pulsar/broker/service/schema/SchemaServiceTest.java
@coderzc coderzc modified the milestones: 3.3.0, 3.4.0 May 8, 2024
@lhotari lhotari modified the milestones: 4.0.0, 4.1.0 Oct 14, 2024
@coderzc coderzc modified the milestones: 4.1.0, 4.2.0 Sep 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/schema doc-not-needed Your PR changes do not impact docs Stale

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[improve][broker] Supplement schema ledger if schema ledger is lost

5 participants