Skip to content

[feat] [broker] PIP-188 Add support to auto create topic resources into green cluster before migration [part-3]#21354

Merged
lhotari merged 1 commit into
apache:masterfrom
rdhabalia:tn_cn
Oct 14, 2023
Merged

[feat] [broker] PIP-188 Add support to auto create topic resources into green cluster before migration [part-3]#21354
lhotari merged 1 commit into
apache:masterfrom
rdhabalia:tn_cn

Conversation

@rdhabalia

@rdhabalia rdhabalia commented Oct 12, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

Motivation

Right now, topic migration requires pre-req manual steps for admin to create migrated topic's resources into green cluster such as tenant, namespaces, namespace policies, and subscriptions. This PR allows admin to automate this process and perform migration without any manual intervention.

Modifications

  • add config to control auto resource creation for topic migration
  • perform topic resource creation at migrated green cluster before starting topic migration
./bin/pulsar-admin clusters update-cluster-migration $blueClusterName \
--service-url ${greenClusterServiceURL} \
--broker-url ${greenClusterURL} \
--broker-url-secure ${greenClusterURL} \
 --migrated

Result

Broker manages topic resource creation at new migrated/green cluster and then perform topic migration without losing data.

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository:

@rdhabalia rdhabalia added this to the 3.2.0 milestone Oct 12, 2023
@rdhabalia rdhabalia self-assigned this Oct 12, 2023
@github-actions github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Oct 12, 2023
This was referenced Oct 13, 2023
Closed
Merged
vineeth1995
vineeth1995 approved these changes Oct 13, 2023
View reviewed changes

@vineeth1995 vineeth1995 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to go.

This was referenced Oct 13, 2023
Merged
Merged
@lhotari lhotari closed this Oct 14, 2023
@lhotari lhotari reopened this Oct 14, 2023
@codecov-commenter

codecov-commenter commented Oct 14, 2023 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

Merging #21354 (039be25) into master (1a352f1) will increase coverage by 39.77%.
Report is 2 commits behind head on master.
The diff coverage is 71.42%.

Impacted file tree graph

@@              Coverage Diff              @@
##             master   #21354       +/-   ##
=============================================
+ Coverage     33.49%   73.26%   +39.77%     
- Complexity    12257    32540    +20283     
=============================================
  Files          1636     1888      +252     
  Lines        127789   140365    +12576     
  Branches      13963    15456     +1493     
=============================================
+ Hits          42798   102843    +60045     
+ Misses        79372    29439    -49933     
- Partials       5619     8083     +2464
Flag Coverage Δ
inttests 24.22% <3.57%> (+0.08%) ⬆️
systests 24.72% <3.57%> (?)
unittests 72.54% <71.42%> (+40.53%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
...org/apache/pulsar/broker/ServiceConfiguration.java 99.38% <100.00%> (+1.43%) ⬆️
...pache/pulsar/common/policies/data/ClusterData.java 77.77% <66.66%> (+63.49%) ⬆️
.../java/org/apache/pulsar/admin/cli/CmdClusters.java 74.07% <0.00%> (+74.07%) ⬆️
...sar/broker/service/persistent/PersistentTopic.java 79.71% <71.79%> (+27.48%) ⬆️

... and 1528 files with indirect coverage changes

@lhotari lhotari merged commit c8183e7 into apache:master Oct 14, 2023
@rdhabalia rdhabalia deleted the tn_cn branch October 14, 2023 18:45
@rdhabalia rdhabalia mentioned this pull request Oct 16, 2023
Merged
15 tasks
vraulji567 pushed a commit to vraulji567/pulsar that referenced this pull request Oct 16, 2023
@rdhabalia
[feat] [broker] PIP-188 Add support to auto create topic resources in…
3cd07b4 
…to green cluster before migration [part-3] (apache#21354)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lhotari lhotari lhotari approved these changes

@eolivelli eolivelli eolivelli approved these changes

@vineeth1995 vineeth1995 vineeth1995 approved these changes

Assignees

@rdhabalia rdhabalia

Labels

area/broker doc-not-needed Your PR changes do not impact docs ready-to-test

Projects

None yet

Milestone

3.2.0

Development

Successfully merging this pull request may close these issues.

5 participants

@rdhabalia @codecov-commenter @lhotari @eolivelli @vineeth1995