feat: remove base repl endpoint from hbase 2.x #3539

sushanb · 2022-03-07T19:14:05Z

Replaces dependency on BaseReplicationEndpoint with ReplicationEndpoint
Adds a dependency on bigtable shaded guava.
Adds tests for scope and custom WAL entry filters
Adds a config for custom wal entry filters for Hbase2.x

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> ☕️

.../com/google/cloud/bigtable/hbase2_x/replication/HbaseToCloudBigtableReplicationEndpoint.java

.../java/com/google/cloud/bigtable/hbase2_x/replication/configuration/CustomWALEntryConfig.java

.../google/cloud/bigtable/hbase2_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

.../com/google/cloud/bigtable/hbase2_x/replication/HbaseToCloudBigtableReplicationEndpoint.java

.../google/cloud/bigtable/hbase2_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

vermas2012 · 2022-03-16T18:35:12Z

.../google/cloud/bigtable/hbase1_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

+    TestUtils.waitForReplication(
+        () -> {
+          //  replicate Entries will be zero
+          return TestReplicationEndpoint.replicatedEntries.get() >= 1;


should be 2

FILTERED_ROW_KEY is not passed under replicateContext

Then explain in the comment that FILTERED KEY will not be passed. A new engineer debugging this test should not need to figure that part out.

vermas2012 · 2022-03-16T18:36:25Z

.../google/cloud/bigtable/hbase1_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

+        "Number of cells , actual cells: " + hbaseResult.listCells(),
+        2,
+        2,
+        hbaseResult.listCells().size());


Not required.

vermas2012 · 2022-03-16T18:39:21Z

.../google/cloud/bigtable/hbase1_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

+    Result cbtResult = cbtTable.get(new Get(ROW_KEY).setMaxVersions());
+    Result hbaseResult = hbaseTable.get(new Get(ROW_KEY).setMaxVersions());
+    Assert.assertFalse(cbtResult.isEmpty());
+    Assert.assertFalse(hbaseResult.isEmpty());


not required

vermas2012 · 2022-03-16T18:40:50Z

.../google/cloud/bigtable/hbase1_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

+    Result hbaseResult = hbaseTable.get(new Get(ROW_KEY).setMaxVersions());
+    Assert.assertFalse(cbtResult.isEmpty());
+    Assert.assertFalse(hbaseResult.isEmpty());
+    Assert.assertEquals(


this method seems weird, what are we validating ? The name should give me a hint, this is not an "assertEquals" method.

vermas2012 · 2022-03-16T18:40:58Z

.../google/cloud/bigtable/hbase1_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

+        1,
+        1,
+        cbtResult.listCells().size());
+    TestUtils.assertEquals(


.../google/cloud/bigtable/hbase2_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

vermas2012 · 2022-03-16T18:45:49Z

.../google/cloud/bigtable/hbase2_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

+        "Number of cells , actual cells: " + hbaseResult.listCells(),
+        2,
+        2,
+        hbaseResult.listCells().size());


not required.

vermas2012 · 2022-03-16T18:46:19Z

.../google/cloud/bigtable/hbase2_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

+    Result cbtResult = cbtTable.get(new Get(ROW_KEY).setMaxVersions());
+    Result hbaseResult = hbaseTable.get(new Get(ROW_KEY).setMaxVersions());
+    Assert.assertFalse(cbtResult.isEmpty());
+    Assert.assertFalse(hbaseResult.isEmpty());


not required.

vermas2012 · 2022-03-16T18:48:04Z

.../google/cloud/bigtable/hbase2_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

+    Result hbaseResult = hbaseTable.get(new Get(ROW_KEY).setMaxVersions());
+    Assert.assertFalse(cbtResult.isEmpty());
+    Assert.assertFalse(hbaseResult.isEmpty());
+    Assert.assertEquals(


this is not calling your method.

vermas2012 · 2022-03-16T18:49:47Z

.../google/cloud/bigtable/hbase2_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

+    TestUtils.assertEquals(
+        hbaseResult,
+        cbtResult,
+        HConstants.REPLICATION_SCOPE_GLOBAL,


check it in test. Use assertEqual on cells.

sushanb

Addressed PR feedbacks

sushanb · 2022-03-18T16:28:18Z

.../google/cloud/bigtable/hbase1_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

+        "Number of cells , actual cells: " + hbaseResult.listCells(),
+        2,
+        2,
+        hbaseResult.listCells().size());


sushanb · 2022-03-18T16:30:03Z

.../google/cloud/bigtable/hbase1_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

+    Result hbaseResult = hbaseTable.get(new Get(ROW_KEY).setMaxVersions());
+    Assert.assertFalse(cbtResult.isEmpty());
+    Assert.assertFalse(hbaseResult.isEmpty());
+    Assert.assertEquals(


sushanb · 2022-03-18T16:30:09Z

.../google/cloud/bigtable/hbase1_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

+        1,
+        1,
+        cbtResult.listCells().size());
+    TestUtils.assertEquals(


.../google/cloud/bigtable/hbase2_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

sushanb · 2022-03-18T16:30:52Z

.../google/cloud/bigtable/hbase2_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

+        "Number of cells , actual cells: " + hbaseResult.listCells(),
+        2,
+        2,
+        hbaseResult.listCells().size());


sushanb · 2022-03-18T16:30:56Z

.../google/cloud/bigtable/hbase2_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

+    Result cbtResult = cbtTable.get(new Get(ROW_KEY).setMaxVersions());
+    Result hbaseResult = hbaseTable.get(new Get(ROW_KEY).setMaxVersions());
+    Assert.assertFalse(cbtResult.isEmpty());
+    Assert.assertFalse(hbaseResult.isEmpty());


sushanb · 2022-03-18T16:31:02Z

.../google/cloud/bigtable/hbase2_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

+    TestUtils.assertEquals(
+        hbaseResult,
+        cbtResult,
+        HConstants.REPLICATION_SCOPE_GLOBAL,


sushanb · 2022-03-18T16:31:12Z

.../google/cloud/bigtable/hbase2_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

+    Result hbaseResult = hbaseTable.get(new Get(ROW_KEY).setMaxVersions());
+    Assert.assertFalse(cbtResult.isEmpty());
+    Assert.assertFalse(hbaseResult.isEmpty());
+    Assert.assertEquals(


vermas2012 · 2022-03-18T19:19:02Z

.../google/cloud/bigtable/hbase1_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

+    TestUtils.waitForReplication(
+        () -> {
+          //  replicate Entries will be zero
+          return TestReplicationEndpoint.replicatedEntries.get() >= 1;


Then explain in the comment that FILTERED KEY will not be passed. A new engineer debugging this test should not need to figure that part out.

.../google/cloud/bigtable/hbase1_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

.../google/cloud/bigtable/hbase2_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java

🤖 I have created a release *beep* *boop* --- ## [2.1.0](v2.0.0...v2.1.0) (2022-04-01) ### Features * [hase-cbt-replication]Add a metrics for hbase<>bigtable timestamp overflow ([#3540](#3540)) ([9c73318](9c73318)) * Adding a detailed readme for HBase replication support. ([#3527](#3527)) ([b406ac5](b406ac5)) * enable dry-run mode for HBase to Cloud Bigtable replication ([#3532](#3532)) ([164738b](164738b)) * Enable HBase to Cloud Bigtable replication ([#3510](#3510)) ([68d4a01](68d4a01)) * flag a put in future timestamp and record a metrics ([#3534](#3534)) ([7dd7653](7dd7653)) * remove base repl endpoint from hbase 2.x ([#3539](#3539)) ([f748c70](f748c70)) ### Bug Fixes * [hbase-cbt-replication] make hbase2.x endpoint context private ([#3559](#3559)) ([5ec3b18](5ec3b18)) * consistent behavior of customwalentry filter across hbase1.x and hbase2.x ([#3562](#3562)) ([f70b0cb](f70b0cb)) * license and notice files for all the shaded artifacts ([#3565](#3565)) ([86a5643](86a5643)) * minor documentation update ([#3572](#3572)) ([68db6c2](68db6c2)) ### Documentation * fix typos in README and add --region option ([#3520](#3520)) ([c189d46](c189d46)) ### Dependencies * update dependency ch.qos.logback:logback-classic to v1.2.11 ([#3538](#3538)) ([92e8b63](92e8b63)) * update dependency com.google.cloud:google-cloud-bigtable-emulator to v0.142.3 ([#3528](#3528)) ([33a8eba](33a8eba)) * update dependency com.google.cloud:google-cloud-bigtable-emulator to v0.143.0 ([#3543](#3543)) ([2d57f14](2d57f14)) * update dependency net.bytebuddy:byte-buddy to v1.12.8 ([#3502](#3502)) ([77ae2f1](77ae2f1)) * update jmh.version to v1.35 ([#3561](#3561)) ([188738d](188738d)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

sushanb requested a review from vermas2012 March 7, 2022 19:14

sushanb requested a review from a team as a code owner March 7, 2022 19:14

product-auto-label bot added the api: bigtable Issues related to the googleapis/java-bigtable-hbase API. label Mar 7, 2022

vermas2012 requested changes Mar 8, 2022

View reviewed changes

sushanb force-pushed the main branch 2 times, most recently from 35e3fc6 to 617034a Compare March 14, 2022 16:39

sushanb added 12 commits March 14, 2022 13:23

fix test

6a1d33f

fix

9afc139

revert

882cd2e

fix

f17f227

fix

4df699a

fix test

d88a007

fix test

30dfe33

fitler

c3de939

fix

1652580

fixalL

10247dd

fix test

6cf5f13

fix test

c9b9495

sushanb force-pushed the main branch from 617034a to c9b9495 Compare March 14, 2022 18:29

sushanb added 4 commits March 14, 2022 14:32

switch

e0772ab

fix err log

2630d2d

fix formatting

f7bfa41

remove TODO

2fc5a17

sushanb requested a review from vermas2012 March 14, 2022 18:50

vermas2012 requested changes Mar 14, 2022

View reviewed changes

sushanb added 5 commits March 15, 2022 11:09

address comments

cd734a5

add test

3def51e

fix:

3ac30ae

fix

41867bf

al

406ff22

add scope filter

9a3b2bb

sushanb requested a review from vermas2012 March 15, 2022 17:44

fix

93f0947

vermas2012 reviewed Mar 16, 2022

View reviewed changes

fix

65e6e01

sushanb force-pushed the main branch from 5f25bae to 65e6e01 Compare March 17, 2022 20:18

sushanb added 6 commits March 17, 2022 16:20

Merge branch 'main' into main

95bc8db

fix trailing cursor

e7f3872

fix comments

2bcc5c4

add logger back

c43baf7

comment out

19621fd

fix

25535ea

sushanb commented Mar 18, 2022

View reviewed changes

sushanb added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Mar 18, 2022

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Mar 18, 2022

sushanb force-pushed the main branch 2 times, most recently from a33222a to a938eac Compare March 18, 2022 16:51

fix

47f8bc7

sushanb force-pushed the main branch from a938eac to 47f8bc7 Compare March 18, 2022 16:52

vermas2012 self-requested a review March 18, 2022 19:24

vermas2012 approved these changes Mar 18, 2022

View reviewed changes

sushanb added 2 commits March 18, 2022 15:34

address PR feedback

f726b65

remove unused value

1d44ab3

sushanb added kokoro:force-run Add this label to force Kokoro to re-run the tests. owlbot:run Add this label to trigger the Owlbot post processor. labels Mar 18, 2022

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 18, 2022

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Mar 18, 2022

sushanb merged commit f748c70 into googleapis:main Mar 18, 2022

release-please bot mentioned this pull request Mar 18, 2022

chore(main): release 2.1.0 #3524

Merged

feat: remove base repl endpoint from hbase 2.x #3539

feat: remove base repl endpoint from hbase 2.x #3539

Uh oh!

Conversation

sushanb commented Mar 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sushanb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sushanb commented Mar 7, 2022 •

edited

Loading