Skip to content

feat: Add a metrics for hbase<>bigtable timestamp overflow #3540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
sushanb merged 30 commits into from
Mar 17, 2022
Merged

feat: Add a metrics for hbase<>bigtable timestamp overflow #3540

sushanb merged 30 commits into from
Mar 17, 2022

Conversation

@sushanb
Copy link
Contributor

@sushanb sushanb commented Mar 7, 2022

Based on PR https://github.com/googleapis/java-bigtable-hbase/pull/3522/files

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> ☕️

@sushanb sushanb requested a review from vermas2012 March 7, 2022 19:42
@sushanb sushanb requested a review from a team as a code owner March 7, 2022 19:42
@product-auto-label product-auto-label bot added the api: bigtable Issues related to the googleapis/java-bigtable-hbase API. label Mar 7, 2022
vermas2012
vermas2012 requested changes Mar 8, 2022
View reviewed changes
...n/java/com/google/cloud/bigtable/hbase/replication/adapters/IncompatibleMutationAdapter.java Outdated
for (int index = 0; index < cellsToAdapt.size(); index++) {
Cell cell = cellsToAdapt.get(index);
// check whether there is timestamp overflow from HBase -> CBT
if (cell.getTimestamp() >= HBASE_EFFECTIVE_MAX_TIMESTAMP) {
Copy link
Member

@vermas2012 vermas2012 Mar 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it will work out of the box like that.

There are valid deletes, like delete row (

java-bigtable-hbase/bigtable-client-core-parent/bigtable-hbase/src/main/java/com/google/cloud/bigtable/hbase/adapters/DeleteAdapter.java

Line 74 in bf999fa

if (operation.getTimeStamp() != HConstants.LATEST_TIMESTAMP) {
) that require the timestmap to be HConstants.LATEST_TIMESTAMP, which is Long.MAX_VALUE (https://github.com/apache/hbase/blob/master/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java#L672)

Copy link
Contributor Author

@sushanb sushanb Mar 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

vermas2012
vermas2012 requested changes Mar 9, 2022
View reviewed changes
@sushanb sushanb requested a review from vermas2012 March 10, 2022 20:39
vermas2012
vermas2012 requested changes Mar 15, 2022
View reviewed changes
Copy link
Member

@vermas2012 vermas2012 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a test with overflow timestamp on 2.x and 1.x integ tests. You can update an existing test or add a new test, up to you.

vermas2012
vermas2012 approved these changes Mar 16, 2022
View reviewed changes
.../google/cloud/bigtable/hbase1_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java Outdated

for (int i = 0; i < expectedCells.size(); i++) {
Assert.assertNotEquals(
"Timestamp mismatch for row " + TestUtils.ROW_KEY,
Copy link
Member

@vermas2012 vermas2012 Mar 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Timestamps should not match?

Copy link
Contributor Author

@sushanb sushanb Mar 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

.../google/cloud/bigtable/hbase1_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java Outdated
hbaseTable.get(new Get(TestUtils.ROW_KEY).setMaxVersions()).listCells();
List<Cell> actualCells = cbtTable.get(new Get(TestUtils.ROW_KEY).setMaxVersions()).listCells();

for (int i = 0; i < expectedCells.size(); i++) {
Copy link
Member

@vermas2012 vermas2012 Mar 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check size =1 and compare result[0]

Copy link
Contributor Author

@sushanb sushanb Mar 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

.../google/cloud/bigtable/hbase2_x/replication/HbaseToCloudBigtableReplicationEndpointTest.java Outdated
List<Cell> actualCells = cbtTable.get(new Get(TestUtils.ROW_KEY).setMaxVersions()).listCells();

for (int i = 0; i < expectedCells.size(); i++) {
Assert.assertNotEquals(
Copy link
Member

@vermas2012 vermas2012 Mar 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same, don't use array.

Copy link
Contributor Author

@sushanb sushanb Mar 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@mutianf mutianf added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Mar 17, 2022
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Mar 17, 2022
@sushanb sushanb added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Mar 17, 2022
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Mar 17, 2022
@sushanb sushanb added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Mar 17, 2022
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Mar 17, 2022
@sushanb sushanb added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 17, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 17, 2022
@sushanb sushanb merged commit 9c73318 into googleapis:main Mar 17, 2022
@sushanb sushanb deleted the timestamp branch March 17, 2022 19:25
@release-please release-please bot mentioned this pull request Mar 17, 2022
Merged
gcf-merge-on-green bot pushed a commit that referenced this pull request Apr 4, 2022
@release-please
chore(main): release 2.1.0 (#3524)
b61d3fd 
🤖 I have created a release *beep* *boop*
---


## [2.1.0](v2.0.0...v2.1.0) (2022-04-01)


### Features

* [hase-cbt-replication]Add a metrics for hbase<>bigtable timestamp overflow ([#3540](#3540)) ([9c73318](9c73318))
* Adding a detailed readme for HBase replication support. ([#3527](#3527)) ([b406ac5](b406ac5))
* enable dry-run mode for HBase to Cloud Bigtable replication ([#3532](#3532)) ([164738b](164738b))
* Enable HBase to Cloud Bigtable replication  ([#3510](#3510)) ([68d4a01](68d4a01))
* flag a put in future timestamp and record a metrics ([#3534](#3534)) ([7dd7653](7dd7653))
* remove base repl endpoint from hbase 2.x ([#3539](#3539)) ([f748c70](f748c70))


### Bug Fixes

* [hbase-cbt-replication] make hbase2.x endpoint context private ([#3559](#3559)) ([5ec3b18](5ec3b18))
* consistent behavior of customwalentry filter across hbase1.x and hbase2.x ([#3562](#3562)) ([f70b0cb](f70b0cb))
* license and notice files for all the shaded artifacts ([#3565](#3565)) ([86a5643](86a5643))
* minor documentation update ([#3572](#3572)) ([68db6c2](68db6c2))


### Documentation

* fix typos in README and add --region option ([#3520](#3520)) ([c189d46](c189d46))


### Dependencies

* update dependency ch.qos.logback:logback-classic to v1.2.11 ([#3538](#3538)) ([92e8b63](92e8b63))
* update dependency com.google.cloud:google-cloud-bigtable-emulator to v0.142.3 ([#3528](#3528)) ([33a8eba](33a8eba))
* update dependency com.google.cloud:google-cloud-bigtable-emulator to v0.143.0 ([#3543](#3543)) ([2d57f14](2d57f14))
* update dependency net.bytebuddy:byte-buddy to v1.12.8 ([#3502](#3502)) ([77ae2f1](77ae2f1))
* update jmh.version to v1.35 ([#3561](#3561)) ([188738d](188738d))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vermas2012 vermas2012 vermas2012 approved these changes

Assignees

No one assigned

Labels

api: bigtable Issues related to the googleapis/java-bigtable-hbase API.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sushanb @vermas2012 @mutianf @yoshi-kokoro