PubSub to Elasticsearch dynamic template, along with refactoring for v2. by pritukam · Pull Request #58 · GoogleCloudPlatform/DataflowTemplates

pritukam · 2019-09-27T13:53:48Z

No description provided.

googlebot · 2019-09-27T13:53:51Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

sabhyankar · 2019-09-27T14:21:51Z

@pritukam - Can you sign the CLA?

pritukam · 2019-09-27T14:55:12Z

@googlebot I signed it!

pritukam · 2019-09-27T14:55:54Z

@sabhyankar signed the CLA

googlebot · 2019-10-02T15:38:12Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

chrlarsen · 2019-10-16T15:07:28Z

@sabhyankar the template has been updated to include udf, deadlettering and field extract support.

sabhyankar · 2019-10-29T19:20:01Z

v2/common/src/main/java/com/google/cloud/teleport/v2/transforms/ElasticsearchTransforms.java

Since this file is being added in PR-61 as well, we will rebase after that PR is merged.

sabhyankar · 2019-10-29T19:24:45Z

v2/common/src/main/java/com/google/cloud/teleport/v2/transforms/JavascriptTextTransformer.java

should we just call these:
udf-transform-success-count
udf-transform-failed-count

Counters have been renamed.

sabhyankar · 2019-10-29T19:29:45Z

v2/pubsub-to-elasticsearch/README.md

We want to mention that an optional UDF can also be applied.

Udf has been added.

sabhyankar · 2019-10-29T19:30:06Z

v2/pubsub-to-elasticsearch/README.md

We only want to read from a subscription. Please remove the option to read from a topic :)

Topic functionality has been removed here and other references.

sabhyankar · 2019-10-30T13:25:09Z

v2/pubsub-to-elasticsearch/README.md

host(s) since it could be a cluster

host(s) has been added.

sabhyankar · 2019-10-30T13:49:45Z

...lasticsearch/src/main/java/com/google/cloud/teleport/v2/templates/PubSubToElasticsearch.java

I am not sure why we would use a stringified attribute map if the payload is empty?

@sabhyankar I believe the original thinking was that if the payload is empty then there would be data in the attribute map. If the message is empty then we could throw an error and catch it in the deadletter table.

sabhyankar · 2019-10-30T13:50:46Z

...lasticsearch/src/main/java/com/google/cloud/teleport/v2/templates/PubSubToElasticsearch.java

Can you check whether we can reuse the Gson object instead of creating a new one for every message?

sabhyankar · 2019-10-30T14:04:26Z

...icsearch/src/test/java/com/google/cloud/teleport/v2/templates/PubSubToElasticsearchTest.java

We are only transforming and not necessarily 'filtering'

sabhyankar · 2019-10-30T14:09:09Z

...icsearch/src/test/java/com/google/cloud/teleport/v2/templates/PubSubToElasticsearchTest.java

Why not directly use FAILSAFE_ELEMENT_CODER instead of assigning it to failsafeElementCoder?

sabhyankar · 2019-10-30T14:09:46Z

...icsearch/src/test/java/com/google/cloud/teleport/v2/templates/PubSubToElasticsearchTest.java

See comment in the previous test

googlebot · 2019-11-06T00:16:06Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

googlebot · 2019-11-06T00:36:25Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

chrlarsen · 2019-11-06T15:40:40Z

@sabhyankar comments have been addressed. PR has been rebased onto PR-61 and commits have been squashed.

chrlarsen

Added error converter.

PiperOrigin-RevId: 281057367

* Dml integration (#53) * Added extensive UT Added extensive UT * Cassandra pr bug fixes (#57) * Cassandra Consolidate Unit Test case and Regression testing fixes (#58) * Added Mapping fixes * Added Spoltles fixes * Added Consolidated fixes * Added TODO * Addess Data and Time * Cassandra pr bug fixes (#64) * Handle TypeHandler Parsing issue fixes (#65) Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> * Added Safe handle (#68) * Handle LocalTime For Time Data Type In Cassandra (#69) * Cassandra pr bug fixes (#70) * Handle Timestamp Fixes (#72) * Added Code Combined in a single way * Address The Unwanted Hop * Cassandra pr bug fixes (#75) * Added PR Review Comments * Remove NamesCol Dependecy as spannerTableName is same as In Given Mapping * Added spannerTableId for fetching Mapping * Removed SpannerToID and also Updated Session file with proper structure * Timestamp in milisecond * removed assertNotNull from UT wherever possible * Added Fixes * Added Note Instead of Question * -- review fixes (#78) * Added Bytes to hex to blob conversion * Handling Bytes as Binary encoded As of now * Passing Null Value to Primary Key as well for cassandra * Added UT fixes * Added UT refectoring * Reverse merge confict fixes --------- Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> Co-authored-by: Akash Thawait <aakash@ollion.com>

googlebot added the cla: no The PR submitter does not have a CLA label Sep 27, 2019

pritukam closed this Sep 27, 2019

pritukam reopened this Sep 27, 2019

sabhyankar self-requested a review October 2, 2019 15:37

googlebot added cla: yes The PR submitter has a CLA and removed cla: no The PR submitter does not have a CLA labels Oct 2, 2019

pritukam force-pushed the master branch from 4177ea2 to 81bf602 Compare October 3, 2019 04:31

chrlarsen force-pushed the master branch from 894350a to 78d872d Compare October 16, 2019 15:05

chrlarsen force-pushed the master branch from 78d872d to 4016d5b Compare October 16, 2019 15:26

sabhyankar requested changes Oct 30, 2019

View reviewed changes

chrlarsen force-pushed the master branch from 4016d5b to bb5efb6 Compare November 6, 2019 00:16

googlebot added cla: no The PR submitter does not have a CLA and removed cla: yes The PR submitter has a CLA labels Nov 6, 2019

chrlarsen force-pushed the master branch from bb5efb6 to f2e89f7 Compare November 6, 2019 00:36

googlebot added cla: yes The PR submitter has a CLA and removed cla: no The PR submitter does not have a CLA labels Nov 6, 2019

PubSub to Elasticsearch dynamic template, along with refactoring for v2

81e3e45

chrlarsen force-pushed the master branch from f2e89f7 to 81e3e45 Compare November 6, 2019 15:34

chrlarsen reviewed Nov 6, 2019

View reviewed changes

sabhyankar approved these changes Nov 8, 2019

View reviewed changes

sabhyankar added the Google LGTM Approval of a pull request to be merged into the repository label Nov 8, 2019

pritukam force-pushed the master branch from b1dc7f7 to 81e3e45 Compare November 18, 2019 14:10

sabhyankar added a commit that referenced this pull request Nov 18, 2019

Merge pull request #58 from pritukam:master

e4dc82c

PiperOrigin-RevId: 281057367

sabhyankar merged commit 81e3e45 into GoogleCloudPlatform:master Nov 18, 2019

Conversation

pritukam commented Sep 27, 2019

Uh oh!

googlebot commented Sep 27, 2019

What to do if you already signed the CLA

Individual signers

Corporate signers

Uh oh!

sabhyankar commented Sep 27, 2019

Uh oh!

pritukam commented Sep 27, 2019

Uh oh!

pritukam commented Sep 27, 2019

Uh oh!

googlebot commented Oct 2, 2019

Uh oh!

chrlarsen commented Oct 16, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

googlebot commented Nov 6, 2019

Uh oh!

googlebot commented Nov 6, 2019

Uh oh!

chrlarsen commented Nov 6, 2019

Uh oh!

chrlarsen left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants