Kafka to PubSub template #176

KhaninArtur · 2020-10-06T15:59:17Z

This PR adds Kafka to PubSub Flex template.

google-cla · 2020-10-06T15:59:23Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

ilya-kozyrev · 2020-10-06T16:02:24Z

@googlebot I fixed it.

google-cla · 2020-10-06T16:02:31Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

ilya-kozyrev · 2020-10-06T16:05:17Z

@googlebot I consent.

v2/kafka-to-pubsub/README.md

v2/kafka-to-pubsub/src/main/java/com/google/cloud/teleport/v2/options/KafkaToPubsubOptions.java

v2/kafka-to-pubsub/src/main/java/com/google/cloud/teleport/v2/templates/KafkaToPubsub.java

v2/kafka-to-pubsub/src/main/java/com/google/cloud/teleport/v2/options/KafkaToPubsubOptions.java

v2/kafka-to-pubsub/src/main/java/com/google/cloud/teleport/v2/templates/KafkaToPubsub.java

v2/kafka-to-pubsub/src/main/resources/kafka_to_pubsub_metadata.json

prathapreddy123

Didn't notice any changes based on review comments but however marked as fixed. Please revisit all review comments.

v2/kafka-to-pubsub/README.md

v2/kafka-to-pubsub/src/main/java/com/google/cloud/teleport/v2/options/KafkaToPubsubOptions.java

v2/kafka-to-pubsub/src/main/java/com/google/cloud/teleport/v2/templates/KafkaToPubsub.java

KhaninArtur · 2020-10-15T12:23:10Z

@prathapreddy123, hi, thank you very much for your review! It helps us a lot.
Seems like something went wrong with changes notices, sorry for that. Could you please take a look at these commits:

We tried to resolve your comments here.
After your review we added new functionality, could you please take a look at the last actual version of the PR.

google-cla · 2020-10-15T17:54:30Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

AKosolapov · 2020-10-15T18:11:25Z

@googlebot I fixed it.

v2/kafka-to-pubsub/src/main/java/com/google/cloud/teleport/v2/options/KafkaToPubsubOptions.java

v2/kafka-to-pubsub/src/main/java/com/google/cloud/teleport/v2/transforms/FormatTransform.java

v2/kafka-to-pubsub/src/main/java/com/google/cloud/teleport/v2/options/KafkaToPubsubOptions.java

v2/kafka-to-pubsub/src/main/resources/kafka_to_pubsub_metadata.json

prathapreddy123

Thanks for addressing earlier comments. Left few additional comments.

v2/kafka-to-pubsub/README.md

...o-pubsub/src/main/java/com/google/cloud/teleport/v2/kafka/consumer/SslConsumerFactoryFn.java

v2/kafka-to-pubsub/src/main/java/com/google/cloud/teleport/v2/options/KafkaToPubsubOptions.java

v2/kafka-to-pubsub/src/main/java/com/google/cloud/teleport/v2/templates/KafkaToPubsub.java

v2/kafka-to-pubsub/pom.xml

prathapreddy123

Thanks for addressing all changes.

v2/kafka-to-pubsub/src/main/java/com/google/cloud/teleport/v2/templates/Constants.java

prathapreddy123

I do see limited options for scope of unit tests. Can you check if any tests can be added.
Note: Unit tests should not have any external resource dependencies (e.g: GCS, Pub/Sub)

...o-pubsub/src/main/java/com/google/cloud/teleport/v2/kafka/consumer/SslConsumerFactoryFn.java

KhaninArtur · 2020-11-17T16:00:18Z

I do see limited options for scope of unit tests. Can you check if any tests can be added.
Note: Unit tests should not have any external resource dependencies (e.g: GCS, Pub/Sub)

I've checked and managed to add unit tests for Utils.

prathapreddy123

Thanks for your contribution.

google-cla · 2020-11-20T13:44:57Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

sabhyankar

Hello - Please add the missing license headers, rebase and squash commits so we can try this again.

sabhyankar · 2020-11-20T15:04:03Z

v2/kafka-to-pubsub/src/test/java/com/google/cloud/teleport/v2/templates/KafkaToPubsubTest.java

@@ -0,0 +1,70 @@
+package com.google.cloud.teleport.v2.templates;


Please add a license header.

sabhyankar · 2020-11-20T15:04:30Z

...fka-to-pubsub/src/main/java/com/google/cloud/teleport/v2/templates/KafkaPubsubConstants.java

@@ -0,0 +1,32 @@
+package com.google.cloud.teleport.v2.templates;


Please add a license header.

sabhyankar · 2020-11-20T15:05:18Z

v2/kafka-to-pubsub/src/main/java/com/google/cloud/teleport/v2/kafka/consumer/Utils.java

@@ -0,0 +1,151 @@
+package com.google.cloud.teleport.v2.kafka.consumer;


Please add a license header.

Co-authored-by: Ilya Kozyrev <[email protected]> Co-authored-by: Alex Kosolapov <[email protected]>

google-cla · 2020-11-20T15:37:49Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

KhaninArtur · 2020-11-20T15:38:47Z

Hello - Please add the missing license headers, rebase and squash commits so we can try this again.

@sabhyankar Added missing licenses

googlebot · 2020-11-20T15:44:44Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

sabhyankar · 2020-11-20T15:45:58Z

Thank you @KhaninArtur - Can you also see if "[email protected]" can sign the CLA since they are one of the commit authors?

KhaninArtur · 2020-11-20T16:23:26Z

@googlebot I fixed it.

google-cla · 2020-11-20T16:23:30Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

AKosolapov · 2020-11-20T16:25:33Z

@googlebot I consent.

google-cla · 2020-11-20T16:25:38Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

KhaninArtur · 2020-11-20T16:26:50Z

@sabhyankar done.

ilya-kozyrev · 2020-11-20T16:27:19Z

@googlebot I consent.

google-cla · 2020-11-20T16:27:23Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

sabhyankar

LGTM

Thank you all!

Please approve this CL. It will be submitted automatically, and its GitHub pull request will be marked as merged. Imported from GitHub PR #176 This PR adds Kafka to PubSub Flex template. Copybara import of the project: - 7aceebf9219d17becb789b6ccff647ee6ded6420 Implemented Kafka to Pub/Sub template by Artur Khanin <[email protected]> COPYBARA_INTEGRATE_REVIEW=#176 from akvelon:KafkaToPubsubTemplate 7aceebf9219d17becb789b6ccff647ee6ded6420 PiperOrigin-RevId: 343484865

Please approve this CL. It will be submitted automatically, and its GitHub pull request will be marked as merged. Imported from GitHub PR #176 This PR adds Kafka to PubSub Flex template. PiperOrigin-RevId: 345486564

* Wide Row IT for Forward Migration (#170) * Added Missing Datastream Private Connectivity * Review comments fixes (#176) * Move to generic to Base Class * Rename method * Moved Spanner Check to base class * Added Ignore For 5000 Tables * FM code Refactored (#188) * Added FM Low Priority WIde Row Fixws * Added Code Refectored * Code Refecoting Fixes * removed 5K table test fix compilation error * revert changes * Added IT PR Stuck Fixes and Ignore 100MB test as it is holding our Pipeline * removed 100MB flaky test --------- Co-authored-by: taherkl <[email protected]> Co-authored-by: Akash Thawait <[email protected]>

* Wide Row IT for Forward Migration (GoogleCloudPlatform#170) * Added Missing Datastream Private Connectivity * Review comments fixes (GoogleCloudPlatform#176) * Move to generic to Base Class * Rename method * Moved Spanner Check to base class * Added Ignore For 5000 Tables * FM code Refactored (GoogleCloudPlatform#188) * Added FM Low Priority WIde Row Fixws * Added Code Refectored * Code Refecoting Fixes * removed 5K table test fix compilation error * revert changes * Added IT PR Stuck Fixes and Ignore 100MB test as it is holding our Pipeline * removed 100MB flaky test --------- Co-authored-by: taherkl <[email protected]> Co-authored-by: Akash Thawait <[email protected]>

google-cla bot added the cla: no The PR submitter does not have a CLA label Oct 6, 2020

google-cla bot added cla: yes The PR submitter has a CLA and removed cla: no The PR submitter does not have a CLA labels Oct 6, 2020

sabhyankar requested a review from prathapreddy123 October 7, 2020 14:39

prathapreddy123 suggested changes Oct 8, 2020

View reviewed changes

prathapreddy123 reviewed Oct 14, 2020

View reviewed changes

google-cla bot added cla: no The PR submitter does not have a CLA and removed cla: yes The PR submitter has a CLA labels Oct 15, 2020

google-cla bot added cla: yes The PR submitter has a CLA and removed cla: no The PR submitter does not have a CLA labels Oct 15, 2020

KhaninArtur requested a review from prathapreddy123 October 16, 2020 08:50

prathapreddy123 reviewed Oct 22, 2020

View reviewed changes

KhaninArtur requested a review from prathapreddy123 October 26, 2020 14:32

KhaninArtur changed the title ~~(WIP) Kafka to PubSub template~~ Kafka to PubSub template Oct 26, 2020

prathapreddy123 suggested changes Nov 7, 2020

View reviewed changes

manavgarg mentioned this pull request Nov 10, 2020

[BEAM-11065] Apache Beam pipeline example to ingest from Apache Kafka to Google Pub/Sub apache/beam#13112

Merged

4 tasks

prathapreddy123 suggested changes Nov 12, 2020

View reviewed changes

v2/kafka-to-pubsub/src/main/java/com/google/cloud/teleport/v2/templates/Constants.java Outdated Show resolved Hide resolved

KhaninArtur requested a review from prathapreddy123 November 16, 2020 08:45

prathapreddy123 reviewed Nov 16, 2020

View reviewed changes

...o-pubsub/src/main/java/com/google/cloud/teleport/v2/kafka/consumer/SslConsumerFactoryFn.java Show resolved Hide resolved

KhaninArtur requested a review from prathapreddy123 November 17, 2020 16:00

prathapreddy123 approved these changes Nov 17, 2020

View reviewed changes

prathapreddy123 added the Google LGTM Approval of a pull request to be merged into the repository label Nov 17, 2020

google-cla bot added cla: yes The PR submitter has a CLA and removed cla: no The PR submitter does not have a CLA labels Nov 20, 2020

KhaninArtur force-pushed the KafkaToPubsubTemplate branch from 7aceebf to 5603d21 Compare November 20, 2020 13:44

google-cla bot added cla: no The PR submitter does not have a CLA and removed cla: yes The PR submitter has a CLA labels Nov 20, 2020

sabhyankar requested changes Nov 20, 2020

View reviewed changes

sabhyankar removed the Google LGTM Approval of a pull request to be merged into the repository label Nov 20, 2020

Implemented Kafka to Pub/Sub template

13570b5

Co-authored-by: Ilya Kozyrev <[email protected]> Co-authored-by: Alex Kosolapov <[email protected]>

KhaninArtur force-pushed the KafkaToPubsubTemplate branch from 5603d21 to 13570b5 Compare November 20, 2020 15:37

google-cla bot added cla: yes The PR submitter has a CLA and removed cla: no The PR submitter does not have a CLA labels Nov 20, 2020

sabhyankar approved these changes Nov 20, 2020

View reviewed changes

sabhyankar added the Google LGTM Approval of a pull request to be merged into the repository label Nov 20, 2020

prathapreddy123 merged commit 3c1cda6 into GoogleCloudPlatform:master Dec 3, 2020

		@@ -0,0 +1,70 @@
		package com.google.cloud.teleport.v2.templates;

		@@ -0,0 +1,32 @@
		package com.google.cloud.teleport.v2.templates;

		@@ -0,0 +1,151 @@
		package com.google.cloud.teleport.v2.kafka.consumer;

Kafka to PubSub template #176

Kafka to PubSub template #176

Uh oh!

Conversation

KhaninArtur commented Oct 6, 2020

Uh oh!

google-cla bot commented Oct 6, 2020

Uh oh!

ilya-kozyrev commented Oct 6, 2020

Uh oh!

google-cla bot commented Oct 6, 2020

Uh oh!

ilya-kozyrev commented Oct 6, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

prathapreddy123 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

KhaninArtur commented Oct 15, 2020

Uh oh!

google-cla bot commented Oct 15, 2020

Uh oh!

AKosolapov commented Oct 15, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

prathapreddy123 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

prathapreddy123 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

prathapreddy123 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

KhaninArtur commented Nov 17, 2020

Uh oh!

prathapreddy123 left a comment

Choose a reason for hiding this comment

Uh oh!

google-cla bot commented Nov 20, 2020

Uh oh!

sabhyankar left a comment

Choose a reason for hiding this comment

Uh oh!

sabhyankar Nov 20, 2020

Choose a reason for hiding this comment

Uh oh!

sabhyankar Nov 20, 2020