Merged
Conversation
* Upgrade upload-artifact version * Fix naming * naming * naming * naming
* Add support for Search Indexes * Reverted order of where clause * Add tests and fix formatting * Fixing tests and addressing comments * Addressing comments and simplifying code * Fixed formatting
* Adding List and Map tables in UT * Support for non-nested Cassandra List and Set
* Testing spanner terraform validator issue * test change * minor fix * removing extra changes
* Added VIRTUAL column support to PG dataflow * Ran spotless:apply to fix formatting * Added a virtual column to the pgSearchIndexes() test
* Fix handling of empty arrays, json arrays and jsonb arrays. * Add support for INTERVAL types. --------- Co-authored-by: Claude <cvandermerwe@google.com>
* Update terraform spanner-to-cassandra (#73) * update readme * pr review comments addressed * pr review fixes * Add tf validator path and revert code * PR review fixes * fix space issue * revert formatting change --------- Co-authored-by: Nasir Nooruddin <nasir@nasir.id> Co-authored-by: taherkl <taher.lakdawala@ollion.com>
* Add UUID support in import/export templates * Add UUID support in import/export templates * Add UUID support in import/export templates * Spotless fix * Separate out UUID Avro integration tests * Separate out UUID CSV import test * Fix ImportPipelineIT tests * Separate out UUID SpannerToTextIT tests * Separate out UUID SpannerToTextIT tests * Add more unit tests * Remove println * Add todos for cleanups * Add null tests * Add comment * Associate bug id with todos * Add todos * Fix test * Add todos
* updated the readme doc (#76) Documentation for Cassandra Reverse Replication * case change * fix rebase * PR review comment fixed --------- Co-authored-by: Vivek Yadav <105432992+VivekY1098@users.noreply.github.com> Co-authored-by: taherkl <taher.lakdawala@ollion.com>
* [Reverse Replication] Adding load test for custom transformation * minor fix * addressing review comments
* Add IT for Cassandra (#86) * Add IT for Cassandra Reverse Replication * Added IT Fixes * Added IT fixes * removed logger * removed unwanted system log * Handle Catch * Handle with safehandler * Running load test * revert unwanted commit * Added IT FIXES * Added UT and removed unwanted SOUT * Added One to Many Datatype Transformation for IT (#90) Added One to Many Datatype Transformation for IT Fix PR review comments * Handle IT for MYSQL * Added ByteBuffer TO BigInteger * Cassandra rr custom transfornation it test (#96) * PR Review Comments (#97) * Splotless fixes * Handle Retry Missing Exception Category * Rebase Issue fixes --------- Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com>
There is no clear value add for the error message, simply pushing the raw data is the best way to handle such custom fields
taherkl
pushed a commit
that referenced
this pull request
Feb 18, 2025
* sync upstream/main (#98) * Custom transformation fixes * Added Custom Transformation * Added Custom Transformation * Added Fixes * Address null to all columns * Added Null Assert * Added Timeout fixes * Added Spotless fixes * reverse merge the main * Added Custom Fixes * Added Drop Keys ---------
pawankashyapollion
added a commit
that referenced
this pull request
Feb 18, 2025
commit aee07b4 Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 12:04:19 2025 +0530 RR custom transformation feature (#102) * sync upstream/main (#98) * Custom transformation fixes * Added Custom Transformation * Added Custom Transformation * Added Fixes * Address null to all columns * Added Null Assert * Added Timeout fixes * Added Spotless fixes * reverse merge the main * Added Custom Fixes * Added Drop Keys ---------
pawankashyapollion
added a commit
that referenced
this pull request
Feb 18, 2025
commit 2360a1b Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 17:35:02 2025 +0530 Added Custom Transformation UT commit 72d104b Merge: 2735241 3f7e766 Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 17:29:22 2025 +0530 Merge branch 'cassandra_rr_custom_transformation_it' into rr-custom-transformation-feature commit 2735241 Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 17:28:24 2025 +0530 Added Null Pointer Fixes commit 3f7e766 Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 16:48:26 2025 +0530 Null fix commit 4a4bdbd Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 15:43:32 2025 +0530 Added Null Fixes commit e721049 Merge: a694901 df151c9 Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 15:38:44 2025 +0530 Merge branch 'cassandra_rr_custom_transformation_it' into rr-custom-transformation-feature commit a694901 Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 15:33:32 2025 +0530 remove unwanted commit df151c9 Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 15:00:34 2025 +0530 Issue fix commit a49b37d Merge: feca407 5b599a4 Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 14:19:22 2025 +0530 Merge branch 'cassandra_rr_custom_transformation_it' into rr-custom-transformation-feature commit feca407 Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 14:18:56 2025 +0530 Added Custom and Null Pointer Exception handling commit dbf7671 Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 14:15:45 2025 +0530 Added Fixes commit 5b599a4 Merge: e9f0beb 810e553 Author: Taher Lakdawala <78196491+taherkl@users.noreply.github.com> Date: Tue Feb 18 13:16:21 2025 +0530 Merge pull request #109 from GoogleCloudPlatform/main Referesh `SpannerToSourceDbCustomTransformationIT` tables for re-runs… commit e9f0beb Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 13:12:40 2025 +0530 Move null test case into previous tests Move null test case into previous tests commit 38dcf46 Merge: c87c4bd aee07b4 Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 13:09:27 2025 +0530 Merge branch 'cassandra_rr_custom_transformation_it' into rr-custom-transformation-feature commit c87c4bd Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 13:06:18 2025 +0530 Squashed commit of the following: commit aee07b4 Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 12:04:19 2025 +0530 RR custom transformation feature (#102) * sync upstream/main (#98) * Custom transformation fixes * Added Custom Transformation * Added Custom Transformation * Added Fixes * Address null to all columns * Added Null Assert * Added Timeout fixes * Added Spotless fixes * reverse merge the main * Added Custom Fixes * Added Drop Keys --------- commit e9e54d0 Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 12:45:38 2025 +0530 Added Check style fixes commit db8c8c1 Merge: 8a24829 dc5e904 Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 12:33:59 2025 +0530 Merge branch 'main' into rr-custom-transformation-feature commit 8a24829 Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 12:28:01 2025 +0530 Cosolidate Null with same test case commit aee07b4 Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 12:04:19 2025 +0530 RR custom transformation feature (#102) * sync upstream/main (#98) * Custom transformation fixes * Added Custom Transformation * Added Custom Transformation * Added Fixes * Address null to all columns * Added Null Assert * Added Timeout fixes * Added Spotless fixes * reverse merge the main * Added Custom Fixes * Added Drop Keys --------- commit 3ba25ed Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 11:39:30 2025 +0530 Added Drop Keys commit 252d5b9 Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Tue Feb 18 11:38:39 2025 +0530 Added Custom Fixes commit 8778c42 Merge: 2ca52bc f54e3de Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Mon Feb 17 18:11:56 2025 +0530 Merge branch 'cassandra_rr_custom_transformation_it' into rr-custom-transformation-feature commit 2ca52bc Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Mon Feb 17 18:07:38 2025 +0530 reverse merge the main commit 1bdca73 Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Mon Feb 17 17:34:55 2025 +0530 Added Spotless fixes commit 33e561e Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Mon Feb 17 17:33:07 2025 +0530 Added Timeout fixes commit c83f2e9 Author: pawankashyapollion <v-pawan.kumar@ollion.com> Date: Mon Feb 17 16:59:08 2025 +0530 Added Null Assert
taherkl
added a commit
that referenced
this pull request
Mar 4, 2025
…latform#2201) * RR custom transformation feature (#102) * sync upstream/main (#98) * Custom transformation fixes * Added Custom Transformation * Added Custom Transformation * Added Fixes * Address null to all columns * Added Null Assert * Added Timeout fixes * Added Spotless fixes * reverse merge the main * Added Custom Fixes * Added Drop Keys --------- * Move null test case into previous tests Move null test case into previous tests * Issue fix * Null fix * Null fix for varint * Removed unwanted code * Added Bytes Fixes * Missing Bytes Error Code Pushed * PR comment fixes (#114) * Regex fixes to validate binary string vs base64 * Added Review Comments (#122) * Added Review Comments * Address TODO * Added POM File * Added Ignore Case in table name check * Reference Mismatch * Added full_name as we are skiping extra column and adding as null * Schema Fixes * Added Missing Alignment issue with CustomTransformation * Removed binaryString encoded Support * removed unwanted log * Keep Log Level As warm --------- Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com>
akashthawaitcc
added a commit
that referenced
this pull request
Mar 17, 2025
* * Addition of Load Tests in SpannerToSourceDB For Cassandra (#89) * Addition of Load Tests in SpannerToSourceDB For Cassandra * Address Merge conflict * Added LT Refectored (#92) * Added POM Dependecies * sync upstream/main (#98) * RR LOAD TEST FIXES (#101) * Resolved PR comments (#115) * Added Module Dependency Fixes * Added Copyrigh * Added missing commit * Enhanced Retry Logic (GoogleCloudPlatform#2196) Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> * Adding support for Cassandra map (GoogleCloudPlatform#2209) * Adding test table for Map types * Adding support for cassandra map * changes (GoogleCloudPlatform#2212) * Fix inf issues in Datastream reader (GoogleCloudPlatform#2213) * add inf fix to ts fixing bug on timestamp type * Update FormatDatastreamRecordToJson.java use equals not == * Upgrade Beam version to 2.63.0 (GoogleCloudPlatform#2206) * Upgrade Beam version to 2.63.0 * add cache factory to local spanner io * remove cache pass to ReadChangeStreamPartitionDoFn * Add unimplemented/unused stubs to TestChangeStreamMutation * Use java Instant instead of threeten Instant * Fix low watermark setter call * fix testchangestreammutation * Recreate validation PR * fix import order * linux-env requirements files * fix v1 pom * SkipShade for Spanner common module (GoogleCloudPlatform#2194) * Add load test for cross db txn (GoogleCloudPlatform#2199) * Add load test for cross db txn * Change test timeout to 2 days * Revert spanner-pr.yml changes * Report Lineage for CsvToBigQuery template (GoogleCloudPlatform#2205) * Report Lineage for CsvToBigQuery template * Reply beampr-32662 to CsvConverters * Spanner Import/Export INTERLEAVE IN (GoogleCloudPlatform#2128) * Changes to write read interleave type from information schema, and write/read the type to/from avro. No change to tests yet. * Fix warnings, bug in InformationSchemaScanner, and bug in AvroSchemaToDdlConverter. Also properly default to IN PARENT when emitting ddl, in case the interleave type is not set (really only necessary for tests, since otherwise it will always be set. * Set interleaveType in InfoSchemaScanner * Style fixes, and only generate INTERLEAVE IN ddl for gsql. * another style fix - remove unused import * Make conditions more readable and add comments * Tests * Fix condition * Add interleave in table to ExportPipelineIT * Add SCRAM-SHA-512 authentication support to Kafka templates (GoogleCloudPlatform#2181) Added SCRAM-SHA-512 authentication support to Kafka to Kafka, Kafka to GCS, and Kafka to BigQuery templates. * Update the required Java version in the base doc, then regenerate docs (GoogleCloudPlatform#2218) * Update java requirement * Generate docs * Post 2.63.0 fixes (GoogleCloudPlatform#2216) * Uncomment kinesis * bump protoc to 4 in v2 * bug-fix: Use jdbc connection properties for reverse migration (GoogleCloudPlatform#2198) * changes * Changes * changes * changes * docs * Support partitioned reads for DateTime column type in JDBC to BigQuery template (GoogleCloudPlatform#2084) * Support partitioned reads for DateTime column type * minor changes * Support backward compatibility and timezone in lower/upper bounds * removed test cases for bounds in unit tests as derby does not support timezone * Fixed typo's in JdbcToBigQuery.java Corrected testcases expected output * Corrected unit test cases expected output. Removed DateTime integration test cases. * Corrected a typo * Added default value for partitionColumnType * spotless apply * Disabling flaky test to unblock dataflow release (GoogleCloudPlatform#2220) * disabling flaky test to unblock dataflow release * minor change * Add logic to skip runnerV2 for the ITs (GoogleCloudPlatform#2219) * skip runnerv2 tests in TemplateTestBase * add logic to skip use_runner_v2 experiment in the launchTemplate * Remove redundant property skipRunnerV2Test in pom and update logic in launchTemplate * Fix format violations using mvn spotless:apply * Add warning about caching with plugin (GoogleCloudPlatform#2221) * [DatastreamToSpanner] Spanner Exception handling (GoogleCloudPlatform#2185) * SpannerExceptionClassifier class and IT * Unit tests and Integration tests * Formatting changes * Correcting UT * Unit test for SpannerResourceManager * Creating Spanner Migration Exception * Formatting changes * Correcting tests * Addressing comments * Correcting a UT * Addressing comments * Changing SpannerMigrationException to extend RuntimeException * Added Cassandra Resource Manager Refectoring and removed Generics * Added Keyspace Voilation fixes * minor changes * Create session for row check --------- Co-authored-by: taherkl <taher.lakdawala@ollion.com> Co-authored-by: Akash Thawait <aakash@ollion.com> Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com> Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com> Co-authored-by: Dylan Hercher <dylanhercher@gmail.com> Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Co-authored-by: Yi Hu <yathu@google.com> Co-authored-by: Deep1998 <deepchowdhury1998@gmail.com> Co-authored-by: jjfox15 <seanfox@google.com> Co-authored-by: vgnanasekaran <venku123@gmail.com> Co-authored-by: Danny McCormick <dannymccormick@google.com> Co-authored-by: Sharan Teja M <sharantej957@gmail.com> Co-authored-by: shreyakhajanchi <92910380+shreyakhajanchi@users.noreply.github.com> Co-authored-by: Rudra-Gujarathi <rudragujarathi@google.com> Co-authored-by: Derrick Williams <derrickaw@google.com> Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com>
akashthawaitcc
added a commit
that referenced
this pull request
Mar 17, 2025
* * Addition of Load Tests in SpannerToSourceDB For Cassandra (#89) * Addition of Load Tests in SpannerToSourceDB For Cassandra * Address Merge conflict * Added LT Refectored (#92) * Added POM Dependecies * sync upstream/main (#98) * RR LOAD TEST FIXES (#101) * Resolved PR comments (#115) * Added Module Dependency Fixes * Added Copyrigh * Added missing commit * Enhanced Retry Logic (GoogleCloudPlatform#2196) Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> * Adding support for Cassandra map (GoogleCloudPlatform#2209) * Adding test table for Map types * Adding support for cassandra map * changes (GoogleCloudPlatform#2212) * Fix inf issues in Datastream reader (GoogleCloudPlatform#2213) * add inf fix to ts fixing bug on timestamp type * Update FormatDatastreamRecordToJson.java use equals not == * Upgrade Beam version to 2.63.0 (GoogleCloudPlatform#2206) * Upgrade Beam version to 2.63.0 * add cache factory to local spanner io * remove cache pass to ReadChangeStreamPartitionDoFn * Add unimplemented/unused stubs to TestChangeStreamMutation * Use java Instant instead of threeten Instant * Fix low watermark setter call * fix testchangestreammutation * Recreate validation PR * fix import order * linux-env requirements files * fix v1 pom * SkipShade for Spanner common module (GoogleCloudPlatform#2194) * Add load test for cross db txn (GoogleCloudPlatform#2199) * Add load test for cross db txn * Change test timeout to 2 days * Revert spanner-pr.yml changes * Report Lineage for CsvToBigQuery template (GoogleCloudPlatform#2205) * Report Lineage for CsvToBigQuery template * Reply beampr-32662 to CsvConverters * Spanner Import/Export INTERLEAVE IN (GoogleCloudPlatform#2128) * Changes to write read interleave type from information schema, and write/read the type to/from avro. No change to tests yet. * Fix warnings, bug in InformationSchemaScanner, and bug in AvroSchemaToDdlConverter. Also properly default to IN PARENT when emitting ddl, in case the interleave type is not set (really only necessary for tests, since otherwise it will always be set. * Set interleaveType in InfoSchemaScanner * Style fixes, and only generate INTERLEAVE IN ddl for gsql. * another style fix - remove unused import * Make conditions more readable and add comments * Tests * Fix condition * Add interleave in table to ExportPipelineIT * Add SCRAM-SHA-512 authentication support to Kafka templates (GoogleCloudPlatform#2181) Added SCRAM-SHA-512 authentication support to Kafka to Kafka, Kafka to GCS, and Kafka to BigQuery templates. * Update the required Java version in the base doc, then regenerate docs (GoogleCloudPlatform#2218) * Update java requirement * Generate docs * Post 2.63.0 fixes (GoogleCloudPlatform#2216) * Uncomment kinesis * bump protoc to 4 in v2 * bug-fix: Use jdbc connection properties for reverse migration (GoogleCloudPlatform#2198) * changes * Changes * changes * changes * docs * Support partitioned reads for DateTime column type in JDBC to BigQuery template (GoogleCloudPlatform#2084) * Support partitioned reads for DateTime column type * minor changes * Support backward compatibility and timezone in lower/upper bounds * removed test cases for bounds in unit tests as derby does not support timezone * Fixed typo's in JdbcToBigQuery.java Corrected testcases expected output * Corrected unit test cases expected output. Removed DateTime integration test cases. * Corrected a typo * Added default value for partitionColumnType * spotless apply * Disabling flaky test to unblock dataflow release (GoogleCloudPlatform#2220) * disabling flaky test to unblock dataflow release * minor change * Add logic to skip runnerV2 for the ITs (GoogleCloudPlatform#2219) * skip runnerv2 tests in TemplateTestBase * add logic to skip use_runner_v2 experiment in the launchTemplate * Remove redundant property skipRunnerV2Test in pom and update logic in launchTemplate * Fix format violations using mvn spotless:apply * Add warning about caching with plugin (GoogleCloudPlatform#2221) * [DatastreamToSpanner] Spanner Exception handling (GoogleCloudPlatform#2185) * SpannerExceptionClassifier class and IT * Unit tests and Integration tests * Formatting changes * Correcting UT * Unit test for SpannerResourceManager * Creating Spanner Migration Exception * Formatting changes * Correcting tests * Addressing comments * Correcting a UT * Addressing comments * Changing SpannerMigrationException to extend RuntimeException * Added Cassandra Resource Manager Refectoring and removed Generics * Added Keyspace Voilation fixes * minor changes * Create session for row check --------- Co-authored-by: taherkl <taher.lakdawala@ollion.com> Co-authored-by: Akash Thawait <aakash@ollion.com> Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com> Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com> Co-authored-by: Dylan Hercher <dylanhercher@gmail.com> Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Co-authored-by: Yi Hu <yathu@google.com> Co-authored-by: Deep1998 <deepchowdhury1998@gmail.com> Co-authored-by: jjfox15 <seanfox@google.com> Co-authored-by: vgnanasekaran <venku123@gmail.com> Co-authored-by: Danny McCormick <dannymccormick@google.com> Co-authored-by: Sharan Teja M <sharantej957@gmail.com> Co-authored-by: shreyakhajanchi <92910380+shreyakhajanchi@users.noreply.github.com> Co-authored-by: Rudra-Gujarathi <rudragujarathi@google.com> Co-authored-by: Derrick Williams <derrickaw@google.com> Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com>
akashthawaitcc
added a commit
that referenced
this pull request
Mar 17, 2025
* sync upstream/main (#98) * Enhanced Retry Logic (GoogleCloudPlatform#2196) Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> * Adding support for Cassandra map (GoogleCloudPlatform#2209) * Adding test table for Map types * Adding support for cassandra map * changes (GoogleCloudPlatform#2212) * Fix inf issues in Datastream reader (GoogleCloudPlatform#2213) * add inf fix to ts fixing bug on timestamp type * Update FormatDatastreamRecordToJson.java use equals not == * Upgrade Beam version to 2.63.0 (GoogleCloudPlatform#2206) * Upgrade Beam version to 2.63.0 * add cache factory to local spanner io * remove cache pass to ReadChangeStreamPartitionDoFn * Add unimplemented/unused stubs to TestChangeStreamMutation * Use java Instant instead of threeten Instant * Fix low watermark setter call * fix testchangestreammutation * Recreate validation PR * fix import order * linux-env requirements files * fix v1 pom * SkipShade for Spanner common module (GoogleCloudPlatform#2194) * Add load test for cross db txn (GoogleCloudPlatform#2199) * Add load test for cross db txn * Change test timeout to 2 days * Revert spanner-pr.yml changes * Report Lineage for CsvToBigQuery template (GoogleCloudPlatform#2205) * Report Lineage for CsvToBigQuery template * Reply beampr-32662 to CsvConverters * Spanner Import/Export INTERLEAVE IN (GoogleCloudPlatform#2128) * Changes to write read interleave type from information schema, and write/read the type to/from avro. No change to tests yet. * Fix warnings, bug in InformationSchemaScanner, and bug in AvroSchemaToDdlConverter. Also properly default to IN PARENT when emitting ddl, in case the interleave type is not set (really only necessary for tests, since otherwise it will always be set. * Set interleaveType in InfoSchemaScanner * Style fixes, and only generate INTERLEAVE IN ddl for gsql. * another style fix - remove unused import * Make conditions more readable and add comments * Tests * Fix condition * Add interleave in table to ExportPipelineIT * Add SCRAM-SHA-512 authentication support to Kafka templates (GoogleCloudPlatform#2181) Added SCRAM-SHA-512 authentication support to Kafka to Kafka, Kafka to GCS, and Kafka to BigQuery templates. * Update the required Java version in the base doc, then regenerate docs (GoogleCloudPlatform#2218) * Update java requirement * Generate docs * Post 2.63.0 fixes (GoogleCloudPlatform#2216) * Uncomment kinesis * bump protoc to 4 in v2 * bug-fix: Use jdbc connection properties for reverse migration (GoogleCloudPlatform#2198) * changes * Changes * changes * changes * docs * Support partitioned reads for DateTime column type in JDBC to BigQuery template (GoogleCloudPlatform#2084) * Support partitioned reads for DateTime column type * minor changes * Support backward compatibility and timezone in lower/upper bounds * removed test cases for bounds in unit tests as derby does not support timezone * Fixed typo's in JdbcToBigQuery.java Corrected testcases expected output * Corrected unit test cases expected output. Removed DateTime integration test cases. * Corrected a typo * Added default value for partitionColumnType * spotless apply * Disabling flaky test to unblock dataflow release (GoogleCloudPlatform#2220) * disabling flaky test to unblock dataflow release * minor change * Add logic to skip runnerV2 for the ITs (GoogleCloudPlatform#2219) * skip runnerv2 tests in TemplateTestBase * add logic to skip use_runner_v2 experiment in the launchTemplate * Remove redundant property skipRunnerV2Test in pom and update logic in launchTemplate * Fix format violations using mvn spotless:apply * Add warning about caching with plugin (GoogleCloudPlatform#2221) * [DatastreamToSpanner] Spanner Exception handling (GoogleCloudPlatform#2185) * SpannerExceptionClassifier class and IT * Unit tests and Integration tests * Formatting changes * Correcting UT * Unit test for SpannerResourceManager * Creating Spanner Migration Exception * Formatting changes * Correcting tests * Addressing comments * Correcting a UT * Addressing comments * Changing SpannerMigrationException to extend RuntimeException * Remove Python version from `pom.xml` (GoogleCloudPlatform#2234) * Update pom.xml * Update pom.xml * Add SkipRunnerV2Test category to JmsToPubsubIT and PubSubCdcToBigQueryIT (GoogleCloudPlatform#2235) * Fix a bug in CSVToBigQuery where commas in fields are not handled correctly. (GoogleCloudPlatform#2229) * Attempt to fix csv bug where commas in fields are not handled correctly. * Replace ImmutableList with Iterable * Add tests to cover the scenario of commas within quotes. * Polish tests and add a test case to cover csv without headers. * Update Dockerfile-template-yaml (GoogleCloudPlatform#2222) * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Adding All Datatypes IT for Cassandra Migration (GoogleCloudPlatform#2230) * Add IF NOT EXISTS clause for spanner ddls used in ITs (GoogleCloudPlatform#2237) * Enable DatastreamToSpannerIT with if not exists clause * Update Datastream to Spanner IT spanner schemas with if not exists * Update BULK IT spanner schemas with if not exists * Update reverse replications ITs spanner schemas with if not exists * Add space after if not exists * Using set of random buckets for spanner ITs (GoogleCloudPlatform#2223) * Using set of random buckets for spanner ITs * reverse replication test * checkstyle fix * adding more buckets * removed ignore for testing * spotless fix * fixing UT * skip the flaky test again * addressing comments * spotless fix * Add promote artifact method in release plugin (GoogleCloudPlatform#2227) * Add promote artifact method * address comments; also fixed stagingArtifactRegistry support us.gcr.io * Consolidate method * fixed default DLQ path (GoogleCloudPlatform#2241) * Fix stagingArtifactRegistry support raw us.gcr.io artifact registry (GoogleCloudPlatform#2243) * Print error response on wget call (GoogleCloudPlatform#2245) * This helps debugging e.g. permission issues * Moving local spanner io to a different namespace (GoogleCloudPlatform#2231) * Removed LocalSpannerIO * Moving all of SpannerIO into Teleport * Copying tests also * Updated tests and excluded SpannerIO changestream from coverage checks * spotless apply * Excluding coverage check * Adding retry settings which were overwritten in LocalSpannerAccessor * Changing deprecated retrysettings function calls * Added warning and TODO comments to remove the local copy * spotless * Adding Cassandra Type Options to IT test (GoogleCloudPlatform#2242) * Bump timeouts for tests involving FKs/interleaved dependenceis (GoogleCloudPlatform#2239) * Bump timeouts for datastream to spanner test * Update FK timeout for reverse template * Bump timeouts for old reverse repl template * Enable SpannerToSourceDbInterleaveMultiShardIT * Revert loadtest timeout * Load Tests - Cassandra Reverse Replication (GoogleCloudPlatform#2163) * * Addition of Load Tests in SpannerToSourceDB For Cassandra (#89) * Addition of Load Tests in SpannerToSourceDB For Cassandra * Address Merge conflict * Added LT Refectored (#92) * Added POM Dependecies * sync upstream/main (#98) * RR LOAD TEST FIXES (#101) * Resolved PR comments (#115) * Added Module Dependency Fixes * Added Copyrigh * Added missing commit * Enhanced Retry Logic (GoogleCloudPlatform#2196) Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> * Adding support for Cassandra map (GoogleCloudPlatform#2209) * Adding test table for Map types * Adding support for cassandra map * changes (GoogleCloudPlatform#2212) * Fix inf issues in Datastream reader (GoogleCloudPlatform#2213) * add inf fix to ts fixing bug on timestamp type * Update FormatDatastreamRecordToJson.java use equals not == * Upgrade Beam version to 2.63.0 (GoogleCloudPlatform#2206) * Upgrade Beam version to 2.63.0 * add cache factory to local spanner io * remove cache pass to ReadChangeStreamPartitionDoFn * Add unimplemented/unused stubs to TestChangeStreamMutation * Use java Instant instead of threeten Instant * Fix low watermark setter call * fix testchangestreammutation * Recreate validation PR * fix import order * linux-env requirements files * fix v1 pom * SkipShade for Spanner common module (GoogleCloudPlatform#2194) * Add load test for cross db txn (GoogleCloudPlatform#2199) * Add load test for cross db txn * Change test timeout to 2 days * Revert spanner-pr.yml changes * Report Lineage for CsvToBigQuery template (GoogleCloudPlatform#2205) * Report Lineage for CsvToBigQuery template * Reply beampr-32662 to CsvConverters * Spanner Import/Export INTERLEAVE IN (GoogleCloudPlatform#2128) * Changes to write read interleave type from information schema, and write/read the type to/from avro. No change to tests yet. * Fix warnings, bug in InformationSchemaScanner, and bug in AvroSchemaToDdlConverter. Also properly default to IN PARENT when emitting ddl, in case the interleave type is not set (really only necessary for tests, since otherwise it will always be set. * Set interleaveType in InfoSchemaScanner * Style fixes, and only generate INTERLEAVE IN ddl for gsql. * another style fix - remove unused import * Make conditions more readable and add comments * Tests * Fix condition * Add interleave in table to ExportPipelineIT * Add SCRAM-SHA-512 authentication support to Kafka templates (GoogleCloudPlatform#2181) Added SCRAM-SHA-512 authentication support to Kafka to Kafka, Kafka to GCS, and Kafka to BigQuery templates. * Update the required Java version in the base doc, then regenerate docs (GoogleCloudPlatform#2218) * Update java requirement * Generate docs * Post 2.63.0 fixes (GoogleCloudPlatform#2216) * Uncomment kinesis * bump protoc to 4 in v2 * bug-fix: Use jdbc connection properties for reverse migration (GoogleCloudPlatform#2198) * changes * Changes * changes * changes * docs * Support partitioned reads for DateTime column type in JDBC to BigQuery template (GoogleCloudPlatform#2084) * Support partitioned reads for DateTime column type * minor changes * Support backward compatibility and timezone in lower/upper bounds * removed test cases for bounds in unit tests as derby does not support timezone * Fixed typo's in JdbcToBigQuery.java Corrected testcases expected output * Corrected unit test cases expected output. Removed DateTime integration test cases. * Corrected a typo * Added default value for partitionColumnType * spotless apply * Disabling flaky test to unblock dataflow release (GoogleCloudPlatform#2220) * disabling flaky test to unblock dataflow release * minor change * Add logic to skip runnerV2 for the ITs (GoogleCloudPlatform#2219) * skip runnerv2 tests in TemplateTestBase * add logic to skip use_runner_v2 experiment in the launchTemplate * Remove redundant property skipRunnerV2Test in pom and update logic in launchTemplate * Fix format violations using mvn spotless:apply * Add warning about caching with plugin (GoogleCloudPlatform#2221) * [DatastreamToSpanner] Spanner Exception handling (GoogleCloudPlatform#2185) * SpannerExceptionClassifier class and IT * Unit tests and Integration tests * Formatting changes * Correcting UT * Unit test for SpannerResourceManager * Creating Spanner Migration Exception * Formatting changes * Correcting tests * Addressing comments * Correcting a UT * Addressing comments * Changing SpannerMigrationException to extend RuntimeException * Added Cassandra Resource Manager Refectoring and removed Generics * Added Keyspace Voilation fixes * minor changes * Create session for row check --------- Co-authored-by: taherkl <taher.lakdawala@ollion.com> Co-authored-by: Akash Thawait <aakash@ollion.com> Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com> Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com> Co-authored-by: Dylan Hercher <dylanhercher@gmail.com> Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Co-authored-by: Yi Hu <yathu@google.com> Co-authored-by: Deep1998 <deepchowdhury1998@gmail.com> Co-authored-by: jjfox15 <seanfox@google.com> Co-authored-by: vgnanasekaran <venku123@gmail.com> Co-authored-by: Danny McCormick <dannymccormick@google.com> Co-authored-by: Sharan Teja M <sharantej957@gmail.com> Co-authored-by: shreyakhajanchi <92910380+shreyakhajanchi@users.noreply.github.com> Co-authored-by: Rudra-Gujarathi <rudragujarathi@google.com> Co-authored-by: Derrick Williams <derrickaw@google.com> Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com> * Cassandra wide row it (#140) --------- Co-authored-by: taherkl <taher.lakdawala@ollion.com> Co-authored-by: Taher Lakdawala <78196491+taherkl@users.noreply.github.com> Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com> Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com> Co-authored-by: Dylan Hercher <dylanhercher@gmail.com> Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Co-authored-by: Yi Hu <yathu@google.com> Co-authored-by: Deep1998 <deepchowdhury1998@gmail.com> Co-authored-by: jjfox15 <seanfox@google.com> Co-authored-by: vgnanasekaran <venku123@gmail.com> Co-authored-by: Danny McCormick <dannymccormick@google.com> Co-authored-by: Sharan Teja M <sharantej957@gmail.com> Co-authored-by: shreyakhajanchi <92910380+shreyakhajanchi@users.noreply.github.com> Co-authored-by: Rudra-Gujarathi <rudragujarathi@google.com> Co-authored-by: Derrick Williams <derrickaw@google.com> Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com> Co-authored-by: Svetak Sundhar <svetaksundhar@google.com> Co-authored-by: Shunping Huang <shunping@google.com> Co-authored-by: Andrej Galad <dedo.cibula@gmail.com>
akashthawaitcc
added a commit
that referenced
this pull request
Mar 22, 2025
* sync upstream/main (#98) * Enhanced Retry Logic (GoogleCloudPlatform#2196) Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> * Adding support for Cassandra map (GoogleCloudPlatform#2209) * Adding test table for Map types * Adding support for cassandra map * changes (GoogleCloudPlatform#2212) * Fix inf issues in Datastream reader (GoogleCloudPlatform#2213) * add inf fix to ts fixing bug on timestamp type * Update FormatDatastreamRecordToJson.java use equals not == * Upgrade Beam version to 2.63.0 (GoogleCloudPlatform#2206) * Upgrade Beam version to 2.63.0 * add cache factory to local spanner io * remove cache pass to ReadChangeStreamPartitionDoFn * Add unimplemented/unused stubs to TestChangeStreamMutation * Use java Instant instead of threeten Instant * Fix low watermark setter call * fix testchangestreammutation * Recreate validation PR * fix import order * linux-env requirements files * fix v1 pom * SkipShade for Spanner common module (GoogleCloudPlatform#2194) * Add load test for cross db txn (GoogleCloudPlatform#2199) * Add load test for cross db txn * Change test timeout to 2 days * Revert spanner-pr.yml changes * Report Lineage for CsvToBigQuery template (GoogleCloudPlatform#2205) * Report Lineage for CsvToBigQuery template * Reply beampr-32662 to CsvConverters * Spanner Import/Export INTERLEAVE IN (GoogleCloudPlatform#2128) * Changes to write read interleave type from information schema, and write/read the type to/from avro. No change to tests yet. * Fix warnings, bug in InformationSchemaScanner, and bug in AvroSchemaToDdlConverter. Also properly default to IN PARENT when emitting ddl, in case the interleave type is not set (really only necessary for tests, since otherwise it will always be set. * Set interleaveType in InfoSchemaScanner * Style fixes, and only generate INTERLEAVE IN ddl for gsql. * another style fix - remove unused import * Make conditions more readable and add comments * Tests * Fix condition * Add interleave in table to ExportPipelineIT * Add SCRAM-SHA-512 authentication support to Kafka templates (GoogleCloudPlatform#2181) Added SCRAM-SHA-512 authentication support to Kafka to Kafka, Kafka to GCS, and Kafka to BigQuery templates. * Update the required Java version in the base doc, then regenerate docs (GoogleCloudPlatform#2218) * Update java requirement * Generate docs * Post 2.63.0 fixes (GoogleCloudPlatform#2216) * Uncomment kinesis * bump protoc to 4 in v2 * bug-fix: Use jdbc connection properties for reverse migration (GoogleCloudPlatform#2198) * changes * Changes * changes * changes * docs * Support partitioned reads for DateTime column type in JDBC to BigQuery template (GoogleCloudPlatform#2084) * Support partitioned reads for DateTime column type * minor changes * Support backward compatibility and timezone in lower/upper bounds * removed test cases for bounds in unit tests as derby does not support timezone * Fixed typo's in JdbcToBigQuery.java Corrected testcases expected output * Corrected unit test cases expected output. Removed DateTime integration test cases. * Corrected a typo * Added default value for partitionColumnType * spotless apply * Disabling flaky test to unblock dataflow release (GoogleCloudPlatform#2220) * disabling flaky test to unblock dataflow release * minor change * Add logic to skip runnerV2 for the ITs (GoogleCloudPlatform#2219) * skip runnerv2 tests in TemplateTestBase * add logic to skip use_runner_v2 experiment in the launchTemplate * Remove redundant property skipRunnerV2Test in pom and update logic in launchTemplate * Fix format violations using mvn spotless:apply * Add warning about caching with plugin (GoogleCloudPlatform#2221) * [DatastreamToSpanner] Spanner Exception handling (GoogleCloudPlatform#2185) * SpannerExceptionClassifier class and IT * Unit tests and Integration tests * Formatting changes * Correcting UT * Unit test for SpannerResourceManager * Creating Spanner Migration Exception * Formatting changes * Correcting tests * Addressing comments * Correcting a UT * Addressing comments * Changing SpannerMigrationException to extend RuntimeException * Remove Python version from `pom.xml` (GoogleCloudPlatform#2234) * Update pom.xml * Update pom.xml * Add SkipRunnerV2Test category to JmsToPubsubIT and PubSubCdcToBigQueryIT (GoogleCloudPlatform#2235) * Fix a bug in CSVToBigQuery where commas in fields are not handled correctly. (GoogleCloudPlatform#2229) * Attempt to fix csv bug where commas in fields are not handled correctly. * Replace ImmutableList with Iterable * Add tests to cover the scenario of commas within quotes. * Polish tests and add a test case to cover csv without headers. * Update Dockerfile-template-yaml (GoogleCloudPlatform#2222) * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Adding All Datatypes IT for Cassandra Migration (GoogleCloudPlatform#2230) * Add IF NOT EXISTS clause for spanner ddls used in ITs (GoogleCloudPlatform#2237) * Enable DatastreamToSpannerIT with if not exists clause * Update Datastream to Spanner IT spanner schemas with if not exists * Update BULK IT spanner schemas with if not exists * Update reverse replications ITs spanner schemas with if not exists * Add space after if not exists * Using set of random buckets for spanner ITs (GoogleCloudPlatform#2223) * Using set of random buckets for spanner ITs * reverse replication test * checkstyle fix * adding more buckets * removed ignore for testing * spotless fix * fixing UT * skip the flaky test again * addressing comments * spotless fix * Add promote artifact method in release plugin (GoogleCloudPlatform#2227) * Add promote artifact method * address comments; also fixed stagingArtifactRegistry support us.gcr.io * Consolidate method * fixed default DLQ path (GoogleCloudPlatform#2241) * Fix stagingArtifactRegistry support raw us.gcr.io artifact registry (GoogleCloudPlatform#2243) * Print error response on wget call (GoogleCloudPlatform#2245) * This helps debugging e.g. permission issues * Moving local spanner io to a different namespace (GoogleCloudPlatform#2231) * Removed LocalSpannerIO * Moving all of SpannerIO into Teleport * Copying tests also * Updated tests and excluded SpannerIO changestream from coverage checks * spotless apply * Excluding coverage check * Adding retry settings which were overwritten in LocalSpannerAccessor * Changing deprecated retrysettings function calls * Added warning and TODO comments to remove the local copy * spotless * Adding Cassandra Type Options to IT test (GoogleCloudPlatform#2242) * Bump timeouts for tests involving FKs/interleaved dependenceis (GoogleCloudPlatform#2239) * Bump timeouts for datastream to spanner test * Update FK timeout for reverse template * Bump timeouts for old reverse repl template * Enable SpannerToSourceDbInterleaveMultiShardIT * Revert loadtest timeout * Load Tests - Cassandra Reverse Replication (GoogleCloudPlatform#2163) * * Addition of Load Tests in SpannerToSourceDB For Cassandra (#89) * Addition of Load Tests in SpannerToSourceDB For Cassandra * Address Merge conflict * Added LT Refectored (#92) * Added POM Dependecies * sync upstream/main (#98) * RR LOAD TEST FIXES (#101) * Resolved PR comments (#115) * Added Module Dependency Fixes * Added Copyrigh * Added missing commit * Enhanced Retry Logic (GoogleCloudPlatform#2196) Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> * Adding support for Cassandra map (GoogleCloudPlatform#2209) * Adding test table for Map types * Adding support for cassandra map * changes (GoogleCloudPlatform#2212) * Fix inf issues in Datastream reader (GoogleCloudPlatform#2213) * add inf fix to ts fixing bug on timestamp type * Update FormatDatastreamRecordToJson.java use equals not == * Upgrade Beam version to 2.63.0 (GoogleCloudPlatform#2206) * Upgrade Beam version to 2.63.0 * add cache factory to local spanner io * remove cache pass to ReadChangeStreamPartitionDoFn * Add unimplemented/unused stubs to TestChangeStreamMutation * Use java Instant instead of threeten Instant * Fix low watermark setter call * fix testchangestreammutation * Recreate validation PR * fix import order * linux-env requirements files * fix v1 pom * SkipShade for Spanner common module (GoogleCloudPlatform#2194) * Add load test for cross db txn (GoogleCloudPlatform#2199) * Add load test for cross db txn * Change test timeout to 2 days * Revert spanner-pr.yml changes * Report Lineage for CsvToBigQuery template (GoogleCloudPlatform#2205) * Report Lineage for CsvToBigQuery template * Reply beampr-32662 to CsvConverters * Spanner Import/Export INTERLEAVE IN (GoogleCloudPlatform#2128) * Changes to write read interleave type from information schema, and write/read the type to/from avro. No change to tests yet. * Fix warnings, bug in InformationSchemaScanner, and bug in AvroSchemaToDdlConverter. Also properly default to IN PARENT when emitting ddl, in case the interleave type is not set (really only necessary for tests, since otherwise it will always be set. * Set interleaveType in InfoSchemaScanner * Style fixes, and only generate INTERLEAVE IN ddl for gsql. * another style fix - remove unused import * Make conditions more readable and add comments * Tests * Fix condition * Add interleave in table to ExportPipelineIT * Add SCRAM-SHA-512 authentication support to Kafka templates (GoogleCloudPlatform#2181) Added SCRAM-SHA-512 authentication support to Kafka to Kafka, Kafka to GCS, and Kafka to BigQuery templates. * Update the required Java version in the base doc, then regenerate docs (GoogleCloudPlatform#2218) * Update java requirement * Generate docs * Post 2.63.0 fixes (GoogleCloudPlatform#2216) * Uncomment kinesis * bump protoc to 4 in v2 * bug-fix: Use jdbc connection properties for reverse migration (GoogleCloudPlatform#2198) * changes * Changes * changes * changes * docs * Support partitioned reads for DateTime column type in JDBC to BigQuery template (GoogleCloudPlatform#2084) * Support partitioned reads for DateTime column type * minor changes * Support backward compatibility and timezone in lower/upper bounds * removed test cases for bounds in unit tests as derby does not support timezone * Fixed typo's in JdbcToBigQuery.java Corrected testcases expected output * Corrected unit test cases expected output. Removed DateTime integration test cases. * Corrected a typo * Added default value for partitionColumnType * spotless apply * Disabling flaky test to unblock dataflow release (GoogleCloudPlatform#2220) * disabling flaky test to unblock dataflow release * minor change * Add logic to skip runnerV2 for the ITs (GoogleCloudPlatform#2219) * skip runnerv2 tests in TemplateTestBase * add logic to skip use_runner_v2 experiment in the launchTemplate * Remove redundant property skipRunnerV2Test in pom and update logic in launchTemplate * Fix format violations using mvn spotless:apply * Add warning about caching with plugin (GoogleCloudPlatform#2221) * [DatastreamToSpanner] Spanner Exception handling (GoogleCloudPlatform#2185) * SpannerExceptionClassifier class and IT * Unit tests and Integration tests * Formatting changes * Correcting UT * Unit test for SpannerResourceManager * Creating Spanner Migration Exception * Formatting changes * Correcting tests * Addressing comments * Correcting a UT * Addressing comments * Changing SpannerMigrationException to extend RuntimeException * Added Cassandra Resource Manager Refectoring and removed Generics * Added Keyspace Voilation fixes * minor changes * Create session for row check --------- Co-authored-by: taherkl <taher.lakdawala@ollion.com> Co-authored-by: Akash Thawait <aakash@ollion.com> Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com> Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com> Co-authored-by: Dylan Hercher <dylanhercher@gmail.com> Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Co-authored-by: Yi Hu <yathu@google.com> Co-authored-by: Deep1998 <deepchowdhury1998@gmail.com> Co-authored-by: jjfox15 <seanfox@google.com> Co-authored-by: vgnanasekaran <venku123@gmail.com> Co-authored-by: Danny McCormick <dannymccormick@google.com> Co-authored-by: Sharan Teja M <sharantej957@gmail.com> Co-authored-by: shreyakhajanchi <92910380+shreyakhajanchi@users.noreply.github.com> Co-authored-by: Rudra-Gujarathi <rudragujarathi@google.com> Co-authored-by: Derrick Williams <derrickaw@google.com> Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com> * Added Missing Case * Added IT for Delete * Added For testing * Added IT for Emty and Custom handling * fixed cassandra schema * Added And removed unwanted Table * Added And removed unwanted * Added Missing case and removed Delete case * Spoltless fixes * Handle Null in custom Transformation * revert delete changes * Handle Checkstyle fixes * removed unwanted toSting --------- Co-authored-by: taherkl <taher.lakdawala@ollion.com> Co-authored-by: Taher Lakdawala <78196491+taherkl@users.noreply.github.com> Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com> Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com> Co-authored-by: Dylan Hercher <dylanhercher@gmail.com> Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Co-authored-by: Yi Hu <yathu@google.com> Co-authored-by: Deep1998 <deepchowdhury1998@gmail.com> Co-authored-by: jjfox15 <seanfox@google.com> Co-authored-by: vgnanasekaran <venku123@gmail.com> Co-authored-by: Danny McCormick <dannymccormick@google.com> Co-authored-by: Sharan Teja M <sharantej957@gmail.com> Co-authored-by: shreyakhajanchi <92910380+shreyakhajanchi@users.noreply.github.com> Co-authored-by: Rudra-Gujarathi <rudragujarathi@google.com> Co-authored-by: Derrick Williams <derrickaw@google.com> Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com> Co-authored-by: Svetak Sundhar <svetaksundhar@google.com> Co-authored-by: Shunping Huang <shunping@google.com> Co-authored-by: Andrej Galad <dedo.cibula@gmail.com>
taherkl
added a commit
that referenced
this pull request
Apr 2, 2025
* sync upstream/main (#98) * Enhanced Retry Logic (GoogleCloudPlatform#2196) Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> * Adding support for Cassandra map (GoogleCloudPlatform#2209) * Adding test table for Map types * Adding support for cassandra map * changes (GoogleCloudPlatform#2212) * Fix inf issues in Datastream reader (GoogleCloudPlatform#2213) * add inf fix to ts fixing bug on timestamp type * Update FormatDatastreamRecordToJson.java use equals not == * Upgrade Beam version to 2.63.0 (GoogleCloudPlatform#2206) * Upgrade Beam version to 2.63.0 * add cache factory to local spanner io * remove cache pass to ReadChangeStreamPartitionDoFn * Add unimplemented/unused stubs to TestChangeStreamMutation * Use java Instant instead of threeten Instant * Fix low watermark setter call * fix testchangestreammutation * Recreate validation PR * fix import order * linux-env requirements files * fix v1 pom * SkipShade for Spanner common module (GoogleCloudPlatform#2194) * Add load test for cross db txn (GoogleCloudPlatform#2199) * Add load test for cross db txn * Change test timeout to 2 days * Revert spanner-pr.yml changes * Report Lineage for CsvToBigQuery template (GoogleCloudPlatform#2205) * Report Lineage for CsvToBigQuery template * Reply beampr-32662 to CsvConverters * Spanner Import/Export INTERLEAVE IN (GoogleCloudPlatform#2128) * Changes to write read interleave type from information schema, and write/read the type to/from avro. No change to tests yet. * Fix warnings, bug in InformationSchemaScanner, and bug in AvroSchemaToDdlConverter. Also properly default to IN PARENT when emitting ddl, in case the interleave type is not set (really only necessary for tests, since otherwise it will always be set. * Set interleaveType in InfoSchemaScanner * Style fixes, and only generate INTERLEAVE IN ddl for gsql. * another style fix - remove unused import * Make conditions more readable and add comments * Tests * Fix condition * Add interleave in table to ExportPipelineIT * Add SCRAM-SHA-512 authentication support to Kafka templates (GoogleCloudPlatform#2181) Added SCRAM-SHA-512 authentication support to Kafka to Kafka, Kafka to GCS, and Kafka to BigQuery templates. * Update the required Java version in the base doc, then regenerate docs (GoogleCloudPlatform#2218) * Update java requirement * Generate docs * Post 2.63.0 fixes (GoogleCloudPlatform#2216) * Uncomment kinesis * bump protoc to 4 in v2 * bug-fix: Use jdbc connection properties for reverse migration (GoogleCloudPlatform#2198) * changes * Changes * changes * changes * docs * Support partitioned reads for DateTime column type in JDBC to BigQuery template (GoogleCloudPlatform#2084) * Support partitioned reads for DateTime column type * minor changes * Support backward compatibility and timezone in lower/upper bounds * removed test cases for bounds in unit tests as derby does not support timezone * Fixed typo's in JdbcToBigQuery.java Corrected testcases expected output * Corrected unit test cases expected output. Removed DateTime integration test cases. * Corrected a typo * Added default value for partitionColumnType * spotless apply * Disabling flaky test to unblock dataflow release (GoogleCloudPlatform#2220) * disabling flaky test to unblock dataflow release * minor change * Add logic to skip runnerV2 for the ITs (GoogleCloudPlatform#2219) * skip runnerv2 tests in TemplateTestBase * add logic to skip use_runner_v2 experiment in the launchTemplate * Remove redundant property skipRunnerV2Test in pom and update logic in launchTemplate * Fix format violations using mvn spotless:apply * Add warning about caching with plugin (GoogleCloudPlatform#2221) * [DatastreamToSpanner] Spanner Exception handling (GoogleCloudPlatform#2185) * SpannerExceptionClassifier class and IT * Unit tests and Integration tests * Formatting changes * Correcting UT * Unit test for SpannerResourceManager * Creating Spanner Migration Exception * Formatting changes * Correcting tests * Addressing comments * Correcting a UT * Addressing comments * Changing SpannerMigrationException to extend RuntimeException * Remove Python version from `pom.xml` (GoogleCloudPlatform#2234) * Update pom.xml * Update pom.xml * Add SkipRunnerV2Test category to JmsToPubsubIT and PubSubCdcToBigQueryIT (GoogleCloudPlatform#2235) * Fix a bug in CSVToBigQuery where commas in fields are not handled correctly. (GoogleCloudPlatform#2229) * Attempt to fix csv bug where commas in fields are not handled correctly. * Replace ImmutableList with Iterable * Add tests to cover the scenario of commas within quotes. * Polish tests and add a test case to cover csv without headers. * Update Dockerfile-template-yaml (GoogleCloudPlatform#2222) * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Adding All Datatypes IT for Cassandra Migration (GoogleCloudPlatform#2230) * Add IF NOT EXISTS clause for spanner ddls used in ITs (GoogleCloudPlatform#2237) * Enable DatastreamToSpannerIT with if not exists clause * Update Datastream to Spanner IT spanner schemas with if not exists * Update BULK IT spanner schemas with if not exists * Update reverse replications ITs spanner schemas with if not exists * Add space after if not exists * Using set of random buckets for spanner ITs (GoogleCloudPlatform#2223) * Using set of random buckets for spanner ITs * reverse replication test * checkstyle fix * adding more buckets * removed ignore for testing * spotless fix * fixing UT * skip the flaky test again * addressing comments * spotless fix * Add promote artifact method in release plugin (GoogleCloudPlatform#2227) * Add promote artifact method * address comments; also fixed stagingArtifactRegistry support us.gcr.io * Consolidate method * fixed default DLQ path (GoogleCloudPlatform#2241) * Fix stagingArtifactRegistry support raw us.gcr.io artifact registry (GoogleCloudPlatform#2243) * Print error response on wget call (GoogleCloudPlatform#2245) * This helps debugging e.g. permission issues * Moving local spanner io to a different namespace (GoogleCloudPlatform#2231) * Removed LocalSpannerIO * Moving all of SpannerIO into Teleport * Copying tests also * Updated tests and excluded SpannerIO changestream from coverage checks * spotless apply * Excluding coverage check * Adding retry settings which were overwritten in LocalSpannerAccessor * Changing deprecated retrysettings function calls * Added warning and TODO comments to remove the local copy * spotless * Adding Cassandra Type Options to IT test (GoogleCloudPlatform#2242) * Bump timeouts for tests involving FKs/interleaved dependenceis (GoogleCloudPlatform#2239) * Bump timeouts for datastream to spanner test * Update FK timeout for reverse template * Bump timeouts for old reverse repl template * Enable SpannerToSourceDbInterleaveMultiShardIT * Revert loadtest timeout * Load Tests - Cassandra Reverse Replication (GoogleCloudPlatform#2163) * * Addition of Load Tests in SpannerToSourceDB For Cassandra (#89) * Addition of Load Tests in SpannerToSourceDB For Cassandra * Address Merge conflict * Added LT Refectored (#92) * Added POM Dependecies * sync upstream/main (#98) * RR LOAD TEST FIXES (#101) * Resolved PR comments (#115) * Added Module Dependency Fixes * Added Copyrigh * Added missing commit * Enhanced Retry Logic (GoogleCloudPlatform#2196) Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> * Adding support for Cassandra map (GoogleCloudPlatform#2209) * Adding test table for Map types * Adding support for cassandra map * changes (GoogleCloudPlatform#2212) * Fix inf issues in Datastream reader (GoogleCloudPlatform#2213) * add inf fix to ts fixing bug on timestamp type * Update FormatDatastreamRecordToJson.java use equals not == * Upgrade Beam version to 2.63.0 (GoogleCloudPlatform#2206) * Upgrade Beam version to 2.63.0 * add cache factory to local spanner io * remove cache pass to ReadChangeStreamPartitionDoFn * Add unimplemented/unused stubs to TestChangeStreamMutation * Use java Instant instead of threeten Instant * Fix low watermark setter call * fix testchangestreammutation * Recreate validation PR * fix import order * linux-env requirements files * fix v1 pom * SkipShade for Spanner common module (GoogleCloudPlatform#2194) * Add load test for cross db txn (GoogleCloudPlatform#2199) * Add load test for cross db txn * Change test timeout to 2 days * Revert spanner-pr.yml changes * Report Lineage for CsvToBigQuery template (GoogleCloudPlatform#2205) * Report Lineage for CsvToBigQuery template * Reply beampr-32662 to CsvConverters * Spanner Import/Export INTERLEAVE IN (GoogleCloudPlatform#2128) * Changes to write read interleave type from information schema, and write/read the type to/from avro. No change to tests yet. * Fix warnings, bug in InformationSchemaScanner, and bug in AvroSchemaToDdlConverter. Also properly default to IN PARENT when emitting ddl, in case the interleave type is not set (really only necessary for tests, since otherwise it will always be set. * Set interleaveType in InfoSchemaScanner * Style fixes, and only generate INTERLEAVE IN ddl for gsql. * another style fix - remove unused import * Make conditions more readable and add comments * Tests * Fix condition * Add interleave in table to ExportPipelineIT * Add SCRAM-SHA-512 authentication support to Kafka templates (GoogleCloudPlatform#2181) Added SCRAM-SHA-512 authentication support to Kafka to Kafka, Kafka to GCS, and Kafka to BigQuery templates. * Update the required Java version in the base doc, then regenerate docs (GoogleCloudPlatform#2218) * Update java requirement * Generate docs * Post 2.63.0 fixes (GoogleCloudPlatform#2216) * Uncomment kinesis * bump protoc to 4 in v2 * bug-fix: Use jdbc connection properties for reverse migration (GoogleCloudPlatform#2198) * changes * Changes * changes * changes * docs * Support partitioned reads for DateTime column type in JDBC to BigQuery template (GoogleCloudPlatform#2084) * Support partitioned reads for DateTime column type * minor changes * Support backward compatibility and timezone in lower/upper bounds * removed test cases for bounds in unit tests as derby does not support timezone * Fixed typo's in JdbcToBigQuery.java Corrected testcases expected output * Corrected unit test cases expected output. Removed DateTime integration test cases. * Corrected a typo * Added default value for partitionColumnType * spotless apply * Disabling flaky test to unblock dataflow release (GoogleCloudPlatform#2220) * disabling flaky test to unblock dataflow release * minor change * Add logic to skip runnerV2 for the ITs (GoogleCloudPlatform#2219) * skip runnerv2 tests in TemplateTestBase * add logic to skip use_runner_v2 experiment in the launchTemplate * Remove redundant property skipRunnerV2Test in pom and update logic in launchTemplate * Fix format violations using mvn spotless:apply * Add warning about caching with plugin (GoogleCloudPlatform#2221) * [DatastreamToSpanner] Spanner Exception handling (GoogleCloudPlatform#2185) * SpannerExceptionClassifier class and IT * Unit tests and Integration tests * Formatting changes * Correcting UT * Unit test for SpannerResourceManager * Creating Spanner Migration Exception * Formatting changes * Correcting tests * Addressing comments * Correcting a UT * Addressing comments * Changing SpannerMigrationException to extend RuntimeException * Added Cassandra Resource Manager Refectoring and removed Generics * Added Keyspace Voilation fixes * minor changes * Create session for row check --------- Co-authored-by: taherkl <taher.lakdawala@ollion.com> Co-authored-by: Akash Thawait <aakash@ollion.com> Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com> Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com> Co-authored-by: Dylan Hercher <dylanhercher@gmail.com> Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Co-authored-by: Yi Hu <yathu@google.com> Co-authored-by: Deep1998 <deepchowdhury1998@gmail.com> Co-authored-by: jjfox15 <seanfox@google.com> Co-authored-by: vgnanasekaran <venku123@gmail.com> Co-authored-by: Danny McCormick <dannymccormick@google.com> Co-authored-by: Sharan Teja M <sharantej957@gmail.com> Co-authored-by: shreyakhajanchi <92910380+shreyakhajanchi@users.noreply.github.com> Co-authored-by: Rudra-Gujarathi <rudragujarathi@google.com> Co-authored-by: Derrick Williams <derrickaw@google.com> Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com> * Added Missing Case * Added IT for Delete * Added For testing * Added IT for Emty and Custom handling * fixed cassandra schema * Added And removed unwanted Table * Added And removed unwanted * Added Missing case and removed Delete case * Spoltless fixes * Handle Null in custom Transformation * revert delete changes * Handle Checkstyle fixes * removed unwanted toSting --------- Co-authored-by: taherkl <taher.lakdawala@ollion.com> Co-authored-by: Taher Lakdawala <78196491+taherkl@users.noreply.github.com> Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com> Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com> Co-authored-by: Dylan Hercher <dylanhercher@gmail.com> Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Co-authored-by: Yi Hu <yathu@google.com> Co-authored-by: Deep1998 <deepchowdhury1998@gmail.com> Co-authored-by: jjfox15 <seanfox@google.com> Co-authored-by: vgnanasekaran <venku123@gmail.com> Co-authored-by: Danny McCormick <dannymccormick@google.com> Co-authored-by: Sharan Teja M <sharantej957@gmail.com> Co-authored-by: shreyakhajanchi <92910380+shreyakhajanchi@users.noreply.github.com> Co-authored-by: Rudra-Gujarathi <rudragujarathi@google.com> Co-authored-by: Derrick Williams <derrickaw@google.com> Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com> Co-authored-by: Svetak Sundhar <svetaksundhar@google.com> Co-authored-by: Shunping Huang <shunping@google.com> Co-authored-by: Andrej Galad <dedo.cibula@gmail.com>
taherkl
added a commit
that referenced
this pull request
Apr 2, 2025
* sync upstream/main (#98) * Enhanced Retry Logic (GoogleCloudPlatform#2196) Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> * Adding support for Cassandra map (GoogleCloudPlatform#2209) * Adding test table for Map types * Adding support for cassandra map * changes (GoogleCloudPlatform#2212) * Fix inf issues in Datastream reader (GoogleCloudPlatform#2213) * add inf fix to ts fixing bug on timestamp type * Update FormatDatastreamRecordToJson.java use equals not == * Upgrade Beam version to 2.63.0 (GoogleCloudPlatform#2206) * Upgrade Beam version to 2.63.0 * add cache factory to local spanner io * remove cache pass to ReadChangeStreamPartitionDoFn * Add unimplemented/unused stubs to TestChangeStreamMutation * Use java Instant instead of threeten Instant * Fix low watermark setter call * fix testchangestreammutation * Recreate validation PR * fix import order * linux-env requirements files * fix v1 pom * SkipShade for Spanner common module (GoogleCloudPlatform#2194) * Add load test for cross db txn (GoogleCloudPlatform#2199) * Add load test for cross db txn * Change test timeout to 2 days * Revert spanner-pr.yml changes * Report Lineage for CsvToBigQuery template (GoogleCloudPlatform#2205) * Report Lineage for CsvToBigQuery template * Reply beampr-32662 to CsvConverters * Spanner Import/Export INTERLEAVE IN (GoogleCloudPlatform#2128) * Changes to write read interleave type from information schema, and write/read the type to/from avro. No change to tests yet. * Fix warnings, bug in InformationSchemaScanner, and bug in AvroSchemaToDdlConverter. Also properly default to IN PARENT when emitting ddl, in case the interleave type is not set (really only necessary for tests, since otherwise it will always be set. * Set interleaveType in InfoSchemaScanner * Style fixes, and only generate INTERLEAVE IN ddl for gsql. * another style fix - remove unused import * Make conditions more readable and add comments * Tests * Fix condition * Add interleave in table to ExportPipelineIT * Add SCRAM-SHA-512 authentication support to Kafka templates (GoogleCloudPlatform#2181) Added SCRAM-SHA-512 authentication support to Kafka to Kafka, Kafka to GCS, and Kafka to BigQuery templates. * Update the required Java version in the base doc, then regenerate docs (GoogleCloudPlatform#2218) * Update java requirement * Generate docs * Post 2.63.0 fixes (GoogleCloudPlatform#2216) * Uncomment kinesis * bump protoc to 4 in v2 * bug-fix: Use jdbc connection properties for reverse migration (GoogleCloudPlatform#2198) * changes * Changes * changes * changes * docs * Support partitioned reads for DateTime column type in JDBC to BigQuery template (GoogleCloudPlatform#2084) * Support partitioned reads for DateTime column type * minor changes * Support backward compatibility and timezone in lower/upper bounds * removed test cases for bounds in unit tests as derby does not support timezone * Fixed typo's in JdbcToBigQuery.java Corrected testcases expected output * Corrected unit test cases expected output. Removed DateTime integration test cases. * Corrected a typo * Added default value for partitionColumnType * spotless apply * Disabling flaky test to unblock dataflow release (GoogleCloudPlatform#2220) * disabling flaky test to unblock dataflow release * minor change * Add logic to skip runnerV2 for the ITs (GoogleCloudPlatform#2219) * skip runnerv2 tests in TemplateTestBase * add logic to skip use_runner_v2 experiment in the launchTemplate * Remove redundant property skipRunnerV2Test in pom and update logic in launchTemplate * Fix format violations using mvn spotless:apply * Add warning about caching with plugin (GoogleCloudPlatform#2221) * [DatastreamToSpanner] Spanner Exception handling (GoogleCloudPlatform#2185) * SpannerExceptionClassifier class and IT * Unit tests and Integration tests * Formatting changes * Correcting UT * Unit test for SpannerResourceManager * Creating Spanner Migration Exception * Formatting changes * Correcting tests * Addressing comments * Correcting a UT * Addressing comments * Changing SpannerMigrationException to extend RuntimeException * Remove Python version from `pom.xml` (GoogleCloudPlatform#2234) * Update pom.xml * Update pom.xml * Add SkipRunnerV2Test category to JmsToPubsubIT and PubSubCdcToBigQueryIT (GoogleCloudPlatform#2235) * Fix a bug in CSVToBigQuery where commas in fields are not handled correctly. (GoogleCloudPlatform#2229) * Attempt to fix csv bug where commas in fields are not handled correctly. * Replace ImmutableList with Iterable * Add tests to cover the scenario of commas within quotes. * Polish tests and add a test case to cover csv without headers. * Update Dockerfile-template-yaml (GoogleCloudPlatform#2222) * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Adding All Datatypes IT for Cassandra Migration (GoogleCloudPlatform#2230) * Add IF NOT EXISTS clause for spanner ddls used in ITs (GoogleCloudPlatform#2237) * Enable DatastreamToSpannerIT with if not exists clause * Update Datastream to Spanner IT spanner schemas with if not exists * Update BULK IT spanner schemas with if not exists * Update reverse replications ITs spanner schemas with if not exists * Add space after if not exists * Using set of random buckets for spanner ITs (GoogleCloudPlatform#2223) * Using set of random buckets for spanner ITs * reverse replication test * checkstyle fix * adding more buckets * removed ignore for testing * spotless fix * fixing UT * skip the flaky test again * addressing comments * spotless fix * Add promote artifact method in release plugin (GoogleCloudPlatform#2227) * Add promote artifact method * address comments; also fixed stagingArtifactRegistry support us.gcr.io * Consolidate method * fixed default DLQ path (GoogleCloudPlatform#2241) * Fix stagingArtifactRegistry support raw us.gcr.io artifact registry (GoogleCloudPlatform#2243) * Print error response on wget call (GoogleCloudPlatform#2245) * This helps debugging e.g. permission issues * Moving local spanner io to a different namespace (GoogleCloudPlatform#2231) * Removed LocalSpannerIO * Moving all of SpannerIO into Teleport * Copying tests also * Updated tests and excluded SpannerIO changestream from coverage checks * spotless apply * Excluding coverage check * Adding retry settings which were overwritten in LocalSpannerAccessor * Changing deprecated retrysettings function calls * Added warning and TODO comments to remove the local copy * spotless * Adding Cassandra Type Options to IT test (GoogleCloudPlatform#2242) * Bump timeouts for tests involving FKs/interleaved dependenceis (GoogleCloudPlatform#2239) * Bump timeouts for datastream to spanner test * Update FK timeout for reverse template * Bump timeouts for old reverse repl template * Enable SpannerToSourceDbInterleaveMultiShardIT * Revert loadtest timeout * Load Tests - Cassandra Reverse Replication (GoogleCloudPlatform#2163) * * Addition of Load Tests in SpannerToSourceDB For Cassandra (#89) * Addition of Load Tests in SpannerToSourceDB For Cassandra * Address Merge conflict * Added LT Refectored (#92) * Added POM Dependecies * sync upstream/main (#98) * RR LOAD TEST FIXES (#101) * Resolved PR comments (#115) * Added Module Dependency Fixes * Added Copyrigh * Added missing commit * Enhanced Retry Logic (GoogleCloudPlatform#2196) Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> * Adding support for Cassandra map (GoogleCloudPlatform#2209) * Adding test table for Map types * Adding support for cassandra map * changes (GoogleCloudPlatform#2212) * Fix inf issues in Datastream reader (GoogleCloudPlatform#2213) * add inf fix to ts fixing bug on timestamp type * Update FormatDatastreamRecordToJson.java use equals not == * Upgrade Beam version to 2.63.0 (GoogleCloudPlatform#2206) * Upgrade Beam version to 2.63.0 * add cache factory to local spanner io * remove cache pass to ReadChangeStreamPartitionDoFn * Add unimplemented/unused stubs to TestChangeStreamMutation * Use java Instant instead of threeten Instant * Fix low watermark setter call * fix testchangestreammutation * Recreate validation PR * fix import order * linux-env requirements files * fix v1 pom * SkipShade for Spanner common module (GoogleCloudPlatform#2194) * Add load test for cross db txn (GoogleCloudPlatform#2199) * Add load test for cross db txn * Change test timeout to 2 days * Revert spanner-pr.yml changes * Report Lineage for CsvToBigQuery template (GoogleCloudPlatform#2205) * Report Lineage for CsvToBigQuery template * Reply beampr-32662 to CsvConverters * Spanner Import/Export INTERLEAVE IN (GoogleCloudPlatform#2128) * Changes to write read interleave type from information schema, and write/read the type to/from avro. No change to tests yet. * Fix warnings, bug in InformationSchemaScanner, and bug in AvroSchemaToDdlConverter. Also properly default to IN PARENT when emitting ddl, in case the interleave type is not set (really only necessary for tests, since otherwise it will always be set. * Set interleaveType in InfoSchemaScanner * Style fixes, and only generate INTERLEAVE IN ddl for gsql. * another style fix - remove unused import * Make conditions more readable and add comments * Tests * Fix condition * Add interleave in table to ExportPipelineIT * Add SCRAM-SHA-512 authentication support to Kafka templates (GoogleCloudPlatform#2181) Added SCRAM-SHA-512 authentication support to Kafka to Kafka, Kafka to GCS, and Kafka to BigQuery templates. * Update the required Java version in the base doc, then regenerate docs (GoogleCloudPlatform#2218) * Update java requirement * Generate docs * Post 2.63.0 fixes (GoogleCloudPlatform#2216) * Uncomment kinesis * bump protoc to 4 in v2 * bug-fix: Use jdbc connection properties for reverse migration (GoogleCloudPlatform#2198) * changes * Changes * changes * changes * docs * Support partitioned reads for DateTime column type in JDBC to BigQuery template (GoogleCloudPlatform#2084) * Support partitioned reads for DateTime column type * minor changes * Support backward compatibility and timezone in lower/upper bounds * removed test cases for bounds in unit tests as derby does not support timezone * Fixed typo's in JdbcToBigQuery.java Corrected testcases expected output * Corrected unit test cases expected output. Removed DateTime integration test cases. * Corrected a typo * Added default value for partitionColumnType * spotless apply * Disabling flaky test to unblock dataflow release (GoogleCloudPlatform#2220) * disabling flaky test to unblock dataflow release * minor change * Add logic to skip runnerV2 for the ITs (GoogleCloudPlatform#2219) * skip runnerv2 tests in TemplateTestBase * add logic to skip use_runner_v2 experiment in the launchTemplate * Remove redundant property skipRunnerV2Test in pom and update logic in launchTemplate * Fix format violations using mvn spotless:apply * Add warning about caching with plugin (GoogleCloudPlatform#2221) * [DatastreamToSpanner] Spanner Exception handling (GoogleCloudPlatform#2185) * SpannerExceptionClassifier class and IT * Unit tests and Integration tests * Formatting changes * Correcting UT * Unit test for SpannerResourceManager * Creating Spanner Migration Exception * Formatting changes * Correcting tests * Addressing comments * Correcting a UT * Addressing comments * Changing SpannerMigrationException to extend RuntimeException * Added Cassandra Resource Manager Refectoring and removed Generics * Added Keyspace Voilation fixes * minor changes * Create session for row check --------- Co-authored-by: taherkl <taher.lakdawala@ollion.com> Co-authored-by: Akash Thawait <aakash@ollion.com> Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com> Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com> Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Co-authored-by: Yi Hu <yathu@google.com> Co-authored-by: jjfox15 <seanfox@google.com> Co-authored-by: vgnanasekaran <venku123@gmail.com> Co-authored-by: Danny McCormick <dannymccormick@google.com> Co-authored-by: shreyakhajanchi <92910380+shreyakhajanchi@users.noreply.github.com> Co-authored-by: Rudra-Gujarathi <rudragujarathi@google.com> Co-authored-by: Derrick Williams <derrickaw@google.com> Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com> * Added Missing Case * Added IT for Delete * Added For testing * Added IT for Emty and Custom handling * fixed cassandra schema * Added And removed unwanted Table * Added And removed unwanted * Added Missing case and removed Delete case * Spoltless fixes * Handle Null in custom Transformation * revert delete changes * Handle Checkstyle fixes * removed unwanted toSting --------- Co-authored-by: taherkl <taher.lakdawala@ollion.com> Co-authored-by: Taher Lakdawala <78196491+taherkl@users.noreply.github.com> Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com> Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com> Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Co-authored-by: Yi Hu <yathu@google.com> Co-authored-by: jjfox15 <seanfox@google.com> Co-authored-by: vgnanasekaran <venku123@gmail.com> Co-authored-by: Danny McCormick <dannymccormick@google.com> Co-authored-by: shreyakhajanchi <92910380+shreyakhajanchi@users.noreply.github.com> Co-authored-by: Rudra-Gujarathi <rudragujarathi@google.com> Co-authored-by: Derrick Williams <derrickaw@google.com> Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com> Co-authored-by: Svetak Sundhar <svetaksundhar@google.com> Co-authored-by: Shunping Huang <shunping@google.com> Co-authored-by: Andrej Galad <dedo.cibula@gmail.com>
taherkl
added a commit
that referenced
this pull request
Apr 3, 2025
…atform#2268) * Acceptance Test Fixes * sync upstream/main (#98) * Enhanced Retry Logic (GoogleCloudPlatform#2196) Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> * Adding support for Cassandra map (GoogleCloudPlatform#2209) * Adding test table for Map types * Adding support for cassandra map * changes (GoogleCloudPlatform#2212) * Fix inf issues in Datastream reader (GoogleCloudPlatform#2213) * add inf fix to ts fixing bug on timestamp type * Update FormatDatastreamRecordToJson.java use equals not == * Upgrade Beam version to 2.63.0 (GoogleCloudPlatform#2206) * Upgrade Beam version to 2.63.0 * add cache factory to local spanner io * remove cache pass to ReadChangeStreamPartitionDoFn * Add unimplemented/unused stubs to TestChangeStreamMutation * Use java Instant instead of threeten Instant * Fix low watermark setter call * fix testchangestreammutation * Recreate validation PR * fix import order * linux-env requirements files * fix v1 pom * SkipShade for Spanner common module (GoogleCloudPlatform#2194) * Add load test for cross db txn (GoogleCloudPlatform#2199) * Add load test for cross db txn * Change test timeout to 2 days * Revert spanner-pr.yml changes * Report Lineage for CsvToBigQuery template (GoogleCloudPlatform#2205) * Report Lineage for CsvToBigQuery template * Reply beampr-32662 to CsvConverters * Spanner Import/Export INTERLEAVE IN (GoogleCloudPlatform#2128) * Changes to write read interleave type from information schema, and write/read the type to/from avro. No change to tests yet. * Fix warnings, bug in InformationSchemaScanner, and bug in AvroSchemaToDdlConverter. Also properly default to IN PARENT when emitting ddl, in case the interleave type is not set (really only necessary for tests, since otherwise it will always be set. * Set interleaveType in InfoSchemaScanner * Style fixes, and only generate INTERLEAVE IN ddl for gsql. * another style fix - remove unused import * Make conditions more readable and add comments * Tests * Fix condition * Add interleave in table to ExportPipelineIT * Add SCRAM-SHA-512 authentication support to Kafka templates (GoogleCloudPlatform#2181) Added SCRAM-SHA-512 authentication support to Kafka to Kafka, Kafka to GCS, and Kafka to BigQuery templates. * Update the required Java version in the base doc, then regenerate docs (GoogleCloudPlatform#2218) * Update java requirement * Generate docs * Post 2.63.0 fixes (GoogleCloudPlatform#2216) * Uncomment kinesis * bump protoc to 4 in v2 * bug-fix: Use jdbc connection properties for reverse migration (GoogleCloudPlatform#2198) * changes * Changes * changes * changes * docs * Support partitioned reads for DateTime column type in JDBC to BigQuery template (GoogleCloudPlatform#2084) * Support partitioned reads for DateTime column type * minor changes * Support backward compatibility and timezone in lower/upper bounds * removed test cases for bounds in unit tests as derby does not support timezone * Fixed typo's in JdbcToBigQuery.java Corrected testcases expected output * Corrected unit test cases expected output. Removed DateTime integration test cases. * Corrected a typo * Added default value for partitionColumnType * spotless apply * Disabling flaky test to unblock dataflow release (GoogleCloudPlatform#2220) * disabling flaky test to unblock dataflow release * minor change * Add logic to skip runnerV2 for the ITs (GoogleCloudPlatform#2219) * skip runnerv2 tests in TemplateTestBase * add logic to skip use_runner_v2 experiment in the launchTemplate * Remove redundant property skipRunnerV2Test in pom and update logic in launchTemplate * Fix format violations using mvn spotless:apply * Add warning about caching with plugin (GoogleCloudPlatform#2221) * [DatastreamToSpanner] Spanner Exception handling (GoogleCloudPlatform#2185) * SpannerExceptionClassifier class and IT * Unit tests and Integration tests * Formatting changes * Correcting UT * Unit test for SpannerResourceManager * Creating Spanner Migration Exception * Formatting changes * Correcting tests * Addressing comments * Correcting a UT * Addressing comments * Changing SpannerMigrationException to extend RuntimeException * Remove Python version from `pom.xml` (GoogleCloudPlatform#2234) * Update pom.xml * Update pom.xml * Add SkipRunnerV2Test category to JmsToPubsubIT and PubSubCdcToBigQueryIT (GoogleCloudPlatform#2235) * Fix a bug in CSVToBigQuery where commas in fields are not handled correctly. (GoogleCloudPlatform#2229) * Attempt to fix csv bug where commas in fields are not handled correctly. * Replace ImmutableList with Iterable * Add tests to cover the scenario of commas within quotes. * Polish tests and add a test case to cover csv without headers. * Update Dockerfile-template-yaml (GoogleCloudPlatform#2222) * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Adding All Datatypes IT for Cassandra Migration (GoogleCloudPlatform#2230) * Add IF NOT EXISTS clause for spanner ddls used in ITs (GoogleCloudPlatform#2237) * Enable DatastreamToSpannerIT with if not exists clause * Update Datastream to Spanner IT spanner schemas with if not exists * Update BULK IT spanner schemas with if not exists * Update reverse replications ITs spanner schemas with if not exists * Add space after if not exists * Using set of random buckets for spanner ITs (GoogleCloudPlatform#2223) * Using set of random buckets for spanner ITs * reverse replication test * checkstyle fix * adding more buckets * removed ignore for testing * spotless fix * fixing UT * skip the flaky test again * addressing comments * spotless fix * Add promote artifact method in release plugin (GoogleCloudPlatform#2227) * Add promote artifact method * address comments; also fixed stagingArtifactRegistry support us.gcr.io * Consolidate method * fixed default DLQ path (GoogleCloudPlatform#2241) * Fix stagingArtifactRegistry support raw us.gcr.io artifact registry (GoogleCloudPlatform#2243) * Print error response on wget call (GoogleCloudPlatform#2245) * This helps debugging e.g. permission issues * Moving local spanner io to a different namespace (GoogleCloudPlatform#2231) * Removed LocalSpannerIO * Moving all of SpannerIO into Teleport * Copying tests also * Updated tests and excluded SpannerIO changestream from coverage checks * spotless apply * Excluding coverage check * Adding retry settings which were overwritten in LocalSpannerAccessor * Changing deprecated retrysettings function calls * Added warning and TODO comments to remove the local copy * spotless * Adding Cassandra Type Options to IT test (GoogleCloudPlatform#2242) * Bump timeouts for tests involving FKs/interleaved dependenceis (GoogleCloudPlatform#2239) * Bump timeouts for datastream to spanner test * Update FK timeout for reverse template * Bump timeouts for old reverse repl template * Enable SpannerToSourceDbInterleaveMultiShardIT * Revert loadtest timeout * Load Tests - Cassandra Reverse Replication (GoogleCloudPlatform#2163) * * Addition of Load Tests in SpannerToSourceDB For Cassandra (#89) * Addition of Load Tests in SpannerToSourceDB For Cassandra * Address Merge conflict * Added LT Refectored (#92) * Added POM Dependecies * sync upstream/main (#98) * RR LOAD TEST FIXES (#101) * Resolved PR comments (#115) * Added Module Dependency Fixes * Added Copyrigh * Added missing commit * Enhanced Retry Logic (GoogleCloudPlatform#2196) Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> * Adding support for Cassandra map (GoogleCloudPlatform#2209) * Adding test table for Map types * Adding support for cassandra map * changes (GoogleCloudPlatform#2212) * Fix inf issues in Datastream reader (GoogleCloudPlatform#2213) * add inf fix to ts fixing bug on timestamp type * Update FormatDatastreamRecordToJson.java use equals not == * Upgrade Beam version to 2.63.0 (GoogleCloudPlatform#2206) * Upgrade Beam version to 2.63.0 * add cache factory to local spanner io * remove cache pass to ReadChangeStreamPartitionDoFn * Add unimplemented/unused stubs to TestChangeStreamMutation * Use java Instant instead of threeten Instant * Fix low watermark setter call * fix testchangestreammutation * Recreate validation PR * fix import order * linux-env requirements files * fix v1 pom * SkipShade for Spanner common module (GoogleCloudPlatform#2194) * Add load test for cross db txn (GoogleCloudPlatform#2199) * Add load test for cross db txn * Change test timeout to 2 days * Revert spanner-pr.yml changes * Report Lineage for CsvToBigQuery template (GoogleCloudPlatform#2205) * Report Lineage for CsvToBigQuery template * Reply beampr-32662 to CsvConverters * Spanner Import/Export INTERLEAVE IN (GoogleCloudPlatform#2128) * Changes to write read interleave type from information schema, and write/read the type to/from avro. No change to tests yet. * Fix warnings, bug in InformationSchemaScanner, and bug in AvroSchemaToDdlConverter. Also properly default to IN PARENT when emitting ddl, in case the interleave type is not set (really only necessary for tests, since otherwise it will always be set. * Set interleaveType in InfoSchemaScanner * Style fixes, and only generate INTERLEAVE IN ddl for gsql. * another style fix - remove unused import * Make conditions more readable and add comments * Tests * Fix condition * Add interleave in table to ExportPipelineIT * Add SCRAM-SHA-512 authentication support to Kafka templates (GoogleCloudPlatform#2181) Added SCRAM-SHA-512 authentication support to Kafka to Kafka, Kafka to GCS, and Kafka to BigQuery templates. * Update the required Java version in the base doc, then regenerate docs (GoogleCloudPlatform#2218) * Update java requirement * Generate docs * Post 2.63.0 fixes (GoogleCloudPlatform#2216) * Uncomment kinesis * bump protoc to 4 in v2 * bug-fix: Use jdbc connection properties for reverse migration (GoogleCloudPlatform#2198) * changes * Changes * changes * changes * docs * Support partitioned reads for DateTime column type in JDBC to BigQuery template (GoogleCloudPlatform#2084) * Support partitioned reads for DateTime column type * minor changes * Support backward compatibility and timezone in lower/upper bounds * removed test cases for bounds in unit tests as derby does not support timezone * Fixed typo's in JdbcToBigQuery.java Corrected testcases expected output * Corrected unit test cases expected output. Removed DateTime integration test cases. * Corrected a typo * Added default value for partitionColumnType * spotless apply * Disabling flaky test to unblock dataflow release (GoogleCloudPlatform#2220) * disabling flaky test to unblock dataflow release * minor change * Add logic to skip runnerV2 for the ITs (GoogleCloudPlatform#2219) * skip runnerv2 tests in TemplateTestBase * add logic to skip use_runner_v2 experiment in the launchTemplate * Remove redundant property skipRunnerV2Test in pom and update logic in launchTemplate * Fix format violations using mvn spotless:apply * Add warning about caching with plugin (GoogleCloudPlatform#2221) * [DatastreamToSpanner] Spanner Exception handling (GoogleCloudPlatform#2185) * SpannerExceptionClassifier class and IT * Unit tests and Integration tests * Formatting changes * Correcting UT * Unit test for SpannerResourceManager * Creating Spanner Migration Exception * Formatting changes * Correcting tests * Addressing comments * Correcting a UT * Addressing comments * Changing SpannerMigrationException to extend RuntimeException * Added Cassandra Resource Manager Refectoring and removed Generics * Added Keyspace Voilation fixes * minor changes * Create session for row check --------- Co-authored-by: taherkl <taher.lakdawala@ollion.com> Co-authored-by: Akash Thawait <aakash@ollion.com> Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com> Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com> Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Co-authored-by: Yi Hu <yathu@google.com> Co-authored-by: jjfox15 <seanfox@google.com> Co-authored-by: vgnanasekaran <venku123@gmail.com> Co-authored-by: Danny McCormick <dannymccormick@google.com> Co-authored-by: shreyakhajanchi <92910380+shreyakhajanchi@users.noreply.github.com> Co-authored-by: Rudra-Gujarathi <rudragujarathi@google.com> Co-authored-by: Derrick Williams <derrickaw@google.com> Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com> * Added Missing Case * Added IT for Delete * Added For testing * Added IT for Emty and Custom handling * fixed cassandra schema * Added And removed unwanted Table * Added And removed unwanted * Added Missing case and removed Delete case * Spoltless fixes * Handle Null in custom Transformation * revert delete changes * Handle Checkstyle fixes * removed unwanted toSting --------- Co-authored-by: taherkl <taher.lakdawala@ollion.com> Co-authored-by: Taher Lakdawala <78196491+taherkl@users.noreply.github.com> Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com> Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com> Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Co-authored-by: Yi Hu <yathu@google.com> Co-authored-by: jjfox15 <seanfox@google.com> Co-authored-by: vgnanasekaran <venku123@gmail.com> Co-authored-by: Danny McCormick <dannymccormick@google.com> Co-authored-by: shreyakhajanchi <92910380+shreyakhajanchi@users.noreply.github.com> Co-authored-by: Rudra-Gujarathi <rudragujarathi@google.com> Co-authored-by: Derrick Williams <derrickaw@google.com> Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com> Co-authored-by: Svetak Sundhar <svetaksundhar@google.com> Co-authored-by: Shunping Huang <shunping@google.com> Co-authored-by: Andrej Galad <dedo.cibula@gmail.com> * ADDED Missing Combination (#160) * Adding missing col combination assert for empty (#161) * ADDED Missing Combination * Address the assert the same * Schema Fixes * Schema FIxes * Added SCHEMA FIXES * Schema Correction * Added Fixes * Added Schema Fixes * Adding InsertOnly Mode for Bulk job (GoogleCloudPlatform#2260) * Adding Lgs * Handle Null Explicit * removed logs * revert ignore change * revert additional changes * Added UT for Null and Empty Case (#171) --------- Co-authored-by: taherkl <taher.lakdawala@ollion.com> Co-authored-by: Taher Lakdawala <78196491+taherkl@users.noreply.github.com> Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com> Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com> Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Co-authored-by: Yi Hu <yathu@google.com> Co-authored-by: jjfox15 <seanfox@google.com> Co-authored-by: vgnanasekaran <venku123@gmail.com> Co-authored-by: Danny McCormick <dannymccormick@google.com> Co-authored-by: shreyakhajanchi <92910380+shreyakhajanchi@users.noreply.github.com> Co-authored-by: Rudra-Gujarathi <rudragujarathi@google.com> Co-authored-by: Derrick Williams <derrickaw@google.com> Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com> Co-authored-by: Svetak Sundhar <svetaksundhar@google.com> Co-authored-by: Shunping Huang <shunping@google.com> Co-authored-by: Andrej Galad <dedo.cibula@gmail.com>
pawankashyapollion
added a commit
that referenced
this pull request
Apr 17, 2025
* sync upstream/main (#98) * Enhanced Retry Logic (GoogleCloudPlatform#2196) Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> * Adding support for Cassandra map (GoogleCloudPlatform#2209) * Adding test table for Map types * Adding support for cassandra map * changes (GoogleCloudPlatform#2212) * Fix inf issues in Datastream reader (GoogleCloudPlatform#2213) * add inf fix to ts fixing bug on timestamp type * Update FormatDatastreamRecordToJson.java use equals not == * Upgrade Beam version to 2.63.0 (GoogleCloudPlatform#2206) * Upgrade Beam version to 2.63.0 * add cache factory to local spanner io * remove cache pass to ReadChangeStreamPartitionDoFn * Add unimplemented/unused stubs to TestChangeStreamMutation * Use java Instant instead of threeten Instant * Fix low watermark setter call * fix testchangestreammutation * Recreate validation PR * fix import order * linux-env requirements files * fix v1 pom * SkipShade for Spanner common module (GoogleCloudPlatform#2194) * Add load test for cross db txn (GoogleCloudPlatform#2199) * Add load test for cross db txn * Change test timeout to 2 days * Revert spanner-pr.yml changes * Report Lineage for CsvToBigQuery template (GoogleCloudPlatform#2205) * Report Lineage for CsvToBigQuery template * Reply beampr-32662 to CsvConverters * Spanner Import/Export INTERLEAVE IN (GoogleCloudPlatform#2128) * Changes to write read interleave type from information schema, and write/read the type to/from avro. No change to tests yet. * Fix warnings, bug in InformationSchemaScanner, and bug in AvroSchemaToDdlConverter. Also properly default to IN PARENT when emitting ddl, in case the interleave type is not set (really only necessary for tests, since otherwise it will always be set. * Set interleaveType in InfoSchemaScanner * Style fixes, and only generate INTERLEAVE IN ddl for gsql. * another style fix - remove unused import * Make conditions more readable and add comments * Tests * Fix condition * Add interleave in table to ExportPipelineIT * Add SCRAM-SHA-512 authentication support to Kafka templates (GoogleCloudPlatform#2181) Added SCRAM-SHA-512 authentication support to Kafka to Kafka, Kafka to GCS, and Kafka to BigQuery templates. * Update the required Java version in the base doc, then regenerate docs (GoogleCloudPlatform#2218) * Update java requirement * Generate docs * Post 2.63.0 fixes (GoogleCloudPlatform#2216) * Uncomment kinesis * bump protoc to 4 in v2 * bug-fix: Use jdbc connection properties for reverse migration (GoogleCloudPlatform#2198) * changes * Changes * changes * changes * docs * Support partitioned reads for DateTime column type in JDBC to BigQuery template (GoogleCloudPlatform#2084) * Support partitioned reads for DateTime column type * minor changes * Support backward compatibility and timezone in lower/upper bounds * removed test cases for bounds in unit tests as derby does not support timezone * Fixed typo's in JdbcToBigQuery.java Corrected testcases expected output * Corrected unit test cases expected output. Removed DateTime integration test cases. * Corrected a typo * Added default value for partitionColumnType * spotless apply * Disabling flaky test to unblock dataflow release (GoogleCloudPlatform#2220) * disabling flaky test to unblock dataflow release * minor change * Add logic to skip runnerV2 for the ITs (GoogleCloudPlatform#2219) * skip runnerv2 tests in TemplateTestBase * add logic to skip use_runner_v2 experiment in the launchTemplate * Remove redundant property skipRunnerV2Test in pom and update logic in launchTemplate * Fix format violations using mvn spotless:apply * Add warning about caching with plugin (GoogleCloudPlatform#2221) * [DatastreamToSpanner] Spanner Exception handling (GoogleCloudPlatform#2185) * SpannerExceptionClassifier class and IT * Unit tests and Integration tests * Formatting changes * Correcting UT * Unit test for SpannerResourceManager * Creating Spanner Migration Exception * Formatting changes * Correcting tests * Addressing comments * Correcting a UT * Addressing comments * Changing SpannerMigrationException to extend RuntimeException * Remove Python version from `pom.xml` (GoogleCloudPlatform#2234) * Update pom.xml * Update pom.xml * Add SkipRunnerV2Test category to JmsToPubsubIT and PubSubCdcToBigQueryIT (GoogleCloudPlatform#2235) * Fix a bug in CSVToBigQuery where commas in fields are not handled correctly. (GoogleCloudPlatform#2229) * Attempt to fix csv bug where commas in fields are not handled correctly. * Replace ImmutableList with Iterable * Add tests to cover the scenario of commas within quotes. * Polish tests and add a test case to cover csv without headers. * Update Dockerfile-template-yaml (GoogleCloudPlatform#2222) * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Adding All Datatypes IT for Cassandra Migration (GoogleCloudPlatform#2230) * Add IF NOT EXISTS clause for spanner ddls used in ITs (GoogleCloudPlatform#2237) * Enable DatastreamToSpannerIT with if not exists clause * Update Datastream to Spanner IT spanner schemas with if not exists * Update BULK IT spanner schemas with if not exists * Update reverse replications ITs spanner schemas with if not exists * Add space after if not exists * Using set of random buckets for spanner ITs (GoogleCloudPlatform#2223) * Using set of random buckets for spanner ITs * reverse replication test * checkstyle fix * adding more buckets * removed ignore for testing * spotless fix * fixing UT * skip the flaky test again * addressing comments * spotless fix * Add promote artifact method in release plugin (GoogleCloudPlatform#2227) * Add promote artifact method * address comments; also fixed stagingArtifactRegistry support us.gcr.io * Consolidate method * fixed default DLQ path (GoogleCloudPlatform#2241) * Fix stagingArtifactRegistry support raw us.gcr.io artifact registry (GoogleCloudPlatform#2243) * Print error response on wget call (GoogleCloudPlatform#2245) * This helps debugging e.g. permission issues * Moving local spanner io to a different namespace (GoogleCloudPlatform#2231) * Removed LocalSpannerIO * Moving all of SpannerIO into Teleport * Copying tests also * Updated tests and excluded SpannerIO changestream from coverage checks * spotless apply * Excluding coverage check * Adding retry settings which were overwritten in LocalSpannerAccessor * Changing deprecated retrysettings function calls * Added warning and TODO comments to remove the local copy * spotless * Adding Cassandra Type Options to IT test (GoogleCloudPlatform#2242) * Bump timeouts for tests involving FKs/interleaved dependenceis (GoogleCloudPlatform#2239) * Bump timeouts for datastream to spanner test * Update FK timeout for reverse template * Bump timeouts for old reverse repl template * Enable SpannerToSourceDbInterleaveMultiShardIT * Revert loadtest timeout * Load Tests - Cassandra Reverse Replication (GoogleCloudPlatform#2163) * * Addition of Load Tests in SpannerToSourceDB For Cassandra (#89) * Addition of Load Tests in SpannerToSourceDB For Cassandra * Address Merge conflict * Added LT Refectored (#92) * Added POM Dependecies * sync upstream/main (#98) * RR LOAD TEST FIXES (#101) * Resolved PR comments (#115) * Added Module Dependency Fixes * Added Copyrigh * Added missing commit * Enhanced Retry Logic (GoogleCloudPlatform#2196) Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> * Adding support for Cassandra map (GoogleCloudPlatform#2209) * Adding test table for Map types * Adding support for cassandra map * changes (GoogleCloudPlatform#2212) * Fix inf issues in Datastream reader (GoogleCloudPlatform#2213) * add inf fix to ts fixing bug on timestamp type * Update FormatDatastreamRecordToJson.java use equals not == * Upgrade Beam version to 2.63.0 (GoogleCloudPlatform#2206) * Upgrade Beam version to 2.63.0 * add cache factory to local spanner io * remove cache pass to ReadChangeStreamPartitionDoFn * Add unimplemented/unused stubs to TestChangeStreamMutation * Use java Instant instead of threeten Instant * Fix low watermark setter call * fix testchangestreammutation * Recreate validation PR * fix import order * linux-env requirements files * fix v1 pom * SkipShade for Spanner common module (GoogleCloudPlatform#2194) * Add load test for cross db txn (GoogleCloudPlatform#2199) * Add load test for cross db txn * Change test timeout to 2 days * Revert spanner-pr.yml changes * Report Lineage for CsvToBigQuery template (GoogleCloudPlatform#2205) * Report Lineage for CsvToBigQuery template * Reply beampr-32662 to CsvConverters * Spanner Import/Export INTERLEAVE IN (GoogleCloudPlatform#2128) * Changes to write read interleave type from information schema, and write/read the type to/from avro. No change to tests yet. * Fix warnings, bug in InformationSchemaScanner, and bug in AvroSchemaToDdlConverter. Also properly default to IN PARENT when emitting ddl, in case the interleave type is not set (really only necessary for tests, since otherwise it will always be set. * Set interleaveType in InfoSchemaScanner * Style fixes, and only generate INTERLEAVE IN ddl for gsql. * another style fix - remove unused import * Make conditions more readable and add comments * Tests * Fix condition * Add interleave in table to ExportPipelineIT * Add SCRAM-SHA-512 authentication support to Kafka templates (GoogleCloudPlatform#2181) Added SCRAM-SHA-512 authentication support to Kafka to Kafka, Kafka to GCS, and Kafka to BigQuery templates. * Update the required Java version in the base doc, then regenerate docs (GoogleCloudPlatform#2218) * Update java requirement * Generate docs * Post 2.63.0 fixes (GoogleCloudPlatform#2216) * Uncomment kinesis * bump protoc to 4 in v2 * bug-fix: Use jdbc connection properties for reverse migration (GoogleCloudPlatform#2198) * changes * Changes * changes * changes * docs * Support partitioned reads for DateTime column type in JDBC to BigQuery template (GoogleCloudPlatform#2084) * Support partitioned reads for DateTime column type * minor changes * Support backward compatibility and timezone in lower/upper bounds * removed test cases for bounds in unit tests as derby does not support timezone * Fixed typo's in JdbcToBigQuery.java Corrected testcases expected output * Corrected unit test cases expected output. Removed DateTime integration test cases. * Corrected a typo * Added default value for partitionColumnType * spotless apply * Disabling flaky test to unblock dataflow release (GoogleCloudPlatform#2220) * disabling flaky test to unblock dataflow release * minor change * Add logic to skip runnerV2 for the ITs (GoogleCloudPlatform#2219) * skip runnerv2 tests in TemplateTestBase * add logic to skip use_runner_v2 experiment in the launchTemplate * Remove redundant property skipRunnerV2Test in pom and update logic in launchTemplate * Fix format violations using mvn spotless:apply * Add warning about caching with plugin (GoogleCloudPlatform#2221) * [DatastreamToSpanner] Spanner Exception handling (GoogleCloudPlatform#2185) * SpannerExceptionClassifier class and IT * Unit tests and Integration tests * Formatting changes * Correcting UT * Unit test for SpannerResourceManager * Creating Spanner Migration Exception * Formatting changes * Correcting tests * Addressing comments * Correcting a UT * Addressing comments * Changing SpannerMigrationException to extend RuntimeException * Added Cassandra Resource Manager Refectoring and removed Generics * Added Keyspace Voilation fixes * minor changes * Create session for row check --------- Co-authored-by: taherkl <taher.lakdawala@ollion.com> Co-authored-by: Akash Thawait <aakash@ollion.com> Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com> Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com> Co-authored-by: Dylan Hercher <dylanhercher@gmail.com> Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Co-authored-by: Yi Hu <yathu@google.com> Co-authored-by: Deep1998 <deepchowdhury1998@gmail.com> Co-authored-by: jjfox15 <seanfox@google.com> Co-authored-by: vgnanasekaran <venku123@gmail.com> Co-authored-by: Danny McCormick <dannymccormick@google.com> Co-authored-by: Sharan Teja M <sharantej957@gmail.com> Co-authored-by: shreyakhajanchi <92910380+shreyakhajanchi@users.noreply.github.com> Co-authored-by: Rudra-Gujarathi <rudragujarathi@google.com> Co-authored-by: Derrick Williams <derrickaw@google.com> Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com> * Cassandra wide row it (#140) --------- Co-authored-by: taherkl <taher.lakdawala@ollion.com> Co-authored-by: Taher Lakdawala <78196491+taherkl@users.noreply.github.com> Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com> Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com> Co-authored-by: Dylan Hercher <dylanhercher@gmail.com> Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Co-authored-by: Yi Hu <yathu@google.com> Co-authored-by: Deep1998 <deepchowdhury1998@gmail.com> Co-authored-by: jjfox15 <seanfox@google.com> Co-authored-by: vgnanasekaran <venku123@gmail.com> Co-authored-by: Danny McCormick <dannymccormick@google.com> Co-authored-by: Sharan Teja M <sharantej957@gmail.com> Co-authored-by: shreyakhajanchi <92910380+shreyakhajanchi@users.noreply.github.com> Co-authored-by: Rudra-Gujarathi <rudragujarathi@google.com> Co-authored-by: Derrick Williams <derrickaw@google.com> Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com> Co-authored-by: Svetak Sundhar <svetaksundhar@google.com> Co-authored-by: Shunping Huang <shunping@google.com> Co-authored-by: Andrej Galad <dedo.cibula@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
12 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.