WideRowHandling: spanner to source db IT #2252

akashthawaitcc · 2025-03-17T14:23:10Z

This PR introduces validation checks to enforce various constraints on table and column creation, ensuring compliance with database limitations.
Changes Implemented

Table Name Length Constraint (1 to 128 characters)

Successfully creates tables with names up to 128 characters.
Fails for names exceeding 128 characters.

Maximum Columns per Table (1,024)

Successfully creates tables with up to 1,024 columns.
Fails for tables exceeding 1,024 columns.

Column Name Length Constraint (1 to 128 characters)

Successfully creates columns with names up to 128 characters.
Fails for column names exceeding 128 characters.

String Cell Size Constraint (Max 2,621,440 Unicode characters)

Successfully creates a table with a string column of 2,621,440 characters.
Fails when exceeding the limit.

Sync master

Sync main branch

tests: Adding Forward Migration Tests (GoogleCloudPlatform#2001)

Sync main branch

Use [self-hosted, it] for prepare java cache workflow (GoogleCloudPlatform#2080)

Sync main branch

Metadata config and pipeline options (GoogleCloudPlatform#2081)

fix: eager source row fetching logic (GoogleCloudPlatform#2071)

sync upstream/main

Sync main with upstream

Sync main branch

Sync main

Support avro arrays for postgres insertion. (GoogleCloudPlatform#2154)

Referesh `SpannerToSourceDbCustomTransformationIT` tables for re-runs…

Sycn main

Sync main

Co-authored-by: pawankashyapollion <[email protected]>

* Adding test table for Map types * Adding support for cassandra map

* add inf fix to ts fixing bug on timestamp type * Update FormatDatastreamRecordToJson.java use equals not ==

* Upgrade Beam version to 2.63.0 * add cache factory to local spanner io * remove cache pass to ReadChangeStreamPartitionDoFn * Add unimplemented/unused stubs to TestChangeStreamMutation * Use java Instant instead of threeten Instant * Fix low watermark setter call * fix testchangestreammutation * Recreate validation PR * fix import order * linux-env requirements files * fix v1 pom

…#2231) * Removed LocalSpannerIO * Moving all of SpannerIO into Teleport * Copying tests also * Updated tests and excluded SpannerIO changestream from coverage checks * spotless apply * Excluding coverage check * Adding retry settings which were overwritten in LocalSpannerAccessor * Changing deprecated retrysettings function calls * Added warning and TODO comments to remove the local copy * spotless

…eCloudPlatform#2239) * Bump timeouts for datastream to spanner test * Update FK timeout for reverse template * Bump timeouts for old reverse repl template * Enable SpannerToSourceDbInterleaveMultiShardIT * Revert loadtest timeout

* * Addition of Load Tests in SpannerToSourceDB For Cassandra (#89) * Addition of Load Tests in SpannerToSourceDB For Cassandra * Address Merge conflict * Added LT Refectored (#92) * Added POM Dependecies * sync upstream/main (#98) * RR LOAD TEST FIXES (#101) * Resolved PR comments (#115) * Added Module Dependency Fixes * Added Copyrigh * Added missing commit * Enhanced Retry Logic (GoogleCloudPlatform#2196) Co-authored-by: pawankashyapollion <[email protected]> * Adding support for Cassandra map (GoogleCloudPlatform#2209) * Adding test table for Map types * Adding support for cassandra map * changes (GoogleCloudPlatform#2212) * Fix inf issues in Datastream reader (GoogleCloudPlatform#2213) * add inf fix to ts fixing bug on timestamp type * Update FormatDatastreamRecordToJson.java use equals not == * Upgrade Beam version to 2.63.0 (GoogleCloudPlatform#2206) * Upgrade Beam version to 2.63.0 * add cache factory to local spanner io * remove cache pass to ReadChangeStreamPartitionDoFn * Add unimplemented/unused stubs to TestChangeStreamMutation * Use java Instant instead of threeten Instant * Fix low watermark setter call * fix testchangestreammutation * Recreate validation PR * fix import order * linux-env requirements files * fix v1 pom * SkipShade for Spanner common module (GoogleCloudPlatform#2194) * Add load test for cross db txn (GoogleCloudPlatform#2199) * Add load test for cross db txn * Change test timeout to 2 days * Revert spanner-pr.yml changes * Report Lineage for CsvToBigQuery template (GoogleCloudPlatform#2205) * Report Lineage for CsvToBigQuery template * Reply beampr-32662 to CsvConverters * Spanner Import/Export INTERLEAVE IN (GoogleCloudPlatform#2128) * Changes to write read interleave type from information schema, and write/read the type to/from avro. No change to tests yet. * Fix warnings, bug in InformationSchemaScanner, and bug in AvroSchemaToDdlConverter. Also properly default to IN PARENT when emitting ddl, in case the interleave type is not set (really only necessary for tests, since otherwise it will always be set. * Set interleaveType in InfoSchemaScanner * Style fixes, and only generate INTERLEAVE IN ddl for gsql. * another style fix - remove unused import * Make conditions more readable and add comments * Tests * Fix condition * Add interleave in table to ExportPipelineIT * Add SCRAM-SHA-512 authentication support to Kafka templates (GoogleCloudPlatform#2181) Added SCRAM-SHA-512 authentication support to Kafka to Kafka, Kafka to GCS, and Kafka to BigQuery templates. * Update the required Java version in the base doc, then regenerate docs (GoogleCloudPlatform#2218) * Update java requirement * Generate docs * Post 2.63.0 fixes (GoogleCloudPlatform#2216) * Uncomment kinesis * bump protoc to 4 in v2 * bug-fix: Use jdbc connection properties for reverse migration (GoogleCloudPlatform#2198) * changes * Changes * changes * changes * docs * Support partitioned reads for DateTime column type in JDBC to BigQuery template (GoogleCloudPlatform#2084) * Support partitioned reads for DateTime column type * minor changes * Support backward compatibility and timezone in lower/upper bounds * removed test cases for bounds in unit tests as derby does not support timezone * Fixed typo's in JdbcToBigQuery.java Corrected testcases expected output * Corrected unit test cases expected output. Removed DateTime integration test cases. * Corrected a typo * Added default value for partitionColumnType * spotless apply * Disabling flaky test to unblock dataflow release (GoogleCloudPlatform#2220) * disabling flaky test to unblock dataflow release * minor change * Add logic to skip runnerV2 for the ITs (GoogleCloudPlatform#2219) * skip runnerv2 tests in TemplateTestBase * add logic to skip use_runner_v2 experiment in the launchTemplate * Remove redundant property skipRunnerV2Test in pom and update logic in launchTemplate * Fix format violations using mvn spotless:apply * Add warning about caching with plugin (GoogleCloudPlatform#2221) * [DatastreamToSpanner] Spanner Exception handling (GoogleCloudPlatform#2185) * SpannerExceptionClassifier class and IT * Unit tests and Integration tests * Formatting changes * Correcting UT * Unit test for SpannerResourceManager * Creating Spanner Migration Exception * Formatting changes * Correcting tests * Addressing comments * Correcting a UT * Addressing comments * Changing SpannerMigrationException to extend RuntimeException * Added Cassandra Resource Manager Refectoring and removed Generics * Added Keyspace Voilation fixes * minor changes * Create session for row check --------- Co-authored-by: taherkl <[email protected]> Co-authored-by: Akash Thawait <[email protected]> Co-authored-by: pawankashyapollion <[email protected]> Co-authored-by: Vardhan Vinay Thigle <[email protected]> Co-authored-by: Astha Mohta <[email protected]> Co-authored-by: Dylan Hercher <[email protected]> Co-authored-by: Jack McCluskey <[email protected]> Co-authored-by: Yi Hu <[email protected]> Co-authored-by: Deep1998 <[email protected]> Co-authored-by: jjfox15 <[email protected]> Co-authored-by: vgnanasekaran <[email protected]> Co-authored-by: Danny McCormick <[email protected]> Co-authored-by: Sharan Teja M <[email protected]> Co-authored-by: shreyakhajanchi <[email protected]> Co-authored-by: Rudra-Gujarathi <[email protected]> Co-authored-by: Derrick Williams <[email protected]> Co-authored-by: darshan-sj <[email protected]>

* sync upstream/main (#98) * Enhanced Retry Logic (GoogleCloudPlatform#2196) Co-authored-by: pawankashyapollion <[email protected]> * Adding support for Cassandra map (GoogleCloudPlatform#2209) * Adding test table for Map types * Adding support for cassandra map * changes (GoogleCloudPlatform#2212) * Fix inf issues in Datastream reader (GoogleCloudPlatform#2213) * add inf fix to ts fixing bug on timestamp type * Update FormatDatastreamRecordToJson.java use equals not == * Upgrade Beam version to 2.63.0 (GoogleCloudPlatform#2206) * Upgrade Beam version to 2.63.0 * add cache factory to local spanner io * remove cache pass to ReadChangeStreamPartitionDoFn * Add unimplemented/unused stubs to TestChangeStreamMutation * Use java Instant instead of threeten Instant * Fix low watermark setter call * fix testchangestreammutation * Recreate validation PR * fix import order * linux-env requirements files * fix v1 pom * SkipShade for Spanner common module (GoogleCloudPlatform#2194) * Add load test for cross db txn (GoogleCloudPlatform#2199) * Add load test for cross db txn * Change test timeout to 2 days * Revert spanner-pr.yml changes * Report Lineage for CsvToBigQuery template (GoogleCloudPlatform#2205) * Report Lineage for CsvToBigQuery template * Reply beampr-32662 to CsvConverters * Spanner Import/Export INTERLEAVE IN (GoogleCloudPlatform#2128) * Changes to write read interleave type from information schema, and write/read the type to/from avro. No change to tests yet. * Fix warnings, bug in InformationSchemaScanner, and bug in AvroSchemaToDdlConverter. Also properly default to IN PARENT when emitting ddl, in case the interleave type is not set (really only necessary for tests, since otherwise it will always be set. * Set interleaveType in InfoSchemaScanner * Style fixes, and only generate INTERLEAVE IN ddl for gsql. * another style fix - remove unused import * Make conditions more readable and add comments * Tests * Fix condition * Add interleave in table to ExportPipelineIT * Add SCRAM-SHA-512 authentication support to Kafka templates (GoogleCloudPlatform#2181) Added SCRAM-SHA-512 authentication support to Kafka to Kafka, Kafka to GCS, and Kafka to BigQuery templates. * Update the required Java version in the base doc, then regenerate docs (GoogleCloudPlatform#2218) * Update java requirement * Generate docs * Post 2.63.0 fixes (GoogleCloudPlatform#2216) * Uncomment kinesis * bump protoc to 4 in v2 * bug-fix: Use jdbc connection properties for reverse migration (GoogleCloudPlatform#2198) * changes * Changes * changes * changes * docs * Support partitioned reads for DateTime column type in JDBC to BigQuery template (GoogleCloudPlatform#2084) * Support partitioned reads for DateTime column type * minor changes * Support backward compatibility and timezone in lower/upper bounds * removed test cases for bounds in unit tests as derby does not support timezone * Fixed typo's in JdbcToBigQuery.java Corrected testcases expected output * Corrected unit test cases expected output. Removed DateTime integration test cases. * Corrected a typo * Added default value for partitionColumnType * spotless apply * Disabling flaky test to unblock dataflow release (GoogleCloudPlatform#2220) * disabling flaky test to unblock dataflow release * minor change * Add logic to skip runnerV2 for the ITs (GoogleCloudPlatform#2219) * skip runnerv2 tests in TemplateTestBase * add logic to skip use_runner_v2 experiment in the launchTemplate * Remove redundant property skipRunnerV2Test in pom and update logic in launchTemplate * Fix format violations using mvn spotless:apply * Add warning about caching with plugin (GoogleCloudPlatform#2221) * [DatastreamToSpanner] Spanner Exception handling (GoogleCloudPlatform#2185) * SpannerExceptionClassifier class and IT * Unit tests and Integration tests * Formatting changes * Correcting UT * Unit test for SpannerResourceManager * Creating Spanner Migration Exception * Formatting changes * Correcting tests * Addressing comments * Correcting a UT * Addressing comments * Changing SpannerMigrationException to extend RuntimeException * Remove Python version from `pom.xml` (GoogleCloudPlatform#2234) * Update pom.xml * Update pom.xml * Add SkipRunnerV2Test category to JmsToPubsubIT and PubSubCdcToBigQueryIT (GoogleCloudPlatform#2235) * Fix a bug in CSVToBigQuery where commas in fields are not handled correctly. (GoogleCloudPlatform#2229) * Attempt to fix csv bug where commas in fields are not handled correctly. * Replace ImmutableList with Iterable * Add tests to cover the scenario of commas within quotes. * Polish tests and add a test case to cover csv without headers. * Update Dockerfile-template-yaml (GoogleCloudPlatform#2222) * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Update Dockerfile-template-yaml * Adding All Datatypes IT for Cassandra Migration (GoogleCloudPlatform#2230) * Add IF NOT EXISTS clause for spanner ddls used in ITs (GoogleCloudPlatform#2237) * Enable DatastreamToSpannerIT with if not exists clause * Update Datastream to Spanner IT spanner schemas with if not exists * Update BULK IT spanner schemas with if not exists * Update reverse replications ITs spanner schemas with if not exists * Add space after if not exists * Using set of random buckets for spanner ITs (GoogleCloudPlatform#2223) * Using set of random buckets for spanner ITs * reverse replication test * checkstyle fix * adding more buckets * removed ignore for testing * spotless fix * fixing UT * skip the flaky test again * addressing comments * spotless fix * Add promote artifact method in release plugin (GoogleCloudPlatform#2227) * Add promote artifact method * address comments; also fixed stagingArtifactRegistry support us.gcr.io * Consolidate method * fixed default DLQ path (GoogleCloudPlatform#2241) * Fix stagingArtifactRegistry support raw us.gcr.io artifact registry (GoogleCloudPlatform#2243) * Print error response on wget call (GoogleCloudPlatform#2245) * This helps debugging e.g. permission issues * Moving local spanner io to a different namespace (GoogleCloudPlatform#2231) * Removed LocalSpannerIO * Moving all of SpannerIO into Teleport * Copying tests also * Updated tests and excluded SpannerIO changestream from coverage checks * spotless apply * Excluding coverage check * Adding retry settings which were overwritten in LocalSpannerAccessor * Changing deprecated retrysettings function calls * Added warning and TODO comments to remove the local copy * spotless * Adding Cassandra Type Options to IT test (GoogleCloudPlatform#2242) * Bump timeouts for tests involving FKs/interleaved dependenceis (GoogleCloudPlatform#2239) * Bump timeouts for datastream to spanner test * Update FK timeout for reverse template * Bump timeouts for old reverse repl template * Enable SpannerToSourceDbInterleaveMultiShardIT * Revert loadtest timeout * Load Tests - Cassandra Reverse Replication (GoogleCloudPlatform#2163) * * Addition of Load Tests in SpannerToSourceDB For Cassandra (#89) * Addition of Load Tests in SpannerToSourceDB For Cassandra * Address Merge conflict * Added LT Refectored (#92) * Added POM Dependecies * sync upstream/main (#98) * RR LOAD TEST FIXES (#101) * Resolved PR comments (#115) * Added Module Dependency Fixes * Added Copyrigh * Added missing commit * Enhanced Retry Logic (GoogleCloudPlatform#2196) Co-authored-by: pawankashyapollion <[email protected]> * Adding support for Cassandra map (GoogleCloudPlatform#2209) * Adding test table for Map types * Adding support for cassandra map * changes (GoogleCloudPlatform#2212) * Fix inf issues in Datastream reader (GoogleCloudPlatform#2213) * add inf fix to ts fixing bug on timestamp type * Update FormatDatastreamRecordToJson.java use equals not == * Upgrade Beam version to 2.63.0 (GoogleCloudPlatform#2206) * Upgrade Beam version to 2.63.0 * add cache factory to local spanner io * remove cache pass to ReadChangeStreamPartitionDoFn * Add unimplemented/unused stubs to TestChangeStreamMutation * Use java Instant instead of threeten Instant * Fix low watermark setter call * fix testchangestreammutation * Recreate validation PR * fix import order * linux-env requirements files * fix v1 pom * SkipShade for Spanner common module (GoogleCloudPlatform#2194) * Add load test for cross db txn (GoogleCloudPlatform#2199) * Add load test for cross db txn * Change test timeout to 2 days * Revert spanner-pr.yml changes * Report Lineage for CsvToBigQuery template (GoogleCloudPlatform#2205) * Report Lineage for CsvToBigQuery template * Reply beampr-32662 to CsvConverters * Spanner Import/Export INTERLEAVE IN (GoogleCloudPlatform#2128) * Changes to write read interleave type from information schema, and write/read the type to/from avro. No change to tests yet. * Fix warnings, bug in InformationSchemaScanner, and bug in AvroSchemaToDdlConverter. Also properly default to IN PARENT when emitting ddl, in case the interleave type is not set (really only necessary for tests, since otherwise it will always be set. * Set interleaveType in InfoSchemaScanner * Style fixes, and only generate INTERLEAVE IN ddl for gsql. * another style fix - remove unused import * Make conditions more readable and add comments * Tests * Fix condition * Add interleave in table to ExportPipelineIT * Add SCRAM-SHA-512 authentication support to Kafka templates (GoogleCloudPlatform#2181) Added SCRAM-SHA-512 authentication support to Kafka to Kafka, Kafka to GCS, and Kafka to BigQuery templates. * Update the required Java version in the base doc, then regenerate docs (GoogleCloudPlatform#2218) * Update java requirement * Generate docs * Post 2.63.0 fixes (GoogleCloudPlatform#2216) * Uncomment kinesis * bump protoc to 4 in v2 * bug-fix: Use jdbc connection properties for reverse migration (GoogleCloudPlatform#2198) * changes * Changes * changes * changes * docs * Support partitioned reads for DateTime column type in JDBC to BigQuery template (GoogleCloudPlatform#2084) * Support partitioned reads for DateTime column type * minor changes * Support backward compatibility and timezone in lower/upper bounds * removed test cases for bounds in unit tests as derby does not support timezone * Fixed typo's in JdbcToBigQuery.java Corrected testcases expected output * Corrected unit test cases expected output. Removed DateTime integration test cases. * Corrected a typo * Added default value for partitionColumnType * spotless apply * Disabling flaky test to unblock dataflow release (GoogleCloudPlatform#2220) * disabling flaky test to unblock dataflow release * minor change * Add logic to skip runnerV2 for the ITs (GoogleCloudPlatform#2219) * skip runnerv2 tests in TemplateTestBase * add logic to skip use_runner_v2 experiment in the launchTemplate * Remove redundant property skipRunnerV2Test in pom and update logic in launchTemplate * Fix format violations using mvn spotless:apply * Add warning about caching with plugin (GoogleCloudPlatform#2221) * [DatastreamToSpanner] Spanner Exception handling (GoogleCloudPlatform#2185) * SpannerExceptionClassifier class and IT * Unit tests and Integration tests * Formatting changes * Correcting UT * Unit test for SpannerResourceManager * Creating Spanner Migration Exception * Formatting changes * Correcting tests * Addressing comments * Correcting a UT * Addressing comments * Changing SpannerMigrationException to extend RuntimeException * Added Cassandra Resource Manager Refectoring and removed Generics * Added Keyspace Voilation fixes * minor changes * Create session for row check --------- Co-authored-by: taherkl <[email protected]> Co-authored-by: Akash Thawait <[email protected]> Co-authored-by: pawankashyapollion <[email protected]> Co-authored-by: Vardhan Vinay Thigle <[email protected]> Co-authored-by: Astha Mohta <[email protected]> Co-authored-by: Dylan Hercher <[email protected]> Co-authored-by: Jack McCluskey <[email protected]> Co-authored-by: Yi Hu <[email protected]> Co-authored-by: Deep1998 <[email protected]> Co-authored-by: jjfox15 <[email protected]> Co-authored-by: vgnanasekaran <[email protected]> Co-authored-by: Danny McCormick <[email protected]> Co-authored-by: Sharan Teja M <[email protected]> Co-authored-by: shreyakhajanchi <[email protected]> Co-authored-by: Rudra-Gujarathi <[email protected]> Co-authored-by: Derrick Williams <[email protected]> Co-authored-by: darshan-sj <[email protected]> * Cassandra wide row it (#140) --------- Co-authored-by: taherkl <[email protected]> Co-authored-by: Taher Lakdawala <[email protected]> Co-authored-by: pawankashyapollion <[email protected]> Co-authored-by: Vardhan Vinay Thigle <[email protected]> Co-authored-by: Astha Mohta <[email protected]> Co-authored-by: Dylan Hercher <[email protected]> Co-authored-by: Jack McCluskey <[email protected]> Co-authored-by: Yi Hu <[email protected]> Co-authored-by: Deep1998 <[email protected]> Co-authored-by: jjfox15 <[email protected]> Co-authored-by: vgnanasekaran <[email protected]> Co-authored-by: Danny McCormick <[email protected]> Co-authored-by: Sharan Teja M <[email protected]> Co-authored-by: shreyakhajanchi <[email protected]> Co-authored-by: Rudra-Gujarathi <[email protected]> Co-authored-by: Derrick Williams <[email protected]> Co-authored-by: darshan-sj <[email protected]> Co-authored-by: Svetak Sundhar <[email protected]> Co-authored-by: Shunping Huang <[email protected]> Co-authored-by: Andrej Galad <[email protected]>

google-cla · 2025-03-17T14:23:28Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

…-fork

codecov · 2025-03-19T14:42:09Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 49.25%. Comparing base (f2950f3) to head (6688c71).
Report is 4 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2252      +/-   ##
============================================
+ Coverage     49.21%   49.25%   +0.03%     
- Complexity     4607     4952     +345     
============================================
  Files           918      924       +6     
  Lines         55722    55777      +55     
  Branches       5985     5992       +7     
============================================
+ Hits          27425    27473      +48     
- Misses        26320    26329       +9     
+ Partials       1977     1975       -2

Components	Coverage Δ
spanner-templates	`70.24% <90.47%> (+0.03%)`	⬆️
spanner-import-export	`68.05% <ø> (+0.01%)`	⬆️
spanner-live-forward-migration	`78.42% <ø> (ø)`
spanner-live-reverse-replication	`80.40% <90.47%> (+0.09%)`	⬆️
spanner-bulk-migration	`87.77% <ø> (ø)`

see 9 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

* Added allowes Packet Size * Added and remove unwanted Boundry Check

* Added allowes Packet Size * Added and remove unwanted Boundry Check * Added Foxes for Max in Size IT * Added Fixes for max cols * Added 10mb * Added Fixes * Added Fixes * Spotless fixes * Added Fixes * Added FIxes * removed unwanted

asthamohta · 2025-03-26T04:51:42Z

.../java/com/google/cloud/teleport/v2/templates/SpannerToCassandraSourceDbMaxColumnsSizeIT.java

+    }
+  }
+
+  /** Writes a row with 1,024 columns in Spanner and verifies replication to Cassandra. */


Fix this comment and add what the actual approximate size of row is here

@asthamohta Updated the comments

Sync main

taherkl and others added 30 commits November 27, 2024 20:19

Merge remote-tracking branch 'upstream/main'

15b621d

Merge pull request #9 from GoogleCloudPlatform/main

d314c70

Sync master

Merge pull request #20 from GoogleCloudPlatform/main

71a6477

Sync main branch

Merge pull request #24 from GoogleCloudPlatform/main

7512825

Sync main branch

Merge pull request #26 from GoogleCloudPlatform/main

5aa21dc

tests: Adding Forward Migration Tests (GoogleCloudPlatform#2001)

Merge pull request #28 from GoogleCloudPlatform/main

04bac39

Sync main branch

Merge pull request #29 from GoogleCloudPlatform/main

f618128

Use [self-hosted, it] for prepare java cache workflow (GoogleCloudPlatform#2080)

Merge pull request #32 from GoogleCloudPlatform/main

e53e8e4

Sync main branch

Merge pull request #36 from GoogleCloudPlatform/main

407a593

Sync main branch

Merge pull request #40 from GoogleCloudPlatform/main

0d140e6

Sync main branch

Merge pull request #46 from GoogleCloudPlatform/main

ae3037a

Sync main branch

Merge pull request #48 from GoogleCloudPlatform/main

ef5ae8d

Metadata config and pipeline options (GoogleCloudPlatform#2081)

Merge pull request #52 from GoogleCloudPlatform/main

c6442ba

fix: eager source row fetching logic (GoogleCloudPlatform#2071)

Merge pull request #55 from GoogleCloudPlatform/main

7c02a96

sync upstream/main

Merge pull request #79 from GoogleCloudPlatform/main

e3f4369

Sync main with upstream

Merge pull request #83 from GoogleCloudPlatform/main

5d5f1a8

Sync main branch

Merge pull request #84 from GoogleCloudPlatform/main

f1a4d1e

Sync main

Merge pull request #87 from GoogleCloudPlatform/main

22e677f

Support avro arrays for postgres insertion. (GoogleCloudPlatform#2154)

sync upstream/main (#98)

ee9c99a

Merge remote-tracking branch 'upstream/main'

186075f

Merge remote-tracking branch 'upstream/main'

c4fff0f

Merge pull request #107 from GoogleCloudPlatform/main

dc5e904

Referesh `SpannerToSourceDbCustomTransformationIT` tables for re-runs…

Merge pull request #116 from GoogleCloudPlatform/main

6a55499

Sycn main

Merge pull request #120 from GoogleCloudPlatform/main

8504aeb

Sync main

Enhanced Retry Logic (GoogleCloudPlatform#2196)

15b906b

Co-authored-by: pawankashyapollion <[email protected]>

Adding support for Cassandra map (GoogleCloudPlatform#2209)

421f5f3

* Adding test table for Map types * Adding support for cassandra map

changes (GoogleCloudPlatform#2212)

afff9c2

Fix inf issues in Datastream reader (GoogleCloudPlatform#2213)

06310d9

* add inf fix to ts fixing bug on timestamp type * Update FormatDatastreamRecordToJson.java use equals not ==

SkipShade for Spanner common module (GoogleCloudPlatform#2194)

9d5a0d2

darshan-sj and others added 5 commits March 17, 2025 10:52

Adding Cassandra Type Options to IT test (GoogleCloudPlatform#2242)

1acbfb9

akashthawaitcc requested a review from a team as a code owner March 17, 2025 14:23

akashthawaitcc requested review from asthamohta and VardhanThigle March 17, 2025 14:23

pull-request-size bot added the size/XXL label Mar 17, 2025

pawankashyapollion and others added 4 commits March 18, 2025 17:13

Merge branch 'main' of https://github.com/ollionorg/DataflowTemplates…

62f0f65

…-fork

Resolve Merge Conflict

6aaeeb3

resolve merge conflict

ed12948

Merge branch 'GoogleCloudPlatform:main' into main

de32155

pawankashyapollion and others added 4 commits March 21, 2025 17:55

Wide row rr fixes (#155)

f5d67a6

* Added allowes Packet Size * Added and remove unwanted Boundry Check

Merge branch 'GoogleCloudPlatform:main' into main

8f4bd23

IT Resource Exhausted Fixes

f2dc699

* Added allowes Packet Size * Added and remove unwanted Boundry Check * Added Foxes for Max in Size IT * Added Fixes for max cols * Added 10mb * Added Fixes * Added Fixes * Spotless fixes * Added Fixes * Added FIxes * removed unwanted

Merge branch 'GoogleCloudPlatform:main' into main

318bbeb

asthamohta reviewed Mar 26, 2025

View reviewed changes

pawankashyapollion and others added 10 commits March 26, 2025 14:26

updated the Comments

9991aa6

Merge branch 'GoogleCloudPlatform:main' into main

c2ffd28

Merge branch 'GoogleCloudPlatform:main' into main

e3f05ca

Merge pull request #173 from GoogleCloudPlatform/main

7be05ba

Sync main

Merge branch 'main' into wide_row/spanner_to_sourcedb_it

4b9be45

Added this Line to fix build issue

6688c71

Added Session file update

c61e90f

Splotless fixes

76e199d

spanner Schamea

d34e7d8

Mysql FIxes

73e279d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WideRowHandling: spanner to source db IT #2252

WideRowHandling: spanner to source db IT #2252

akashthawaitcc commented Mar 17, 2025

google-cla bot commented Mar 17, 2025

codecov bot commented Mar 19, 2025 •

edited

Loading

asthamohta Mar 26, 2025

pawankashyapollion Mar 26, 2025

WideRowHandling: spanner to source db IT #2252

Are you sure you want to change the base?

WideRowHandling: spanner to source db IT #2252

Conversation

akashthawaitcc commented Mar 17, 2025

google-cla bot commented Mar 17, 2025

codecov bot commented Mar 19, 2025 • edited Loading

Codecov Report

asthamohta Mar 26, 2025

Choose a reason for hiding this comment

pawankashyapollion Mar 26, 2025

Choose a reason for hiding this comment

codecov bot commented Mar 19, 2025 •

edited

Loading