Skip to content

Cassandra Schema and Value Mapping#2048

Merged
VardhanThigle merged 1 commit intoGoogleCloudPlatform:mainfrom
VardhanThigle:cassandra-mapping
Dec 14, 2024
Merged

Cassandra Schema and Value Mapping#2048
VardhanThigle merged 1 commit intoGoogleCloudPlatform:mainfrom
VardhanThigle:cassandra-mapping

Conversation

@VardhanThigle
Copy link
Copy Markdown
Contributor

@VardhanThigle VardhanThigle commented Dec 4, 2024 •
edited
Loading

Cassandra Schema and Data mapping for primitive types.

This PR covers mapping of primitive types to Avro for Cassandra.
As of this PR, all the non-primitive types are mapped to null.

TODOS:

  • end to end integration with pipeline options and CassandraIO.
  • integration and scale testing
  • support for non-primitive types.

@VardhanThigle VardhanThigle added addition New feature or request ignore-for-release labels Dec 4, 2024
@codecov
Copy link
Copy Markdown

codecov bot commented Dec 4, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 99.40828% with 1 line in your changes missing coverage. Please review.

Project coverage is 46.16%. Comparing base (ff60a5f) to head (b9af015).

Files with missing lines Patch % Lines
.../cassandra/mappings/CassandraMappingsProvider.java 98.00% 1 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##               main    #2048      +/-   ##
============================================
+ Coverage     45.98%   46.16%   +0.17%     
- Complexity     3812     4167     +355     
============================================
  Files           851      856       +5     
  Lines         50844    51010     +166     
  Branches       5359     5365       +6     
============================================
+ Hits          23383    23547     +164     
- Misses        25759    25760       +1     
- Partials       1702     1703       +1
Components Coverage Δ
spanner-templates 67.89% <99.32%> (+0.27%) ⬆️
spanner-import-export 65.57% <ø> (-0.02%) ⬇️
spanner-live-forward-migration 76.42% <100.00%> (+0.11%) ⬆️
spanner-live-reverse-replication 77.35% <100.00%> (+0.09%) ⬆️
spanner-bulk-migration 87.16% <99.32%> (+0.29%) ⬆️
Files with missing lines Coverage Δ
...tream/transforms/FormatDatastreamRecordToJson.java 42.58% <100.00%> (+3.95%) ⬆️
...eader/io/cassandra/mappings/CassandraMappings.java 100.00% <100.00%> (ø)
...r/io/cassandra/rowmapper/CassandraFieldMapper.java 100.00% <100.00%> (ø)
...der/io/cassandra/rowmapper/CassandraRowMapper.java 100.00% <100.00%> (ø)
.../cassandra/rowmapper/CassandraSourceRowMapper.java 100.00% <100.00%> (ø)
.../io/cassandra/schema/CassandraSchemaDiscovery.java 98.24% <ø> (ø)
...eader/io/schema/typemapping/UnifiedTypeMapper.java 100.00% <100.00%> (ø)
...ema/typemapping/provider/unified/CustomSchema.java 100.00% <100.00%> (ø)
...pping/provider/unified/UnifiedMappingProvider.java 100.00% <100.00%> (ø)
...er/migrations/avro/GenericRecordTypeConvertor.java 93.58% <100.00%> (+0.46%) ⬆️
... and 1 more

... and 1 file with indirect coverage changes

@VardhanThigle VardhanThigle force-pushed the cassandra-mapping branch 9 times, most recently from 9a82e35 to 3d23485 Compare December 11, 2024 10:03
@VardhanThigle VardhanThigle force-pushed the cassandra-mapping branch 6 times, most recently from 46039b0 to 452f060 Compare December 12, 2024 04:53
@VardhanThigle VardhanThigle marked this pull request as ready for review December 12, 2024 04:53
@VardhanThigle VardhanThigle requested a review from a team as a code owner December 12, 2024 04:53
@VardhanThigle VardhanThigle force-pushed the cassandra-mapping branch 2 times, most recently from 8118ab4 to 677859f Compare December 12, 2024 08:45
@VardhanThigle VardhanThigle requested a review from a team as a code owner December 12, 2024 08:45
@Deep1998
Copy link
Copy Markdown
Contributor

Deep1998 commented Dec 12, 2024

LGTM

bharadwaj-aditya
bharadwaj-aditya reviewed Dec 12, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@bharadwaj-aditya bharadwaj-aditya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine overall. Please address the comments before merging.

@VardhanThigle VardhanThigle force-pushed the cassandra-mapping branch 3 times, most recently from 37a95ec to b9af015 Compare December 13, 2024 04:31
VardhanThigle
VardhanThigle commented Dec 14, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor Author

@VardhanThigle VardhanThigle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IT tests are failing, due to

{"errors":[{"code":"TOOMANYREQUESTS","message":"Quota exceeded for quota metric 'Requests per project in the US multi-region' and limit 'Requests per project in the US multi-region per minute' of service 'artifactregistry.googleapis.com' for consumer 'project_number:167633336322'."}]}

The same workflow had passed on this PR 13 hours ago.

Even the scheduled workflow on main seems to be flaky.

@VardhanThigle VardhanThigle merged commit cd28d01 into GoogleCloudPlatform:main Dec 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bharadwaj-aditya bharadwaj-aditya bharadwaj-aditya left review comments

@darshan-sj darshan-sj Awaiting requested review from darshan-sj darshan-sj is a code owner automatically assigned from GoogleCloudPlatform/spanner-migrations-team

@shreyakhajanchi shreyakhajanchi Awaiting requested review from shreyakhajanchi shreyakhajanchi is a code owner automatically assigned from GoogleCloudPlatform/spanner-migrations-team

+1 more reviewer

@Deep1998 Deep1998 Deep1998 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

addition New feature or request ignore-for-release size/XXL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@VardhanThigle @Deep1998 @bharadwaj-aditya