Skip to content

feat: introduce new configuration properties for bigquery source#2049

Merged
Abacn merged 1 commit intoGoogleCloudPlatform:mainfrom
Emrehzl94:query-temp-project-and-dataset
Dec 9, 2024
Merged

feat: introduce new configuration properties for bigquery source#2049
Abacn merged 1 commit intoGoogleCloudPlatform:mainfrom
Emrehzl94:query-temp-project-and-dataset

Conversation

@Emrehzl94
Copy link
Copy Markdown
Contributor

@Emrehzl94 Emrehzl94 commented Dec 4, 2024

This PR introduces query_temp_project and query_temp_dataset configuration properties, allowing temporary data to be written to a specified project and dataset for BigQuery sources.

Example configuration;

{
     "type": "bigquery",
     "name": "persons",
     "query": "SELECT person_tmdbId, name, bornIn, born, died FROM team-connectors-dev.movies.persons",
     "query_temp_project": "team-connectors-dev",
     "query_temp_dataset": "test_temp_dataset_beam"
}

venikkin
venikkin approved these changes Dec 5, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@venikkin venikkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@liferoad liferoad added the improvement Making existing code better label Dec 9, 2024
@codecov
Copy link
Copy Markdown

codecov bot commented Dec 9, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 40.67797% with 35 lines in your changes missing coverage. Please review.

Project coverage is 45.95%. Comparing base (91b5be3) to head (ee6150c).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...port/v2/neo4j/providers/bigquery/BqQueryToRow.java 0.00% 12 Missing ⚠️
.../teleport/v2/neo4j/model/helpers/BigQuerySpec.java 0.00% 11 Missing ⚠️
...port/v2/neo4j/providers/bigquery/BigQueryImpl.java 0.00% 10 Missing ⚠️
...eleport/v2/neo4j/model/sources/BigQuerySource.java 87.50% 1 Missing ⚠️
...idation/BigQuerySourceProjectDatasetValidator.java 92.30% 0 Missing and 1 partial ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##               main    #2049      +/-   ##
============================================
- Coverage     45.95%   45.95%   -0.01%     
- Complexity     3792     3799       +7     
============================================
  Files           850      851       +1     
  Lines         50764    50805      +41     
  Branches       5340     5343       +3     
============================================
+ Hits          23331    23348      +17     
- Misses        25741    25763      +22     
- Partials       1692     1694       +2
Components Coverage Δ
spanner-templates 67.57% <ø> (-0.02%) ⬇️
spanner-import-export 65.47% <ø> (-0.04%) ⬇️
spanner-live-forward-migration 76.30% <ø> (ø)
spanner-live-reverse-replication 77.25% <ø> (ø)
spanner-bulk-migration 86.87% <ø> (ø)
Files with missing lines Coverage Δ
...v2/neo4j/model/sources/BigQuerySourceProvider.java 100.00% <100.00%> (ø)
...eleport/v2/neo4j/model/sources/BigQuerySource.java 61.90% <87.50%> (+15.23%) ⬆️
...idation/BigQuerySourceProjectDatasetValidator.java 92.30% <92.30%> (ø)
...port/v2/neo4j/providers/bigquery/BigQueryImpl.java 0.00% <0.00%> (ø)
.../teleport/v2/neo4j/model/helpers/BigQuerySpec.java 0.00% <0.00%> (ø)
...port/v2/neo4j/providers/bigquery/BqQueryToRow.java 0.00% <0.00%> (ø)

... and 3 files with indirect coverage changes

@Abacn Abacn merged commit ae17903 into GoogleCloudPlatform:main Dec 9, 2024
@Emrehzl94 Emrehzl94 deleted the query-temp-project-and-dataset branch December 26, 2024 18:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liferoad liferoad liferoad approved these changes

+1 more reviewer

@venikkin venikkin venikkin approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

improvement Making existing code better size/L

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Emrehzl94 @venikkin @liferoad @Abacn