Releases · GoogleCloudDataproc/spark-bigquery-connector

Added new connector, spark-4.1-bigquery aimed to be used in Spark 4.1. Like Spark 4.1, this connector requires at
least Java 17 runtime. It is currently in preview mode.
spark-4.0-bigquery is generally available!
PR #1445: Add streaming support for Spark DS v2 indirect write.
PR #1452: Improved the performance of the dynamic partition overwrite for RANGE_BUCKET partitioned tables.
PR #1434: Support per-catalog configuration for project and location for BigQueryCatalog.

Added new connector, spark-4.0-bigquery aimed to be used in Spark 4.0. Like Spark 4.0, this connector requires at
least Java 17 runtime. It is currently in preview mode.
PR #1367: Query Pushdown is no longer supported.
PR #1369: Catalog enhancements
PR #1374: Ensure TableId includes project ID if not explicitly set
PR #1376: materializationDataset is now optional to read from views or queries.
PR #1380: Fixed ImpersonatedCredentials serialization
PR #1381: Added the option to set custom credentials scopes
PR #1411: Added Support for SparkSession#executeCommand
BigQuery API has been upgraded to version 2.54.0
BigQuery Storage API has been upgraded to version 3.16.1
GAX has been upgraded to version 2.68.2
gRPC has been upgraded to version 1.74.0

@ddebowczyk92

@tom-s-powell

CVE-2025-24970, CVE-2025-25193: Upgrading netty to version 4.1.119.Final
PR #1284: Making BigQueryClientFactory Kryo serializable. Thanks @tom-s-powell !
PR #1345: materializationDataset is no longer needed to read from views or queries.

Issue #1254: fix getting partitioning fields for pseudo columns

Releases: GoogleCloudDataproc/spark-bigquery-connector