Skip to content
This repository was archived by the owner on Feb 24, 2026. It is now read-only.
This repository was archived by the owner on Feb 24, 2026. It is now read-only.

Stream writer with custom mapper #2039

@arturowczarek

Description

@arturowczarek

Background: After migrating one of the services from Insert API to Write API, we've observed a significant performance decline. The JSON to Protobuf conversion step is using 2/3 of CPU time. Our application internally uses Avro records, transforms them to JSON and sends with JsonStreamWriter. The transformation to and from JSON is unnecessary and we should skip it.

There should be a generic class similar to JsonStreamWriter where the developer could provide their own mapper. The mapper should have access to Protobuf (table) schema. It should be able to make not only direct transformations but also some adjustments (for example transforming maps to lists). The generic writer class should take care of schema refreshes. The mapper should inform in a uniform manner that retriable error occurred (and schema should be retried).

Metadata

Metadata

Assignees

No one assigned

    Labels

    api: bigquerystorageIssues related to the googleapis/java-bigquerystorage API.type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions