Storage write api - support default stream#226
Conversation
…t-stream Storage write api support default stream
|
+1 to this - @goccy is this repo actively maintained still? |
|
+1 actively need this as well as the default streaming method uses the "_default" stream and you can't use connections pool without that : I tried a workaround for this consisting of creating the "_default" write stream after the table but It seems like the API doesn't allow us to set the name of the stream -> With the following code, the stream created have a randomly generated name. |
|
+1 i need this with go client. Here's my example Go code using the managedwriter package to request the bigquery emulator: ctx := context.Background()
grpcCtx, _ := context.WithTimeout(ctx, 1*time.Second)
conn, _ := grpc.DialContext(grpcCtx, "localhost:9060",
grpc.WithTransportCredentials(insecure.NewCredentials()),
grpc.WithIdleTimeout(1*time.Second),
)
client, err := managedwriter.NewClient(ctx, projectID,
option.WithGRPCConn(conn),
option.WithoutAuthentication(),
) // OK
pb := &v1.pb{}
descriptorProto, err := adapt.NormalizeDescriptor(pb.ProtoReflect().Descriptor())
tableName := fmt.Sprintf("projects/%s/datasets/%s/tables/%s", projectID, datasetID, "{table_name}")
stream, err := client.NewManagedStream(ctx,
managedwriter.WithDestinationTable(tableName),
managedwriter.WithSchemaDescriptor(descriptorProto),
managedwriter.WithAppendRowsCallOption(gax.WithTimeout(1*time.Second)),
) // return error
// rpc error: code = Unknown desc = failed to find stream from projects/{project}/datasets/{dataset}/tables/{table}/streams/_defaultWhen calling NewManagedStream() without specifying the stream type option, it internally invokes getWriteStream with the stream name argument IMO, the API should automatically create a _default stream for each table if it doesn't exist at the time of the call, or alternatively, prepare a _default stream for all tables during the initialization phase. Here's a quick fix I tried as an alternative, but I'm not sure if it's the right approach: araj-dev#1 |
|
+1 for default straem support |
goccy
left a comment
There was a problem hiding this comment.
Please add test cases for this fixing
# Conflicts: # server/storage_handler.go
|
@goccy added test case for default stream |
|
@goccy Could you please merge this PR into the main branch? I also need it. |
|
@MatanLevy I would like to check the CI results, so could you push an empty commit to trigger the CI ? |
|
@goccy pushed now an empty commit |
|
Thank you for your contribution !!! LGTM 👍 |
* storage write api support default stream * commit in case of commited stream * Storage write api - support default stream: add test cases * empty commit --------- Co-authored-by: Matan Levy <malevy@paloaltonetworks.com>
* storage write api support default stream * commit in case of commited stream * Storage write api - support default stream: add test cases * empty commit --------- Co-authored-by: Matan Levy <malevy@paloaltonetworks.com>
This pull request aims to support bigquery emulator for clients using storage write api with the default stream.
Two main requirements (based on google documentation) for default stream that are part of that pull request:
default stream creation
commited stream in AppendRows