-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Description
I'm facing an issue with BigQuery streaming inserts (Table.insert(...), specifically insert(Iterable<InsertAllRequest.RowToInsert> rows, boolean skipInvalidRows, boolean ignoreUnknownValues) with skipInvalidRows = false and ignoreUnknownValues = false) where (sometimes) records don't seem to be available after one or more insert requests. The InsertAllRequests complete successfully, i.e. no exceptions are thrown and no errors are reported (InsertAllResponse.hasErrors returns false). I checked availability of streamed data in the BigQuery Web UI and using the Table.list(...) API. According to https://cloud.google.com/bigquery/streaming-data-into-bigquery I would expect streamed data to be available for query a few seconds after insertion. In cases where some records were missing after the initial check, I tried again after 10s, 30s, 60s, 1h, ... but to no avail. So it looks like the records have been dropped for some reason.