Bug Description
When DBCode is doing introspection on a large dataset where there are more than 40K columns, it often fails with the following error:
"Invalid response body while trying to fetch https://bigquery.googleapis.com/bigquery/v2/projects/xxxxxx/queries/xxxxxx?location=US&formatOptions.useInt64Timestamp=true&startIndex=0&prettyPrint=false: Premature close"
I extracted the introspection query and ran it in BigQuery Console, it works fine--50K rows returned. If I run the query in DBCode, it fails with the same error mentioned above. If I include the "LIMIT 40000" clause in the query to return only the first 40K rows, it works fine.
It looks like the standard BigQuery REST API can only handle queries up to a certain size. Beyond that you need to use the Storage API. Could using pagination with introspection query help? Most corporate users don't have the access to run Storage API queries so that's not a practical option.
Repro steps
- Refresh a dataset with more than 40K columns to trigger introspection.
Expected Behavior
DBCode retrieves metadata for all columns in the dataset during introspection. No introspection error.
Actual Behavior
Introspection error.
Environment
- DBCode version: 1.26.10
- VS Code (or fork) version: 1.108.1
- OS: Windows 11
- Database: BigQuery
- Connection: (direct/SSH/SSL) direct
Error Messages (if applicable)
Screenshots
If applicable, add screenshots to help explain your problem.
Logs
If available, paste relevant logs from the Output panel (View > Output > DBCode).
Bug Description
When DBCode is doing introspection on a large dataset where there are more than 40K columns, it often fails with the following error:
"Invalid response body while trying to fetch https://bigquery.googleapis.com/bigquery/v2/projects/xxxxxx/queries/xxxxxx?location=US&formatOptions.useInt64Timestamp=true&startIndex=0&prettyPrint=false: Premature close"
I extracted the introspection query and ran it in BigQuery Console, it works fine--50K rows returned. If I run the query in DBCode, it fails with the same error mentioned above. If I include the "LIMIT 40000" clause in the query to return only the first 40K rows, it works fine.
It looks like the standard BigQuery REST API can only handle queries up to a certain size. Beyond that you need to use the Storage API. Could using pagination with introspection query help? Most corporate users don't have the access to run Storage API queries so that's not a practical option.
Repro steps
Expected Behavior
DBCode retrieves metadata for all columns in the dataset during introspection. No introspection error.
Actual Behavior
Introspection error.
Environment
Error Messages (if applicable)
Screenshots
If applicable, add screenshots to help explain your problem.
Logs
If available, paste relevant logs from the Output panel (View > Output > DBCode).