Skip to content

BigQuery introspection error when dataset has more than 40K columns #925

@cassinium

Description

@cassinium

Bug Description

When DBCode is doing introspection on a large dataset where there are more than 40K columns, it often fails with the following error:

"Invalid response body while trying to fetch https://bigquery.googleapis.com/bigquery/v2/projects/xxxxxx/queries/xxxxxx?location=US&formatOptions.useInt64Timestamp=true&startIndex=0&prettyPrint=false: Premature close"

I extracted the introspection query and ran it in BigQuery Console, it works fine--50K rows returned. If I run the query in DBCode, it fails with the same error mentioned above. If I include the "LIMIT 40000" clause in the query to return only the first 40K rows, it works fine.

It looks like the standard BigQuery REST API can only handle queries up to a certain size. Beyond that you need to use the Storage API. Could using pagination with introspection query help? Most corporate users don't have the access to run Storage API queries so that's not a practical option.

Repro steps

  1. Refresh a dataset with more than 40K columns to trigger introspection.

Expected Behavior

DBCode retrieves metadata for all columns in the dataset during introspection. No introspection error.

Actual Behavior

Introspection error.

Environment

  • DBCode version: 1.26.10
  • VS Code (or fork) version: 1.108.1
  • OS: Windows 11
  • Database: BigQuery
  • Connection: (direct/SSH/SSL) direct

Error Messages (if applicable)


Screenshots

If applicable, add screenshots to help explain your problem.

Logs

If available, paste relevant logs from the Output panel (View > Output > DBCode).


Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions