Skip to content
This repository was archived by the owner on Mar 6, 2026. It is now read-only.

feat: add support for BigQuery DataFrames. Set context.engine to 'bigframes' to support query results larger than 10 GB#58

Merged
tswast merged 18 commits intomainfrom
issue16-context
Sep 20, 2024
Merged

feat: add support for BigQuery DataFrames. Set context.engine to 'bigframes' to support query results larger than 10 GB#58
tswast merged 18 commits intomainfrom
issue16-context

Conversation

@sycai
Copy link
Copy Markdown
Contributor

@sycai sycai commented Sep 19, 2024

No description provided.

@sycai sycai requested review from a team and tswast September 19, 2024 00:13
@product-auto-label product-auto-label bot added the size: m Pull request size is medium. label Sep 19, 2024
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-magics API. label Sep 19, 2024
@sycai sycai changed the title feat: support BigFrames API by setting context.engine to 'bigframes' feat: support BigFrames queries by setting context.engine to 'bigframes' Sep 19, 2024
setup.py Outdated
# 'Development Status :: 5 - Production/Stable'``
release_status = "Development Status :: 4 - Beta"
dependencies = [
"bigframes >= 1.17.0",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't.

In the design I shared that this should be an optional dependency. That means it should be an "extra".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack. Moved it to extra.

@product-auto-label product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Sep 19, 2024
@product-auto-label product-auto-label bot added size: m Pull request size is medium. and removed size: l Pull request size is large. labels Sep 19, 2024
@product-auto-label product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Sep 19, 2024
@product-auto-label product-auto-label bot added size: m Pull request size is medium. and removed size: l Pull request size is large. labels Sep 19, 2024
@product-auto-label product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Sep 19, 2024
@product-auto-label product-auto-label bot added the size: m Pull request size is medium. label Sep 19, 2024
@product-auto-label product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Sep 19, 2024
@product-auto-label product-auto-label bot added size: m Pull request size is medium. and removed size: l Pull request size is large. labels Sep 19, 2024
@product-auto-label product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Sep 19, 2024
@product-auto-label product-auto-label bot added size: m Pull request size is medium. and removed size: l Pull request size is large. labels Sep 19, 2024
@product-auto-label product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Sep 19, 2024
@sycai sycai requested a review from tswast September 19, 2024 19:58
owlbot.py Outdated
# Multi-processing note isn't relevant, as bigquery-magics is responsible for
# creating clients, not the end user.
"docs/multiprocessing.rst",
"noxfile.py",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't modify the noxfile in this way. There is a feature in the template that does what we want.

https://github.com/googleapis/synthtool/blob/master/synthtool/gcp/templates/python_library/noxfile.py.j2#L75C1-L75C27

Use unit_test_extras_by_python to make sure we have some Python versions where we test without extras and some where we test with.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change reverted

noxfile.py Outdated
session.install("protobuf<4")

if install_bigframes:
session.install("bigframes >= 1.17.0")
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use the "extras". Don't install this manually.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. Updated owlbot.py instead

"grpcio >= 1.47.0, < 2.0dev",
"grpcio >= 1.49.1, < 2.0dev; python_version>='3.11'",
],
"bigframes": ["bigframes >= 1.17.0"],
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pip will always install the latest version of bigframes. Let's make sure we test our compatibility with the minimum supported bigframes by adding bigframes==1.17.0 to https://github.com/googleapis/python-bigquery-magics/blob/main/testing/constraints-3.9.txt

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@sycai sycai requested a review from tswast September 19, 2024 22:17
Copy link
Copy Markdown
Contributor

@tswast tswast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Comment on lines +454 to +455
bpd.options.bigquery.project = context.project
bpd.options.bigquery.credentials = context.credentials
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm worried these might fail if the bigframes session has already started. Maybe that's what we want to happen?

Alternatively, we could explicitly create a bigframes Session object, but then whatever DataFrame we produce won't be compatible with other DataFrames in the notebook.

@tswast tswast changed the title feat: support BigFrames queries by setting context.engine to 'bigframes' feat: add support for BigQuery DataFrames. Set context.engine to 'bigframes' to support query results larger than 10 GB Sep 20, 2024
@tswast tswast merged commit 90ba05f into main Sep 20, 2024
@tswast tswast deleted the issue16-context branch September 20, 2024 21:36
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

api: bigquery Issues related to the googleapis/python-bigquery-magics API. size: l Pull request size is large.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants