Skip to content

feat: add QueryJobConfig properties to bigquery backend specified at query time #11255

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

Wopple
Copy link

@Wopple Wopple commented May 27, 2025

Description of changes

Add a parameter to the query level methods in the Big Query backend to specify the QueryJobConfig. The following methods are affected:

  • raw_sql
  • _to_query
  • to_pyarrow
  • to_pyarrow_batches
  • execute

Each is given an optional named parameter query_job_config. If params is also provided, the query parameters in both will be merged where params takes precedence.

@github-actions github-actions bot added the bigquery The BigQuery backend label May 27, 2025
@dlstadther
Copy link
Contributor

Resolves #11231 . Right?

Copy link
Member

@cpcloud cpcloud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, just a quick note about naming, but overall this looks like a good change.

query: str,
params: dict | None = None,
job_id: str | None = None,
query_job_props: dict | None = None,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's call it query_job_config to match the Google API naming convention.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated to a different approach which is better. My rationale for using the props naming is no longer relevant, so I also updated to query_job_config.

@github-actions github-actions bot added the tests Issues or PRs related to tests label May 28, 2025
@dtran-im
Copy link
Contributor

We would need to do something with the read_ functions to enable adding properties (labels, etc., though not params) to any LoadJobConfigs, right?

@Wopple
Copy link
Author

Wopple commented May 28, 2025

@dlstadther I believe it does close that one.

@dtran-im the read methods allow for **kwargs which are sent to the LoadJobConfig.

@Wopple
Copy link
Author

Wopple commented May 28, 2025

@cpcloud It appears the linting is failing both with and without a blank line. Do you know how to properly format this to make ruff happy?

@Wopple
Copy link
Author

Wopple commented May 29, 2025

@cpcloud I think I figured out the lint issue, but I'm still not understanding the MySQL backend test failures:

=================================== FAILURES ===================================
_________________ test_impure_uncorrelated_same_id[mysql-uuid] _________________
[gw2] linux -- Python 3.13.3 /home/runner/work/ibis/ibis/.venv/bin/python
[XPASS(strict)] instances are correlated
---------------------------- Captured stderr setup -----------------------------
 ibis-mysql-1 copy /home/runner/work/ibis/ibis/ci/ibis-testing-data/csv/geo.csv to ibis-mysql-1:/data/geo.csv Copying
 ibis-mysql-1 copy /home/runner/work/ibis/ibis/ci/ibis-testing-data/csv/geo.csv to ibis-mysql-1:/data/geo.csv Copied
 ibis-mysql-1 copy /home/runner/work/ibis/ibis/ci/ibis-testing-data/csv/awards_players.csv to ibis-mysql-1:/data/awards_players.csv Copying
 ibis-mysql-1 copy /home/runner/work/ibis/ibis/ci/ibis-testing-data/csv/diamonds.csv to ibis-mysql-1:/data/diamonds.csv Copying
 ibis-mysql-1 copy /home/runner/work/ibis/ibis/ci/ibis-testing-data/csv/astronauts.csv to ibis-mysql-1:/data/astronauts.csv Copying
 ibis-mysql-1 copy /home/runner/work/ibis/ibis/ci/ibis-testing-data/csv/functional_alltypes.csv to ibis-mysql-1:/data/functional_alltypes.csv Copying
 ibis-mysql-1 copy /home/runner/work/ibis/ibis/ci/ibis-testing-data/csv/awards_players.csv to ibis-mysql-1:/data/awards_players.csv Copied
 ibis-mysql-1 copy /home/runner/work/ibis/ibis/ci/ibis-testing-data/csv/diamonds.csv to ibis-mysql-1:/data/diamonds.csv Copied
 ibis-mysql-1 copy /home/runner/work/ibis/ibis/ci/ibis-testing-data/csv/astronauts.csv to ibis-mysql-1:/data/astronauts.csv Copied
 ibis-mysql-1 copy /home/runner/work/ibis/ibis/ci/ibis-testing-data/csv/batting.csv to ibis-mysql-1:/data/batting.csv Copying
 ibis-mysql-1 copy /home/runner/work/ibis/ibis/ci/ibis-testing-data/csv/functional_alltypes.csv to ibis-mysql-1:/data/functional_alltypes.csv Copied
 ibis-mysql-1 copy /home/runner/work/ibis/ibis/ci/ibis-testing-data/csv/batting.csv to ibis-mysql-1:/data/batting.csv Copied
______________ test_impure_uncorrelated_different_id[mysql-uuid] _______________
[gw2] linux -- Python 3.13.3 /home/runner/work/ibis/ibis/.venv/bin/python
[XPASS(strict)] instances are correlated

It doesn't feel like I touched that code.

As for the SQLite failure: I don't see any test failures, but the tests are timing out when nearly complete. I expect the timeout duration will need to be extended.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bigquery The BigQuery backend tests Issues or PRs related to tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants