feat: Support dry_run in to_pandas()

[sycai]

No description provided.

[tswast]

Heads up: I suspect we might encounter some conflicts after #1448 merges. I think that allow_large_results change is more urgent, so let's refrain from merging this until that's in.

[tswast]

I find this pretty confusing. I think it would be more typical to have the job properties as separate columns.

Alternatively, we could return a pd.Series if we do want a 1 dimension object.

[tswast]

Why always [index]? Sometimes the index has a name. Also, sometimes the index is a multi-index.

[tswast]

This feels a bit risky to me. What if the columns has the name total_bytes_processed? I would rather see dtypes as it's own object column/row that contains the predicted dtypes as a single object.

[tswast]

Same here, maybe introduce index_dtypes and have that be the whole object of 0+ dtypes for a potential multi-index.

[sycai]

I used a DataFrame for dry run stats, just like what describe() does.

It looks like this: https://screenshot.googleplex.com/gEEBU7aMqUy93SB

Plus, I applied some "@overload" magic to make mypy happy.

Let me know your thoughts @tswast

[tswast]

Here's what I had in mind:

import copy

import pandas as pd
from google.colab import auth
from google.cloud import bigquery

# Authenticate the user
auth.authenticate_user()

# Initialize a BigQuery client
client = bigquery.Client(project='bigframes-dev') # Replace with your project ID

job_config = bigquery.QueryJobConfig()
job_config.dry_run = True

query = """
SELECT
  name,
  SUM(number) AS total
FROM
  `bigquery-public-data.usa_names.usa_1910_2013`
GROUP BY
  name
ORDER BY
  total DESC
LIMIT
  10;
"""

job = client.query(query, job_config=job_config)

job_api_repr = copy.deepcopy(job._properties)
print(job_api_repr)
index = []
values = []
query_configuration = job_api_repr['configuration'].pop("query")
index.extend(query_configuration.keys())
values.extend(query_configuration.values())

query_statistics = job_api_repr['statistics'].pop("query")
index.extend(query_statistics.keys())
values.extend(query_statistics.values())

remaining_statistics = job_api_repr['statistics']
index.extend(remaining_statistics.keys())
values.extend(remaining_statistics.values())

series = pd.Series(values, index=index)
print(series)

[tswast]

Re: #1436 (comment)

It would need

• some careful checks in case some of those keys don't exist,
• maybe we want an allowlist of keys so we only expose the job stats we've validated are useful to a bigframes user,
• add dtypes and index_dtypes for the expected output types (possibly remove schema).

[sycai]

Now the stats look like these: https://screenshot.googleplex.com/BhaUb8uTYrdz7iM

I hard-coded all the keys for value lookup in the dry run job.

@tswast

[tswast]

Please find another way to make the type checker happy. I dislike the repeated code.

[tswast]

Suggested change

	raise NotImplementedError("Dry run with sampling is not supproted")
	raise NotImplementedError("Dry run with sampling is not supported")

[tswast]

The select_columns([]) confuses me, but I see that was here before. Please refactor these a bit so that self._block.select_columns([]) is saved to a variable since it is in common with both.

[tswast]

Alternatively, we can get rid of this if statement and rename the variable df_or_series.

[tswast]

I think allow_large_results shouldn't have an effect on dry run queries, as that controls the destination table property.

[tswast]

Why include the dry_run argument here if we know it's false?

[tswast]

Why include the dry_run argument here if we know it's false?

[tswast]

Why don't you just use bool consistently, then?

[sycai]

Please find another way to make the type checker happy. I dislike the repeated code.

Yeah that's true... I moved the if dry_run then block._compute_dry_run code from the block.to_pandas() to DataFrames/Series/Index.to_pandas(), that should clean up the code a bit. Plus, no more signature override to block.to_pandas()

[tswast]

Presubmits failed:

FAILED tests/system/small/test_dataframe.py::test_df_peek[partial] - assert (...
FAILED tests/system/small/test_dataframe.py::test_df_peek[strict] - assert (6...

Hard to say if it's related to this PR.