feat: (Preview) Support automatic load of timedelta from BQ tables. #1429
Conversation
product-auto-label
bot
added
size: m
sycai
requested review from
tswast,
TrevorBergeron and
chelsea-lin
and removed request for
GarrettWu
gcf-owl-bot
bot
removed
the
owlbot:run
|
Closing the PR for now to think about annotations for RECORD and ARRAY. I will prepare another PR once everything falls into place |
bigframes/session/executor.py
Outdated
| return query_job | ||
|
|
||
| def _attach_timedelta_metadata( |
There was a problem hiding this comment.
why not put this logic instead in convert_to_schema_field? it is the natural inverse of convert_schema_field
There was a problem hiding this comment.
Good call! That makes the code cleaner.
There was a problem hiding this comment.
Though interestingly, in export_gbq we rely on the BigQuery engine to figure out the schema based on the provided SQL. I need to update the schema again afterwards to include the tag
bigframes/dtypes.py
Outdated
| elif ( | ||
| field.field_type == "INTEGER" | ||
| and field.description is not None | ||
| and TIMEDELTA_DESCRIPTION_TAG in field.description |
There was a problem hiding this comment.
nit: but I do wonder if once in a while, we might catch a user description that accidentally uses '#microseconds' somewhere? maybe restrict to beginning, or put some odds character sequence to ensure non-accidental triggering
There was a problem hiding this comment.
SG. Changed the logic to check whether the description ends with the tag.
This works only with
read_gbq_table()under the hood, becauseread_gbq_query()erases column descriptions.