Skip to content

Destination BigQuery: timestamps without timezone should be mapped to DATETIME #12660

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Tracked by #10148
yurii-bidiuk opened this issue May 6, 2022 · 5 comments
Closed
Tracked by #10148
Assignees
Labels
team/destinations Destinations team's backlog type/enhancement New feature or request

Comments

@yurii-bidiuk
Copy link
Contributor

yurii-bidiuk commented May 6, 2022

Due to new updates in the building of JSON schema, a new property "airbyte_type" was added to the field description.
One of the possible values for "airbyte_type" is "timestamp_without_timezone". Now "airbyte_type" is ignored and all "timestamp_without_timezone" fields with "format":"date-time" are mapped as TIMESTAMP in destination BigQuery.

According to #11090 and this comment all fields with "airbyte_type": "timestamp_without_timezone" should be mapped as DATETIME in destination BigQuery.

Input JSON schema example:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "type_datetime_without_timezone": {
      "type": "string",
      "format": "date-time",
      "airbyte_type": "timestamp_without_timezone"
    },
    
  }
}

Input data example:

{
  "type_datetime_without_timezone": "2022-11-22T01:23:45"}
@yurii-bidiuk
Copy link
Contributor Author

As datatype mapping for destiantion-bigquery should be done on the normalization side, I'm moving this ticket to the board of Phyton team
cc: @alexandr-shegeda

@grishick
Copy link
Contributor

grishick commented Nov 9, 2022

@edgao does this issue depend (should it depend) on us implementing changes for handling data types using the new protocol version?

@edgao
Copy link
Contributor

edgao commented Nov 9, 2022

No dependency. Probably should not happen in parallel with #17647 / #17642 to avoid merge conflicts, but they can happen in either order.

@etsybaev etsybaev self-assigned this Nov 11, 2022
@etsybaev
Copy link
Contributor

@grishick
Copy link
Contributor

@etsybaev could you please run a test (manual test is fine in this case) to confirm that this is working as expected?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
team/destinations Destinations team's backlog type/enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

6 participants