Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Destination S3 (avro/parquet format): handle empty schemas #44933

Merged
merged 7 commits into from
Aug 30, 2024
Merged

Destination S3 (avro/parquet format): handle empty schemas #44933

merged 7 commits into from
Aug 30, 2024

Conversation

edgao
Copy link
Contributor

@edgao edgao commented Aug 30, 2024

from https://github.com/airbytehq/oncall/issues/6299

currently, schemas like {type: array, items: {}} throw an illegal arg exception because of the empty {} nested schema. This PR:

  • Adds a JSON type to represent that schema
  • adds functionality to the mappers to handle that schema
  • updates the avro schema+record mappers to convert JSON to string

(.... eventually we should just use AirbyteType here 😅 which has much nicer handling for array/object/primitive types in general)

@edgao edgao requested a review from johnny-schmidt August 30, 2024 16:37
@edgao edgao marked this pull request as ready for review August 30, 2024 16:37
@vercel Vercel
Copy link

vercel bot commented Aug 30, 2024 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Aug 30, 2024 6:06pm

@edgao edgao requested a review from a team as a code owner August 30, 2024 16:37
@octavia-squidington-iii octavia-squidington-iii added the CDK Connector Development Kit label Aug 30, 2024
johnny-schmidt
johnny-schmidt reviewed Aug 30, 2024
View reviewed changes
...c/main/kotlin/io/airbyte/cdk/integrations/destination/s3/jsonschema/AirbyteJsonSchemaType.kt
@@ -175,7 +177,7 @@ enum class AirbyteJsonSchemaType {
// Usually the root node
return OBJECT_WITH_PROPERTIES
} else {
throw IllegalArgumentException("Unspecified schema type")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be explicit about the fact that we're matching an empty {}? Or do you want to use this as a catch-all

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

more a catchall. So, I guess I should rename it to unknown in that case 🤔

johnny-schmidt
johnny-schmidt reviewed Aug 30, 2024
View reviewed changes
...n/kotlin/io/airbyte/cdk/integrations/destination/s3/parquet/JsonSchemaParquetPreprocessor.kt Outdated
@@ -36,6 +36,11 @@ class JsonSchemaParquetPreprocessor : JsonSchemaIdentityMapper() {
AirbyteJsonSchemaType.UNION,
AirbyteJsonSchemaType.COMBINED ->
throw IllegalStateException("Nested unions are not supported")
// TODO is this true?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is true in the sense that Parquet doesn't have a Json type per se. It has a json-logical type that can be applied to strings, which we don't use, because there's no way to represent it in the Avro schema.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gotcha. I'll update the comment to reflect that

@johnny-schmidt
Copy link
Contributor

This looks correct to me. You could also consider adding these types to the DATs for avro and parquet.

The relevant test method is DestinationAcceptanceTest.testProblematicTypes (I'm bad at naming sorry).

And the workflow for adding them would be

  • add fields of that type to the end of the test catalog file v0/problematic_types_configured_catalog.json (you can step into it from the test in intellij)
  • add corresponding fields to the end of v0/problematic_types_messages_in.txt. There are only three rows, which are basically Variant1, Variant2 (if applicable), and Nulled
  • add the expected avro output to the end of the three rows in v0/problematic_types_coerced_schemaless_schema.json (should really be renamed)
  • add the expected parquet output to the end of the three rows in v0/problematic_types_disjoint_union_schema.json
  • (expected output in this case would be the same: stringified json blob)

johnny-schmidt
johnny-schmidt approved these changes Aug 30, 2024
View reviewed changes
Copy link
Contributor

@johnny-schmidt johnny-schmidt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than possibly amending the DATs, this is good to go.

@edgao edgao requested a review from a team as a code owner August 30, 2024 17:43
@edgao
Copy link
Contributor Author

edgao commented Aug 30, 2024 •
edited by github-actions bot
Loading

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/10636936796
✅ Successfully published Java CDK version=0.44.20!

@edgao edgao enabled auto-merge (squash) August 30, 2024 18:02
@octavia-squidington-iii octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Aug 30, 2024
@edgao edgao merged commit 0050274 into master Aug 30, 2024
36 checks passed
@edgao edgao deleted the edgao/json_type_in_avro branch August 30, 2024 18:17
@manuito
Copy link

manuito commented Sep 2, 2024 •
edited
Loading

Hi. Since this release on Airbyte Cloud we are facing issues : the sync_id is not anymore in destination file name. Seen by some other users => https://airbytehq.slack.com/archives/C021JANJ6TY/p1725177438711539

I have opened issue #45084
It's a very impactful regression for us 🙏

@edgao edgao mentioned this pull request Sep 3, 2024
Closed
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@johnny-schmidt johnny-schmidt johnny-schmidt approved these changes

Assignees
No one assigned
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation CDK Connector Development Kit connectors/destination/s3
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@edgao @johnny-schmidt @manuito @octavia-squidington-iii