Source S3: fix reading jsonl files with nested data #16607

davydov-d · 2022-09-12T19:01:48Z

What

https://github.com/airbytehq/oncall/issues/531

Reading a file is a three step process:

first, each file's schema is discovered in pyarrow data types
then all of them are converted to json schemas and merged with a user-provided schema that was filled in during the connection setup into a master schema
master schema is converted back to pyarrow schema and passed in to a reader as an explicit schema

The problem is the connector can not convert pyarrow struct type to object and back. It uses large_string instead therefore pyarrow fails when there's a type mismatch

How

Skip objects in master schema when reading json files. Because we do not distinguish different object types and do not provide a way for the user to describe the object's schema as for now so it can be literally any object. In this case (by default) pyarrow still reads the data and its type remains inferred. Another option could be ignoring mismatched data types but this would impact scalar types as well.

davydov-d · 2022-09-12T19:02:44Z

/test connector=conenctors/source-s3

davydov-d · 2022-09-13T07:06:04Z

/test connector=conenctors/source-s3

🕑 conenctors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/3043007587
❌ conenctors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/3043007587
🐛

davydov-d · 2022-09-13T07:19:23Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/3043080729
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/3043080729
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1325    463    65%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/jsonl_spec.py                13      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/avro_spec.py                  5      0   100%
source_s3/s3file.py                                                  37      0   100%
source_s3/s3_utils.py                                                19      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source.py                                                  29      1    97%
source_s3/source_files_abstract/storagefile.py                       23      1    96%
source_s3/source_files_abstract/stream.py                           218     13    94%
source_s3/stream.py                                                  43      3    93%
source_s3/source_files_abstract/formats/abstract_file_parser.py      39      3    92%
source_s3/source_files_abstract/formats/csv_parser.py                76     18    76%
source_s3/source_files_abstract/file_info.py                         26      8    69%
source_s3/utils.py                                                   31     10    68%
source_s3/source_files_abstract/source.py                            37     14    62%
source_s3/source_files_abstract/spec.py                              44     22    50%
source_s3/source_files_abstract/formats/jsonl_parser.py              42     25    40%
source_s3/source_files_abstract/formats/avro_parser.py               38     25    34%
source_s3/source_files_abstract/formats/parquet_parser.py            61     44    28%
-------------------------------------------------------------------------------------
TOTAL                                                               808    187    77%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/storagefile.py                       23      0   100%
source_s3/source_files_abstract/spec.py                              44      0   100%
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/jsonl_spec.py                13      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/avro_spec.py                  5      0   100%
source_s3/source.py                                                  29      0   100%
source_s3/s3file.py                                                  37      0   100%
source_s3/s3_utils.py                                                19      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source_files_abstract/formats/parquet_parser.py            61      1    98%
source_s3/stream.py                                                  43      1    98%
source_s3/source_files_abstract/formats/jsonl_parser.py              42      1    98%
source_s3/source_files_abstract/formats/abstract_file_parser.py      39      1    97%
source_s3/source_files_abstract/source.py                            37      2    95%
source_s3/source_files_abstract/formats/avro_parser.py               38      3    92%
source_s3/source_files_abstract/file_info.py                         26      3    88%
source_s3/source_files_abstract/stream.py                           218     36    83%
source_s3/source_files_abstract/formats/csv_parser.py                76     18    76%
source_s3/utils.py                                                   31      8    74%
-------------------------------------------------------------------------------------
TOTAL                                                               808     74    91%

Build Passed

Test summary info:

All Passed

alafanechere

Thank you for digging into this but I'm not sure of really understanding what happens there 😄

...tions/connectors/source-s3/unit_tests/sample_files/jsonl/test_file_10_nested_structure.jsonl

...rations/connectors/source-s3/source_s3/source_files_abstract/formats/abstract_file_parser.py

...te-integrations/connectors/source-s3/source_s3/source_files_abstract/formats/jsonl_parser.py

…tructures

alafanechere

I approve because this fixes the OC issue. As @davydov-d mentioned, there's a lot of fragility in the schema inference mechanism. I hope we get to rework this as part of our efforts in abstracting the file sources.

…tructures

davydov-d · 2022-09-19T08:32:59Z

/publish connector=connectors/source-s3

🕑 Publishing the following connectors:
connectors/source-s3
https://github.com/airbytehq/airbyte/actions/runs/3080988595

Connector	Did it publish?	Were definitions generated?
connectors/source-s3	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

* airbytehq#531 source s3: fix reading nested jsonl files * airbytehq#531 source s3: upd changelog * oncall airbytehq#531 source s3: fix sample file * auto-bump connector version [ci skip] Co-authored-by: Octavia Squidington III <[email protected]>

#531 source s3: fix reading nested jsonl files

8458210

octavia-squidington-iii added the connectors/source/s3 label Sep 12, 2022

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Sep 12, 2022

#531 source s3: upd changelog

6422eb5

davydov-d requested review from alafanechere, arsenlosenko, grubberr and lazebnyi September 12, 2022 19:20

lazebnyi approved these changes Sep 12, 2022

View reviewed changes

arsenlosenko approved these changes Sep 13, 2022

View reviewed changes

davydov-d requested a review from sherifnada September 13, 2022 16:46

alafanechere suggested changes Sep 14, 2022

View reviewed changes

davydov-d added 2 commits September 15, 2022 10:50

Merge branch 'master' into ddavydov/#531-source-s3-fix-jsonl-nested-s…

30caed9

…tructures

oncall #531 source s3: fix sample file

f682947

davydov-d requested review from alafanechere and pedroslopez September 15, 2022 09:00

alafanechere approved these changes Sep 16, 2022

View reviewed changes

sherifnada removed request for pedroslopez and sherifnada September 17, 2022 20:39

Merge branch 'master' into ddavydov/#531-source-s3-fix-jsonl-nested-s…

018b1e1

…tructures

auto-bump connector version [ci skip]

9c8115e

davydov-d merged commit 4dc394c into master Sep 19, 2022

davydov-d deleted the ddavydov/#531-source-s3-fix-jsonl-nested-structures branch September 19, 2022 09:09

octavia-squidington-iii mentioned this pull request Sep 21, 2022

Bump Airbyte version from 0.40.7 to 0.40.8 #17009

Merged

davydov-d mentioned this pull request Oct 14, 2022

Source S3 - fix schema inference #17991

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Source S3: fix reading jsonl files with nested data #16607

Source S3: fix reading jsonl files with nested data #16607

Uh oh!

davydov-d commented Sep 12, 2022 •

edited

Loading

Uh oh!

davydov-d commented Sep 12, 2022

Uh oh!

davydov-d commented Sep 13, 2022 •

edited by github-actions bot

Loading

Uh oh!

davydov-d commented Sep 13, 2022 •

edited by github-actions bot

Loading

Uh oh!

alafanechere left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alafanechere left a comment

Uh oh!

davydov-d commented Sep 19, 2022 •

edited by github-actions bot

Loading

Uh oh!

Uh oh!

Source S3: fix reading jsonl files with nested data #16607

Source S3: fix reading jsonl files with nested data #16607

Uh oh!

Conversation

davydov-d commented Sep 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Uh oh!

davydov-d commented Sep 12, 2022

Uh oh!

davydov-d commented Sep 13, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davydov-d commented Sep 13, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

alafanechere left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alafanechere left a comment

Choose a reason for hiding this comment

Uh oh!

davydov-d commented Sep 19, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

davydov-d commented Sep 12, 2022 •

edited

Loading

davydov-d commented Sep 13, 2022 •

edited by github-actions bot

Loading

davydov-d commented Sep 13, 2022 •

edited by github-actions bot

Loading

davydov-d commented Sep 19, 2022 •

edited by github-actions bot

Loading