🐛 Source Shopify: fix for mismatched number of tables during normalization #8597

bazarnov · 2021-12-07T18:02:27Z

What

#8353

How

edited scheams: orders, abandoned_checkouts, draft_orders, metafields
forced the DataTypeEnforcer to set string as string, in some cases.
renamed orders_refunds to order_refunds to avoid name conflicts during normalisation process.
fixed the issue, when AirbyteLogger exposes the data coming from stream_slice
increased the performance of order_refunds stream, by polling out refund-ready information from the order.
implemented re-use of shop stream in the check_connection method.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

sherifnada · 2021-12-08T05:40:21Z

@bazarnov what was the root cause for the linked issue?

bazarnov · 2021-12-08T08:57:57Z

@bazarnov what was the root cause for the linked issue?

There was a conflict between orders stream schema and orders_refunds stream schema.
The orders schema contains the refunds property which is the same as the whole orders_refunds schema, it's duplicating as nested structure. While creating the tables for orders_refunds there was a conflict between of two and the tables were not created properly.

…fy-normalization-issue

…alization-issue

bazarnov · 2021-12-14T10:40:19Z

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1577396628
✅ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1577396628
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        76      8    89%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              235     95    60%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     24    56%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  946    442    53%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                          Stmts   Miss  Cover
	 -------------------------------------------------
	 source_shopify/__init__.py        2      2     0%
	 -------------------------------------------------/actions-runner/_work/airbyte/airbyte/airbyte-integrations/connectors/source-shopify/.venv/lib/python3.8/site-packages/coverage/control.py:768: CoverageWarning: No data was collected. (no-data-collected)
	 source_shopify/auth.py           20     20     0%
	 source_shopify/source.py        249    249     0%
	 source_shopify/transform.py      58     58     0%
	 source_shopify/utils.py          57     57     0%
	 -------------------------------------------------
	 TOTAL                           386    386     0%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                          Stmts   Miss  Cover
	 -------------------------------------------------
	 source_shopify/__init__.py        2      0   100%
	 source_shopify/auth.py           20     11    45%
	 source_shopify/source.py        249    101    59%
	 source_shopify/transform.py      58      3    95%
	 source_shopify/utils.py          57      6    89%
	 -------------------------------------------------
	 TOTAL                           386    121    69%

bazarnov · 2021-12-14T13:03:04Z

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1577938332
✅ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1577938332
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        76      8    89%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              235     95    60%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     24    56%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  946    442    53%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                          Stmts   Miss  Cover
	 -------------------------------------------------
	 source_shopify/__init__.py        2      2     0%
	 source_shopify/auth.py           20     20     0%
	 source_shopify/source.py        250    250     0%
	 source_shopify/transform.py      58     58     0%
	 source_shopify/utils.py          57     57     0%
	 -------------------------------------------------
	 TOTAL                           387    387     0%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                          Stmts   Miss  Cover
	 -------------------------------------------------
	 source_shopify/__init__.py        2      0   100%
	 source_shopify/auth.py           20     11    45%
	 source_shopify/source.py        250    102    59%
	 source_shopify/transform.py      58      3    95%
	 source_shopify/utils.py          57      6    89%
	 -------------------------------------------------
	 TOTAL                           387    122    68%

…fy-normalization-issue

antixar

Names of 2 streams were changed. will it be able to break exist user syncs?
Can we keep old ones? Or create new streams separately or add names' aliases

bazarnov · 2021-12-15T11:07:22Z

Names of 2 streams were changed. will it be able to break exist user syncs? Can we keep old ones? Or create new streams separately or add names' aliases

As far as we edited the schemas, the users will have to redo the whole sync again, unfortunately. We cannot keep the old ones here.

antixar · 2021-12-15T12:54:20Z

Names of 2 streams were changed. will it be able to break exist user syncs? Can we keep old ones? Or create new streams separately or add names' aliases

As far as we edited the schemas, the users will have to redo the whole sync again, unfortunately. We cannot keep the old ones here.
@bazarnov ,
Please add comment about this possible problems with old streams and obvious errors as log message for end users. With a short instruction how they have to migrate to the new ones. Maybe add a link to this PR where will be more details

…fy-normalization-issue

bazarnov · 2021-12-16T12:14:28Z

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1587391227
✅ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1587391227
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        76      8    89%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              235     95    60%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     24    56%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py       101     65    36%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  965    458    53%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	   self._warn("No data was collected.", slug="no-data-collected")
	 Name                          Stmts   Miss  Cover
	 -------------------------------------------------
	 source_shopify/__init__.py        2      2     0%
	 source_shopify/auth.py           20     20     0%
	 source_shopify/source.py        250    250     0%
	 source_shopify/transform.py      58     58     0%
	 source_shopify/utils.py          57     57     0%
	 -------------------------------------------------
	 TOTAL                           387    387     0%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                          Stmts   Miss  Cover
	 -------------------------------------------------
	 source_shopify/__init__.py        2      0   100%
	 source_shopify/auth.py           20     11    45%
	 source_shopify/source.py        250    102    59%
	 source_shopify/transform.py      58      3    95%
	 source_shopify/utils.py          57      6    89%
	 -------------------------------------------------
	 TOTAL                           387    122    68%

antixar

LGTM!
Please publish your changes

bazarnov · 2021-12-16T12:58:55Z

/publish connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1587550260
✅ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1587550260

…fy-normalization-issue

…ation (airbytehq#8597)

fixed schemas

35c2398

bazarnov self-assigned this Dec 7, 2021

github-actions bot added the area/connectors Connector related issues label Dec 7, 2021

bazarnov linked an issue Dec 7, 2021 that may be closed by this pull request

Source Shopify throwAssertionError: Mismatched number of tables during normalization #8353

Closed

Merge remote-tracking branch 'origin/master' into bazarnov/8353-shopi…

650df7f

…fy-normalization-issue

bazarnov mentioned this pull request Dec 9, 2021

🐛 Destination BigQuery-Denormalized: cannot format root of NULL with nested objects #8665

Closed

bazarnov linked an issue Dec 10, 2021 that may be closed by this pull request

Sync failure between Shopify and BigQuery (Denormalized) #8420

Closed

Merge remote-tracking branch 'origin/master' into bazarnov/8353-shopi…

7c5b332

…fy-normalization-issue

bazarnov removed a link to an issue Dec 13, 2021

Sync failure between Shopify and BigQuery (Denormalized) #8420

Closed

bazarnov requested review from antixar, grubberr and lazebnyi and removed request for grubberr and lazebnyi December 13, 2021 13:08

added auth_method to required

9f5f710

bazarnov temporarily deployed to more-secrets December 13, 2021 14:28 Inactive

Merge remote-tracking branch 'origin' into bazarnov/8353-shopify-norm…

f9d9d76

…alization-issue

jrhizor temporarily deployed to more-secrets December 14, 2021 10:42 Inactive

increased order_refunds stream performance, bumped the version

6b94dca

github-actions bot added the area/documentation Improvements or additions to documentation label Dec 14, 2021

bazarnov temporarily deployed to more-secrets December 14, 2021 13:04 Inactive

jrhizor temporarily deployed to more-secrets December 14, 2021 13:05 Inactive

bazarnov mentioned this pull request Dec 14, 2021

🐛 Source Shopify: slow orders_refunds read + InvalidChunkLength error #8487

Closed

Merge remote-tracking branch 'origin/master' into bazarnov/8353-shopi…

91ca37c

…fy-normalization-issue

bazarnov linked an issue Dec 14, 2021 that may be closed by this pull request

🐛 Source Shopify: slow orders_refunds read + InvalidChunkLength error #8487

Closed

antixar reviewed Dec 15, 2021

View reviewed changes

updated after review

40f05bd

bazarnov temporarily deployed to more-secrets December 16, 2021 12:04 Inactive

jrhizor temporarily deployed to more-secrets December 16, 2021 12:05 Inactive

Merge remote-tracking branch 'origin/master' into bazarnov/8353-shopi…

f7d5da2

…fy-normalization-issue

bazarnov temporarily deployed to more-secrets December 16, 2021 12:15 Inactive

jrhizor temporarily deployed to more-secrets December 16, 2021 12:16 Inactive

antixar approved these changes Dec 16, 2021

View reviewed changes

jrhizor temporarily deployed to more-secrets December 16, 2021 13:00 Inactive

Oleksandr Bazarnov added 2 commits December 16, 2021 15:45

Merge remote-tracking branch 'origin/master' into bazarnov/8353-shopi…

cbdda7c

…fy-normalization-issue

generated new specs

db7d9a0

bazarnov temporarily deployed to more-secrets December 16, 2021 13:48 Inactive

lazebnyi approved these changes Dec 16, 2021

View reviewed changes

bazarnov temporarily deployed to more-secrets December 16, 2021 15:21 Inactive

bazarnov merged commit fb834c9 into master Dec 16, 2021

bazarnov deleted the bazarnov/8353-shopify-normalization-issue branch December 16, 2021 15:50

jrhizor mentioned this pull request Dec 17, 2021

Bump Airbyte version from 0.34.0-alpha to 0.34.1-alpha #8860

Merged

schlattk pushed a commit to schlattk/airbyte that referenced this pull request Jan 4, 2022

🐛 Source Shopify: fix for mismatched number of tables during normaliz…

8a9a6c2

…ation (airbytehq#8597)

dsnam mentioned this pull request Feb 2, 2022

🐛 Source shopify: v0.1.28 breaking config change #9994

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

🐛 Source Shopify: fix for mismatched number of tables during normalization #8597

🐛 Source Shopify: fix for mismatched number of tables during normalization #8597

Uh oh!

bazarnov commented Dec 7, 2021 •

edited

Loading

Uh oh!

sherifnada commented Dec 8, 2021

Uh oh!

bazarnov commented Dec 8, 2021

Uh oh!

bazarnov commented Dec 14, 2021 •

edited by github-actions bot

Loading

Uh oh!

bazarnov commented Dec 14, 2021 •

edited by github-actions bot

Loading

Uh oh!

antixar left a comment

Uh oh!

bazarnov commented Dec 15, 2021

Uh oh!

antixar commented Dec 15, 2021 •

edited

Loading

Uh oh!

bazarnov commented Dec 16, 2021 •

edited by github-actions bot

Loading

Uh oh!

antixar left a comment

Uh oh!

bazarnov commented Dec 16, 2021 •

edited by github-actions bot

Loading

Uh oh!

Uh oh!

🐛 Source Shopify: fix for mismatched number of tables during normalization #8597

🐛 Source Shopify: fix for mismatched number of tables during normalization #8597

Uh oh!

Conversation

bazarnov commented Dec 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Pre-merge Checklist

Community member or Airbyter

Airbyter

Uh oh!

sherifnada commented Dec 8, 2021

Uh oh!

bazarnov commented Dec 8, 2021

Uh oh!

bazarnov commented Dec 14, 2021 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bazarnov commented Dec 14, 2021 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

antixar left a comment

Choose a reason for hiding this comment

Uh oh!

bazarnov commented Dec 15, 2021

Uh oh!

antixar commented Dec 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bazarnov commented Dec 16, 2021 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

antixar left a comment

Choose a reason for hiding this comment

Uh oh!

bazarnov commented Dec 16, 2021 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

bazarnov commented Dec 7, 2021 •

edited

Loading

bazarnov commented Dec 14, 2021 •

edited by github-actions bot

Loading

bazarnov commented Dec 14, 2021 •

edited by github-actions bot

Loading

antixar commented Dec 15, 2021 •

edited

Loading

bazarnov commented Dec 16, 2021 •

edited by github-actions bot

Loading

bazarnov commented Dec 16, 2021 •

edited by github-actions bot

Loading