🐛 Source Shopify: Fix FAILED scenario for BULK streams, fixed deleted events STATE collision

[bazarnov]

What

Resolving:

• https://github.com/airbytehq/oncall/issues/6098

How

There are a couple of issues faced during the investigation of the OC:

• BULK streams that don't support bulk checkpointing with FAILED Job status, could run into an infinity loop, with no further resolutions, in this case, we should raise an Exception and retry with the next sync attempt (INCOMPLETE status for the stream)
• BULK Streams that support bulk checkpointing now can catch the partialDataUrl and proceed with the partial results collected correctly.
• BULK stream metafields_product_images has changed the way we query the metafields due to the fact they suddenly changed the ownership of the Metafield resource we pull (metafield product images), now we query from media, instead of images. More information about this here. This is super annoying because Shopify changed the way they store the metafields for product images for future 2024-10 but applied the change to the 2024-04 (current shopify API version the source uses). Thus, the id assigned to the metafield has changed because the ownership has changed from prodcut_image to media_image.
  Failed CAT test, with 0 records returned for metafield_product_images
• For REST streams, such as Orders and others that support Deleted Events fetch, the STATE overlap/collision was fixed; now, the STATE emitted from the main stream's part and the STATE from the deleted events are resolved correctly.
• added unit test to cover the changes
• remove pointless unit tests that no longer cover the behavior

User Impact

This is not a breaking change! No impact is expected.

Can this PR be safely reverted and rolled back?

• YES 💚
• NO ❌

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 5, 2024 3:14pm

[bazarnov]

Regression Tests fail is expected in:

• TestDataIntegrity
  • test_record_count_with_state:
    • due to the fact current 2.14.16 version doesn't pull the metafield_product_images records at all
  • test_record_schema_match_with_state and test_all_records_are_the_same_with_state:
    • due to the fact the cursor overlap for the deleted events is fixed; thus, the fail is expected. Previously it had updated_at field, for the deleted record, now it's not, and should not since the deleted record could not be updated on the Shopify side, moreover this field was synthetic, we don't need it now.

All the rest - it 1:1.

[bazarnov]

/approve-regression-tests

Check job output.

✅ Approving regression tests

[maxi297]

Just a couple questions to make sure I understand properly

[maxi297]

Thanks for the added context. I'm good with those changes!

[bazarnov]

/approve-regression-tests

Check job output.

✅ Approving regression tests

[tolik0]

/approve-regression-tests

Check job output.

✅ Approving regression tests

[xavier-lai]

Thanks @bazarnov for those fix