🐛 Source Stripe: missing data in Incremental sync mode

[kev-datams]

Connector Name

source-stripe

Connector Version

5.0.1

What step the error happened?

During the sync

Relevant information

Hello Airbyte's community,

I identified an issue on source Stripe Charges stream using Incremental sync mode, leading to missing data in destination.

The concerned missing payments looks to systematically have:

• Blocked outcome type
• empty failure message

NB: some other Blocked payments with filled failure message are properly retrieved.

Please find attached an example:


Another common pattern is the 402 Stripe API Error. It could probably be an interesting path to follow...
If this path is confirmed, it possibly impacts other streams also.

To be noticed:

• I tried with many destinations and the behavior is always the same:
  • Google Cloud Storage in AVRO
  • Google Cloud Storage in JSONL
  • BigQuery with GCS staging
  • BigQuery with standards inserts
• I tried a manual Incremental run in local from source-stripe Airbyte's repo using below command, and concerned payment is still NOT retrieved (contrary to others) :

python main.py read --config secrets/config.json --catalog secrets/configured_catalog.json --state secrets/state.json | grep py_***my_id***

• if I run a Full sync, data is properly retrieved BUT with wrong updated field value, for which right value is available thanks to Incremental sync only. So, this workaround does not look acceptable.

@davydov-d as Stripe expert, you could probably help to investigate ? :)
I would be happy to contribute also, but not sure from where to start...
This is currently a blocking point on our side to consume Stripe data via Airbyte.

Thanks a lot for your help ! 👍

EDIT: please check below comment with new elements to go further...

Relevant log output

No response

Contribute

• Yes, I want to contribute

[kev-datams]

Few new elements after deeper testing Full refresh vs Incremental: it seems to be an issue regarding pagination management in Incremental sync.

Considering a start date set at 2023-11-29T00:00:00Z (in config.json for Full refresh), equivalent to timestamp 1701216000 (in state.json for Incremental), I have below logs:

Full refresh

python main.py read --config secrets/config.json --catalog secrets/configured_catalog.json --state secrets/state.json | grep py_***my_id***

=> FOUND 👍

python main.py read --config secrets/config.json --catalog secrets/configured_catalog.json --state secrets/state.json --debug | grep "https://api.stripe.com/v1/charges"

=> generates:

{"type": "DEBUG", "message": "Making outbound API request", "data": {"headers": "{'User-Agent': 'python-requests/2.28.2', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Stripe-Version': '2022-11-15', 'Stripe-Account': 'xxx', 'Authorization': 'Bearer ****'}", "request_body": "None", "url": "https://api.stripe.com/v1/charges?created%5Bgte%5D=1701216000&created%5Blte%5D=1701302400&limit=100&expand%5B%5D=data.refunds"}}
{"type": "DEBUG", "message": "Making outbound API request", "data": {"headers": "{'User-Agent': 'python-requests/2.28.2', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Stripe-Version': '2022-11-15', 'Stripe-Account': 'xxx', 'Authorization': 'Bearer ****'}", "request_body": "None", "url": "https://api.stripe.com/v1/charges?created%5Bgte%5D=1701302401&created%5Blte%5D=1701388801&limit=100&expand%5B%5D=data.refunds"}}
{"type": "DEBUG", "message": "Making outbound API request", "data": {"headers": "{'User-Agent': 'python-requests/2.28.2', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Stripe-Version': '2022-11-15', 'Stripe-Account': 'xxx', 'Authorization': 'Bearer ****'}", "request_body": "None", "url": "https://api.stripe.com/v1/charges?created%5Bgte%5D=1701388802&created%5Blte%5D=1701452651&limit=100&expand%5B%5D=data.refunds"}}
{"type": "DEBUG", "message": "Making outbound API request", "data": {"headers": "{'User-Agent': 'python-requests/2.28.2', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Stripe-Version': '2022-11-15', 'Stripe-Account': 'xxx', 'Authorization': 'Bearer ****'}", "request_body": "None", "url": "https://api.stripe.com/v1/charges?created%5Bgte%5D=1701216000&created%5Blte%5D=1701302400&limit=100&expand%5B%5D=data.refunds"}}
{"type": "DEBUG", "message": "Making outbound API request", "data": {"headers": "{'User-Agent': 'python-requests/2.28.2', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Stripe-Version': '2022-11-15', 'Stripe-Account': 'xxx', 'Authorization': 'Bearer ****'}", "request_body": "None", "url": "https://api.stripe.com/v1/charges?created%5Bgte%5D=1701302401&created%5Blte%5D=1701388801&limit=100&expand%5B%5D=data.refunds&starting_after=py_3OI1uCCBk8J2diIe1k9P14b5"}}

=> pagination management looks fine 👍

Incremental

python main.py read --config secrets/config.json --catalog secrets/configured_catalog.json --state secrets/state.json | grep py_***my_id***

=> NOT FOUND 👎

python main.py read --config secrets/config.json --catalog secrets/configured_catalog.json --state secrets/state.json --debug | grep "https://api.stripe.com/v1/charges"

=> generates:

{"type": "DEBUG", "message": "Making outbound API request", "data": {"headers": "{'User-Agent': 'python-requests/2.28.2', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Stripe-Version': '2022-11-15', 'Stripe-Account': 'xxx', 'Authorization': 'Bearer ****'}", "url": "https://api.stripe.com/v1/charges?created%5Bgte%5D=1701216000&created%5Blte%5D=1701302400&limit=100&expand%5B%5D=data.refunds", "request_body": "None"}}

=> pagination management looks broken 👎

... or is it simply a different logging display in case of enabled debugging ?

---

Alternative: bug on Stripe ?

Other idea: any issue on Stripe side to log the event on charge creation following the PaymentIntent into the Events object, even if the charge really exists in the Charges object ?

=> to prevent from similar issues in a general way, in case of Incremental load, the connector would have to check into the Stream object (here Charges) for occurrences in the same temporal scope complementary to Events object ? (99% it will produce same objects duplicated, but 1% it would help to workaround the identified issue)

[kev-datams]

or perhaps @girarda as you seem to master Stripe connector 🙏