Instagram connector failing for all user-related tables

[jim-barlow]

Enviroment

• Airbyte version: example is 0.27.0-alpha
• OS Version / Instance: GCP Compute Engine (n1-standard-1)
• Deployment: Docker
• Source Connector and version: Instagram 0.1.6
• Destination Connector and version: BigQuery 0.3.6
• Severity: High
• Step where error happened: Sync job

Current Behavior

I upgraded both Airbyte and the Instagram Connector on 9th July for housekeeping, however since that date the user_insights, user_lifetime_insights and users have been failing.

Expected Behavior

Sync all tables as normal

Logs

logs-268-2.txt
logs-268-1.txt
logs-268-0.txt

LOG

replace this with
your long log
output here

Steps to Reproduce

1. Set up Instagram Source
2. Set up BigQuery Destination
3. Sync (all defaults)

[sherifnada]

thanks for creating the report @jimbeepbeep , will look into it!

[keu]

@jimbeepbeep, I tracked the issue, it is because of different format of the state files that the new version uses. If that is possible for you, could you please remove states and try again?

[jim-barlow]

Thanks for the quick response - I've just upgraded Airbyte and Connector versions today and am running a full sync, however we have 140+ accounts so it takes a while. When it is complete (success or fail) I will post the new logs here.

[jim-barlow]

It's been running for about 12hrs already, which normally means it'll fail... logs are here:

logs-285-0.txt

[keu]

Thanks, @jimbeepbeep , just to clarify, did you reset the connection from the UI? because I still can see errors related to old state file format.

[jim-barlow]

I didn't as I don't want to reset old data - is there a a way of doing that but preserving historic data?

[sherifnada]

@keu I think we will have to add backwards compatibility for the old format in the connector itself, manually changing state is not going to be very doable here

[jim-barlow]

Just leaving the logs here - this is the full failed fun from yesterday, there should be ~120k records but it says it's failed with ~360k records, looking like it's processing the media stream three times:

logs-285-0 (1).txt