Connectors should emit multiple error trace messages for each failing stream

[girarda]

Problem

https://docs.google.com/document/d/1yvJrxdjNNHVRvA-dsQQnnUeOFiMroG21SSSNSdWLWzI

The problem has been well documented in the one-pager. TLDR is that although the platform does not currently read anything but the first error trace message, we can modify the airbyte-cdk so that connectors continue syncing after one stream fails and they emit error messages for each stream with failures.

Implementation

Most of the changes should be able to be made within abstract_source.py:read(). Instead of collecting the messages in the dictionary and emitting it at the end, the errors should just be emitted as they come in so that there is parity with the old behavior where the sync stopped after the first failure.

We originally added a new interface method continue_sync_on_stream_failure() which could be overwritten on a per-connector basis. We only enabled this for source-klaviyo. We want the new error handling to be the default behavior so this method can be removed.

The code changes aren't that complex, but we want to buffer a little extra time for testing since we'll have to do this in prod using pre-release versions:

• Using an existing connector like klaviyo, change 3 streams that automatically fail with an unhandled 400 error
• Publish it as a pre-release version
• Update LaunchDarkly to use this version for your own personal workspace
• Run a sync from your workspace
• Verify in Sentry that the first stream that failed is logged as an error in Sentry

Open questions:

• Should we add a new interface method stop_sync_on_stream_failure() so connectors can disable this feature as needed? We could just wait and see if this is needed as opposed to adding it proactively.

Acceptance Criteria

• Connectors continue syncing even if the first stream raises an error
• Connectors emit an error trace message for each stream that has a failure
• New connectors should use this behavior once they get republished
• Add a log message at the end of the stream for which streams failed
• New error flow has been tested using pre-release versions

[brianjlai]

grooming notes:

• we should keep this interface flag
• as part of acceptance go through the top 10-20 connectors to see if they need to be overwritten (i.e. salesforce shared daily limit). Write this into a doc to list out which connectors should stop syncs early if an error occurs