Description
What
We want GA connectors to make use of an AvailabilityStrategy in their streams to be able to perform error handling on the
stream level. The base use case supported by the HttpAvailabilityStrategy
is for stream-based permissions,
in order to be able to skip a stream that the connector no longer has access to (because of e.g. scope
changes on an Oauth user config). An AvailabilityStrategy
can be used to address any error that may occur
on a stream level, to avoid hitting errors, or having to except the same error for every record in a stream,
instead of skipping that stream altogether.
How
There are a few different ways to go about this, usually depending on how much custom error handling we already
do in a connector, but also due to how the connector works.
- If we have no custom error handling: Inherit the default HttpAvailabilityStrategy. This is as simple as removing the override setting that currently sets
the AvailabilityStrategy for the streams toNone
. This is useful if we don't currently do much custom error handling
in the connector. - If we have custom error handling for 403: Forbidden errors: Do the above to inherit the HttpAvailabilityStrategy,
removing the custom error handling, but continuing any testing we have that 403s are handled gracefully by the connector - If we have custom error handling for lots of per-stream errors: Extend the HttpAvailabilityStrategy. You can subclass
the HttpAvailabilityStrategy to change thereasons_for_unavailable_status_codes
method to except more error codes and
provide messages for those error codes. - If there is a smarter way to determine availability than trying to read the first record and excepting errors: Inherit
the AvailabilityStrategy base class. For example, if there are clearly defined scopes required to access each stream in
a given source, you might instead determine whether a stream is available by checking the scopes on the authenticator to
see if it has the scopes required by the stream.
Note that the AvailabilityStrategy is defined on each stream. For most cases, it is easiest to set it in your stream's
base stream class, where the override is, but for some use cases, different streams might have different availability strategies -
and that's totally fine.