[source-opsgenie] No format in ['%Y-%m-%dT%H:%M:%S.%fZ'] matching 2020-11-11T17:46:02Z

[dipth]

Connector Name

source-opsgenie

Connector Version

0.3.0

What step the error happened?

During the sync

Relevant information

This error happens when syncing alerts from the Opsgenie EU Api using the Opsgenie source connector.

Configuration:

• api_token: [REDACTED]
• endpoint: api.eu.opsgenie.com
• start_date: 2024-01-01T00:00:00Z

We have tried both using both "Incremental - Append" and "Incremental - Append + Deduped" replication modes. Both result in the same error.

Furthermore it looks like the connector is making too many requests against the Opsgenie API given that the log also contains a bunch of 429 responses:

2024-02-08 14:20:41 source > Backing off _send(...) for 5.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException: Request URL: https://api.eu.opsgenie.com/v2/alerts?limit=20&sort=createdAt&offset=160&order=asc, Response Code: 429, Response Text: {"requestId":"f73c22b2-68ca-48a6-9b16-3186ec4135c5","message":"You are making too many requests! To avoid errors, we recommend you limit requests.","took":0.002})
2024-02-08 14:20:41 source > Caught retryable error 'Request URL: https://api.eu.opsgenie.com/v2/alerts?limit=20&sort=createdAt&offset=160&order=asc, Response Code: 429, Response Text: {"requestId":"f73c22b2-68ca-48a6-9b16-3186ec4135c5","message":"You are making too many requests! To avoid errors, we recommend you limit requests.","took":0.002}' after 1 tries. Waiting 5 seconds then retrying...
2024-02-08 14:20:47 source > Backing off _send(...) for 5.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException: Request URL: https://api.eu.opsgenie.com/v2/alerts?limit=20&sort=createdAt&offset=240&order=asc, Response Code: 429, Response Text: {"requestId":"95911c6c-6a8e-4b03-8b62-03a813219326","message":"You are making too many requests! To avoid errors, we recommend you limit requests.","took":0.001})
2024-02-08 14:20:47 source > Caught retryable error 'Request URL: https://api.eu.opsgenie.com/v2/alerts?limit=20&sort=createdAt&offset=240&order=asc, Response Code: 429, Response Text: {"requestId":"95911c6c-6a8e-4b03-8b62-03a813219326","message":"You are making too many requests! To avoid errors, we recommend you limit requests.","took":0.001}' after 1 tries. Waiting 5 seconds then retrying...
2024-02-08 14:20:53 source > Backing off _send(...) for 5.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException: Request URL: https://api.eu.opsgenie.com/v2/alerts?limit=20&sort=createdAt&offset=340&order=asc, Response Code: 429, Response Text: {"requestId":"9a6da3a2-04ed-4505-ab07-ffeb332e7821","message":"You are making too many requests! To avoid errors, we recommend you limit requests.","took":0.001})
2024-02-08 14:20:53 source > Caught retryable error 'Request URL: https://api.eu.opsgenie.com/v2/alerts?limit=20&sort=createdAt&offset=340&order=asc, Response Code: 429, Response Text: {"requestId":"9a6da3a2-04ed-4505-ab07-ffeb332e7821","message":"You are making too many requests! To avoid errors, we recommend you limit requests.","took":0.001}' after 1 tries. Waiting 5 seconds then retrying...
2024-02-08 14:20:59 source > Backing off _send(...) for 5.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException: Request URL: https://api.eu.opsgenie.com/v2/alerts?limit=20&sort=createdAt&offset=440&order=asc, Response Code: 429, Response Text: {"requestId":"a7178a16-7dae-4ec4-aa9c-7dc78bee34f8","message":"You are making too many requests! To avoid errors, we recommend you limit requests.","took":0.002})
2024-02-08 14:20:59 source > Caught retryable error 'Request URL: https://api.eu.opsgenie.com/v2/alerts?limit=20&sort=createdAt&offset=440&order=asc, Response Code: 429, Response Text: {"requestId":"a7178a16-7dae-4ec4-aa9c-7dc78bee34f8","message":"You are making too many requests! To avoid errors, we recommend you limit requests.","took":0.002}' after 1 tries. Waiting 5 seconds then retrying...
2024-02-08 14:21:06 source > Backing off _send(...) for 5.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException: Request URL: https://api.eu.opsgenie.com/v2/alerts?limit=20&sort=createdAt&offset=620&order=asc, Response Code: 429, Response Text: {"requestId":"aec9963f-832d-4ab5-9c63-737702edc206","message":"You are making too many requests! To avoid errors, we recommend you limit requests.","took":0.002})
...

Relevant log output

2024-02-08 14:24:26 source > Encountered an exception while reading stream alerts
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 122, in read
    yield from self._read_stream(
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 194, in _read_stream
    for record in record_iterator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 230, in _read_incremental
    for record_data_or_message in stream_instance.read_incremental(
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/core.py", line 156, in read_incremental
    for record_data_or_message in records:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py", line 104, in read_records
    yield from self.retriever.read_records(stream_slice)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 305, in read_records
    most_recent_record_from_slice = self._get_most_recent_record(most_recent_record_from_slice, stream_data, stream_slice)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 319, in _get_most_recent_record
    return current_most_recent if self.cursor.is_greater_than_or_equal(current_most_recent, record) else record
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/incremental/datetime_based_cursor.py", line 283, in is_greater_than_or_equal
    return self.parse_date(first_cursor_value) >= self.parse_date(second_cursor_value)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/incremental/datetime_based_cursor.py", line 196, in parse_date
    raise ValueError(f"No format in {self.cursor_datetime_formats} matching {date}")
ValueError: No format in ['%Y-%m-%dT%H:%M:%S.%fZ'] matching 2020-11-11T17:46:02Z
2024-02-08 14:24:26 source > Marking stream alerts as STOPPED

Contribute

• Yes, I want to contribute

[dipth]

A theory could be that there's a bug in the OpsGenie API that causes the updated_at timestamp (which is used for the cursor based pagination) to be truncated if it falls exactly on the second.
So for instance instead of returning 2020-11-11T17:46:02.000Z the API returns 2020-11-11T17:46:02Z.
As far as I can see, the only cursor_datetime_formats defined for this connector is "%Y-%m-%dT%H:%M:%S.%fZ".
So changing:

      cursor_datetime_formats:
        - "%Y-%m-%dT%H:%M:%S.%fZ"

to

      cursor_datetime_formats:
        - "%Y-%m-%dT%H:%M:%S.%fZ"
        - "%Y-%m-%dT%H:%M:%SZ"

in https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-opsgenie/source_opsgenie/manifest.yaml#L108C1-L109C34 would probably fix the problem

[dipth]

I've been able to verify that for the particular alert that Airbyte is choking on, the updated_at timestamp is indeed truncated:

    "lastOccurredAt": "2020-11-11T15:34:03.541Z",
    "createdAt": "2020-11-11T15:34:03.541Z",
    "updatedAt": "2020-11-11T17:46:02Z",
    "source": "New Relic",

[dipth]

The alerts immediately preceding and following the problematic one has the correct format for the updated_at timestamp:

    "lastOccurredAt": "2020-11-11T16:24:41.267Z",
    "createdAt": "2020-11-11T15:19:41.263Z",
    "updatedAt": "2020-11-11T16:29:41.205Z",
    "source": "https://[REDACTED]/alertmanager/#/alerts?receiver=opsgenie-warning",

    "lastOccurredAt": "2020-11-11T17:06:41.281Z",
    "createdAt": "2020-11-11T16:01:41.243Z",
    "updatedAt": "2020-11-11T17:21:41.184Z",
    "source": "https://[REDACTED]/alertmanager/#/alerts?receiver=opsgenie-warning",

I've also been able to verify that the issue is not related to the alert source as here is another one from New Relic that has the correct format:

    "lastOccurredAt": "2023-10-20T12:04:52.368Z",
    "createdAt": "2023-10-20T12:04:52.368Z",
    "updatedAt": "2023-10-20T12:05:43.176Z",
    "source": "New Relic",

This all points to my theory being correct.