[source-pardot] 504 gateway timeout on syncs with very far start dates

[maxestorr]

Connector Name

source-pardot

Connector Version

1.0.1

What step the error happened?

During the sync

Relevant information

Pardot sync for the visitor activities stream fails (504 gateway timeout) when setting start date to 2022, but works if I set it to the start of the current month.

All of the other streams I was interested in are capable of backfilling from 2022, it's only visitor activites that has this issue.

It's hard for me to locate the "relevant" parts of the log output, I've changed the log level to trace and still can't find an explanation in the logs as to why it fails, and copying the whole log output is too large - I will find a way to host the file and share it here.

Relevant log output

2025-03-24 15:38:48 source INFO Starting syncing
2025-03-24 15:38:48 source INFO Marking stream visitor_activities as STARTED
2025-03-24 15:38:48 source INFO Syncing stream: visitor_activities 
2025-03-24 15:38:48 source INFO Backing off _send(...) for 0.0s (airbyte_cdk.sources.streams.http.exceptions.UserDefinedBackoffException: Gateway timeout.)
2025-03-24 15:38:48 source INFO Retrying. Sleeping for 30.0 seconds
2025-03-24 15:38:48 source ERROR Giving up _send(...) after 2 tries (airbyte_cdk.sources.streams.http.exceptions.UserDefinedBackoffException: Gateway timeout.)
2025-03-24 15:38:48 source ERROR Max retry limit reached in 181.225222s. Request: <PreparedRequest [GET]>, Response: <Response [504]>
2025-03-24 15:38:48 source ERROR Exception while syncing stream visitor_activities
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/streams/concurrent/partition_reader.py", line 40, in process_partition
    for record in partition.read():
  File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/declarative/stream_slicers/declarative_partition_generator.py", line 59, in read
    for stream_data in self._retriever.read_records(self._json_schema, self._stream_slice):
  File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 469, in read_records
    for stream_data in self._read_pages(record_generator, self.state, _slice):
  File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 366, in _read_pages
    response = self._fetch_next_page(stream_state, stream_slice, next_page_token)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 322, in _fetch_next_page
    return self.requester.send_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/declarative/requesters/http_requester.py", line 400, in send_request
    request, response = self._http_client.send_request(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/streams/http/http_client.py", line 524, in send_request
    response: requests.Response = self._send_with_retry(
                                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/streams/http/http_client.py", line 270, in _send_with_retry
    response = backoff_handler(rate_limit_backoff_handler(user_backoff_handler))(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/streams/http/http_client.py", line 340, in _send
    self._handle_error_resolution(
  File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/streams/http/http_client.py", line 466, in _handle_error_resolution
    raise UserDefinedBackoffException(

Contribute

• Yes, I want to contribute

[maxestorr]

It actually won't even let me drag and drop the 216 KB log file into this comment, I'm not sure why...

[maxestorr]

logs: https://pastebin.com/kg9WkwuQ
password: FdftzVzPW2

[maxestorr]

Doing some of my own tests interacting with the API I’ve run into the same issue.

Requesting data updated after a timestamp long ago (e.g. 2022) with the parameter orderBy set to updatedAt ASC results in a 504 timeout, but not including the orderBy results in a successful response - albeit unordered, so not very useful for a backfill where we want to recover from failures.

I suppose the Pardot API times out trying to order such a large quantity of data. I have no idea why this is only an issue for visitor_activities when plenty of other streams worked fine requesting data from 2022.

I believe a potential solution would be to batch such a request using the updatedAtBeforeOrEqual parameter so that the orderBy doesn’t cause the Pardot API to time out.