Survey Monkey Response Stream (Incremental Setting) does not pull all responses for a survey

[danieldiamond]

Environment

• Airbyte version: 0.35.50-alpha
• OS Version / Instance: AWS EC2
• Deployment: Docker
• Source Connector and version: Survey Monkey 0.1.7
• Destination Connector and version: 0.4.17
• Severity: Critical
• Step where error happened: Sync

Current Behavior

When a response stream is configured for "full-refresh" / "overwrite", all responses are pulled for a survey. However, when the response stream is configured for "incremental", it does not pull all responses, even if it is the first sync.

Expected Behavior

I expect the same amount of responses pulled for the first sync irrespective of the stream configuration.

Logs

Steps to Reproduce

1. Configure response stream as incremental
2. Sync
3. Reset
4. Configure response stream as full-refresh/overwrite,
5. Sync
6. Obverve difference

Are you willing to submit a PR?

No

[danieldiamond]

@marcosmarxm is there someone that understands incremental streams that can apply the same logic for the survey monkey stream?

All other sources have incremental streams working correctly so I imagine it's just that this specific one is outdated e.g. applying GoogleAds Incremental logic here.

Taking a step back, it's probably worth creating some base classes for the incremental stream (appreciating that each source has it's on limitations/configurations).

[alafanechere]

Hi @danieldiamond, the most up-to-date way of implementing incremental streams with our CDK is to use IncrementalMixinclasses.
In the SurveyMonkey context the SurveyResponse stream is overriding the default (and now legacy) get_updated_state function defined in IncrementalSurveymonkeyStream and tries to manage nested cursor (one cursor per Survey). I think the error lies in this function and this logic. I would suggest to rewrite this stream by leveraging IncrementalMixin and HTTPSubstreams. The Stripe connector could be a good example for the use of HttpSubstream.