You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Facebook Marketing is a good example of why the state should be an explicit attribute of the stream.
An incremental stream is a cursor slicing through the remote data, it is wrong to assume that the state (cursor position) depends only on the latest record that we read. Therefore we should use explicit state get and set constructions (the version that we used in base python before introducing CDK). This will allow setting the state explicitly before any reading operation (stream_slices depends on state value because we save state for each slice individually in FB Marketing).
Another example:
Let's say we have reversed incremental stream, that stream updates its state only once - at the end of the reading.
This makes it impossible to update the state with get_updated_state AFTER the last record has been emitted.
Later state attribute could be used to unambiguously detect if the stream is incremental (the current way is to check cursor_field, which is rather weird).
The text was updated successfully, but these errors were encountered:
Tell us about the problem you're trying to solve
Facebook Marketing is a good example of why the state should be an explicit attribute of the stream.
An incremental stream is a cursor slicing through the remote data, it is wrong to assume that the state (cursor position) depends only on the latest record that we read. Therefore we should use explicit state get and set constructions (the version that we used in base python before introducing CDK). This will allow setting the state explicitly before any reading operation (
stream_slices
depends on state value because we save state for each slice individually in FB Marketing).Another example:
Let's say we have reversed incremental stream, that stream updates its state only once - at the end of the reading.
This makes it impossible to update the state with
get_updated_state
AFTER the last record has been emitted.Describe the solution you’d like
in source
Later
state
attribute could be used to unambiguously detect if the stream is incremental (the current way is to check cursor_field, which is rather weird).The text was updated successfully, but these errors were encountered: