Closed
Description
We'd like a guide that shows how to set up a scheduled pipeline that can read a sliding window of data.
It’s already clear how to set up a pipeline that will read the exact same source data each time it runs. For scheduled pipelines users often want some way (such as a regex) to specify how to read a sliding window of data. This capability exists, but needs to be documented.
A good place for the guide is in the tutorials section:
https://www.kubeflow.org/docs/pipelines/tutorials/
It'd be good if this guide is generic rather than being specific to GCP or any other platform.