Implementing a timeout on the runners for long-running pipelines

[daniddelrio]

In our organization, we have issues on some workflows intermittently and unintentionally running for a few hours. Since there's no option of globally setting timeouts on the workflows (see Discussion), we were wondering if there's a way to implement this timeout on the runner-level or in ARC.

Our first idea was to:

1. Using a pre-start hook, annotate our runners with the start time of the job
2. Create a cronjob that constantly checks for this running job and for the time elapsed since the annotated starting time. If it surpasses a certain amount of time, then we kill the runner

However, maybe there's an easier way that's already implemented natively? Something like a TTL controller or an existing process that can handle this.

I see that there was an issue #2360 posted about this, but it didn't get any traction

[salob]

We are hitting this issue too but in our case we want to increase the 6hr limit. It was the reason we moved from hosted to self-hosted. I will try and explicitly set the timeout to more than 6hrs in the workflow but I am not confident