feat(previews): retry errored previews if feature flag enabled

[iainsproat]

Description & motivation

For various reasons the preview service can return an error response. It will not be retried.

This provides a simple retry mechanism. Running as a scheduled task ("cronjob"), every 5 minutes it finds the most recent errored preview (one for each connected regional database). It resets the preview request to pending and queues a job to be processed by the preview service.

It maintains a simple in-memory record of the last errored preview it retried for each database, ensuring it progresses to the next most recent on the next scheduled run.

Changes:

• adds a new feature flag FF_RETRY_ERRORED_PREVIEWS_ENABLED

To-do before merge:

• test in a testing environment
  • test it works for no previews to retry
  • test it works for some errored previews to retry
• test in latest

Screenshots:

Validation of changes:

Checklist:

• My pull request follows the guidelines in the Contributing guide?
• My pull request does not duplicate any other open Pull Requests for the same update/change?
• My commits are related to the pull request and do not amend unrelated code or documentation.
• My code follows a similar style to existing code.
• I have added appropriate tests.
• I have updated or added relevant documentation.

References

[linear]

WEB-2979 Scheduled task to search for old failed previews and re-do them