Add background task to monitor and cancel stuck pending pods

[APErebus]

Background

We have situation where the script pods can't be scheduled and remain in a Pending state. This is a problem because unless you have a step level timeout configured, the deployment will run for ever.

Results

Adds a new PendingPodWatchDog task that monitors all pods and if they are pending for more than a set period, then they are marked as timedout (which then causes them to be deleted).

Will be controlled by a value in the Helm Chart, but if this is not set, the watchdog does not run

After

How to review this PR

Quality ✔️

Pre-requisites

• I have read How we use GitHub Issues for help deciding when and where it's appropriate to make an issue.
• I have considered informing or consulting the right people, according to the ownership map.
• I have considered appropriate testing for my change.

[zentron]

Makes sense.
Is there any information or event data that would be useful logging?
If it is marked as completed and cleaned up, it may be harder to diagnose why it was hanging.