When an instance of uptimer's pushability measurement fails it should continue to delete the pushed app

[sunjayBhatia]

When uptimer is targeting a foundation that cf push fails consistently on, it does not clean up the pushability measurement app that it pushed.

The cmdRunner.RunInSequenceWithContext method will return early if any of the commands to run fail. This is problematic for the pushability measurement because it can leave applications around in the pushability measurement org/space that fill up the quota and prevent further apps from being pushed.

For example, we had a deployment recently where uptimer failed to push an app 17 times, however the last 7 of these failed with You have exceeded your organization's memory limit: app requested more memory than available. We expected that all 17 failures would be the same failure reason as the first 10 failures.

This could be fixed by always running all commands in cmdRunner. RunInSequenceWithContext and collecting errors, though this might break other measurements/cause a bad UX. It would be preferable if the pushability measurement had a cleanup command that always ran to delete the pushed app instead.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/162612805

The labels on this github issue will be updated when the story is started.

[ljfranklin]

This bit the PAS RelEng team recently as well. @heyjcollins is this something that y'all would have the bandwidth to address or would you rather take a PR?

[heyjcollins]

Sorry for the delay here - the team's focused on another tract of work, but we would be open to a PR to address this issue.

cc @Syerram (rolling on as RelInt PM).