Fix Periodic jobs

[kadel]

Currently, Periodic jobs are all failing. From time to time, one of the successes but that is quite rare :-/
And as far as I know, we are not investigating those errors :-(

This means that we can't rely on those jobs and we don't have any tests that are verifying odo on different openshift versions.

Goal:
Get periodic jobs into the state when each day each test passes at least once.

/area testing
/kind flake
/priority high

[prietyc123]

Almost everytime we are hitting platform issue on periodic job (4.2, 4.3 and 4.4 cluster).

error: some steps failed:
  * could not run steps: step integration-e2e-periodic failed: "integration-e2e-periodic" pre steps failed: "integration-e2e-periodic" pod "integration-e2e-periodic-ipi-conf" failed: pod didn't start running within 30 minutes: 
* Container artifacts is not ready with reason Completed and message 
* Container test is not ready with reason ImagePullBackOff and message Back-off pulling image "docker-registry.default.svc:5000/ci-op-xdyz9gp5/stable:tools"
Link to step on registry info site: https://steps.ci.openshift.org/reference/ipi-conf
Link to job on registry info site: https://steps.ci.openshift.org/job?org=openshift&repo=odo&branch=master&test=integration-e2e-periodic&variant=v4.3
time="2020-08-30T06:36:24Z" level=info msg="Reporting job state 'failed' with reason 'executing_graph:step_failed:utilizing_lease:executing_test:executing_multi_stage_test'"

Details : https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-odo-master-v4.3-integration-e2e-periodic/1300312385071878144#1:build-log.txt%3A60

Already reported to the 4 dev triage team. Will update the failure reason soon.

[prietyc123]

Added few more debug logs in the discussion thread in test platform channel regarding our periodic job failure.
Analysis information:

2020/08/31 06:00:23 Resolved ocp/4.3:cli to sha256:7d4f7a758df94828e4f315f400bc7be5228fe89c1452fed28a8745a18dc8541c
2020/08/31 06:00:23 Resolved ocp/4.5:upi-installer to sha256:eab670e4c84cee13872bc7186b466097a3390353a96298619061f2e8e4ed6359

As the jobs are running on 4.3 cluster but somehow 4.5 upi-installer has been reported in the log which looks weird to me.

Now this issue has been fixed through openshift/release#11447

Good news is that our periodic jobs are now passing since last run. No platform issue has been hit after that fix
Thanks @amitkrout for sharing CI job tips.

[prietyc123]

We have one more platform failure but that is not consistent. I need to follow up on this with test platform folks.

[prietyc123]

Fixed frequent periodic failure on operatorhub #3256 (comment) with pr openshift/release#11695

Now I feel we are in a state where each day each test passes at least once on periodic jobs. Achieving our described goals, So closing this issue.

/close

[kadel]

🎉 This is great. Thank you @prietyc123