Gate ovn-controller starting post reboot until ovnkube controller syncs

TODO; Must investigate if this is the right approach for all deployments.
We need IC enabled.

This commit fixes OCPBUGS-42303.
If ovn-controller starts before ovnkube-controller syncs and the changes
propagated to SB DB, then ovn-controller will consume stale SB DB data.
This PR gates starting ovn-controller until ovnkube controller syncs.
ovnkube-controller emits a file to non-persistent storage and we
predicate the start on this.

Signed-off-by: Martin Kennelly <[email protected]>

Author: martinkennelly

bindata/network/ovn-kubernetes/common/008-script-lib.yaml

@@ -59,6 +59,17 @@ data:
         exit 1
       fi
 
+      echo "$(date -Iseconds) - waiting for 5 minutes for ovnkube-controller to sync once post reboot by ensuring a file exits..."
+      local retries=0
+      # ten minutes timeout
+      while [[ 3000 -gt "${retries}" ]]; do
+        (( retries += 1 ))
+        if [[ -f /tmp/ovnkube-controller-sync-post-boot ]]; then
+          break
+        fi
+        sleep .2
+      done
+
       echo "$(date -Iseconds) - starting ovn-controller"
       exec ovn-controller \
         unix:${vswitch_dbsock} \