Create pipeline element status tables

[nguyent8]

Problem & Solution Overview

Currently users can only see statuses of pipelines through the "k8s".pipelines metadata table but there are cases, such as debugging, where they would like to see more details of the elements deployed for the pipelines. As of now, they need do multiple Kubernetes queries directly to get those information.

This PR is to provide a more convenient way to get status of pipelines and their direct elements by:

• adding new fields (ready, failed) to the existing new fields "k8s".pipelines
• adding new tables:
  • "k8s".pipeline_elements table: provides statuses of the elements specified in the spec field of the pipeline K8s objects
  • "k8s".pipeline_element_map : provides n:n mapping between pipeline elements and their pipelines

In the future, the above tables can be expanded to provide statuses of indirect elements, i.e. the ones deployed based on spec fields of other elements.

Testing Done

• Integration test: com.linkedin.hoptimator.k8s.TestSqlScripts.k8sMetadataTables

• Manual verification with the hoptimator cli:

  • deploy dev environment

  • create a couple of pipelines

  • verify statuses of pipelines and their elements by querying "k8s".pipelines and the new tables "k8s".pipeline_elements and "k8s".pipeline_element_map, e.g.:

select pl.name as pipeline_name, pl.ready as pipeline_ready, pl.status as pipeline_status, pe.element_name, pe.ready as element_ready, pe.failed as element_failed, pe.status as element_status from "k8s".pipelines pl inner join (select t2.element_name, t1.ready, t1.failed, t1.status, t2.pipeline_name from "k8s".pipeline_elements t1 inner join "k8s".pipeline_element_map t2 on t1.name = t2.element_name) pe on pl.name = pe.pipeline_name;

[ryannedolan]

🔥 🔥 🔥

[ryannedolan]

This is amazing!