Add bash script that extracts images from manifests repо #2419

difince · 2023-03-24T08:30:16Z

Which issue is resolved by this Pull Request:
Partially resolves #kubeflow/community#599

Description of your changes:
The PR contains:

a bash script that extracts images from the manifests repo currently for the following directories:

./apps 
../common 
../example 
../contrib/metacontroller 
../contrib/seldon

The resulted image list itself placed in /docs directory

Checklist:

Unit tests pass:
Make sure you have installed kustomize == 3.2.1
1. make generate-changed-only
2. make test

google-oss-prow · 2023-03-24T08:30:28Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: difince
Once this PR has been reviewed and has the lgtm label, please assign yanniszark for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

The PR contains: - the script that extracts the images - the image list itslef Signed-off-by: difince <[email protected]>

difince · 2023-03-24T09:19:33Z

Thanks to @annajung for noticing the followings:

kustomize build command fails for the following directories:

 ../apps/kfp-tekton/upstream/third-party/kfp-csi-s3.
 ../apps/kfp-tekton/upstream/env/platform-agnostic.
 ../apps/kfp-tekton/upstream/env/plain-multi-user.
 ../apps/kfp-tekton/upstream/env/plain.
 ../apps/kfp-tekton/upstream/env/kfp-template.
 ../apps/kfp-tekton/upstream/env/platform-agnostic-emissary.
 ../apps/kfp-tekton/upstream/env/platform-agnostic-multi-user.
 ../apps/kfp-tekton/upstream/env/aws.
 ../apps/kfp-tekton/upstream/env/platform-agnostic-multi-user-emissary.
 ../apps/kfp-tekton/upstream/env/dev.
 ../apps/kfp-tekton/upstream/env/cert-manager/platform-agnostic-multi-user.
 ../apps/kfp-tekton/upstream/env/cert-manager/dev.
 ../common/dex/overlays/ldap.
 ../common/dex/overlays/github.

Some images does not specify imageTag:

docker.io/kubeflowkatib/cert-generator
docker.io/kubeflowkatib/katib-ui
docker.io/kubeflownotebookswg/notebook-controller
docker.io/kubeflownotebookswg/profile-controller
docker.io/kubeflownotebookswg/tensorboard-controller
kubeflow/training-operator
minio/minio
prom/prometheus

hack/extract_manifests_images.sh

annajung · 2023-03-24T13:21:21Z

Some images does not specify imageTag:

docker.io/kubeflowkatib/cert-generator
docker.io/kubeflowkatib/katib-ui
docker.io/kubeflownotebookswg/notebook-controller
docker.io/kubeflownotebookswg/profile-controller
docker.io/kubeflownotebookswg/tensorboard-controller
kubeflow/training-operator
minio/minio
prom/prometheus

To clarify, the script captures both non tag and tag images for the list above.

For example, docker.io/kubeflowkatib/cert-generator lists both

docker.io/kubeflowkatib/cert-generator
docker.io/kubeflowkatib/cert-generator:v0.15.0-rc.0

After a further look into it, running kustomize build apps/katib/upstream/components/cert-generator/ produces the following with no image tag appended to the image name

apiVersion: v1
kind: ServiceAccount
metadata:
  name: katib-cert-generator
  namespace: kubeflow
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: katib-cert-generator
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  - services
  verbs:
  - get
  - create
  - delete
- apiGroups:
  - batch
  resources:
  - jobs
  verbs:
  - get
- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - validatingwebhookconfigurations
  - mutatingwebhookconfigurations
  verbs:
  - get
  - patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: katib-cert-generator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: katib-cert-generator
subjects:
- kind: ServiceAccount
  name: katib-cert-generator
  namespace: kubeflow
---
apiVersion: batch/v1
kind: Job
metadata:
  labels:
    katib.kubeflow.org/component: cert-generator
  name: katib-cert-generator
  namespace: kubeflow
spec:
  backoffLimit: 4
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
      containers:
      - args:
        - generate
        - --namespace=$(KATIB_CORE_NAMESPACE)
        command:
        - ./katib-cert-generator
        env:
        - name: KATIB_CORE_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        image: docker.io/kubeflowkatib/cert-generator
        name: cert-generator
      restartPolicy: Never
      serviceAccountName: katib-cert-generator

difince · 2023-03-24T14:55:12Z

Yes, exactly Anna..thank you for being more explicit.
Below I am sharing the mapping between the images without a tag and the relevant directories

image: docker.io/kubeflowkatib/cert-generator
kustomize build apps/katib/upstream/components/cert-generator
image: docker.io/kubeflowkatib/katib-ui
kustomize build apps/katib/upstream/components/ui
image: docker.io/kubeflownotebookswg/notebook-controller
kustomize build apps/jupyter/notebook-controller/upstream/manager
image: docker.io/kubeflownotebookswg/tensorboard-controller
kustomize build apps/profiles/upstream/manager
image: docker.io/kubeflownotebookswg/tensorboard-controller
kustomize build apps/tensorboard/tensorboard-controller/upstream/manager
image: kubeflow/training-operator
kustomize build apps/training-operator/upstream/base/crds
image: minio/minio
kustomize build apps/kfp-tekton/upstream/third-party/argo/upstream/manifests/quick-start/base/minio
kustomize build apps/pipeline/upstream/third-party/argo/upstream/manifests/quick-start/base/minio
image: prom/prometheus
kustomize build apps/kfp-tekton/upstream/third-party/argo/upstream/manifests/quick-start/base/prometheus
kustomize build apps/pipeline/upstream/third-party/argo/upstream/manifests/quick-start/base/prometheus

create a string array of all the directories the script itterates over update the list of image to include bentoml related one Signed-off-by: difince <[email protected]>

difince · 2023-03-24T15:20:56Z

/assign @StefanoFioravanzo
/assign @elikatsis

annajung · 2023-03-28T14:51:59Z

With the release being scheduled for tomorrow, a friendly reminder to review @DomFleischmann @kimwnasptd

DomFleischmann · 2023-03-28T15:58:35Z

This is a great effort @difince ! Would you mind updating the list with the final 1.7 images once #2428 is merged?

difince · 2023-03-29T10:59:56Z

Closing the PR as a new PR containing the same code is open against v1.7-branch

google-oss-prow bot added the size/M label Mar 24, 2023

google-oss-prow bot requested review from annajung, elikatsis and StefanoFioravanzo March 24, 2023 08:30

google-oss-prow bot added size/L and removed size/M labels Mar 24, 2023

difince force-pushed the extract_images branch from 5730f3f to 60a4f3e Compare March 24, 2023 08:57

Add a script that extracts all images used by main KF components

995ce7a

The PR contains: - the script that extracts the images - the image list itslef Signed-off-by: difince <[email protected]>

difince force-pushed the extract_images branch from 582d4f4 to 995ce7a Compare March 24, 2023 09:01

annajung reviewed Mar 24, 2023

View reviewed changes

hack/extract_manifests_images.sh Outdated Show resolved Hide resolved

Add "../contrib/bentoml" dir to extract files from

4d7507f

create a string array of all the directories the script itterates over update the list of image to include bentoml related one Signed-off-by: difince <[email protected]>

difince requested review from annajung and removed request for StefanoFioravanzo and elikatsis March 24, 2023 15:14

google-oss-prow bot assigned elikatsis and StefanoFioravanzo Mar 24, 2023

difince closed this Mar 29, 2023

annajung mentioned this pull request Mar 31, 2023

Apply new release tags to all controller images kubeflow/kubeflow#7078

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add bash script that extracts images from manifests repо #2419

Add bash script that extracts images from manifests repо #2419

difince commented Mar 24, 2023 •

edited

Loading

google-oss-prow bot commented Mar 24, 2023

difince commented Mar 24, 2023

annajung commented Mar 24, 2023

difince commented Mar 24, 2023 •

edited

Loading

difince commented Mar 24, 2023

annajung commented Mar 28, 2023

DomFleischmann commented Mar 28, 2023

difince commented Mar 29, 2023

Add bash script that extracts images from manifests repо #2419

Add bash script that extracts images from manifests repо #2419

Conversation

difince commented Mar 24, 2023 • edited Loading

google-oss-prow bot commented Mar 24, 2023

difince commented Mar 24, 2023

annajung commented Mar 24, 2023

difince commented Mar 24, 2023 • edited Loading

difince commented Mar 24, 2023

annajung commented Mar 28, 2023

DomFleischmann commented Mar 28, 2023

difince commented Mar 29, 2023

difince commented Mar 24, 2023 •

edited

Loading

difince commented Mar 24, 2023 •

edited

Loading