[feature] migrate images to ghcr #2455

mahdikhashan · 2025-02-27T12:17:39Z

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #2446

ref: kubeflow/manifests#3010 (comment)

Checklist:

Docs included if any changes are user facing

Signed-off-by: mahdikhashan <[email protected]>

mahdikhashan · 2025-02-27T12:20:03Z

/cc @kubeflow/wg-automl-leads @Electronic-Waste @varodrig PTAL!

andreyvelich

Thank you for updating this @mahdikhashan!
Should we do the same for the Training Operator v1.9 ?

andreyvelich · 2025-02-27T13:07:02Z

.github/workflows/build-and-push-images.yaml

@@ -45,7 +46,7 @@ jobs:
        id: publish
        uses: ./.github/workflows/template-publish-image
        with:
-          image: docker.io/kubeflow/${{ matrix.component-name }}
+          image: ghcr.io/kubeflow/${{ matrix.component-name }}


Does GHCR support additional paths so we can distinguish images by Kubeflow Project name:

docker pull ghcr.io/kubeflow/trainer/trainer-controller-manager

https://github.com/orgs/kubeflow/packages/container/package/training%2Ftraining-operator

yes, you are right - done.

tenzen-y

Thank you for this.
Could we check if uploading time does does not exceed 10min?

https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry

For example images (mostly releae-1.9 branch), I'm wondering if we take 10 or more time to upload images.

Signed-off-by: mahdikhashan <[email protected]>

Electronic-Waste

@mahdikhashan Thanks for this! We may also need to update the image under manifests/ dir:

trainer/manifests/base/manager/manager.yaml

Line 25 in a6b4840

image: kubeflow/trainer-controller-manager

trainer/manifests/overlays/manager/kustomization.yaml

Lines 15 to 18 in a6b4840

    
           # Update the Kubeflow Trainer controller manager image tag. 
        
           images: 
        
             - name: kubeflow/trainer-controller-manager 
        
               newTag: latest

Not sure if there are other image definitions elsewhere.

varodrig · 2025-03-03T13:58:15Z

@mahdikhashan thank you for working on this.

varodrig · 2025-03-03T16:22:48Z

! We may also need to update the image under manifests/ dir:

@mahdikhashan keep us posted on this updates to approve the PR. thank you

varodrig · 2025-03-03T16:24:15Z

/hold waiting for updates

andreyvelich · 2025-03-04T16:47:09Z

We discussed with @thesuperzapper whether we should push images to both DockerHub and GHCR.
Maybe we can consider it but by default we will use the GHCR image ?

andreyvelich · 2025-03-04T16:47:52Z

@mahdikhashan We also need to migrate Training Operator 1.9 images to GHCR, do you have time to help us with it as well?

tenzen-y · 2025-03-04T21:03:52Z

We discussed with @thesuperzapper whether we should push images to both DockerHub and GHCR. Maybe we can consider it but by default we will use the GHCR image ?

+1 on DockerHub and GHCR since we can increase Googlability.

mahdikhashan · 2025-03-05T09:00:18Z

thank you all for your time, i'll free up some time and manage this pr.

andreyvelich

As we discussed in this PR, @mahdikhashan please can we continue to publish images to GHCR and DockerHub, but use GHCR in the default manifests: #2491

mahdikhashan · 2025-03-13T13:31:08Z

As we discussed in this PR, @mahdikhashan please can we continue to publish images to GHCR and DockerHub, but use GHCR in the default manifests: #2491

I'll dedicate time today and finalize it. sorry for the delay.

mahdikhashan · 2025-03-16T09:44:43Z

@mahdikhashan We also need to migrate Training Operator 1.9 images to GHCR, do you have time to help us with it as well?

it is addressed here: #2491

mahdikhashan · 2025-03-16T09:45:22Z

Thank you for this. Could we check if uploading time does does not exceed 10min?

https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry

For example images (mostly releae-1.9 branch), I'm wondering if we take 10 or more time to upload images.

I'll check it on my own user-account and share the result here.

mahdikhashan · 2025-03-16T12:05:31Z

@tenzen-y

please check the following run:

https://github.com/mahdikhashan/trainer/actions/runs/13882367194/job/38842581119#step:4:1315

the build takes almost 30 min, but the push to ghcr is around 11 seconds.

Signed-off-by: mahdikhashan <[email protected]>

mahdikhashan · 2025-03-16T13:56:19Z

As we discussed in this PR, @mahdikhashan please can we continue to publish images to GHCR and DockerHub, but use GHCR in the default manifests: #2491

done. please take a look: https://github.com/mahdikhashan/trainer/actions/runs/13884081580

Signed-off-by: Mahdi Khashan <[email protected]>

Signed-off-by: mahdikhashan <[email protected]>

mahdikhashan · 2025-03-16T14:02:31Z

requested changes addressed.

/cc @kubeflow/wg-automl-leads @Electronic-Waste @varodrig PTAL!

tenzen-y · 2025-03-16T15:03:24Z

@tenzen-y

please check the following run:

https://github.com/mahdikhashan/trainer/actions/runs/13882367194/job/38842581119#step:4:1315

the build takes almost 30 min, but the push to ghcr is around 11 seconds.

Sounds great, thanks

tenzen-y · 2025-03-16T15:04:34Z

@mahdikhashan We need to fix E2E testings so that we can use ghcr imegin inside KinD cluster.
Or, we can use dockerhub image for E2E.

Failed to pull image "ghcr.io/kubeflow/trainer/trainer-controller-manager:latest": failed to pull and unpack image "ghcr.io/kubeflow/trainer/trainer-controller-manager:latest": failed to resolve reference "ghcr.io/kubeflow/trainer/trainer-controller-manager:latest": failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://ghcr.io/token?scope=repository%3Akubeflow%2Ftrainer%2Ftrainer-controller-manager%3Apull&service=ghcr.io: 403 Forbidden

mahdikhashan · 2025-03-16T15:06:32Z

@mahdikhashan We need to fix E2E testings so that we can use ghcr imegin inside KinD cluster. Or, we can use dockerhub image for E2E.

Failed to pull image "ghcr.io/kubeflow/trainer/trainer-controller-manager:latest": failed to pull and unpack image "ghcr.io/kubeflow/trainer/trainer-controller-manager:latest": failed to resolve reference "ghcr.io/kubeflow/trainer/trainer-controller-manager:latest": failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://ghcr.io/token?scope=repository%3Akubeflow%2Ftrainer%2Ftrainer-controller-manager%3Apull&service=ghcr.io: 403 Forbidden

the first option is not possible since there are no images on ghcr, for dockerhub it will become a problem due to pull limit by 1st of April. I'd suggest we fix them in another pr. open to your suggestion though.

tenzen-y · 2025-03-16T15:09:10Z

@mahdikhashan We need to fix E2E testings so that we can use ghcr imegin inside KinD cluster. Or, we can use dockerhub image for E2E.

Failed to pull image "ghcr.io/kubeflow/trainer/trainer-controller-manager:latest": failed to pull and unpack image "ghcr.io/kubeflow/trainer/trainer-controller-manager:latest": failed to resolve reference "ghcr.io/kubeflow/trainer/trainer-controller-manager:latest": failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://ghcr.io/token?scope=repository%3Akubeflow%2Ftrainer%2Ftrainer-controller-manager%3Apull&service=ghcr.io: 403 Forbidden

the first option is not possible since there are no images on ghcr, for dockerhub it will become a problem due to pull limit by 1st of April. I'd suggest we fix them in another pr. open to your suggestion though.

In that case, we can keep using the dockerhub image for manifests in this PR and focus only on gh actions change.
In the next PR, we can replace dockerhub image with ghcr one for manifests.

mahdikhashan · 2025-03-16T15:10:02Z

@mahdikhashan We need to fix E2E testings so that we can use ghcr imegin inside KinD cluster. Or, we can use dockerhub image for E2E.

Failed to pull image "ghcr.io/kubeflow/trainer/trainer-controller-manager:latest": failed to pull and unpack image "ghcr.io/kubeflow/trainer/trainer-controller-manager:latest": failed to resolve reference "ghcr.io/kubeflow/trainer/trainer-controller-manager:latest": failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://ghcr.io/token?scope=repository%3Akubeflow%2Ftrainer%2Ftrainer-controller-manager%3Apull&service=ghcr.io: 403 Forbidden

the first option is not possible since there are no images on ghcr, for dockerhub it will become a problem due to pull limit by 1st of April. I'd suggest we fix them in another pr. open to your suggestion though.

In that case, we can keep using the dockerhub image for manifests in this PR and focus only on gh actions change. In the next PR, we can replace dockerhub image with ghcr one for manifests.

ok, i'll revert changes for manifest here and open an accompanying pr for manifests.

Signed-off-by: mahdikhashan <[email protected]>

mahdikhashan · 2025-03-16T15:18:24Z

@tenzen-y ptal #2529

tenzen-y

Thank you
/lgtm
/approve

tenzen-y · 2025-03-16T15:32:57Z

.github/workflows/build-and-push-images.yaml

+          context: ${{ matrix.context }}
+          push: false
+
+  build-and-publish-to-dockerhub:


I think this and GHCR can consolidate into one.

- name: Checkout - name: GHCR Login - name: Docker Hub Login - name: Publish Component ${{ matrix.component-name }} - name: Test Build For Component ${{ matrix.component-name }}

Because images can be specified multiple name: https://github.com/docker/metadata-action?tab=readme-ov-file#inputs

@mahdikhashan could you open an issue or work on this improvement?

you are right, i'm more leaned toward current change since it separates path for publishing and has less mental effort for contribution. final decision is yours, i can merge/consolidate them in this pr.

you are right, i'm more leaned toward current change since it separates path for publishing and has less mental effort for contribution. final decision is yours, i can merge them in this pr.

I do not want to block this PR by this my comment. So, I'm happy if you can work on my additional suggestion in the next PR, thanks.

/hold cancel

Please let me just know if you can work on it or just open an issue.

google-oss-prow · 2025-03-16T15:33:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

andreyvelich · 2025-03-16T15:45:52Z

.github/workflows/build-and-push-images.yaml

+          registry: docker.io
+          username: ${{ secrets.DOCKER_USERNAME }}
+          password: ${{ secrets.DOCKER_PASSWORD }}


This needs to be fixed @mahdikhashan:
DOCKERHUB_USERNAME

I'll open a hotfix.

I mentioned fixess in follow-up PR as well

* migrate Signed-off-by: mahdikhashan <[email protected]> * update path Signed-off-by: mahdikhashan <[email protected]> * pull manifest from ghcr Signed-off-by: mahdikhashan <[email protected]> * pull image from ghcr Signed-off-by: mahdikhashan <[email protected]> * add dockerhub publish as a job Signed-off-by: mahdikhashan <[email protected]> * update images and runner Signed-off-by: mahdikhashan <[email protected]> * revert: manifest changes Signed-off-by: mahdikhashan <[email protected]> --------- Signed-off-by: mahdikhashan <[email protected]> Signed-off-by: Mahdi Khashan <[email protected]>

google-oss-prow bot requested review from jinchihe and kuizhiqing February 27, 2025 12:17

google-oss-prow bot added the size/S label Feb 27, 2025

migrate

900ca0a

Signed-off-by: mahdikhashan <[email protected]>

mahdikhashan force-pushed the migrate-images-to-ghcr branch from 9699402 to 900ca0a Compare February 27, 2025 12:18

andreyvelich reviewed Feb 27, 2025

View reviewed changes

tenzen-y reviewed Feb 27, 2025

View reviewed changes

update path

e151070

Signed-off-by: mahdikhashan <[email protected]>

mahdikhashan force-pushed the migrate-images-to-ghcr branch from fafd8d6 to e151070 Compare February 28, 2025 19:13

This comment was marked as outdated.

Sign in to view

Electronic-Waste reviewed Mar 3, 2025

View reviewed changes

google-oss-prow bot assigned varodrig Mar 3, 2025

google-oss-prow bot added the lgtm label Mar 3, 2025

google-oss-prow bot added the do-not-merge/hold label Mar 3, 2025

saileshd1402 mentioned this pull request Mar 10, 2025

Push images to GHCR for release-1.9 #2491

Merged

1 task

andreyvelich reviewed Mar 13, 2025

View reviewed changes

pull manifest from ghcr

d36be2a

Signed-off-by: mahdikhashan <[email protected]>

google-oss-prow bot added size/M and removed size/S labels Mar 16, 2025

mahdikhashan and others added 2 commits March 16, 2025 14:57

Merge branch 'master' into migrate-images-to-ghcr

1a4cac2

Signed-off-by: Mahdi Khashan <[email protected]>

update images and runner

6c863e6

Signed-off-by: mahdikhashan <[email protected]>

mahdikhashan requested review from andreyvelich, tenzen-y and Electronic-Waste March 16, 2025 14:02

revert: manifest changes

8ae887a

Signed-off-by: mahdikhashan <[email protected]>

mahdikhashan mentioned this pull request Mar 16, 2025

[feature] pull images in manifest from ghcr #2529

Merged

1 task

tenzen-y reviewed Mar 16, 2025

View reviewed changes

google-oss-prow bot assigned tenzen-y Mar 16, 2025

google-oss-prow bot added the lgtm label Mar 16, 2025

google-oss-prow bot added approved and removed do-not-merge/hold labels Mar 16, 2025

google-oss-prow bot merged commit 1adef36 into kubeflow:master Mar 16, 2025
19 checks passed

andreyvelich reviewed Mar 16, 2025

View reviewed changes

andreyvelich mentioned this pull request Mar 16, 2025

[hotfix] fix checkout on workflow #2531

Merged

1 task

mahdikhashan mentioned this pull request Mar 16, 2025

[feature] consolidate workflow jobs into one #2532

Closed

ashwinr64 mentioned this pull request Mar 17, 2025

[feature] merge GHCR and DockerHub CI jobs #2537

Merged

1 task

thesuperzapper mentioned this pull request Apr 4, 2025

Upload container images to Github Container Registry kubeflow/mpi-operator#684

Open

	# Update the Kubeflow Trainer controller manager image tag.
	images:
	- name: kubeflow/trainer-controller-manager
	newTag: latest

[feature] migrate images to ghcr #2455

[feature] migrate images to ghcr #2455

Uh oh!

Conversation

mahdikhashan commented Feb 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mahdikhashan commented Feb 27, 2025

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tenzen-y left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

Electronic-Waste left a comment

Choose a reason for hiding this comment

Uh oh!

varodrig commented Mar 3, 2025

Uh oh!

varodrig commented Mar 3, 2025

Uh oh!

varodrig commented Mar 3, 2025

Uh oh!

andreyvelich commented Mar 4, 2025

Uh oh!

andreyvelich commented Mar 4, 2025

Uh oh!

tenzen-y commented Mar 4, 2025

Uh oh!

mahdikhashan commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

mahdikhashan commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mahdikhashan commented Mar 16, 2025

Uh oh!

mahdikhashan commented Mar 16, 2025

Uh oh!

mahdikhashan commented Mar 16, 2025

Uh oh!

mahdikhashan commented Mar 16, 2025

Uh oh!

mahdikhashan commented Mar 16, 2025

Uh oh!

tenzen-y commented Mar 16, 2025

Uh oh!

tenzen-y commented Mar 16, 2025

Uh oh!

mahdikhashan commented Mar 16, 2025

Uh oh!

tenzen-y commented Mar 16, 2025

Uh oh!

mahdikhashan commented Mar 16, 2025

Uh oh!

mahdikhashan commented Mar 16, 2025

Uh oh!

tenzen-y left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mahdikhashan Mar 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tenzen-y Mar 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mahdikhashan commented Feb 27, 2025 •

edited

Loading

mahdikhashan commented Mar 5, 2025 •

edited

Loading

mahdikhashan commented Mar 13, 2025 •

edited

Loading

mahdikhashan Mar 16, 2025 •

edited

Loading

tenzen-y Mar 16, 2025 •

edited

Loading