on-pr docker build stuck with `user is not authorized to BatchGetImage`

[malfet]

🐛 Describe the bug

Build https://github.com/pytorch/pytorch/actions/runs/13724713611/job/38388317099?pr=148740 stuck at Calculate docker image step trying to check if such image already exists or not

+ [[ 1741362495 -lt 1741364292 ]]
+ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc11:c097a94c03da3be2f692f9ff22e3963e933633cf
denied: User: arn:aws:sts::391835788720:assumed-role/ghci-lf-github-action-runners-runner-role/i-0e98877505f067739 is not authorized to perform: ecr:BatchGetImage on resource: arn:aws:ecr:us-east-1:308535385114:repository/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc11 because no resource-based policy allows the ecr:BatchGetImage action
+ '[' false == true ']'
+ sleep 300
++ date +%s

This logic was added by pytorch/test-infra#6013 but looks like it does not work right now due to some sort of security considerations. (Though all runners should have read access to ECR, shouldn't they?)

Versions

CI

cc @seemethere @pytorch/pytorch-dev-infra

[jeanschmidt]

Hi, yes, all runners should have access to them, but I would advise you to make sure you are not relying on it and you do have a OIDC step in your workflow to acquire this permission.

But I believe this is a new repository right?

If so, it is important to coordinate with pytorch dev infra team to make sure we're replicating this permission to runners in all our partners. Like Linux Foundation.

[malfet]

Hi, yes, all runners should have access to them, but I would advise you to make sure you are not relying on it and you do have a OIDC step in your workflow to acquire this permission.

Why? Read-only access should be there by default for all authorized accounts?

But I believe this is a new repository right?

Correct

If so, it is important to coordinate with pytorch dev infra team to make sure we're replicating this permission to runners in all our partners. Like Linux Foundation.

Is it documented somewhere?

[malfet]

I remember hitting those errors in the past, when adding new ECR

[ZainRizvi]

@jeanschmidt @malfet Is this still occurring?

[ZainRizvi]

Real errors:

• The ECR repository didn't acutally exist, but the error message was very misleading.
• Permission denied error still resulted in retries for 3 hours even though such a failure would never succeed on retry (tracked in Don't retry on permission denied errors for ECR #151353)

Note: This should be split into two issues

Ideal solution might be to change the ECR naming convention where you don't frequently need to be making new ECRs, and instead you set different tags on the same ECR repo to handle the version matching