image-promo job is hitting quota limits

[schrej]

The image-promo prow job is failing a lot more frequently than usual (6x in 5h): https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/post-k8sio-image-promo

It looks like it's hitting quota limits on various regional GCP registries. Maybe the total amount of images is too high for the current limits.

two examples:

11347b163438dc07aabfb3ddb651c785b4830f379c1ca6b57bd4939e7e29c895.sig: fetching \"asia-south1-docker.pkg.dev/k8s-artifacts-prod/images/kube-proxy-ppc64le:sha256-11347b163438dc07aabfb3ddb651c785b4830f379c1ca6b57bd4939e7e29c895.sig\": GET https://asia-south1-docker.pkg.dev/v2/k8s-artifacts-prod/images/kube-proxy-ppc64le/manifests/sha256-11347b163438dc07aabfb3ddb651c785b4830f379c1ca6b57bd4939e7e29c895.sig: TOOMANYREQUESTS: Quota exceeded for quota metric 'Requests per project per user' and limit 'Requests per project per user per minute per user' of service 'artifactregistry.googleapis.com' for consumer 'project_number:388270116193'." diff=12ms

time="15:40:40.771" level=fatal msg="run `cip run`: promote images: signing images: replicating signatures: copying signature europe-west9-docker.pkg.dev/k8s-artifacts-prod/images/capi-ipam-ic/cluster-api-ipam-in-cluster-controller:sha256-2fa62384935b0233f68acf75fcb12bbe149b7f122e83d4e5f67f157e73998732.sig to australia-southeast1-docker.pkg.dev/k8s-artifacts-prod/images/capi-ipam-ic/cluster-api-ipam-in-cluster-controller:sha256-2fa62384935b0233f68acf75fcb12bbe149b7f122e83d4e5f67f157e73998732.sig: HEAD https://australia-southeast1-docker.pkg.dev/v2/k8s-artifacts-prod/images/capi-ipam-ic/cluster-api-ipam-in-cluster-controller/manifests/sha256-2fa62384935b0233f68acf75fcb12bbe149b7f122e83d4e5f67f157e73998732.sig: unexpected status code 429 Too Many Requests (HEAD responses have no body, use GET for details)" diff=958ms

[schrej]

/area registry.k8s.io
/area infra/gcp

[BenTheElder]

The image promoter makes a really high amount of API calls because of the approach to image signatures.

We have not changed the quotas in the infrastructure projects.

/transfer kubernetes-sigs/promo-tools
/sig release

[k8s-ci-robot]

@BenTheElder: Something went wrong or the destination repo kubernetes/kubernetes-sigs/promo-tools does not exist.

In response to this:

The image promoter makes a really high amount of API calls because of the approach to image signatures.

We have not changed the quotas in the infrastructure projects.

/transfer kubernetes-sigs/promo-tools
/sig release

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[BenTheElder]

The image promoter is at https://github.com/kubernetes-sigs/promo-tools

[chrischdi]

Happened today too for a CAPV release:

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-k8sio-image-promo/1776261613632884736

[BenTheElder]

This is not getting visibility to the right people, please file a bug with kubernetes-sigs/promo-tools or kubernetes/sig-release.

K8s-Infra is not changing the limits on the backing registries, they are roughly equivalent to the previous non-configurable GCR limits and they're necessary to prevent trivial DOS, even the project itself must not be permitted to self-DOS, and there is no way to exempt a single user.

[chrischdi]

Opened the issue at promo-tools. Maybe I can find some time to take a look where it could get optimised. Thanks for pointing to the right direction @BenTheElder ! Really appreciating it 🚀

[ameukam]

/close

Please check:

• image-promotion hits 429 quota limits kubernetes-sigs/promo-tools#1271

[k8s-ci-robot]

@ameukam: Closing this issue.

In response to this:

/close

Please check:

• image-promotion hits 429 quota limits kubernetes-sigs/promo-tools#1271

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.