Prometheus adapter not able to extract GPU metrics, getting "apiserver was unable to write a JSON response: http2: stream closed"

[Vijaygawate]

I am trying to extract custom GPU metric using prometheus adapter.
But when I am running below command, I am getting error

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq -r . | grep DCGM_FI_DEV_GPU_UTIL

Error from server (NotFound): the server could not find the metric DCGM_FI_DEV_GPU_UTIL pods

I then checked Prometheus adapter logs and found out below logs

I0830 06:59:41.308791 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/DCGM_FI_DEV_GPU_UTIL" latency="6.829796ms" userAgent="kubectl/v1.27.3 (linux/amd64) kubernetes/25b4e43" audit-ID="868c4e60-baee-4213-af4a-eab17b882e46" srcIP="10.1.108.126:53018" resp=404
E0830 06:59:43.515770 1 writers.go:122] apiserver was unable to write a JSON response: http2: stream closed
E0830 06:59:43.515798 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed

EKS version : 1.30
Prometheus adapter version: v0.12.0

Please help here
Thanks!

[dashpole]

/triage accepted
This repo doesn't have a lot of bandwidth of people who are able to investigate issues right now. Sorry if there is a slow response