SQS aws_sqs_approximate_number_of_messages_visible_average - Need your help!

[DPrakhin]

Good day, Guys!

I want to ask for your assistance with configuring of Prometheus adapter for collecting required metrics from AWS CloudWatch for SQS.

Currently, from my side I have the next:

1. Prometheus server is up and running (http://prometheus-server);
2. CloudWatch exporter is up & running and confiugerd;
3. Prometheus adapter is up & running.

What do I have:

1. CloudWatch exporter is configured to collect only three metrics:
   aws_namespace: AWS/EC2
   aws_metric_name: CPUUtilization
   aws_statistics: Average

   aws_namespace: AWS/SQS
   aws_metric_name: ApproximateNumberOfMessagesVisible
   aws_statistics: Average

   aws_namespace: AWS/SQS
   aws_metric_name: ApproximateNumberOfMessagesVisible
   aws_statistics: Average

2. I can view all these metrics in Prometheus server.

3. I can get data and values about these metrics from Prometheus server via it's API:
   $ curl http://prometheus-server/api/v1/label/__name__/values
   name = aws_ec2_cpuutilization_average
   name = aws_sqs_approximate_number_of_messages_visible_average

$ curl -g 'http://prometheus-server/api/v1/series?' --data-urlencode 'match[]=aws_ec2_cpuutilization_average' \
--data-urlencode 'match[]=aws_sqs_approximate_number_of_messages_visible_average'

{"status":"success",
"data":[
{"name":"aws_ec2_cpuutilization_average",
"exported_job":"aws_ec2",
"instance":"cloudwatch-exporter-prometheus-cloudwatch-exporter:9106",
"instance_id":"i-075ddced357e55d4a",
"job":"cloudwatch"},

{"__name__":"aws_sqs_approximate_number_of_messages_visible_average",
 "exported_job":"aws_sqs",
 "instance":"cloudwatch-exporter-prometheus-cloudwatch-exporter:9106",
 "job":"cloudwatch",
 "queue_name":"cosmos-dev-delivery-highPrio.fifo"},

{"__name__":"aws_sqs_approximate_number_of_messages_visible_average",
 "exported_job":"aws_sqs",
 "instance":"cloudwatch-exporter-prometheus-cloudwatch-exporter:9106",
 "job":"cloudwatch",
 "queue_name":"cosmos-dev-delivery.fifo"}]}

$ curl 'http://prometheus-server/api/v1/query_range?query=aws_ec2_cpuutilization_average&start=2024-09-25T00:01:00Z&&end=2024-09-25T00:02:00Z&step=15s'
{"status":"success","data":
{"resultType":
"matrix","result":
[{"metric": {
"name":"aws_ec2_cpuutilization_average",
"exported_job":"aws_ec2",
"instance":"cloudwatch-exporter-prometheus-cloudwatch-exporter:9106",
"instance_id":"i-075ddced357e55d4a",
"job":"cloudwatch"},
"values":[[1727222460,"1.5928961748595722"],
[1727222475,"1.5928961748595722"],
[1727222490,"1.5928961748595722"],
[1727222505,"1.5928961748595722"],
[1727222520,"1.374314384894086"]]}]}}

Currently, for Prometheus adapter I have next configuration (just for one metric):
externalRules:
- seriesQuery: 'name="aws_sqs_approximate_number_of_messages_visible_average"'
name:
matches: ""
as: "aws_sqs_dev"
resources:
namespaced: false
metricsQuery: sum(rate(aws_sqs_approximate_number_of_messages_visible_average{"exported_job":"aws_sqs","instance":"cloudwatch-exporter-prometheus-cloudwatch-exporter:9106","job":"cloudwatch","queue_name":"cosmos-dev-delivery-highPrio.fifo"}))

- metricsQuery: sum(aws_ec2_cpuutilization_average{"exported_job":"aws_ec2","instance":"cloudwatch-exporter-prometheus-cloudwatch-exporter:9106","instance_id":"i-075ddced357e55d4a","job":"cloudwatch"})
  name:
    as: aws_ec2_metric
    matches: ""
  resources:
    namespaced: false
  seriesQuery: '__name__="aws_ec2_cpuutilization_average"'

- metricsQuery: sum(aws_sqs_approximate_number_of_messages_visible_average{job="cloudwatch"}[15s]) by (queue_name)
  name:
    as: aws_sqs_metric
    matches: ""
  resources:
    namespaced: false
  seriesQuery: '__name__="aws_sqs_approximate_number_of_messages_visible_average"'

Unfortunately, I don't have any results for it:
$ kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 | jq
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "external.metrics.k8s.io/v1beta1",
"resources": []
}

Guys, could you, please, help me to create appropriate configuration for Prometheus adapter to get this metrics for future use with HPA (SQS)? You are my last chance to get it done....

Thank you in advance!
BR,
Daniil

[DPrakhin]

Good day, Guys!

Just a small update regarding this issue.
Currently, I'm using the next external metrics rule for Prometheus-adapter:
externalRules:
- metricsQuery: sum(rate(aws_sqs_approximate_number_of_messages_visible_average{queue_name="cosmos-dev-delivery.fifo"}))
name:
as: aws_sqs_approximate_number_of_messages_visible_average
matches: ""
resources:
template: <<.Resource>>
seriesQuery: '{name=~"aws_sqs_approximate_number_of_messages_visible_average",queue_name="cosmos-dev-delivery.fifo"}'

Using this rule Prometheus-adapter sends next request to the Prometheus-server:
I0926 13:32:43.773115 1 api.go:88] GET http://prometheus-server:80/api/v1/series?match%5B%5D=%7B__name__%3D~%22aws_sqs_approximate_number_of_messages_visible_average%22%2Cqueue_name%3D%22cosmos-dev-delivery.fifo%22%7D&start=1727357503.771 200 OK

I've made a test and sent the same request to the Prometheus-server. It's the response from the server:
{"status":"success",
"data":[{
"name":"aws_sqs_approximate_number_of_messages_visible_average",
"exported_job":"aws_sqs",
"instance":"cloudwatch-exporter-prometheus-cloudwatch-exporter:9106",
"job":"cloudwatch",
"queue_name":"cosmos-dev-delivery.fifo"
}]
}

Hope this information will help!
BR,
Daniil

[dgrisonnet]

/triage accepted

We sadly don't have enough resources to look after support issues for prometheus-adapter right now.

[dgrisonnet]

/triage accepted

[james-callahan]

We (@samuelebstein) had a similar issue; it turned out to be the max-age (

prometheus-adapter/cmd/adapter/adapter.go

Lines 327 to 330 in c2ae4cd

	// if --metrics-max-age is not set, make it equal to --metrics-relist-interval
	if cmd.MetricsMaxAge == 0*time.Second {
	cmd.MetricsMaxAge = cmd.MetricsRelistInterval
	}

)

Once we increased the max age on the prometheus-adapter command line, the SQS metric was available for use.