awsemf - CloudWatch logs unsupported protocol scheme error

[lzac]

Component(s)

exporter/awsemf

What happened?

Description

We've been using the awsemf exporter with no major issues since a long time but it looks like it's no longer working as expected starting from v0.125.0.
My guess is that something broke because of the upgrade aws-sdk-go-v2 with #39314 as we're now getting the following error with no configuration changes on our side.

scraper-collector-opentelemetry-collector-0 opentelemetry-collector 2025-05-27T07:00:48.102Z warn batchprocessor@v0.126.0/batch_processor.go:264 Sender failed {"resource": {}, "otelcol.component.id": "batch/cloudwatch", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "metrics/awsemfkubestate", "otelcol.signal": "metrics", "error": "operation error CloudWatch Logs: PutLogEvents, exceeded maximum number of attempts, 2, https response error StatusCode: 0, RequestID: , request send failed, Post \"/\": unsupported protocol scheme \"\""}

Steps to Reproduce

We've upgraded our helm chart from 0.123.0 to 0.124.0, which actually upgrade the collector to 0.125.0, and started getting the error.

Expected Result

Logs should be properly sent to CloudWatch.

Actual Result

CloudWatch does not receive any log.

Collector version

v0.125.0

Environment information

No response

OpenTelemetry Collector configuration

config:
  "service":
    "pipelines":
      "metrics/awsemfkubestate":
        "receivers":
        - "prometheus"
        "processors":
        - "filter/awsemfkubestate"
        - "batch"
        "exporters":
        - "awsemf"
  "processors":
    "batch": {}
    "filter/awsemfkubestate":
      "metrics":
        "include":
          "match_type": "strict"
          "metric_names":
          - "kube_deployment_spec_replicas"
          - "kube_deployment_status_replicas_available"
  "exporters":
    "awsemf":
      "region": "${REGION}"
      "output_destination": "cloudwatch"
      "dimension_rollup_option": "NoDimensionRollup"
      "resource_to_telemetry_conversion":
          "enabled": true
      "namespace": "${EMF_NAMESPACE}"
      "log_group_name": "${LOG_GROUP_NAME}"
      "log_retention": 7
      "log_stream_name": "${LOG_STREAM_NAME}"
      "metric_declarations":
        - "dimensions": [[namespace, deployment]]
          "metric_name_selectors":
            - "kube_deployment_spec_replicas"
            - "kube_deployment_status_replicas_available"

Log output

scraper-collector-opentelemetry-collector-0 opentelemetry-collector 2025-05-27T07:00:48.102Z	warn	batchprocessor@v0.126.0/batch_processor.go:264	Sender failed	{"resource": {}, "otelcol.component.id": "batch/cloudwatch", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "metrics/awsemfkubestate", "otelcol.signal": "metrics", "error": "operation error CloudWatch Logs: PutLogEvents, exceeded maximum number of attempts, 2, https response error StatusCode: 0, RequestID: , request send failed, Post \"/\": unsupported protocol scheme \"\""}

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/awsemf: @Aneurysm9 @mxiamxia

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[avaitla]

We are hitting this issue as well, I was a bit surprised at this error at first, but reverting to v0.123.0 resolves it. Something is broken in the awsemf exporter. It seems the amazon otel collector works however.

[al-alison]

We got the same error when upgrading to the latest version, in awsemfexporter, awscloudwatchexporter and even in the awsxrayexporter
The workaround we found was adding the "endpoint" property in the exporters configuration, with the protocol included, like this:

awsemf:
    region: 'us-east-1'
    endpoint: https://logs.us-east-1.amazonaws.com
    log_group_name: '/otel/metrics/{ClusterName}'
    log_stream_name: '{TaskId}'
    log_retention: 7
    resource_to_telemetry_conversion:
        enabled: true

  awscloudwatchlogs:
    region: 'us-east-1'
    endpoint: https://logs.us-east-1.amazonaws.com
    log_group_name: '/otel/logs/{ClusterName}'
    log_stream_name: '{TaskId}'
    log_retention: 7

  awsxray:
    region: 'us-east-1'
    endpoint: https://xray.us-east-1.amazonaws.com
    telemetry:
      enabled: true

[mxiamxia]

Will take a look when get the chance, aiming for the end of the week.

[mjnowen]

+1 I am hitting this issue in v0.127.0 as well. The temp workaround of declaring both "region" and "endpoint" seems to fix it. Looks like calls to IMDS are currently broken, as IMDS should be able to retrieve the current "region" and thus, form the correct "endpoint/protocol" as well to use (and as a result, you shouldn't have to declare either "region" or "endpoint" in your config).

[lzac]

We got the same error when upgrading to the latest version, in awsemfexporter, awscloudwatchexporter and even in the awsxrayexporter The workaround we found was adding the "endpoint" property in the exporters configuration, with the protocol included, like this:

awsemf:
    region: 'us-east-1'
    endpoint: https://logs.us-east-1.amazonaws.com
    log_group_name: '/otel/metrics/{ClusterName}'
    log_stream_name: '{TaskId}'
    log_retention: 7
    resource_to_telemetry_conversion:
        enabled: true

  awscloudwatchlogs:
    region: 'us-east-1'
    endpoint: https://logs.us-east-1.amazonaws.com
    log_group_name: '/otel/logs/{ClusterName}'
    log_stream_name: '{TaskId}'
    log_retention: 7

  awsxray:
    region: 'us-east-1'
    endpoint: https://xray.us-east-1.amazonaws.com
    telemetry:
      enabled: true

Thanks @al-alison , what you suggested has fixed the issue.

[mjlshen]

Hi everyone, author of the original v1->v2 PR here. Thanks for the detailed report and workaround descriptions - sorry, I just noticed this issue!

tl;dr I'm fairly certain that this issue should be fixed on v0.128.0 - I will try to validate myself in the next few days unless someone beats me to it and I want to apologize in general for causing this!

Here's what I understand happened:

• In

  opentelemetry-collector-contrib/internal/aws/awsutil/conn.go

  Lines 166 to 194 in f4c0600

  	// GetAWSConfig returns AWS config and session instances.
  	func GetAWSConfig(logger *zap.Logger, settings *AWSSessionSettings) (awsv2.Config, error) {
  	http, err := newHTTPClient(logger, settings.NumberOfWorkers, settings.RequestTimeoutSeconds, settings.NoVerifySSL, settings.ProxyAddress)
  	if err != nil {
  	logger.Error("unable to obtain proxy URL", zap.Error(err))
  	return awsv2.Config{}, err
  	}
  	cfg, err := config.LoadDefaultConfig(context.Background())
  	if err != nil {
  	return awsv2.Config{}, err
  	}
  	if settings.Region != "" {
  	cfg.Region = settings.Region
  	logger.Debug("Fetch region from commandline/config file", zap.String("region", settings.Region))
  	}
  	if cfg.Region == "" {
  	logger.Error("cannot fetch region variable from config file, environment variables and ec2 metadata")
  	return awsv2.Config{}, errors.New("cannot fetch region variable from config file, environment variables and ec2 metadata")
  	}
  	cfg.HTTPClient = http
  	cfg.RetryMaxAttempts = settings.MaxRetries
  	cfg.BaseEndpoint = aws.String(settings.Endpoint)
  	return cfg, nil
  	}

  I added GetAWSConfig() which unfortunately could not detect AWS region from IMDS. (Compare to the previous logic in GetAWSConfigSession()

  opentelemetry-collector-contrib/internal/aws/awsutil/conn.go

  Lines 136 to 142 in f4c0600

  	} else {
  	awsRegion, err = cn.getEC2Region(es)
  	if err != nil {
  	logger.Error("Unable to retrieve the region from the EC2 instance", zap.Error(err))
  	} else {
  	logger.Debug("Fetch region from ec2 metadata", zap.String("region", awsRegion))
  	}

  )
• In [exporter/awscloudwatchlogsexporter] update to aws-sdk-go-v2 #39314, which was linked in the original issue description, I went to migrate awscloudwatchexporter and later others to leverage GetAWSConfig(), so when these exporters were running on EC2 instances, the (totally fair!) expectation was that the region would be detected from IMDS was no longer working.
• We noticed this when we finally went to remove GetAWSConfigSession() [internal/aws] migrate to AWS v2 SDK #40123 (comment), but didn't realize the impact it already had! This commit is included in v0.128.0.