Token is not renewing after few times when we use oauth2clientauthextension

[saikalyanbhagavathula]

Component(s)

extension/oauth2clientauth

What happened?

Description

When we use oauth2clientauthextension token is not getting regenerated after couple of times.
The request is still sending with old token, even that is expired. How is new token is fetched by opentelemetry collector?

Steps to Reproduce

Expected Result

Token to renew after every time current token usage expires.

Actual Result

Token is not renewing after few times.

Collector version

v0.122.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")

OpenTelemetry Collector configuration

extensions:
  health_check: {}
  zpages: {}
  oauth2client:
    client_id: "<environment_clientId"
    client_secret: "<environment_client_secret>"
    token_url: "<Endpoint_to_get_token>"
    expiry_buffer: 10s

Log output

2025-04-07T04:31:51.570Z	error	internal/queue_sender.go:46	Exporting failed. Dropping data.	{"otelcol.component.id": "otlp/<exporter_name>", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics", "error": "not retryable error: Permanent error: rpc error: code = Unauthenticated desc = Error response message from server", "dropped_items": 1867}

Additional context

No response

[github-actions]

Pinging code owners:

• extension/oauth2clientauth: @pavankrish123 @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[saikalyanbhagavathula]

@pavankrish123 @jpkrohling
I wanted to follow up regarding the issue with the oauth2clientauthextension. We are observing that the token is not being regenerated after a few uses. Instead, the requests continue to use the expired token, leading to repeated authentication errors.

Could you please clarify how the OpenTelemetry Collector fetches a new token in this scenario? Is there any specific mechanism or configuration that governs token renewal in the collector?

Let me know if further details are needed from our side.

[mmarin-ent]

We observed similar behaviour in our setup. Out of 30+ otel Collector instances, a subset of 2-5 behave this way.
We have a setup with 3 exporters (2 with basic auth, 1 with OAuth2).
Exporting to Oauth2 authenticated destination works well for few hours, but after some time it begins to fail to renew the token. The increment in otelcol_exporter_send_failed_metric_points_total metric is visible only for OAuth2 authenticated destination.

[saikalyan-bhagavathula]

@pavankrish123 @jpkrohling
I wanted to follow up regarding the issue with the oauth2clientauthextension. We are observing that the token is not being regenerated after a few uses. Instead, the requests continue to use the expired token, leading to repeated authentication errors.

Could you please clarify how the OpenTelemetry Collector fetches a new token in this scenario? Is there any specific mechanism or configuration that governs token renewal in the collector?

Let me know if further details are needed from our side.

[saikalyan-bhagavathula]

We observed similar behaviour in our setup. Out of 30+ otel Collector instances, a subset of 2-5 behave this way. We have a setup with 3 exporters (2 with basic auth, 1 with OAuth2). Exporting to Oauth2 authenticated destination works well for few hours, but after some time it begins to fail to renew the token. The increment in otelcol_exporter_send_failed_metric_points_total metric is visible only for OAuth2 authenticated destination.

@mmarin-ent You still facing an issue or got any solution?

[mmarin-ent]

@mmarin-ent You still facing an issue or got any solution?

We identified an internal configuration issue on our end. The expires_in returned by our identity API was in ms and the OAuth 2.0 implemented by this extension is in seconds.