fix otlp metric export when using http protocol #7595

garypen · 2025-05-30T15:31:29Z

Ensure that the path handling for an OTLP over HTTP metrics endpoint is correct.

In 1.x we didn't need to perform additional processing. I think, although it's difficult to confirm, that this was auto-magically handled by the opentelemetry-otlp crate we use which is at version 0.13. I note that we were explicitly modifying endpoints for traces, so not sure why we had to do it for traces, but not metrics.

In 2.x we updated our version of the opentelemetry-otlp crate to 0.17 and it looks as though the "magic" no longer happens.

In addition to these problems it looks like we had a number of issues using the opentelemetry_otlp crate which:

Prevented correct use of default endpoint (by always setting one)
Prevented the use of the OTEL ENDPOINT environment variables
Ignored configured protocol settings and took the default from the opentelemetry_otlp crate

I've tried to fix as many issues as I can in a broadly compatible way.

Note: I've only fixed the OTEL ENDPOINT environment variables. Others continue to work/not work as they do currently.

Checklist

Complete the checklist (and note appropriate exceptions) before the PR is marked ready-for-review.

Changes are compatible¹
Documentation² completed
Performance impact assessed and acceptable
Metrics and logs are added³ and documented
Tests added and passing⁴
- Unit Tests
- Integration Tests
- Manual Tests

Exceptions

I've added a new manual test to more rigorously test the various endpoint manipulations that we make.
I manually tested the PR against the major APM providers to ensure that it was still possible to submit O11Y data to them.

Notes

It may be appropriate to bring upcoming changes to the attention of other (impacted) groups. Please endeavour to do this before seeking PR approval. The mechanism for doing this will vary considerably, so use your judgement as to how and when to do this. ↩
Configuration is an important part of many changes. Where applicable please try to document configuration examples. ↩
A lot of (if not most) features benefit from built-in observability and debug-level logs. Please read this guidance on metrics best-practices. ↩
Tick whichever testing boxes are applicable. If you are adding Manual Tests, please document the manual testing (extensively) in the Exceptions. ↩

We use the rust opentelemetry crate to export metrics and traces. In 1.x we use v 0.13 of that crate. In 2.x we use v 0.17 of that crate. It seems as though in the earlier version, exporting otlp metrics over http would work without the correct path (/v1/metrics). This does not work in v0.17. This means that exporting metrics over http has never worked in router 2.x I've added code to explicitly force the path to be correct (as we already do for traces).

I just went through the test configuration and converted localhost/metrics to localhost/v1/metrics wherever I saw it. All the tests are passing.

svc-apollo-docs · 2025-05-30T15:32:16Z

✅ Docs preview ready

The preview is ready to be viewed. View the preview

File Changes

0 new, 19 changed, 0 removed

* graphos/routing/(latest)/cloud/subscriptions.mdx
* graphos/routing/(latest)/configuration/yaml.mdx
* graphos/routing/(latest)/observability/telemetry/metrics-exporters/prometheus.mdx
* graphos/routing/(latest)/observability/telemetry/trace-exporters/overview.mdx
* graphos/routing/(latest)/operations/subscriptions/overview.mdx
* graphos/routing/(latest)/operations/subscriptions/configuration.mdx
* graphos/routing/(latest)/operations/defer.mdx
* graphos/routing/(latest)/operations/file-upload.mdx
* graphos/routing/(latest)/performance/caching/entity.mdx
* graphos/routing/(latest)/performance/caching/in-memory.mdx
* graphos/routing/(latest)/query-planning/native-query-planner.mdx
* graphos/routing/(latest)/security/csrf.mdx
* graphos/routing/(latest)/security/demand-control.mdx
* graphos/routing/(latest)/security/authorization.mdx
* graphos/routing/(latest)/security/subgraph-authentication.mdx
* graphos/routing/(latest)/security/persisted-queries.mdx
* graphos/routing/(latest)/graphos-reporting.mdx
* graphos/routing/(latest)/uplink.mdx
* graphos/routing/(latest)/license.mdx

Build ID: 1346ae499f293fd97dc8e99a

URL: https://www.apollographql.com/docs/deploy-preview/1346ae499f293fd97dc8e99a

I'm not sure if updating the configuration is correct or if the code still needs more tweaking. This is a good first step to help me figure that out.

We need to do some work to process endpoints to accomodate decisions we made in the past. Let's see what this looks like.

So that existing tests don't need to change. I think this is simpler than before and it now works for gRPC/HTTP and metrics/traces, which it didn't do before.

Forget about checking query of url (you can't have an endpoint that contains a query...), but check the various realistic transforms that we might make.

Otherwise we get "empty string" errors...

Add the commit hash for the test changes.

We can't parse our endpoint as a URI unless we have a scheme. If none is specified, prepend "http://". Extend our test to include this very common configuration use case.

BrynCooke · 2025-06-04T14:21:06Z

@garypen Can we do a quick manual test against honeycomb and new relic?
I just want to make sure we can actually connect to these.

We'd like the standard OTLP environment variables for endpoint to work (I think), or at least our customers would, so let's try to help them out.

After more research into how New Relix processes OTLP traffic, which verified they do expect `/v1/metrics` and `/v1/traces` when using the `http` protocol, add a test.

apollo-librarian · 2025-06-09T07:16:57Z

✅ Docs preview has no changes

The preview was not built because there were no changes.

Build ID: 3bbf65816a42074898251b72

apollo-librarian · 2025-06-09T07:16:57Z

🛠️ Docs preview building...

The preview is currently being built.

Build ID: 604162c29c8252e9523f3cd4

apollo-librarian · 2025-06-09T07:16:57Z

🛠️ Docs preview building...

The preview is currently being built.

Build ID: 604162c29c8252e9523f3cd4

When I merged #7595, I squashed it and so my .gitleaks.toml updates were no longer correct. This fixes things.

garypen added 2 commits May 30, 2025 13:04

Fixing the otlp/metrics/http exporter required test changes

97a539a

I just went through the test configuration and converted localhost/metrics to localhost/v1/metrics wherever I saw it. All the tests are passing.

garypen self-assigned this May 30, 2025

garypen requested a review from a team as a code owner May 30, 2025 15:31

This comment has been minimized.

Sign in to view

garypen added 12 commits May 30, 2025 16:34

Merge branch 'dev' into garypen/fix-otlp-http-metrics

082674e

Try to remove the endpoint massaging logic and see if things work

f43f939

Make lint changes

2db12ab

Update all the test fixtures to have the correct configuration

ff2c09e

I'm not sure if updating the configuration is correct or if the code still needs more tweaking. This is a good first step to help me figure that out.

Merge branch 'dev' into garypen/fix-otlp-http-metrics

7e68002

Extract the process_endpoint() function

b52f511

We need to do some work to process endpoints to accomodate decisions we made in the past. Let's see what this looks like.

Add comment for process_endpoint

d2c570f

Figure out the correct combination of endpoint processing

a957161

So that existing tests don't need to change. I think this is simpler than before and it now works for gRPC/HTTP and metrics/traces, which it didn't do before.

Merge branch 'dev' into garypen/fix-otlp-http-metrics

973b7da

Re-instate unit test (but better) for the endpoint processing

94fa95a

Forget about checking query of url (you can't have an endpoint that contains a query...), but check the various realistic transforms that we might make.

Only try to create the tls url if we have an endpoint to process

c81c38e

Otherwise we get "empty string" errors...

Add a changeset

2d51897

garypen requested a review from a team as a code owner June 4, 2025 07:02

garypen added 3 commits June 4, 2025 08:18

Prevent false positive on gitleaks detection

38bfd73

Add the commit hash for the test changes.

Fix one more glitch. If we don't have a scheme we must prepend one

d86afeb

We can't parse our endpoint as a URI unless we have a scheme. If none is specified, prepend "http://". Extend our test to include this very common configuration use case.

Restore the no newline to the end of files to minimise PR noise

d87f4f4

Velfi approved these changes Jun 4, 2025

View reviewed changes

Try cleaning up OTLP configuration processing some more

ac911b6

We'd like the standard OTLP environment variables for endpoint to work (I think), or at least our customers would, so let's try to help them out.

garypen requested review from a team as code owners June 6, 2025 07:50

Finalise processing logic and add test for New Relic

b37a6b3

After more research into how New Relix processes OTLP traffic, which verified they do expect `/v1/metrics` and `/v1/traces` when using the `http` protocol, add a test.

garypen force-pushed the garypen/fix-otlp-http-metrics branch from dd5fde5 to b37a6b3 Compare June 6, 2025 08:02

BrynCooke approved these changes Jun 6, 2025

View reviewed changes

garypen added 2 commits June 6, 2025 16:19

Improve the changeset

49b64a4

Clarify the cause of the warning messages with http protocol

7671e15

garypen added 2 commits June 9, 2025 08:24

Merge branch 'dev' into garypen/fix-otlp-http-metrics

99d290c

Clarify documentation comments in source

2ecafde

garypen merged commit c8ae92e into dev Jun 9, 2025
14 checks passed

garypen deleted the garypen/fix-otlp-http-metrics branch June 9, 2025 07:50

garypen added a commit that referenced this pull request Jun 9, 2025

Fix the commit for gitleaks

1ce1342

When I merged #7595, I squashed it and so my .gitleaks.toml updates were no longer correct. This fixes things.

garypen mentioned this pull request Jun 9, 2025

Fix the commit for gitleaks #7644

Merged

garypen added a commit that referenced this pull request Jun 9, 2025

Fix the commit for gitleaks (#7644)

5e75e60

When I merged #7595, I squashed it and so my .gitleaks.toml updates were no longer correct. This fixes things.

This was referenced Jul 1, 2025

prep release: v2.4.0 #7789

Merged

router v2.1.0 fails to start, error on otlp metrics service not implementated #7157

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix otlp metric export when using http protocol #7595

fix otlp metric export when using http protocol #7595

Uh oh!

garypen commented May 30, 2025 •

edited

Loading

Uh oh!

This comment has been minimized.

svc-apollo-docs commented May 30, 2025 •

edited

Loading

Uh oh!

BrynCooke commented Jun 4, 2025

Uh oh!

apollo-librarian bot commented Jun 9, 2025 •

edited

Loading

Uh oh!

apollo-librarian bot commented Jun 9, 2025

Uh oh!

apollo-librarian bot commented Jun 9, 2025

Uh oh!

Uh oh!

Uh oh!

fix otlp metric export when using http protocol #7595

fix otlp metric export when using http protocol #7595

Uh oh!

Conversation

garypen commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Footnotes

Uh oh!

This comment has been minimized.

svc-apollo-docs commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Docs preview ready

Uh oh!

BrynCooke commented Jun 4, 2025

Uh oh!

apollo-librarian bot commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Docs preview has no changes

Uh oh!

apollo-librarian bot commented Jun 9, 2025

🛠️ Docs preview building...

Uh oh!

apollo-librarian bot commented Jun 9, 2025

🛠️ Docs preview building...

Uh oh!

Uh oh!

Uh oh!

garypen commented May 30, 2025 •

edited

Loading

svc-apollo-docs commented May 30, 2025 •

edited

Loading

apollo-librarian bot commented Jun 9, 2025 •

edited

Loading