Help to config otel colletor to send traces to different newrelic account based on attibute value

[sambestsolutions]

Component(s)

No response

Describe the issue you're reporting

HI

I want to configure the OpenTelemetry (OTel) Collector to send traces to different New Relic accounts based on the value of the tenantId attribute.

For example, if tenantId="t0001", the trace should be sent to one New Relic account, and if tenantId="t0002", it should be sent to another New Relic account. If neither of these values is present, the trace should be sent to a default third New Relic account.

I tried to achieve this using the routing processor, but all traces ended up in every account.

Can someone provide the correct configuration and specify which OTel Collector version I should use for this?

Below is the config I used for this purpose

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
        cors:
          allowed_origins:
            - http://*:5080


processors:
  batch: {}
  routing:
    from_attribute: tenantId
    default_exporters: [otlphttp/newrelic3]
    table:
    - value: t0001
      exporters: [otlphttp]
    - value: t0002
      exporters: [otlphttp/newrelic2]


exporters:
  debug:
    verbosity: detailed
    sampling_initial: 5
    sampling_thereafter: 200

  otlphttp:
    endpoint: https://otlp.nr-data.net:4317
    headers:
      api-key: 

  otlphttp/newrelic2:
    endpoint: https://otlp.nr-data.net:4317
    headers:
      api-key: 
  
  otlphttp/newrelic3:
    endpoint: https://otlp.nr-data.net:4317
    headers:
      api-key: 

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, otlphttp, otlphttp/newrelic2, otlphttp/newrelic3]

[crobert-1]

Hello @sambestsolutions, thanks for filing this issue! Your best bet here is probably to use the routing connector, as the routing processor has been deprecated in favor of the connector.

[github-actions]

Pinging code owners for connector/routingconnector: @jpkrohling @mwear. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[sambestsolutions]

Hello @sambestsolutions, thanks for filing this issue! Your best bet here is probably to use the routing connector, as the routing processor has been deprecated in favor of the connector.

Hi @crobert-1 Thank you for you response and as suggested I used routing connectors and now this is sending all the traces to all the environments i.e this is brodcasting to all configured instances, I have attached the config I used for this, Is there anything I have missed here

Note- I am using otel/opentelemetry-collector-contrib:0.122.0 for this

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
        cors:
          allowed_origins:
            - http://*:5080


connectors:
  routing:
    table:
      - context: resource
        condition: attributes["tenantId"] == "t0001"
        pipelines: [traces/dev]
      - context: resource
        condition: attributes["tenantId"] == "t0002"
        pipelines: [traces/jumjum]


exporters:
  otlphttp/dev:
    endpoint: https://otlp.nr-data.net:4317
    headers:
      api-key: 

  otlphttp/jumjum:
    endpoint: https://otlp.nr-data.net:4317
    headers:
      api-key: 
  
  otlphttp/stage:
    endpoint: https://otlp.nr-data.net:4317
    headers:
      api-key: 


service:
  pipelines:
    traces/dev:
      receivers: [otlp]
      exporters: [otlphttp/dev]
    traces/jumjum:
      receivers: [otlp]
      exporters: [otlphttp/jumjum]
    traces/stage:
      receivers: [otlp]
      exporters: [otlphttp/stage]

[crobert-1]

Just a quick glance, the trace pipelines need to include the routing connector. Right now the receivers are sending data directly to the exporter without going through the connector.

[sambestsolutions]

Just a quick glance, the trace pipelines need to include the routing connector. Right now the receivers are sending data directly to the exporter without going through the connector.

Hi Thanks for the reply, I modified the config as below and now no traces are getting uploaded, Please note that I am very new to this and trying to get it to work, Help is really appreciated.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
        cors:
          allowed_origins:
            - http://*:5080


connectors:
  routing:
    table:
      - context: resource
        condition: attributes["tenantId"] == "t0001"
        pipelines: [traces/dev]
      - context: resource
        condition: attributes["tenantId"] == "t0002"
        pipelines: [traces/jumjum]


exporters:
  otlphttp/dev:
    endpoint: https://otlp.nr-data.net:4317
    headers:
      api-key: 

  otlphttp/jumjum:
    endpoint: https://otlp.nr-data.net:4317
    headers:
      api-key: 
  
  otlphttp/stage:
    endpoint: https://otlp.nr-data.net:4317
    headers:
      api-key: 


service:
  pipelines:
    traces/in:
      receivers: [otlp]
      exporters: [routing]
    traces/dev:
      receivers: [routing]
      exporters: [otlphttp/dev]
    traces/jumjum:
      receivers: [routing]
      exporters: [otlphttp/jumjum]
    traces/stage:
      receivers: [routing]
      exporters: [otlphttp/stage]

[crobert-1]

No worries at all @sambestsolutions! You've said you were getting data in your backend before adding the connectors to your pipeline, so we can probably rule out your receiver and exporter configuration as being the problem. This must mean that the filters you've configured in the connector table aren't matching your data.

My usual debugging step here is to add the debug exporter to your traces/in pipeline with verbosity: detailed, to see what the data looks like. This will help you manually confirm if the resource attribute values are what you expect or not at the time the connector filter is applied.

One thing to look for is whether the attribute you're routing based on is a resource attribute, or a trace attribute. The context needs to be set properly for which type of attribute you're interested in, or the match will fail, and no traces will be sent.

[sambestsolutions]

No worries at all @sambestsolutions! You've said you were getting data in your backend before adding the connectors to your pipeline, so we can probably rule out your receiver and exporter configuration as being the problem. This must mean that the filters you've configured in the connector table aren't matching your data.

My usual debugging step here is to add the debug exporter to your traces/in pipeline with verbosity: detailed, to see what the data looks like. This will help you manually confirm if the resource attribute values are what you expect or not at the time the connector filter is applied.

One thing to look for is whether the attribute you're routing based on is a resource attribute, or a trace attribute. The context needs to be set properly for which type of attribute you're interested in, or the match will fail, and no traces will be sent.

Traces are there in all three environements. And Seems no filterration is applied.

I added debug exporter as you suggested and below is the new configuration and log output after enabling debug logs

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
        cors:
          allowed_origins:
            - http://*:5080


connectors:
  routing:
    table:
      - context: resource
        condition: attributes["tenantId"] == "t0001"
        pipelines: [traces/dev]
      - context: resource
        condition: attributes["tenantId"] == "t0002"
        pipelines: [traces/jumjum]


exporters:
  otlphttp/dev:
    endpoint: https://otlp.nr-data.net:4317
    headers:
      api-key: 

  otlphttp/jumjum:
    endpoint: https://otlp.nr-data.net:4317
    headers:
      api-key: 
  
  otlphttp/stage:
    endpoint: https://otlp.nr-data.net:4317
    headers:
      api-key: 

  debug:
    verbosity: detailed
    sampling_initial: 5
    sampling_thereafter: 200


service:
  pipelines:
    traces/in:
      receivers: [otlp]
      exporters: [debug, routing]
    traces/dev:
      receivers: [routing]
      exporters: [otlphttp/dev]
    traces/jumjum:
      receivers: [routing]
      exporters: [otlphttp/jumjum]
    traces/stage:
      receivers: [routing]
      exporters: [otlphttp/stage]

Logoutput for sample traces

otel-collector-c4f9dffc4-bm2g5 2025-03-20T20:25:13.516Z    info    service@v0.122.0/service.go:193    Setting up own telemetry...
otel-collector-c4f9dffc4-bm2g5 2025-03-20T20:25:13.516Z    info    builders/builders.go:26    Development component. May change in the future.    {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces"}
otel-collector-c4f9dffc4-bm2g5 2025-03-20T20:25:13.517Z    info    service@v0.122.0/service.go:260    Starting otelcol-contrib...    {"Version": "0.122.0", "NumCPU": 16}
otel-collector-c4f9dffc4-bm2g5 2025-03-20T20:25:13.518Z    info    extensions/extensions.go:40    Starting extensions...
otel-collector-c4f9dffc4-bm2g5 2025-03-20T20:25:13.518Z    info    otlpreceiver@v0.122.0/otlp.go:116    Starting GRPC server    {"otelcol.component.id": "otlp", "otelcol.component.kind": "Receiver", "endpoint": "0.0.0.0:4317"}
otel-collector-c4f9dffc4-bm2g5 2025-03-20T20:25:13.518Z    info    otlpreceiver@v0.122.0/otlp.go:173    Starting HTTP server    {"otelcol.component.id": "otlp", "otelcol.component.kind": "Receiver", "endpoint": "0.0.0.0:4318"}
otel-collector-c4f9dffc4-bm2g5 2025-03-20T20:25:13.518Z    info    service@v0.122.0/service.go:283    Everything is ready. Begin running and processing data.
otel-collector-c4f9dffc4-7rmjn 2025-03-20T20:24:54.146Z    info    service@v0.122.0/service.go:193    Setting up own telemetry...
otel-collector-c4f9dffc4-7rmjn 2025-03-20T20:24:54.147Z    info    builders/builders.go:26    Development component. May change in the future.    {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces"}
otel-collector-c4f9dffc4-7rmjn 2025-03-20T20:24:54.148Z    info    service@v0.122.0/service.go:260    Starting otelcol-contrib...    {"Version": "0.122.0", "NumCPU": 16}
otel-collector-c4f9dffc4-7rmjn 2025-03-20T20:24:54.148Z    info    extensions/extensions.go:40    Starting extensions...
otel-collector-c4f9dffc4-7rmjn 2025-03-20T20:24:54.148Z    info    otlpreceiver@v0.122.0/otlp.go:116    Starting GRPC server    {"otelcol.component.id": "otlp", "otelcol.component.kind": "Receiver", "endpoint": "0.0.0.0:4317"}
otel-collector-c4f9dffc4-7rmjn 2025-03-20T20:24:54.148Z    info    otlpreceiver@v0.122.0/otlp.go:173    Starting HTTP server    {"otelcol.component.id": "otlp", "otelcol.component.kind": "Receiver", "endpoint": "0.0.0.0:4318"}
otel-collector-c4f9dffc4-7rmjn 2025-03-20T20:24:54.148Z    info    service@v0.122.0/service.go:283    Everything is ready. Begin running and processing data.
otel-collector-c4f9dffc4-bm2g5 2025-03-20T20:25:50.758Z    info    Traces    {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 1}
otel-collector-c4f9dffc4-bm2g5 2025-03-20T20:25:50.758Z    info    ResourceSpans #0
otel-collector-c4f9dffc4-bm2g5 Resource SchemaURL: 
otel-collector-c4f9dffc4-bm2g5 Resource attributes:
otel-collector-c4f9dffc4-bm2g5      -> telemetry.sdk.language: Str(python)
otel-collector-c4f9dffc4-bm2g5      -> telemetry.sdk.name: Str(opentelemetry)
otel-collector-c4f9dffc4-bm2g5      -> telemetry.sdk.version: Str(1.30.0)
otel-collector-c4f9dffc4-bm2g5      -> service.name: Str(test_tenant_33)
otel-collector-c4f9dffc4-bm2g5      -> service.version: Str(1.0.0)
otel-collector-c4f9dffc4-bm2g5 ScopeSpans #0
otel-collector-c4f9dffc4-bm2g5 ScopeSpans SchemaURL: 
otel-collector-c4f9dffc4-bm2g5 InstrumentationScope __main__ 
otel-collector-c4f9dffc4-bm2g5 Span #0
otel-collector-c4f9dffc4-bm2g5     Trace ID       : 273945bb70834ef9a71312875cbd9a75
otel-collector-c4f9dffc4-bm2g5     Parent ID      : 
otel-collector-c4f9dffc4-bm2g5     ID             : b8e984f1945cc223
otel-collector-c4f9dffc4-bm2g5     Name           : tenant_t0002_operation
otel-collector-c4f9dffc4-bm2g5     Kind           : Internal
otel-collector-c4f9dffc4-bm2g5     Start time     : 2025-03-20 20:25:49.742086479 +0000 UTC
otel-collector-c4f9dffc4-bm2g5     End time       : 2025-03-20 20:25:49.74222083 +0000 UTC
otel-collector-c4f9dffc4-bm2g5     Status code    : Unset
otel-collector-c4f9dffc4-bm2g5     Status message : 
otel-collector-c4f9dffc4-bm2g5 Attributes:
otel-collector-c4f9dffc4-bm2g5      -> tenantId: Str(t0002)
otel-collector-c4f9dffc4-bm2g5     {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces"}

[crobert-1]

From the debug exporter the tenantId is an attribute on the span, not a resource attribute.

Resource attributes:

otel-collector-c4f9dffc4-bm2g5 Resource attributes:
otel-collector-c4f9dffc4-bm2g5      -> telemetry.sdk.language: Str(python)
otel-collector-c4f9dffc4-bm2g5      -> telemetry.sdk.name: Str(opentelemetry)
otel-collector-c4f9dffc4-bm2g5      -> telemetry.sdk.version: Str(1.30.0)
otel-collector-c4f9dffc4-bm2g5      -> service.name: Str(test_tenant_33)
otel-collector-c4f9dffc4-bm2g5      -> service.version: Str(1.0.0)

Span:

otel-collector-c4f9dffc4-bm2g5 Span #0
otel-collector-c4f9dffc4-bm2g5     Trace ID       : 273945bb70834ef9a71312875cbd9a75
otel-collector-c4f9dffc4-bm2g5     Parent ID      : 
otel-collector-c4f9dffc4-bm2g5     ID             : b8e984f1945cc223
otel-collector-c4f9dffc4-bm2g5     Name           : tenant_t0002_operation
otel-collector-c4f9dffc4-bm2g5     Kind           : Internal
otel-collector-c4f9dffc4-bm2g5     Start time     : 2025-03-20 20:25:49.742086479 +0000 UTC
otel-collector-c4f9dffc4-bm2g5     End time       : 2025-03-20 20:25:49.74222083 +0000 UTC
otel-collector-c4f9dffc4-bm2g5     Status code    : Unset
otel-collector-c4f9dffc4-bm2g5     Status message : 
otel-collector-c4f9dffc4-bm2g5 Attributes:
otel-collector-c4f9dffc4-bm2g5      -> tenantId: Str(t0002)

From your configuration, you're looking at the context: resource, which means the connector isn't seeing the desired attribute. Can you try changing context to be span and see if it works?

The connector README shows different valid values for context, which links to OTTL context. From there you can see the valid contexts, and why the span context may be the right one here. (More information on the span context to show how to access its attributes.)

[sambestsolutions]

From the debug exporter the tenantId is an attribute on the span, not a resource attribute.

Resource attributes:

otel-collector-c4f9dffc4-bm2g5 Resource attributes:
otel-collector-c4f9dffc4-bm2g5      -> telemetry.sdk.language: Str(python)
otel-collector-c4f9dffc4-bm2g5      -> telemetry.sdk.name: Str(opentelemetry)
otel-collector-c4f9dffc4-bm2g5      -> telemetry.sdk.version: Str(1.30.0)
otel-collector-c4f9dffc4-bm2g5      -> service.name: Str(test_tenant_33)
otel-collector-c4f9dffc4-bm2g5      -> service.version: Str(1.0.0)

Span:

otel-collector-c4f9dffc4-bm2g5 Span #0
otel-collector-c4f9dffc4-bm2g5     Trace ID       : 273945bb70834ef9a71312875cbd9a75
otel-collector-c4f9dffc4-bm2g5     Parent ID      : 
otel-collector-c4f9dffc4-bm2g5     ID             : b8e984f1945cc223
otel-collector-c4f9dffc4-bm2g5     Name           : tenant_t0002_operation
otel-collector-c4f9dffc4-bm2g5     Kind           : Internal
otel-collector-c4f9dffc4-bm2g5     Start time     : 2025-03-20 20:25:49.742086479 +0000 UTC
otel-collector-c4f9dffc4-bm2g5     End time       : 2025-03-20 20:25:49.74222083 +0000 UTC
otel-collector-c4f9dffc4-bm2g5     Status code    : Unset
otel-collector-c4f9dffc4-bm2g5     Status message : 
otel-collector-c4f9dffc4-bm2g5 Attributes:
otel-collector-c4f9dffc4-bm2g5      -> tenantId: Str(t0002)

From your configuration, you're looking at the context: resource, which means the connector isn't seeing the desired attribute. Can you try changing context to be span and see if it works?

The connector README shows different valid values for context, which links to OTTL context. From there you can see the valid contexts, and why the span context may be the right one here. (More information on the span context to show how to access its attributes.)

Hi @crobert-1 , That was the issue changing Context to span worked,
Thank you very much for helping to get this one to work and really appreciate your support

[crobert-1]

Perfect, glad to hear it worked! I'm going to close for now but feel free to reopen, or open a new issue if anything else comes up.