ARM64 - Could not load file or assembly Microsoft.Extensions.Logging.Abstractions

[Kielek]

I have relate problem, with err message:
Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Extensions.Logging.Abstractions, Version=7.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

File name: 'Microsoft.Extensions.Logging.Abstractions, Version=7.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
at Microsoft.Extensions.DependencyInjection.LoggingServiceCollectionExtensions.AddLogging(IServiceCollection services, Action`1 configure)
at Microsoft.Extensions.DependencyInjection.LoggingServiceCollectionExtensions.AddLogging(IServiceCollection services)
at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider()
at Microsoft.Extensions.Hosting.HostBuilder.Build()

Originally posted by @lumbans in #2295 (comment)

[Kielek]

Hi @lumbans, We need more information to fully understand your issue. At least what we need is to have:

• AutoInstrumetnation version
• Operating system.
• Upload Core Host logs like described under: https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/blob/v0.6.0/docs/troubleshooting.md#assembly-in-additionaldeps-was-not-found
• verify if you application is not referencing (directly or indirectly) any ohter/older versions of Microsoft.Extensions.Logging.Abstractions - the easiest way is to list all dll files in the folder and check the version. If there is version different that 7.0.0. You should add reference to Microsoft.Extensions.Logging.Abstractions to your project file.\
• It will be great if you can create Minimal, Reproducible Example.

Similar issue was discussed here: #2269
Please let me know if it helps you.

[lumbans]

Hi @Kielek, thanks for the response. here's for the details:

• Auto Instrumentation DotNet v0.6.0
• using container images aspnet-6.0 arm64 chipset

[Kielek]

arm64 - we are not ready to support if out of the box. Mostly, we lack of ARM64 environment to execute continuous integration.
See #2167 (comment) - it is still up to date.
You can try to use workaround under #2167 (comment)

When you solve it this part of my comment can be relevant: "verify if you application is not referencing (directly or indirectly) any ohter/older versions of Microsoft.Extensions.Logging.Abstractions - the easiest way is to list all dll files in the folder and check the version. If there is version different that 7.0.0. You should add reference to Microsoft.Extensions.Logging.Abstractions to your project file."

[lumbans]

Hi @Kielek ,
The error not shown anymore, I change the methode.

• using cusom image with v0.5.0 docker file and change the folder in store/x64 to store/arm64
• inject instrumentation in my deployment manifest
  Here's the dockerfile:

FROM busybox

WORKDIR /autoinstrumentation
ADD https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/releases/download/v0.5.0/opentelemetry-dotnet-instrumentation-linux-glibc.zip .
RUN unzip opentelemetry-dotnet-instrumentation-linux-glibc.zip && rm -f opentelemetry-dotnet-instrumentation-linux-glibc.zip && mv store/x64 store/arm64
RUN chmod -R go+r .

My Instrumenation manifest:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  labels:
    app.kubernetes.io/managed-by: opentelemetry-operator
  name: my-instrumentation
  namespace: default
spec:
  dotnet:
    image: xxxxx.dkr.ecr.ap-southeast-1.amazonaws.com/experiment-k8s:0.5.2
  exporter:
    endpoint: http://otel-collector.default.svc.cluster.local:4317
    propagators:
  - tracecontext
  - baggage
  - b3
 sampler:
    argument: "0.25"
    type: parentbased_traceidratio`

my deployment manifests:

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    annotations:
      deployment.kubernetes.io/revision: "5"
    name: hello-world-api
    namespace: dotnet-api
  spec:
    replicas: 2
    selector:
      matchLabels:
        app.kubernetes.io/name: helloworldapi-app
    strategy:
      rollingUpdate:
        maxSurge: 25%
        maxUnavailable: 25%
      type: RollingUpdate
    template:
      metadata:
        annotations:
          instrumentation.opentelemetry.io/inject-dotnet: default/my-instrumentation
        labels:
          app.kubernetes.io/name: helloworldapi-app
      spec:
        containers:
        - env:
          - name: AWS_REGION
            value: ap-southeast-1
          image: xxxxx.dkr.ecr.ap-southeast-1.amazonaws.com/hello-world-api:2.6.8
          name: hello-world-api
          ports:
          - containerPort: 80
            protocol: TCP

but I didnt see any trace data on Jaeger, here's the pod manifest output from kubectl exec pod:

Defaulted container "hello-world-api" out of: hello-world-api, opentelemetry-auto-instrumentation (init)
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=hello-world-api-6b95658567-k2txr
ASPNETCORE_URLS=http://+:80
DOTNET_RUNNING_IN_CONTAINER=true
DOTNET_VERSION=6.0.14
ASPNET_VERSION=6.0.14
CORECLR_ENABLE_PROFILING=1
CORECLR_PROFILER={918728DD-259F-4A6A-AC2B-B85E1B658318}
CORECLR_PROFILER_PATH=/otel-auto-instrumentation/OpenTelemetry.AutoInstrumentation.Native.so
DOTNET_ADDITIONAL_DEPS=/otel-auto-instrumentation/AdditionalDeps
DOTNET_SHARED_STORE=/otel-auto-instrumentation/store
DOTNET_STARTUP_HOOKS=/otel-auto-instrumentation/net/OpenTelemetry.AutoInstrumentation.StartupHook.dll
OTEL_DOTNET_AUTO_HOME=/otel-auto-instrumentation
OTEL_DOTNET_AUTO_INTEGRATIONS_FILE=/App/otel-dotnet-auto/integrations.json
OTEL_SERVICE_NAME=hello-world-api
OTEL_RESOURCE_ATTRIBUTES=k8s.container.name=hello-world-api,k8s.deployment.name=hello-world-api,k8s.namespace.name=dotnet-api,k8s.node.name=ip-192-168-x-x.ap-southeast-1.compute.internal,k8s.pod.name=hello-world-api-6b95658567-k2txr,k8s.replicaset.name=hello-world-api-6b95658567
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.25
OTEL_PROPAGATORS=tracecontext,baggage,b3
AWS_REGION=ap-southeast-1
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.default.svc.cluster.local:4317
OTEL_RESOURCE_ATTRIBUTES_POD_NAME=hello-world-api-6b95658567-k2txr
OTEL_RESOURCE_ATTRIBUTES_NODE_NAME=ip-192-168-x-x.ap-southeast-1.compute.internal
TRAFFIC_GENERATOR_PORT_80_TCP_PORT=80
HELLO_WORLD_API_SERVICE_PORT=80
HELLO_WORLD_API_PORT_80_TCP_ADDR=10.x.x.x
KUBERNETES_PORT_443_TCP_PORT=443
TRAFFIC_GENERATOR_PORT_80_TCP=tcp://10.x.x.x:80
TRAFFIC_GENERATOR_PORT=tcp://10.x.x.x:80
TRAFFIC_GENERATOR_PORT_80_TCP_ADDR=10.x.x.x
HELLO_WORLD_API_SERVICE_HOST=10.x.x.x
HELLO_WORLD_API_PORT_80_TCP_PORT=80
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP_ADDR=10.x.x.x
TRAFFIC_GENERATOR_SERVICE_HOST=10.x.x.x
TRAFFIC_GENERATOR_SERVICE_PORT_80=80
TRAFFIC_GENERATOR_PORT_80_TCP_PROTO=tcp
HELLO_WORLD_API_PORT_80_TCP_PROTO=tcp
KUBERNETES_SERVICE_HOST=10.x.x.x
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT_443_TCP=tcp://10.x.x.x:443
TRAFFIC_GENERATOR_SERVICE_PORT=80
HELLO_WORLD_API_PORT=tcp://10.x.x.x:80
HELLO_WORLD_API_PORT_80_TCP=tcp://10.x.x.x:80
KUBERNETES_PORT=tcp://10.x.x.x:443
KUBERNETES_PORT_443_TCP_PROTO=tcp

how to debug to ensure the trace data send to the opentelemetry collector?

[Kielek]

Logs can be found under /var/log/opentelemetry/dotnet on your container. You can verify if base things are working fine.

I suppose it is related to open-telemetry/opentelemetry-operator#1465 (comment)

@lachmatt promises to double check.

[lachmatt]

@Kielek is right - either Instrumentation CR exporter's endpoint should be changed to http://otel-collector.default.svc.cluster.local:4318, or dotnet specific env that overrides OTEL_EXPORTER_OTLP_ENDPOINT should be added.

[lumbans]

Thanks @Kielek @lachmatt ,
Now opentelemetry can get event trace in our API dotnet, here's the changes of manifest Instrumentations:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  labels:
    app.kubernetes.io/managed-by: opentelemetry-operator
  name: my-instrumentation
  namespace: default
spec:
  dotnet:
      env:
      - name: OTEL_LOG_LEVEL
         value: debug
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://otel-collector.default.svc.cluster.local:4318
      - name: OTEL_EXPORTER_OTLP_TIMEOUT
        value: "20"
      image: xxxxx.dkr.ecr.ap-southeast-1.amazonaws.com/experiment-k8s:0.5.2
  exporter:
    endpoint: http://otel-collector.default.svc.cluster.local:4318
    propagators:
  - tracecontext
  - baggage
  - b3
 sampler:
    argument: "0.25"
    type: parentbased_traceidratio`

[Kielek]

Great to hear that it is working for you. Official support for arm64 is tracked under following items #2181, #1865 and #2314.

Feel free to create new issue/reopen this one if you need any help.