Support log exports to Jaeger

[tillig]

Feature Request

There is support in the OpenTelemetry.Exporter.OpenTelemetryProtocol.Logs library for taking log messages and exporting them as part of a trace span to an OpenTelemetry endpoint.

It does not appear that the same functionality is supported for Jaeger.

Is your feature request related to a problem?

When using Jaeger it's not currently possible to see logs in context with the collected traces.

Describe the solution you'd like:

Jaeger support similar to the OT support for shipping logs to Jaeger.

Describe alternatives you've considered.

I don't think there's an alternative. Maybe just... not having logs in the trace?

Additional Context

It appears the OpenTelemetry.Exporter.OpenTelemetryProtocol.Logs library has the extensions to register the log exporter but the log exporter itself lives, unused and internal, in the OpenTelemetry.Exporter.OpenTelemetryProtocol library.

1. Should those be combined?
2. If not, should the OtlpLogExporter move to that ...Logs library?

I figure however the OTLP version is done, the Jaeger version would follow the same pattern.

[cijothomas]

When you say "log messages", do you mean the standalone logs generated from ILogger, or the ActivityEvent which are embedded logs inside Activity?

[cijothomas]

The embedded logs, (AKA, ActivityEvents) are already supported in OTLP, Jaeger exporters.

The OTLP Log Exporter is for exporting the "standalone logs".

[tillig]

Hmmm. Well, I think I mean standalone logs.

In our current project we've tied into OpenTracing to emit log events using a log sink, like this:

public class OpenTracingSink : ILogEventSink
{
    private readonly ITracer _tracer;
    private readonly IFormatProvider _formatProvider;

    public OpenTracingSink(ITracer tracer, IFormatProvider formatProvider)
    {
        this._tracer = tracer ?? throw new ArgumentNullException(nameof(tracer));
        this._formatProvider = formatProvider;
    }

    public void Emit(LogEvent logEvent)
    {
        if (logEvent == null)
        {
            return;
        }

        var span = this._tracer.ActiveSpan;

        if (span == null)
        {
            return;
        }

        var fields = new Dictionary<string, object>
        {
            { "level", logEvent.Level.ToString() },
            { LogFields.Event, logEvent.MessageTemplate.Text },
        };

        if (logEvent.Properties.TryGetValue("SourceContext", out var sourceContext))
        {
            fields["component"] = sourceContext.ToString().TrimStart('"').Trim('"');
        }

        try
        {
            fields[LogFields.Message] = logEvent.RenderMessage(this._formatProvider);

            if (logEvent.Exception != null)
            {
                fields[LogFields.ErrorKind] = logEvent.Exception.GetType().FullName;
                fields[LogFields.ErrorObject] = logEvent.Exception;
            }

            if (logEvent.Properties != null)
            {
                foreach (var property in logEvent.Properties)
                {
                    fields[property.Key] = property.Value.ToString().TrimStart('"').Trim('"');
                }
            }
        }
        catch (Exception logException)
        {
            fields["mbv.common.logging.error"] = logException.ToString();
        }

        span.Log(fields);
    }
}

In doing that, when you look at a distributed trace (emitted to Jaeger via the OpenTracing library) you not only see a trace span, but you see all of the logs associated with that span right inside the span. It's really helpful.

This is what it looks like the OTLP exporter for logs is doing - exporting the log messages, via OpenTelemetry protocol, to something like Dynatrace or some other OTLP sink, so you can see logs in context of the trace.

It could be my fundamental understanding of this is off.

Under the assumption I'm right, the thing I'm hoping to see is a Jaeger version of this - a way to get logs into Jaeger in the context of the trace.

If I'm wrong then... Hmmm. I guess I'll have to figure out how to do this myself.

[tillig]

I just realized there's an important bit of context here that may affect this whole thing, and I didn't think about it until just now:

We're using Serilog.

So, as it turns out, even if we get this built into this project, I'm still going to need to implement the Serilog version of the sinks (which is why the above snippet is an ILogEventSink and not an ILoggerProvider).

This doesn't mean it isn't interesting at all, just that I probably won't be able to contribute it myself due to the Serilog implication. It doesn't look like there's already one out there so I have some work to do.

[cijothomas]

There is a contrib package, capable of taking ILogger logs, and attaching them to to Activity as ActivityEvent, (which would then be exported to Jaeger.)

https://github.com/open-telemetry/opentelemetry-dotnet-contrib/tree/main/src/OpenTelemetry.Contrib.Preview

(Its preview package. And I think there are interesting uses of doing logs -> traces and traces -> logs conversions)

[cijothomas]

We're using Serilog.

If you are using Serilog's logging API, then not much can be done here. If you use the ILogger from Microsoft.Extensions.Logging.Abstrations, but use Serilog as a Provider, then you are covered, by the preview package shared in above comment.

[tillig]

Yeah, that's why I sort of backtracked a bit - I think there's value in having logs go to trace, and maybe that's something this preview library is providing, but I realized the Serilog thing won't be solved here.

That said, I'm working through the API spec around logging and it seems like this log correlation thing is something it's intended to solve, but maybe only for OpenTelemetry backends.

How I'm reading it - and I very well could be wrong - is that the logging library is supposed to enrich log information with correlation data like the trace ID, and that ends up getting sent to an OpenTelemetry collector along with traces and metrics, which then correlates everything together.

That's kinda what Jaeger will do, minus the metrics.

But, again, I recognize the solution here would be more based on standard Microsoft logging and not Serilog. That's fine, I can solve the Serilog problem. I think it might still be interesting to solve it for the Jaeger situation in standard Microsoft logging, which is what this issue/enhancement could be about... but if that's not interesting, or if it should be pushed to the contrib repo or something, I guess that's fine, too.

I will say, the whole handling of logging (reading the spec, trying to unwind what exactly it's intended to do) is pretty confusing. There's a format, logs get annotated with that, then [insert picture of cloud and some hand-waving] the logs get correlated. I don't totally get it.

[fschmied]

The embedded logs, (AKA, ActivityEvents) are already supported in OTLP, Jaeger exporters.

The OTLP Log Exporter is for exporting the "standalone logs".

@cijothomas Assuming I have Open Telemetry integrated and configured to export traces to Jaeger. I also have it integrated to create Log Records, correlated with my Activities, via ILogger (as shown here and in the ASP.NET Core sample application. Now, what's the recommended way to get the Log Records created via ILogger into that Jaeger instance?

(I also have the same question about the Azure Monitor exporter, but I'll start here, with the official exporters.)

[cijothomas]

@fschmied There is a preview package, which allows ILogger logs to be attached to Activity Events, (which will then be exported to Jaeger.)
https://github.com/open-telemetry/opentelemetry-dotnet-contrib/tree/main/src/OpenTelemetry.Extensions

[cijothomas]

@fschmied AzMonExporter already supports iLogger logs (package may not be released yet to nuget.org). Please use AzMonitor repos for questions about that.

[tillig]

Be aware, attaching logs as activity events will keep the log messages in memory until the trace is sent to the collector. I ended up not going this way because it was kind of a memory burden.

[fschmied]

There is a preview package, which allows ILogger logs to be attached to Activity Events, (which will then be exported to Jaeger.)

As @tillig said, this might be problematic from a memory PoV (the corresponding .NET Core docs also warn about that), but it might fit for some scenarios, I guess.

@cijothomas Do you know whether it's on the roadmap for OpenTelemetry .NET to support log export in more exporters?

(Regarding AzMonExporter - you're right, shouldn't have mentioned it. Here is some info from their repo: Azure/azure-sdk-for-net#23207 (comment) - they're planning to work on it.)

[julealgon]

Should this be closed based on the fact that Jaeger is deprecating support for their custom transport in favor of OTLP?

Related:

• marking jaeger exporters as deprecated opentelemetry-collector-contrib#18503
• Deprecate jaeger exporters. opentelemetry-specification#2858

https://www.jaegertracing.io/docs/1.38/apis/#opentelemetry-protocol-stable

[tillig]

Yes, probably so. I'll close it.