[FEATURE REQ] Unable to override specific field in Application Insights schema

[TimothyMothra]

I'm opening this issue to track and consolidate all other reported instances of this problem.
Currently, the Azure Monitor Exporter does not offer any mechanism to override every attribute in the Application Insights schema. While several attributes are mapped today, several others are not. My team is investigating how to better document this and how to close this feature gap.

This affects users manually creating telemetry and users wishing to override attributes created by an Instrumentation library.

Affected Libraries

• Azure.Monitor.OpenTelemetry.Exporter
• Azure.Monitor.OpenTelemetry.AspNetCore

Reported Issues

• [FEATURE REQ] Support for extending app insights Dependency.Type values #45112
  Override Dependency.Type because unsupported dependencies show as "OTHER" and displays in UX. (E2E transactions)

• [FEATURE REQ] Set, or support setting, App Insights session ID and user ID via otel exporter #45089
  [BUG] Azure.Monitor.OpenTelemetry.AspNetCore Anonymouse user ID and Session ID not stored as expected. #49412
  Set sessionId and userId
  Can these values be set automatically, consistent with classic AI SDK?

• [BUG] OpenTelemetryExporter prevents setting the span name for Request telemetry #44971
  Override span name for Request telemetry.

• [FEATURE REQ] How to set UserAccountId in AppInsights with Azure.Monitor.OpenTelemetry.Exporter #44837
  Set user AccountId

• [FEATURE REQ] Controlling the "type" name of an OTEL activity in Application Insights #44762
  Overriding Dependency.Type currently shows as "InProc" or "HTTP". Wants to set name of api (ie: AcmeApi)

• [FEATURE REQ] Open Telemetry Sample Request #44587
  Set operationName

• [BUG] The operation_Name field is not getting populated property. #41946 (comment)
  Override operationName

• [FEATURE REQ] How to set Application Insights Metrics Namespace in AzureMonitorOpenTelemetryExporter #38270
  Set Metrics Namespace

• [FEATURE REQ] Support Activity.StatusDescription in Exporter #41057
  Support for Activity.StatusDescription

• [FEATURE REQ] Unable to override specific field in Application Insights schema #46021 (comment)
  Set ai.operation.syntheticSource

• [FEATURE REQ] Unable to override specific field in Application Insights schema #46021 (comment)
  Set ai.device.id, ai.device.type

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @cijothomas @rajkumar-rangaraj @reyang @TimothyMothra.

[johncrim]

For some of the linked issues, there are multiple changes needed.

1. Provide a way to set a field and not have it overridden or ignored (this issue - makes sense to consolidate)
2. Decide whether to set the value automatically, eg for behavioral consistency with AI classic

For example, the AI session ID and user ID issue includes both.

[TimothyMothra]

2. Decide whether to set the value automatically, eg for behavioral consistency with AI classic

That's fair, I'll update my summary above so we don't lose sight of this.
Our immediate focus is getting these fields mapped. We wanted to take a dependency on the OpenTelemetry community specification but it's moving slow so we have to explore alternatives.

[gastonmuijtjens]

+1 for this ability. Unfortunately, there is currently no way to set the synthetic source property of Application Insights (ai.operation.syntheticSource) from the Azure Monitor Exporter OpenTelemetry SDK.

[mgaffigan]

It would be great to be able to specify ai.device.id=browser / ai.device.type=Browser when we are proxying front-end data.

[jeremy-morren]

This is a blocker for migrating from App Insights where custom properties are set e.g. DeviceType, custom OS information etc.

This could be accomplished by an additional tags processor which extracts a known series of tags from Resource and items (Activity, Metric, Log). These tags would be those tags that are not extracted from anywhere else.

/// <summary>
/// Processor that handles known resource attributes (mapped to <see cref="ContextTagKeys"/>).
/// </summary>
internal static class KnownResourceAttributesProcessor
{
    /// <summary>
    /// A map of attributes to their corresponding <see cref="ContextTagKeys"/> and maximum length.
    /// </summary>
    public static readonly Dictionary<string, (ContextTagKeys Key, int MaxLength)> Attributes = new()
    {
        // device.* attributes
        {
            SemanticConventions.AttributeDeviceType,
            (ContextTagKeys.AiDeviceType, SchemaConstants.Tags_AiDeviceType_MaxLength)
        },
        {
            SemanticConventions.AttributeDeviceId,
            (ContextTagKeys.AiDeviceId, SchemaConstants.Tags_AiDeviceId_MaxLength)
        },
        {
            SemanticConventions.AttributeDeviceManufacturer,
            (ContextTagKeys.AiDeviceOemName, SchemaConstants.Tags_AiDeviceOemName_MaxLength)
        },
        {
            SemanticConventions.AttributeDeviceModelId,
            (ContextTagKeys.AiDeviceModel, SchemaConstants.Tags_AiDeviceModel_MaxLength)
        },

        // browser.* attributes
        {
            SemanticConventions.AttributeBrowserLanguage,
            (ContextTagKeys.AiDeviceLocale, SchemaConstants.Tags_AiDeviceLocale_MaxLength)
        },

        // os.* attributes
        {
            SemanticConventions.AttributeOsDescription,
            (ContextTagKeys.AiDeviceOsVersion, SchemaConstants.Tags_AiDeviceOsVersion_MaxLength)
        },
    };
}

These attributes could then be added to the resource:

ResourceBuilder.CreateDefault().AddAttributes([
    new KeyValuePair<string, object>("device.type", "Phone"),
    new KeyValuePair<string, object>("device.model.identifier", "SM-G920F"),
]);

Implementation at https://github.com/jeremy-morren/azure-sdk-for-net/blob/fieldOverride/sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/Internals/KnownResourceAttributesProcessor.cs

[PraveenVerma17]

Actually, we need the ability to override specific fields through middleware. The code given above needs to be set once for the resource, but the session ID and user ID need to be set on a per-request basis. Currently, in middleware, all new tags (using activity.settag) move those properties to custom dimensions.

[johncrim]

@PraveenVerma17 - See my comment here - it works if you're willing to put in the work. You can add either an Activity processor or a LogRecord processor the same way - you have to do both if you want the session ID and user ID in both types of telemetry. Here's an example:

/// <summary>
/// Adds tags to activity that are stored in request baggage.
/// </summary>
/// <remarks>
/// The objective here is to duplicate key tags in all downstream activities, eg
/// app insights session.id and user.id are useful in the whole downstream tree.
/// </remarks>
internal sealed class AddRequestBaggageActivityProcessor : BaseProcessor<Activity>
{

    public override void OnStart(Activity activity)
    {
        AddTagsFromBaggage(activity);
    }

    internal static bool CreateTagFromBaggage(string key)
    {
        return key.StartsWith("ai.")
               || key.StartsWith("user.");
    }

    internal static void AddTagsFromBaggage(Activity activity)
    {
        if (activity.IsAllDataRequested)
        {
            foreach (var kvp in Baggage.GetBaggage())
            {
                if (CreateTagFromBaggage(kvp.Key))
                {
                    activity.SetTag(kvp.Key, kvp.Value);
                }
            }
        }
    }

}

Then you have to set the values you want in Baggage (using request middleware), and the trickiest part is you need to set the BaseProcessor.PipelineWeight to a negative value, so it runs before the Otel exporter. Unfortunately there's no way in the Otel SDK to set a negative PipelineWeight, but you can do it via reflection.

[PraveenVerma17]

@johncrim
This isn’t exactly what I was looking for. I’m able to manage end-to-end traceability, but I’m facing an issue where session.id and user.id are being set as custom dimensions instead of being populated in the predefined columns in Application Insights.

We have numerous dashboards built using the existing standard columns, so changing them all would require significant effort. I'm looking for a solution that maintains backward compatibility before we transition fully to Azure Monitor with OpenTelemetry.