Description
Is your feature request related to a problem?
The current visibility into ML-Commons agent behavior and performance is limited. The existing tracing implementation using memory in ML-Commons cannot effectively capture complex agent workflows, particularly when dealing with non-linear executions like branching, loops, and parallel tool calls. Users struggle to understand relationships between planner agents, executor agents, and their tool invocations. There is no intuitive visualization system - making it impossible to easily track execution steps, sequence, or duration. Furthermore, developers cannot effectively compare executions between runs or debug slow/failed tool invocations, significantly hampering development and optimization efforts.
What solution would you like?
We propose implementing a comprehensive agent tracing and observability system using OpenTelemetry. The solution would integrate OpenTelemetry to collect detailed execution data, including tool usage, model interactions, execution phases, and relationships between agents and sub-agents, along with crucial performance metrics and metadata. Users would be able to inspect execution metadata through this interface.
What alternatives have you considered?
We evaluated extending the existing memory/messages/traces API, which would leverage existing code paths but would not be compatible with industry standards and would make visualization transformations difficult.
Do you have any additional context?
Similar solutions exist in tools like LangSmith, demonstrating proven value in this approach. Importantly, OpenTelemetry support is already being utilized in OpenSearch core, query-insights, and performance analyzer, providing a solid foundation for this implementation. This context suggests both the feasibility and strategic alignment of the proposed solution within the existing OpenSearch ecosystem.