AI Agent framework Semantic Convention

[gyliu513]

Area(s)

area:gen-ai

Is your change request related to a problem? Please describe.

There are now many agent frameworks for GenAI, including ibm bee stack, ibm wxflow, crewai, autogen, langgraph etc. Those agents will include many different components, like agent, tools, tasks etc, we need a semantic convention for those resources as well for agents.

@lmolkova @nirga @drewby @karthikscale3 ^^

Describe the solution you'd like

Extend the GenAI Semantic Convention to cover agents

Describe alternatives you've considered

No response

Additional context

No response

[lmolkova]

FYI: I'm baking some early prototype here microsoft#3, I'll re-send the PR against this repo after a bit more prorotyping

[gyliu513]

I did some test with crewai and Instana observability at https://gyliu513.github.io/jekyll/update/2024/10/22/crewai-observability.html, we may need some agent framework level semantic convention.

Also did some test with langtrace, it can also capture some tracing and metrics when I was using crewai based on langchain.

@karthikscale3 has some demo for agent observability with langtrace as well.

@lmolkova it is great you have a PR under-going, will take a look, thanks!

[lmolkova]

wow, this looks awesome!

I'm struggling with a couple of things and hope to gets your thoughts on a few big things:

• what's the scope: client framework, client to service-side agent, multi-agent story. How may layers do we need?
• the level of unification - e.g. task execution is the same as openai assistant run and will be the same as azure AI agent run - how far do we need to go in attempts to unify

Essentially my main worry is that I don't fully understand what exactly we want to unify in semconv or what we want the conventions for.

There is always an alternative that the LLM client level is unified but frameworks do some extra stuff which does not need much consistency. LMK what you think.

[karthikscale3]

Thanks for starting the thread @gyliu513 . Shown below the span graph on Langtrace for CrewAI. Generally speaking, here's a general pattern that's emerging across agentic frameworks:

[1] Sessions - Each session can have multiple agents working independently or together to perform a bunch of tasks.
[2] Agents - Different agents that can do different tasks
[3] Agent Config - Sequential, Hierarchical, Networked
[4] Tools - Tools that agents have access to
[5] Tasks - Tasks defined for agents

Based on our experience at Langtrace, developers like to see:
[1] traces that are isolated and grouped at these high level constructs (agents, tools, tasks etc.)
[2] see the relationship between these constructs per session
[3] see metadata related to each one of these constructs

With all the above requirements in mind, we designed our instrumentation for crewai and other agentic frameworks we support. I think we can come up with sem conv for these high level constructs that are common for these agentic frameworks. Let me know what you all think.

[gyliu513]

[1] Sessions - Each session can have multiple agents working independently or together to perform a bunch of tasks.
[2] Agents - Different agents that can do different tasks
[3] Agent Config - Sequential, Hierarchical, Networked
[4] Tools - Tools that agents have access to
[5] Tasks - Tasks defined for agents

@karthikscale3 good summary, thanks!

@lmolkova comments for this? I think may answer your above question, hope it is clear.

[drewby]

This paper may be useful inputs for this discussion: https://arxiv.org/abs/2411.05285

In particular, Fig 7 takes a stab at generalizing the various steps that may appear in a Trace.

[gyliu513]

I was now checking the white paper for agent from google https://www.kaggle.com/whitepaper-agents , I was wondering maybe we need two issues for agent, one is Agent semantic convention and another is Agent Framework semantic convention.

• Agent was composed by model and tools
• There are different Agent frameworks, those framework have concepts like Agent, Task, Agent Orchestration framework, like CrewAI, AutoGen, Langgraph etc (Some comparison for different agent frameworks https://gyliu513.github.io/jekyll/update/2024/12/17/battle-of-ai-agent-frameworks.html)

Comments? Thanks!

@lmolkova @nirga @drewby @karthikscale3 ^^

[marklysze]

At AG2 (based on AutoGen) we're currently implementing OpenTelemetry support and would like to align with the conventions.

I think a separation between agent and framework could be useful.

[TaoChenOSU]

We currently have the LLM conventions, which should be able to capture most of what is happening within an agent, i.e. chat messages should contain the thought process and tool calls carried out by an agent.

However, we don't have standard ways to capture useful information when multiple agents are working together, i.e. how the next agent is selected, or how the task is determined successful. These are probably more framework oriented.

[gyliu513]

An issue for AI Agent Semantic Convention was created here #1732