LlamaIndex

init() registers an event handler and a span handler on LlamaIndex's instrumentation dispatcher, so the model calls, embeddings, retrievals, queries and reranks LlamaIndex reports become spans. This page builds a support agent with FunctionAgent in Python, runs it once, and reads the trace. The TypeScript SDK also patches the llamaindex package when it is installed; this page covers Python.

What you get

• Each model call is one llm_call span. With the OpenAI LLM it is the OpenAI client's record, chat.completions.create, with the exact token counts, evalkit.framework: llama_index and the LLM class in llama_index.class. A model whose client the SDK does not patch is recorded from LlamaIndex's events, as chat or complete. An embedding is an llm_call named embedding.
• A retrieval is a db_query span named retrieve with the query as its prompt; a query is a function_call named query, and a rerank a function_call named rerank.
• LlamaIndex reports no tool events, so a tool is a span only when you wrap its function with @evalkit.trace_tool, as the tutorial does.
• The OpenAI client's HTTP requests show as POST spans under chat.completions.create.

support-turn                 root span from start_trace
  chat.completions.create    llm_call        evalkit.framework=llama_index
    POST                     http_call
  lookup_order               tool_call       from @evalkit.trace_tool
  chat.completions.create    llm_call        the answer
    POST                     http_call

Install

pip install syntropylabs-evalkit llama-index-core llama-index-llms-openai

Set up

import os

import evalkit

evalkit.init(subscription_key=os.environ["EVALKIT_SUBSCRIPTION_KEY"], service_name="my-agent")

# Import LlamaIndex after init(), so its dispatcher reports to EvalKit from the first call.
from llama_index.llms.openai import OpenAI

Where init() goes

• Call init() once, before LlamaIndex is imported. It adds its handlers to LlamaIndex's root dispatcher and patches the OpenAI client.
• Apply @evalkit.trace_tool after init(): before it, the decorator raises RuntimeError where it is applied. The wrapper keeps the function's name, docstring and signature, which FunctionTool.from_defaults reads.

Tutorial: a support agent with FunctionAgent

The agent has one tool, lookup_order. One question makes two model calls and one tool call, inside one root span for the turn.

1. 1

   Install the packages

   Run the install line above.

2. 2

   Set the keys

   Export the environment key of your Environment and your OpenAI key.

3. 3

   Save the code

   Save it as agent.py.

4. 4

   Run it

   python agent.py prints the model's answer. The SDK sends the queued spans when the script exits.

5. 5

   Open the trace

   In Traces, open the newest trace of the service llamaindex-support.

export EVALKIT_SUBSCRIPTION_KEY=tk_live_...   # Settings → This project → Environments → Copy key
export OPENAI_API_KEY=sk-...

import asyncio
import os

import evalkit

# init() runs before LlamaIndex is imported, so its instrumentation dispatcher reports to EvalKit.
evalkit.init(
    subscription_key=os.environ["EVALKIT_SUBSCRIPTION_KEY"],
    service_name="llamaindex-support",
    environment="development",
)

from llama_index.core.agent.workflow import FunctionAgent  # noqa: E402
from llama_index.core.tools import FunctionTool  # noqa: E402
from llama_index.llms.openai import OpenAI  # noqa: E402


@evalkit.trace_tool(name="lookup_order")  # LlamaIndex reports no tool events, so the tool is wrapped here
def lookup_order(query: str) -> dict:
    """Find an order from the customer's words."""
    return {"order_id": "1001", "status": "shipped", "arrives": "Thursday"}


agent = FunctionAgent(
    tools=[FunctionTool.from_defaults(fn=lookup_order)],
    llm=OpenAI(model="gpt-4o-mini"),  # reads OPENAI_API_KEY
    system_prompt="You are a support agent for an online shop. Use lookup_order for questions about an order.",
    streaming=False,  # whole replies: a streamed OpenAI reply carries no token counts unless include_usage is set
)


async def main() -> None:
    evalkit.set_agent("support-bot")  # the trace lists under this agent in Traces > Agents
    with evalkit.session("conv_llamaindex_1", user_id="u_42"):  # groups the turns of one conversation
        _, end, ctx = evalkit.start_trace("support-turn")  # one root span for the turn
        with evalkit.with_trace(ctx):
            print(await agent.run(user_msg="Where is my order 1001?"))
        end()


asyncio.run(main())

python agent.py

It prints the model's answer, for example:

Order 1001 shipped yesterday and arrives on Thursday.

Tested with syntropylabs-evalkit 0.3.5, llama-index-core 0.14.25, llama-index-llms-openai 0.8.3 and openai 2.54: one trace of 6 spans.

See the trace

• The waterfall shows the tree under What you get. Click chat.completions.create to read the messages and token counts, and lookup_order for the tool's arguments and result.
• Traces → Sessions lists conv_llamaindex_1, and Traces → Agents lists support-bot.

Sessions, users and agents

• Name the agent with evalkit.set_agent("support-bot") and run the turn inside with evalkit.session(session_id, user_id=...). Every span opened inside the block carries the session and user.
• One root span per turn, evalkit.start_trace("support-turn") entered with evalkit.with_trace(ctx), keeps the turn's spans in one trace.
• In a web app, run the agent in the request handler and open the session there, as the page of your framework shows, such as FastAPI.

Pitfalls

• Streamed agent steps report 0 tokens. FunctionAgent streams its model calls by default, and LlamaIndex's OpenAI LLM does not ask for usage in the stream, so OpenAI sends no token counts. The SDK records 0 tokens and logs a warning once per process. Pass streaming=False to the agent, as the tutorial does, to record real counts.
• No tool spans. LlamaIndex reports no tool events. Wrap each tool function with @evalkit.trace_tool(name="...") before you hand it to FunctionTool.
• A warning about SQLAlchemy at startup. llama-index-core installs SQLAlchemy, and with SQLAlchemy 2.1 OpenTelemetry's SQLAlchemy instrumentation, which init() registers, prints SQLAlchemyInstrumentor only instruments "sqlalchemy >= 1.0.0, < 2.1.0". It is harmless.
• Log lines become spans; your log output stays as it was. Each record that propagates to the root logger is recorded as a log span, as a handler there would record it: at the level of the logger that wrote it, which is the root level, WARNING, unless that logger or your app sets another. init() changes no handler, formatter or level, so your console output stays as it was. Every record carries trace_id and span_id, so a format of your own can show them, as in %(asctime)s %(levelname)s [%(trace_id)s] %(name)s: %(message)s. On 0.3.4 and earlier, init() changes the format of the root logger's console handlers and sets the root level to INFO; configure logging after init() there.

Safe by design

• Spans are queued in memory and sent from a background thread, so a request never waits for the trace service.
• If the trace service is down or unreachable, your app answers as before. The SDK retries each export 3 times, logs evalkit: export failed after 3 retries as a warning and drops those spans.
• If the key is wrong, the trace service refuses the spans. After 3 refusals in a row the SDK logs one warning and stops exporting until the process restarts; your app keeps answering.
• A streamed model reply reaches your code chunk by chunk, unchanged. The SDK reads the same chunks on the side and writes the span when the stream ends.
• Tracing never breaks the call it records. An error inside the SDK is logged at debug level and the call runs as if untraced, and an exception from the provider reaches your code unchanged.
• A failed export never raises in your code, and evalkit.flush() returns normally too. At exit the SDK sends what is queued within 3 seconds in all, whether or not the trace service answers, so an outage never holds up a deploy.

Next steps

• Choose what is recorded, from prompts to function arguments, and mask what must not leave the process: Configuration and privacy.
• Every library the SDK traces without code, and how streamed calls are counted: What is traced automatically.
• Follow one request across services, or run beside an OpenTelemetry setup you already have: OpenTelemetry and distributed tracing.
• Read the trace you just sent: Your first trace, and group turns into conversations: Sessions and users.