Sessions and users

A session is one conversation: the traces that share a session id, read in the order they started. The Sessions, Agents and Users views group traces by the ids your code sets.

Set the ids from the SDK

Every span can carry session_id (one conversation), user_id (the end user your app authenticated) and device_id (the installation or browser, for grouping before sign-in). A trace without them is still recorded, but it cannot be grouped. Most apps need one line, a session around each conversation, so every span opened inside it carries the id:

with evalkit.session("conv_55", user_id="u_8123"):
    agent.run(prompt)

# or as a decorator, for a handler that is one conversation turn
@evalkit.session("conv_55", user_id="u_8123")
async def handle(message): ...

From OpenTelemetry or another SDK, set session.id (or conversation.id) on the root span only. The child spans of that trace count in the session automatically, tokens included. To set the ids for one trace, or once for everything that follows in the current request or task:

trace_id, end, ctx = evalkit.start_trace(
    "chat-turn", user_id="u_8123", session_id="conv_55", device_id="ios-9f2",
)

# or once, for everything that follows in this request / task
evalkit.set_user(user_id="u_8123", session_id="conv_55")
...
evalkit.set_user()          # clears all three
evalkit.current_user()      # what is in effect right now

In Python, set_user() writes to a context variable, so a value set in one request or asyncio task never leaks into another. Fields merge, and clear_user(token) restores what was there before. The Django middleware fills user_id from request.user, and the FastAPI and Starlette middleware from scope["user"], when the user is authenticated.

In TypeScript, setUser inside an active trace binds to that trace only. Outside a trace it sets an ambient value: per request through AsyncLocalStorage in Node, or for the whole module where that is missing, as in browsers. Child spans inherit the ids of their trace, including spans from third-party OpenTelemetry instrumentation.

Sessions view

Traces → Sessions lists one row per session id in the time range. Its columns are user, agent, verdict, score, traces, turns, tool calls, error spans, tokens in and out, total latency and last seen.

• A turn is a trace that made at least one model call in the session.
• Error spans counts spans with status ERROR across the traces of the session, so one failing trace can add several.
• Agent is the most recent agent declared in the session, and links to its agent page.
• Verdict and Score are the latest session verdict and need the trace-evaluation plan feature. A dash means never evaluated, and a muted "earlier" means traces arrived after the verdict.

Filter by user or agent, which match exact ids, or search session and user ids. Simulation sessions are left out; only the API includes them, with includeSimulations=true. The time range is the shared picker, so a link keeps it.

The list loads 100 sessions at a time with Load more. Every header sorts the loaded rows in the browser, as the footer says. Sort by Score to put the worst conversations first.

Click a row to open the session page, which has its own address. Its header links the user and the agent and shows first and last seen, traces, turns, tool calls, error spans, tokens, cost and time in traces.

Cost is the sum of the gen_ai.usage.cost_usd your instrumentation reported per turn. A turn it did not price stays unknown, and the label says how many turns were priced.

Below are the session evaluation and the turns in order. Each turn shows its number, prompt, status, duration, LLM calls and tokens, and Open trace opens it. Once the session is evaluated, a Flagged by judges column lists the failed per-item verdicts that name each turn. Read a session evaluation walks through the page, and Export report downloads it as Markdown.

One trace can hold several conversations, as when a simulation runs several scenario sessions under one root. That trace is scored one part per conversation and has no score of its own; its Evaluation tab shows the part for each conversation. On a turn, the session line steps through the conversation, and Back returns to the session page.

Every evaluation of the session is kept. History, above the verdict, steps through the runs, and All runs lists them newest first with each run’s change from the one before, so a session judged again shows whether it moved. An earlier run opens read-only in the same layout, with Back to latest.

An empty list says which of four things is wrong:

• Sessions exist, but are older than the window. The screen offers Look back 7 days.
• No traces arrived in the window.
• Traces arrived without a session id.
• Traces carry session ids, but the trace service returned no sessions. That is worth reporting.

For the last three, the screen shows the missing line, with evalkit.session(...) or withSession(...). The session id comes from your application, so it is the one thing the SDK cannot set for you.

Evaluate a session

A session verdict judges the conversation as a whole: did it reach the user's goal, stay coherent across turns and recover from its own errors. It is a judgment of its own, not an average of the trace scores of its turns.

Click Evaluate session on the session page, or select sessions in the list or on an agent page and click Evaluate N sessions. Choose a Judge model and Evaluators, a collection or Choose individually…. Conversation evaluators that fit any agent are preselected; coding, MCP and memory evaluators are yours to add.

• The judge reads the newest 50 traces of the session, in the order they started. A longer session says so on its evaluation: "The newest 50 of 73 traces scored".
• From a trace that also held other sessions, the judge reads only the spans of this session, the same spans its part is scored on. The shared root, such as a simulation harness, belongs to neither, so the failure of one conversation never lands on the verdict of another.
• The verdict reads passed, failed, score only, no verdict or judge error. The session page shows the overall score, a card per scored evaluator, the evaluators with no score and why, the judge's summary and when it was judged; see Read a session evaluation.
• Each evaluation is kept. When a session gets new traces after it was judged, a notice says so, and Evaluate session judges the conversation as it is now. The earlier verdict stays in History, and judging the session again with no new traces adds a run as well.
• One run judges up to 50 sessions; with more selected, the dialog says only the first 50 will be judged. The result gives scored, passed, failed and the average score. Every session follows: failed, score only, then passed, each lowest first, and last no verdict and judge error.
• An online rule with session scope does the same continuously, optionally for one agent only, and its results land on the same session pages. Online rules leave simulation sessions out.
• A simulation's session can be evaluated by hand too, such as a scenario of an SDK run or of a run against an endpoint or AWS AgentCore agent: open it from its trace. The simulated user's turns are never judged as the agent's. A tool call the agent both reported in its reply and traced with the SDK is judged once, from its own span, and the session page counts it once.
• When a session also holds the agents of an evaluation tool, such as its user simulator and evaluators, only the session's own agent system is judged: the agent named on the session, the agent the list is filtered to, or the agent page you started from. See Which agent a session evaluation judges.

Agents

An agent is a name your code declares; you do not create agents in the console. Name it once and every span after it carries the name, so sessions, traces and verdicts can be read per agent under Traces → Agents. A session that ran several agents is found under each, and its row shows the most recent one.

evalkit.set_agent("support-bot")   # every span from here on, per thread and asyncio task
evalkit.set_agent(None)            # clears it

# or for one trace
evalkit.start_trace("chat-turn", agent="support-bot")

The tab opens on Named agents, one row per name for the 200 most recent. Each row shows the pass rate (for example 67 %, from 12 of 18), average score, traces, sessions, versions, error traces, tokens and last seen. Pass rate is passed sessions over passed and failed ones, by the latest verdict of each; sessions without a verdict are left out.

All versions switches the tab to every configuration fingerprinted in the range, named agent or not. An app that never names an agent still sees its versions there. With no named agent in the range, the page shows the missing line for Python, TypeScript and OpenTelemetry (gen_ai.agent.name on the root span).

• An agent page shows its traces, sessions, error traces and last seen. View traces and View sessions open those lists filtered to the agent.
• Its session verdicts give the pass rate, the average score and the number of sessions evaluated, with a daily trend once verdicts span two days.
• The evaluators that fail it most, its versions, and its sessions newest first, each with its latest verdict, follow.
• Select sessions and click Evaluate N sessions, or click Evaluate the latest N sessions to judge those seen most recently. One run takes at most 50.

Users view

The Users page (…/traces/users) lists one row per user id with sessions, traces, errors, error %, tokens in and out, and first and last seen. The trace service aggregates the rows, so the counts match the trace list.

Error % is error spans over the traces of the user. The rollup counts spans, so a trace with several failing spans can push it past 100 %, and the column is not colored. The token columns appear only when a listed user has model-call tokens, and a user without any is blank there.

Search user id narrows the list, but only the time range is kept in the URL. Users load 100 at a time with Load more, and every header sorts the loaded rows in the browser. Sessions and Traces on a row open those views scoped to the user, as a removable chip, with the range kept.

There is no cost column. The rollup returns tokens per user, not dollars and not tokens per model, so nothing could be priced exactly. End-user thumbs and ratings are stored as scores on traces and are not joined onto the user rollup yet, so there is no feedback column either.

Add a session to a dataset

A whole conversation can become a row of a conversation dataset, with the turns kept as context and the final reply as the expected output. See Golden datasets for the multi-turn shape. The session page cannot build such a row yet, so import the conversation as JSONL or add its traces one at a time.