Core Concepts
The vocabulary used throughout these docs and the dashboard.
| Term | Definition |
|---|---|
| Trace | A tree of spans representing a single end-to-end request or job. Identified by a W3C-format trace ID. |
| Span | A single timed operation within a trace — an LLM call, a tool invocation, an HTTP request, a DB query. Types: llm_call, tool_call, http_call, function_call, db_query, log. |
| Trace Project | A named group of traces with its own subscription key, scoped per environment (production / staging / development). |
| Subscription Key | A secret token (tk_live_...) that authenticates the SDK to the trace ingestion endpoint. Rotatable with no downtime. |
| Session ID | Groups all LLM calls in a single user conversation into one session view. |
| Device ID | Identifies the originating client for APM-style per-device analytics. |
| Evaluation Rule | A prompt template (or statistical/custom rule) a judge model uses to score a trace on a specific dimension — relevancy, groundedness, safety, tone, etc. |
| Evaluator Collection | A named, reusable bundle of evaluation rules. Apply a whole collection at once, to any project. |
| Online Evaluation | Automatic evaluation the backend triggers on every new trace that arrives in a configured Trace Project. |
| Offline Evaluation | Manual evaluation triggered by selecting traces (or a dataset) in the dashboard, or by calling evalkit.evaluate() locally. |
| Judge Model | The LLM used to score traces/outputs against evaluation rules. Bring-your-own-key for most judge calls. |
| Dataset | A CSV of input/target rows used for batch generation + evaluation, or synthetic-dataset generation. |
| Playground (Pointwise / Pairwise) | Interactive single-model generation, or side-by-side two-model comparison with an LLM-as-judge winner. |
| Scenario | A generated multi-turn conversation with a synthetic user persona, including expected tools and scoring constraints. |
| Simulation | A run of scenarios against your real agent via simulate_user / simulateUser, scored automatically and shown under Simulations. |
All spans are W3C-compatible — trace IDs and span IDs follow the W3C traceparent spec, so they interoperate with OpenTelemetry collectors and Jaeger, and propagate correctly across service boundaries. See Distributed Tracing.