Get started
Your first evaluation
Pick evaluators from the managed library, judge your recent traces with a Model, and read the scores that land on them.
Evaluating a trace runs evaluators over it, and an LLM judge needs a Model in your project to run on. A Model comes from a Provider, so a first evaluation usually starts with connecting one.
Evaluating traces is a plan feature (trace-evaluation), and so is the Evaluators library (evaluators). If Evaluators in the sidebar shows a lock, the panel it opens explains what the plan includes and lets you request an upgrade. See Plan, usage and locks.
1. Connect a Provider and add a Model
- 1
Connect a Provider
Open Settings in the sidebar footer, then Organization → Providers, and click Connect provider. Pick the vendor, paste the credential and click Test & save. The platform validates it with one real call and lists the models it discovered.
- 2
Add Models to the project
Open This project → Models and click Add models. The dialog lists the models of your Provider connections. Tick the ones you want in model dropdowns and add them.
A self-hosted provider that the platform cannot reach can be saved without the test. Turn on Skip validation, and the button becomes Save without testing.
An empty model picker anywhere in the app says no Model is configured for the project and links to project settings. Models and providers has the details.
2. Pick evaluators
Open Evaluators. The page has four tabs:
| Tab | What it holds |
|---|---|
| Agent | Managed evaluators for how an agent behaved over a run: its plan, turns, tool calls, handoffs and memory. This tab opens first. |
| Model | Managed evaluators for what a model wrote or generated, read on its own, such as whether an answer is grounded in the documents retrieved for it, whether it is safe or how a video holds up. |
| Custom | Evaluators written in your organization, for agents and models alike, the most recently changed first. |
| Collections | Named sets of evaluators, with New collection to create one. |
Each card shows the evaluator's name and what it checks. A "needs expected answer" pill marks one that is skipped on traces without an expected answer. Live traces have none, so pick evaluators without that pill for a first run.
A managed evaluator cannot be deleted. To change its prompt, open the card menu and click Duplicate. You do not need a collection of your own, because the Evaluate dialog takes a managed collection or evaluators picked one by one. See The managed library.
3. Evaluate your recent traces
- 1
Select traces
In Traces, tick the rows to score. The selection bar shows the count with Evaluate, plus Add to dataset, Annotate and Export.
- 2
Pick the judge and evaluators
Click Evaluate and choose the Judge model. Then pick a collection, or Choose individually… to pick evaluators one by one. Evaluators that read one trace are selected to start, and the others are labeled "reads the conversation".
- 3
Run the evaluation
Click Run evaluation. The dialog scores the traces one at a time and shows its progress. A trace that holds several conversations is scored once per conversation.
- 4
Read the scores
Click Done. The scores appear in the Score column of the trace list.
When you evaluate a single trace that is one turn of a session, the dialog offers to evaluate the whole session, which suits conversation evaluators better.
What a score cell can show
A score cell shows one of three things. A number is the score. No verdict, drawn dashed, means the judge could not decide, for example because the evaluator needs an expected answer and the trace has none. Judge error means the judge call failed. Neither is ever drawn as zero.
Open a trace's Evaluation tab for the detail: the overall score, whether it passed, the judge cost, when it was evaluated, and a row per evaluator with its score and the judge's reason. A score from an online rule says so. Re-evaluate scores the trace again.
What changed elsewhere
The checklist's last step is done, so the Getting started card leaves Home. Judge spend appears on Cost, separate from model spend.
To score live traces on a schedule, with sampling and a daily budget, set up an online rule. To score a dataset instead, start with Upload a CSV and Create a run.