Your first evaluation

After this page your recent traces carry scores with a reason per evaluator, Home shows a pass rate, and you know the one prerequisite most first evaluations trip over: a judge needs a Model, and a Model needs a Provider.

1. Connect a Provider and add a Model

A judge is a Model in your project. Models come from a Provider connected once on the organization: Settings in the sidebar footer → Organization → Providers → Connect provider, pick the vendor, paste the credential, Test & save (the platform makes a real call and lists the models it discovered). Then Project settings → Models → Add models and tick the ones you want in dropdowns. Any empty model picker in the app offers the same guided path and returns you where you were. Details in Models and providers.

2. Add evaluators

Open Evaluators. The Library tab lists the managed evaluators the platform seeds for every organization, filterable by category and scope; each card shows the kind, the scope (whether it needs a dataset row or a whole trace), its κ agreement badge once humans have labelled enough, and where it is used. From the checklist, Add to project creates a collection named after your project with three recommended evaluators preselected. You can also open a card, read what it checks and its threshold, and use it directly. See The managed library.

3. Evaluate your recent traces

1. 1

   Select traces

   In Traces, tick rows. The selection bar shows the count and Evaluate, Add to dataset, Annotate, Export.

2. 2

   Evaluate (n)

   Choose a collection or individual evaluators, the Judge model, and optionally a reference answer. Evaluators that require a reference are flagged, because live traces carry no golden answer. The dialog shows an estimated judge cost computed from the price sheet.

3. 3

   Run

   One evaluation request is made per selected trace and the table polls until each settles. Scores appear in the Score column as they finish.

4. 4

   Read the result

   A score cell is one of three things: a number, no verdict (dashed and neutral — the judge could not decide, for example a reference was required), or error (the judge call failed). A failed judge is never drawn as zero. Open a trace’s Evaluation tab for the overall score against its threshold, the source (manual or online), a row per evaluator with score, pass and the judge’s reason, the judge model and its cost.

What changed elsewhere

The checklist reaches five of five and Home switches to its health layout: traces, error rate, span p95 and ≈cost over the time range you pick (the last 24 hours by default, compared to the previous 24), a chart of spans, error spans and p95, and, when the window has model calls, tokens, models and time to first token. The judge spend appears on Cost as judge spend, separate from model spend. The evaluate dialog offers Make this continuous — the same evaluators on a schedule with sampling and a daily budget — which is an online rule. To score a dataset instead of live traces, start with Upload a CSV and Create a run.