Build a dataset from traces

After this page real failures from Traces are regression rows in a dataset, each linking back to the trace it came from, and the New run form already knows which evaluators can score them.

From the trace list

1. 1

   Select traces

   Filter (for example status:ERROR service:checkout), tick rows, then Add to dataset in the selection bar. Up to 200 traces per add; the dialog shows which are already in a dataset.

2. 2

   Choose the target

   An existing dataset (name · rows · type) or a new one. The dialog reads the dataset’s column layout.

3. 3

   Map fields

   input ← the first user message (or the last); output ← the final assistant reply (or none); target ← empty, to be filled later, or the final assistant reply when the trace is known good. Extra columns such as session, user and app version come along.

4. 4

   Preview and add

   Rows are built server-side from the traces’ llm_call spans. Traces with no user message are listed as skipped with the reason. Each new row is draft, records sourceTraceId (and the span), and shows a source link that opens the waterfall.

Afterwards

The dataset header notes the new rows and the changed hash — comparisons with earlier runs will need a fresh baseline, or a snapshot first. A banner offers Fill targets to enable reference-based evaluators and, on paid plans, Send to review to verify expected outputs. Opening Run evaluation preselects the dataset; evaluators that require a reference are marked will skip when rows have no target. A trace whose content was not captured cannot be added (there is no text to curate).

The same row builder is used by the automatic path — an auto-add rule that samples matching production traces into a dataset every few minutes, optionally through a review queue — described in Curate and maintain a golden dataset. Adding a whole session as one conversation row from the Sessions drawer is not wired yet.