Build a dataset from traces

Rows built from traces turn real failures into regression cases, and each row keeps a link back to its trace.

From the trace list

1. 1

   Select traces

   On the Traces page, filter the list (for example status:ERROR service:checkout), tick the traces, then click Add to dataset in the selection bar, for up to 200 traces at a time.

2. 2

   Choose the dataset

   Pick an existing dataset, or create a new one with a name and a type.

3. 3

   Map the fields

   Input is the last user message (the default) or the first. Output is the final assistant reply, or left empty. Expected is left empty for you to verify later, or set to the final assistant reply for you to correct.

4. 4

   Add the rows

   Click Add rows. The server builds each row from the trace's llm_call spans, as a draft row with sourceTraceId set, and lists each trace it skipped with the reason.

Afterwards

New rows change the dataset hash, so comparing with earlier runs needs a new baseline, or a snapshot taken before the rows were added. In New run, evaluators that need a reference are marked needs an expected answer, and they skip rows without one.

An auto-add rule uses the same row builder: every five minutes it samples matching production traces into the dataset, optionally through a review queue. See Curate and maintain a golden dataset. A whole session cannot yet be added as one conversation row.