Text Evaluation
Datasets, the Playground, evaluator collections, online evaluation, and batch runs.
Datasets
- Upload CSV files, configure input and target columns
- View and edit rows in-app
- Or generate a synthetic dataset from a prompt, right from the dashboard
Playground — Pointwise
- Single-model generation with full parameter control (temperature, system prompt, max tokens)
- File attachments — images, video, documents — for multimodal prompts
- Function tool definitions (JSON schema) and MCP server connections
- LLM-as-judge scoring per response: relevancy, groundedness, safety, custom metrics
Playground — Pairwise Comparison
- Run the same prompt against two models simultaneously
- LLM-as-judge picks a winner, with reasoning
- Full comparison history, per project
Online Evaluation
Attach an evaluator collection to a tracing project and every production trace is scored automatically as it arrives — no manual step between deploy and quality signal. Configure a polling interval (1–60 minutes) per project; auto-evaluated traces show a green auto badge next to their score.
Evaluator Collections
- Pre-built evaluators: relevancy, groundedness, coherence, toxicity, PII detection, custom
- Three rule types: model-graded, custom prompt, statistical
- Reusable collections — apply to any project, or set as a project default
Batch Evaluation
- Run generation + scoring across an entire dataset in one job
- Per-row status tracking:
pending→generating→evaluating→complete - Regenerate or re-evaluate individual rows without rerunning the whole job
For programmatic (in-code) scoring with no LLM cost, see Offline Evaluation —
evalkit.evaluate() runs deterministic checks against tool trajectories and constraints. For synthetic-user testing of a live agent, see Scenario Simulation.