Offline Evaluation

Deterministic, in-process scoring — no LLM cost, no network round trip.

Score an output locally against expected tool calls and constraints; the result is pushed as an evaluation span next to your trace, and also shown in the trace’s Evaluation tab.

scores = evalkit.evaluate(
    output=agent_reply,
    expected_tools=["search", "summarize"],
    tool_calls=[{"name": "search"}, {"name": "summarize"}],
    constraints={"required_terms": ["citation"], "has_citations": True},
)   # → {"tool_trajectory": 1.0, "tool_f1": 1.0, "response_match": 1.0, ...}

This is the same scorer simulate_user calls automatically after each scenario turn — see Scenario Simulation for the multi-turn version, and Selecting a judge model for when to reach for LLM-as-judge instead.