EvalKit Docs

    Production Debugging

    Replay any failing conversation and see the exact prompt, response, tool calls, and latency at every step.

    Once init() is running in production (see Quickstart), every request your service handles is a trace, and every LLM call, tool invocation, HTTP request, and DB query inside it is a span — automatically, with no manual instrumentation to keep in sync with the code.

    1

    Find the request

    Search the dashboard by session ID, device ID, time range, or error status.

    2

    Open the waterfall

    See the full call tree — which LLM call ran, what prompt it received, what it returned, which tools it invoked and with what arguments, and where the time went.

    3

    Spot the fault

    A bad tool call, a malformed prompt, an unexpectedly slow downstream HTTP call, or a DB query that returned the wrong rows — all visible in one place, not scattered across log lines.

    4

    Fix and confirm

    Ship the fix, then use Scenario Simulation or a batch evaluation run to confirm the class of failure is gone before the next deploy.

    PreviousScenario SimulationNextRegression Detection

    EvalKit is built by Syntropylabs. Published on PyPI and npm.