Docs

    Frameworks

    LangChain and LangGraph

    Trace LangChain chains and LangGraph graphs in Python and TypeScript with no wrapper: graph nodes, chat model calls and tools become spans, with a tested tutorial agent in both languages.

    EvalKit traces LangChain and LangGraph in both SDKs without a wrapper. init() adds a callback handler to every LangChain run, so chains, graph nodes, chat model calls, tools and retrievers become spans with their inputs, outputs and token counts. This page builds the same support agent as a LangGraph graph in Python and in TypeScript, runs it once, and reads the trace.

    What you get

    • Each chain run is a function_call span named after it: LangGraph for a compiled graph, then one span per node, such as agent and tools.
    • In Python, each chat model call is an llm_call span named chat.generate, with the model, the messages as the prompt, the completion and the token counts.
    • In TypeScript, a chat model call that a traced provider client makes is recorded once, as that client's span, such as chat.completions.create, under the node it ran in. A model no provider patch records keeps the chat.generate span of LangChain's callback.
    • Each tool is a tool_call span named after the tool, with tool.arguments and tool.result. LangChain tools need no trace_tool or traceTool.
    • A retriever is a db_query span with retriever.query and retriever.document_count.
    • Spans inside a LangGraph node carry where they ran: langgraph.node, langgraph.step, langgraph.triggers (a JSON array in a string, such as ["branch:to:agent"]) and langgraph.checkpoint_ns.
    • In Python, the HTTP requests the OpenAI client sends show as POST spans at the top of the trace, beside the graph rather than under its model call.
    • In TypeScript, a graph run outside any span, in a script or a worker, opens a root span of its own named after the graph, so the run is one trace.
    LangGraph                    function_call
  agent                      function_call   langgraph.node=agent
    chat.generate            llm_call        tokens 28 in, 35 out
    tools_condition          function_call
  tools                      function_call   langgraph.node=tools
    lookup_order             tool_call
  agent                      function_call   langgraph.node=agent
    chat.generate            llm_call        tokens 44 in, 13 out
    tools_condition          function_call
POST                         http_call       the OpenAI client's request
POST                         http_call

    Install

    pip install syntropylabs-evalkit langchain-core langchain-openai langgraph

    Set up

    import os

import evalkit

evalkit.init(subscription_key=os.environ["EVALKIT_SUBSCRIPTION_KEY"], service_name="my-agent")

# Import LangChain after init(): every run from here on carries EvalKit's callback handler.
from langchain_openai import ChatOpenAI

    TypeScript also exports evalkit.langchainHandler(), which resolves to a callback handler you can pass in callbacks yourself. With init() in place you do not need it.

    Where init() goes

    • Python: call init() once, before LangChain is imported. It replaces the constructor of LangChain's callback manager, so every run started after it carries the handler, and it patches the OpenAI client at the same time.
    • TypeScript: call init() in a module of its own, tracing.ts, and import it first. ES modules run their imports in order, so init() runs before @langchain/* and openai are evaluated.
    • Create models and compile graphs after init(), at module level or later, never before it.

    Tutorial: a support agent as a LangGraph graph

    The graph has two nodes: agent calls the model with one tool, lookup_order, and tools runs the tool when the model asks for it. One question makes two model calls and one tool call.

    1. 1

      Install the packages

      Run the install line above for your language.

    2. 2

      Set the keys

      Export the environment key of your Environment and your OpenAI key in the shell that runs the agent.

    3. 3

      Save the code

      Python: agent.py. TypeScript: src/tracing.ts and src/agent.ts.

    4. 4

      Run it

      It prints the model's answer. Python sends the queued spans when the script exits; the TypeScript script flushes them before it ends.

    5. 5

      Open the trace

      In Traces, open the newest trace of the service langgraph-support or langgraph-ts-support.

    Shell
    export EVALKIT_SUBSCRIPTION_KEY=tk_live_...   # Settings → This project → Environments → Copy key
export OPENAI_API_KEY=sk-...
    agent.py
    import os

import evalkit

# init() runs before LangChain is imported, so its callback manager and the OpenAI client are patched.
evalkit.init(
    subscription_key=os.environ["EVALKIT_SUBSCRIPTION_KEY"],
    service_name="langgraph-support",
    environment="development",
)

from langchain_core.messages import HumanMessage, SystemMessage  # noqa: E402
from langchain_core.tools import tool  # noqa: E402
from langchain_openai import ChatOpenAI  # noqa: E402
from langgraph.graph import START, MessagesState, StateGraph  # noqa: E402
from langgraph.prebuilt import ToolNode, tools_condition  # noqa: E402


@tool
def lookup_order(query: str) -> dict:
    """Find an order from the customer's words."""
    return {"order_id": "1001", "status": "shipped", "arrives": "Thursday"}


llm = ChatOpenAI(model="gpt-4o-mini").bind_tools([lookup_order])  # reads OPENAI_API_KEY


def agent(state: MessagesState) -> dict:
    return {"messages": [llm.invoke(state["messages"])]}


graph = StateGraph(MessagesState)
graph.add_node("agent", agent)
graph.add_node("tools", ToolNode([lookup_order]))
graph.add_edge(START, "agent")
graph.add_conditional_edges("agent", tools_condition)  # to "tools" when the model asked for one, else end
graph.add_edge("tools", "agent")
app = graph.compile()

SYSTEM = "You are a support agent for an online shop. Use lookup_order for questions about an order."

evalkit.set_agent("support-bot")  # every trace this script records lists under this agent
with evalkit.session("conv_langgraph_1", user_id="u_42"):  # groups the turns of one conversation
    result = app.invoke({"messages": [SystemMessage(SYSTEM), HumanMessage("Where is my order 1001?")]})
print(result["messages"][-1].content)
    src/tracing.ts
    import evalkit from "syntropylabs-evalkit";

evalkit.init({
  subscriptionKey: process.env.EVALKIT_SUBSCRIPTION_KEY!,
  serviceName: "langgraph-ts-support",
  environment: "development",
});
    src/agent.ts
    import "./tracing.js"; // first import: init() runs before LangChain and OpenAI load
import { HumanMessage, SystemMessage } from "@langchain/core/messages";
import { tool } from "@langchain/core/tools";
import { MessagesAnnotation, START, StateGraph } from "@langchain/langgraph";
import { ToolNode, toolsCondition } from "@langchain/langgraph/prebuilt";
import { ChatOpenAI } from "@langchain/openai";
import evalkit from "syntropylabs-evalkit";
import { z } from "zod";

const lookupOrder = tool(async ({ query }) => ({ order_id: "1001", status: "shipped", arrives: "Thursday" }), {
  name: "lookup_order",
  description: "Find an order from the customer's words.",
  schema: z.object({ query: z.string() }),
});

const llm = new ChatOpenAI({ model: "gpt-4o-mini" }).bindTools([lookupOrder]); // reads OPENAI_API_KEY

const graph = new StateGraph(MessagesAnnotation)
  .addNode("agent", async (state) => ({ messages: [await llm.invoke(state.messages)] }))
  .addNode("tools", new ToolNode([lookupOrder]))
  .addEdge(START, "agent")
  .addConditionalEdges("agent", toolsCondition) // to "tools" when the model asked for one, else end
  .addEdge("tools", "agent")
  .compile();

const SYSTEM = "You are a support agent for an online shop. Use lookup_order for questions about an order.";

evalkit.setAgent("support-bot"); // every trace this script records lists under this agent
// The run is one trace, under a root span named after the graph. withSession groups the turns of one conversation.
const result = await evalkit.withSession({ sessionId: "conv_langgraph_ts_1", userId: "u_42" }, () =>
  graph.invoke({ messages: [new SystemMessage(SYSTEM), new HumanMessage("Where is my order 1001?")] }),
);
console.log(result.messages.at(-1)?.content);

await evalkit.flush(); // Node does not send queued spans on exit

    The TypeScript files are ES modules that use top-level await: give the project a package.json with "type": "module" (npm pkg set type=module) and run them with tsx.

    python agent.py

    Both print the model's answer, for example:

    text
    Order 1001 shipped yesterday and arrives on Thursday.

    Tested with syntropylabs-evalkit 0.3.5, langchain-core 1.6.10, langchain-openai 1.7.2 and langgraph 1.2.15 (Python: one trace of 11 spans), and @langchain/core 1.2.17, @langchain/openai 1.6.2 and @langchain/langgraph 1.5.0 (TypeScript: one trace of 9 spans, plus the printed answer as a log span of its own).

    See the trace

    • The waterfall shows the trees under What you get. Click a chat.generate span to read the messages, the completion and the token counts, and lookup_order to read the tool's arguments and result.
    • Traces → Sessions lists the conversation conv_langgraph_1 (Python) or conv_langgraph_ts_1 (TypeScript), and Traces → Agents lists support-bot.

    Sessions, users and agents

    • Python: name the agent with evalkit.set_agent("support-bot") and run the graph inside with evalkit.session(session_id, user_id=...), as the tutorial does. Every span inside the block carries the session and user, nodes and model calls included.
    • TypeScript: name the agent with evalkit.setAgent("support-bot") and run the graph inside evalkit.withSession({ sessionId, userId }, ...), as the tutorial does. Every span of the run carries the session, user and agent, the root span the SDK opens included.
    • In a web app, run the graph in the request handler and open the session there, as the page of your framework shows, such as FastAPI or Express.

    Pitfalls

    • TypeScript: each graph node lands in a trace of its own, on 0.3.4 and earlier. There a graph run outside a request has no root span, and in a test run its nodes and model calls were spread over 6 traces. Start a root span with evalkit.startTrace("support-turn", { sessionId, userId, agentName }) and run the graph inside evalkit.withTrace(turn.ctx, ...). From 0.3.5 the SDK opens the root span itself.
    • Python: the model's HTTP requests sit outside the graph. The POST spans of the OpenAI client hang off the top of the trace, not off chat.generate. Nothing is lost; read the request and response bodies there.
    • Python: log lines become spans; your log output stays as it was. Each record that propagates to the root logger is recorded as a log span, as a handler there would record it: at the level of the logger that wrote it, which is the root level, WARNING, unless that logger or your app sets another. init() changes no handler, formatter or level, so your console output stays as it was. Every record carries trace_id and span_id, so a format of your own can show them, as in %(asctime)s %(levelname)s [%(trace_id)s] %(name)s: %(message)s. On 0.3.4 and earlier, init() changes the format of the root logger's console handlers and sets the root level to INFO; configure logging after init() there.
    • TypeScript: a script ends before its spans are sent. Node does not send queued spans on exit. End a script with await evalkit.flush(), which never rejects (on 0.3.4 and earlier, add .catch(() => {})); Python flushes at exit on its own.

    Safe by design

    In Python:

    • Spans are queued in memory and sent from a background thread, so a request never waits for the trace service.
    • If the trace service is down or unreachable, your app answers as before. The SDK retries each export 3 times, logs evalkit: export failed after 3 retries as a warning and drops those spans.
    • If the key is wrong, the trace service refuses the spans. After 3 refusals in a row the SDK logs one warning and stops exporting until the process restarts; your app keeps answering.
    • A streamed model reply reaches your code chunk by chunk, unchanged. The SDK reads the same chunks on the side and writes the span when the stream ends.
    • Tracing never breaks the call it records. An error inside the SDK is logged at debug level and the call runs as if untraced, and an exception from the provider reaches your code unchanged.
    • A failed export never raises in your code, and evalkit.flush() returns normally too. At exit the SDK sends what is queued within 3 seconds in all, whether or not the trace service answers, so an outage never holds up a deploy.

    In TypeScript:

    • Spans are batched in memory and exported in the background, so a request never waits for the trace service.
    • If the trace service is down or unreachable, your app answers as before. The SDK writes [evalkit] export network error to the console and drops those spans.
    • If the key is wrong, the trace service refuses the spans and they are dropped; your app keeps answering.
    • A streamed model reply reaches your code chunk by chunk, unchanged, as the provider SDK's own stream object: tee(), withResponse(), finalMessage() and the other helpers work as without the SDK. The SDK reads the same chunks on the side and ends the span when the stream ends.
    • Tracing never breaks the call it records. An error inside the SDK's hooks is swallowed and the call goes on as if untraced, and the provider's own errors reach your code unchanged.
    • await evalkit.flush() never rejects: it resolves to true when the spans were sent and false when the export failed or init() has not run, so a trace service outage never fails a job or a serverless invocation. On 0.3.4 and earlier it rejects when the export fails; write await evalkit.flush().catch(() => {}) there.

    Next steps

    EvalKit is built by SyntropyLabs. Published on PyPI and npm.