AgentTrace

Record, replay and diff LLM agent runs. A from-scratch Myers diff over agent steps finds the first step where two runs part ways, and replay pins tool responses so a run can be re-checked offline. Works with LangGraph and as an MCP proxy, at 8.6 µs of overhead per recorded step.

Built with Python, LangGraph, MCP, React, TypeScript, and 1 more.

AgentTrace records model calls, tool calls and graph nodes to JSONL from three places: a LangChain/LangGraph callback, an MCP proxy that sits between a client and any MCP server over stdio or streamable HTTP, or a plain Recorder API with a @traced decorator, per-thread binding and a redaction hook. Two recorded runs are aligned with a Myers diff written from scratch over step signatures, which pairs removed and added steps into "changed" ones and skips container nodes whose output differs only because a child's did, then reports the first divergence with prompt, argument and output deltas. Because providers assign random tool-call IDs, calls are recorded by position so diffs aren't polluted by ID churn. Replay comes in two modes: tools pinned to their recorded responses with a live model, or fully offline, where strict mode raises a ReplayDivergence with a prompt diff the moment the agent strays; tool arguments are matched after normalising them through each tool's schema, and a recorded MCP server can be replayed without starting it. The diff command has JSON output and exits non-zero when runs differ, so it works as a CI gate, and traces export to OTLP for Jaeger or Tempo. A React viewer shows two runs as a forked rail, and its TypeScript diff is tested against the Python implementation. Overhead is measured with paired, alternating benchmark runs: 8.6 µs per recorded step through the Recorder API and 35.2 µs through the LangGraph callback, 0.39 ms on a 1.6 ms run. In the committed case study (one scenario on llama3.1:8b), two runs first diverge at step 5 of 7, at a lookup_order tool call, and all six replay controls pass. There are 340 tests, 300 of them seeds of a property test that checks the diff's minimality against brute-force longest-common-subsequence. Built by Archit with Claude as the AI coding assistant.

• Three ways to record: a LangGraph callback, an MCP proxy (stdio and streamable HTTP), or the Recorder API with @traced, thread binding and a redaction hook
• First-divergence diff with prompt, argument and output deltas, JSON output, and a non-zero exit code for CI
• Tools-only replay with a live model, or fully offline replay where strict mode raises on the first divergence
• Replays a recorded MCP server without starting it
• OTLP export using the GenAI semantic conventions, for Jaeger or Tempo
• React 19 viewer that draws two runs as a forked rail, its diff tested against the Python implementation

Every project hits walls. Here's what nearly broke AgentTrace — and how it shipped anyway.

• Provider tool-call IDs are random, so steps are recorded by position to keep diffs meaningful
• Finding the true first divergence: step signatures, pairing removed and added steps, and ignoring parent nodes that differ only because a child did
• Matching replayed tool arguments through each tool's pydantic schema so harmless formatting differences don't count
• MCP over HTTP: request IDs scoped per request and server-sent events decoded incrementally
• Measuring overhead credibly with paired, alternating benchmark runs instead of single timings

Every build leaves residue. The takeaways from AgentTrace that carry forward.

• Agent regressions usually start several steps before the wrong answer; diffing the trajectory finds them faster than reading transcripts
• A property test against a brute-force oracle is the cheapest way to trust a hand-written diff algorithm
• Determinism has to be designed in: random IDs and timestamps need recording by position and redaction hooks
• Limits stated plainly: traces are plain files (no server or search), integrations are LangGraph and MCP only, and the case study is one scenario