Coder Eval — evaluate and benchmark AI coding agents and their skills

Playwright for coding agents — one declarative test file, any agent runtime, a real sandbox, and a pass/fail gate in CI.

Coder Eval (pip install coder-eval / uv tool install coder-eval) is an open-source, agent-agnostic framework for evaluating and benchmarking AI coding agents and their skills — built for benchmark authors, CLI builders, and skill builders — with sandboxing, reproducibility, and data-driven analysis. It runs a real agent — Claude Code, OpenAI Codex, Google Antigravity (Gemini), OpenCode, or Pi — in a sandbox against declarative YAML tasks, then scores the files and commands it actually produced. Changing harness is one field (agent.type); the tasks, criteria, scoring, telemetry, and reports stay the same.

Reach for it when you want to benchmark agents on your own domain tasks, test whether a skill triggers in the agent you ship for, A/B-test Claude Code vs. Codex vs. Gemini vs. OpenCode vs. Pi (or model vs. model, prompt vs. prompt), or gate CI on coding-agent quality. It is not a fixed leaderboard: unlike SWE-bench or SkillsBench, which rank models on a shared task set, you bring the tasks and you bring the scoring — weighted 0.0–1.0 criteria, a skill_triggered activation check, an A/B experiment layer, and per-tool cost telemetry, over whatever work you care about. See How it compares. 📚 Full docs: coder-eval.com/docs.

• Declarative YAML tasks with pinned dependencies and clear success criteria
• Sandboxed execution in isolated environments with resource limits
• Weighted, continuous scoring (0.0–1.0) with fractional credit and thresholds
• Many criterion types — from file checks to code similarity and LLM-graded rubrics
• Agent-agnostic by design — Claude Code, OpenAI Codex, Antigravity (Gemini), OpenCode, and Pi today; add your own harness through the plugin SPI
• Experiment layer — A/B agent configs (models, tools, prompts) side-by-side
• Full telemetry — every tool call, token counts, and cost, with real-time streaming

Watch the intro

🔊 Turn the sound on — GitHub's inline player always starts muted.

▶ Also on YouTube: Coder Eval: UiPath open-source framework to test AI Coding Agents — what the framework does, and how a run works end to end.

What you can do with it

• Benchmark coding agents — score an agent across a suite of tasks with weighted scoring and pass/fail thresholds
• Compare models & configs — A/B-test Claude vs. Codex vs. Gemini vs. OpenCode vs. Pi, model vs. model, tool-on vs. tool-off, prompt vs. prompt
• Evaluate skills — verify an agent actually engages a target skill (skill_triggered) and score skill-driven suites (SkillsBench-style), on whichever harness your users run
• Keep skills up to date in CI — re-validate your skills on every change or on a schedule; catch silent regressions when models, prompts, or the skills themselves drift
• Gate CI on agent quality — run the suite in GitHub Actions and fail the build on regressions
• Bring your own dataset — fan one task out over many rows for larger benchmark suites

Keeping skills fresh? Run Coder Eval as a scheduled GitHub Actions job so your skills are continuously re-evaluated against the latest model — a skill that quietly stops triggering surfaces as a failing criterion before your users hit it. See Tutorial 02 — Running Coder Eval in CI.

Quick Start

Prerequisites: Python 3.13+, uv 0.8+, and the runtime of at least one coding agent — plus that agent's own model credentials. Pick the agent you want to evaluate; two of the four ship with a Coder Eval extra, the other two are separate CLIs you install yourself:

Agent	agent.type	Runtime	Guide
Claude Code (default)	claude-code	brew install claude — separate CLI	Claude Code
OpenAI Codex	codex	uv sync --extra codex — the extra ships the Codex SDK + CLI	Codex
Google Antigravity (Gemini)	antigravity	uv sync --extra antigravity — the extra ships the harness binary	Antigravity
OpenCode (open-weight models)	opencode	npm install -g opencode-ai — separate CLI	OpenCode

The examples below use the default claude-code agent. Developed on macOS; CI runs on Linux.

git clone https://github.com/UiPath/coder_eval.git
cd coder_eval

uv sync                      # install the framework (add --extra codex /
                             # --extra antigravity for those runtimes)
cp .env.example .env         # then set ANTHROPIC_API_KEY — or skip that: an
                             # existing Claude Code login (`claude login`) is
                             # picked up automatically

uv run coder-eval plan tasks/hello_date.yaml   # validate (no tokens spent)
uv run coder-eval run  tasks/hello_date.yaml   # run your first evaluation
uv run coder-eval report runs/latest           # view the result

New here? Follow Tutorial 01 — Your First Evaluation.

Two more extras you only need on purpose: --extra dev adds the contributor toolchain (pytest, ruff, pyright, pre-commit — see CONTRIBUTING.md), and --extra uipath adds the in-host uipath SDK for local sandbox parity (public PyPI, no credentials). Without either, the framework still runs end-to-end.

Just want the CLI, without cloning? Install the pu