skillcheck

Controlled A/B testing for AI agent skills, .cursorrules, CLAUDE.md, and system prompts — measure whether agent instructions actually improve model performance or just waste context tokens.

Not a syntax linter or basic assertion check — a controlled scientific experiment that measures statistical effect size.

Most published SKILL.md files, .cursorrules, and agent prompts have never been tested against an unprompted baseline. You cannot tell whether they help your model, make it worse, or are purely decorative prompt bloat. Skillcheck replaces vibe checks with empirical AI evaluation: paired control vs. treatment trials, blind grading, and bootstrap confidence intervals.

Point it at any Markdown skill or rule file and it runs an automated A/B experiment: it generates fresh domain-specific tasks, has the model solve each task with and without the skill injected, grades both arms blind, and reports the measured effect with a 95% bootstrap confidence interval and a 0–100 satisfaction score.

[!NOTE] Empirical AI Evaluation with Controlled A/B Testing

Evaluate your own agent skills, prompt instructions, and rule files against an unprompted baseline:

• Paired evaluation arms: Runs identical tasks with and without the skill injected.
• Blind grading: Evaluates outputs blind (the grader is blind to which arm produced an output, the subject model cannot be) with candidate order shuffled using a seeded hash.
• Bootstrap confidence intervals: 1,000 paired resamples produce statistical effect sizes and clear verdicts (HELPS, PLACEBO, or HARMS).

📊 View the Skillcheck Example Report and Format Demo →

https://github.com/user-attachments/assets/76f0c267-5e64-4de8-a336-714e070f36c1

$ skillcheck

✓ Evaluation tasks ready · 12s
✓ Trials complete (30/30) · 1m 38s
✓ Grading complete (30/30) · 41s
✓ Analysis complete · 2m 33s

╭────────────────────────────────────────────────────────╮
│ SKILLCHECK RESULT                                      │
├────────────────────────────────────────────────────────┤
│ Skill         API Documentation                        │
│ Run size      5 tasks × 3 trials                       │
│                                                        │
│ Verdict       HELPS                                    │
│ The skill HELPED — model passed 80% of tasks with it   │
│ vs 55% without.                                        │
│                                                        │
│ With skill    80.0% of tasks passed                    │
│ Without skill 55.0% of tasks passed                    │
│ Skill effect  +25.0 pp change in pass rate             │
│ Confidence    +8.0 pp to +42.0 pp (95% range)          │
│ Token cost    +480 tokens to include the skill         │
├────────────────────────────────────────────────────────┤
│ Satisfaction  ██████████████░░░░  75.0/100  GOOD       │
╰────────────────────────────────────────────────────────╯

Table of contents

• Install
• Quick start
• Supported formats & agents
• Why skillcheck
• Skillcheck vs. traditional linters
• How it works
• Architecture
• Commands
• CI/CD integration
• Badges for your repository
• Reading the result
• Effort levels
• Terminal experience
• Configuration
• Model choice
• Self-hosting
• Development
• Star history
• License

Install

npm install -g @sx4im/skillcheck@latest

# or run it without installing:
npx @sx4im/skillcheck@latest

Requires Node.js 20+. Works on Linux, macOS, and Windows.

Skillcheck checks npm for a newer version about once a day and offers to update (like the Codex and Gemini CLIs). Disable with SKILLCHECK_NO_UPDATE_CHECK=1.

Quick start

1. Instant 5-Second Demo (No API Keys Required)

Experience Skillcheck's interactive step tracker and satisfaction scorecard immediately:

npx @sx4im/skillcheck demo

2. Zero-Signup Evaluation (Bring Your Own Key)

If you already have any standard model API key set in your environment, Skillcheck runs out of the box with zero registration:

# Works instantly with OpenAI, Anthropic, Gemini, Groq, Mistral, OpenRouter, or NVIDIA NIM:
export OPENAI_API_KEY="sk-..."       # or ANTHROPIC_API_KEY, GEMINI_API_KEY, etc.
npx @sx4im/skillcheck check ./SKILL.md

3. Interactive Mode & Hosted Cloud (10 Free Checks Included)

Don't have API keys handy? Run the interactive assistant:

skillcheck

On first run without an existing key, it offers to use Skillcheck Cloud (grab a free token from skillcheck.page — includes 10 free cloud evaluations). Key entry is masked, verified securely before saving, and launches a full-screen terminal file picker: navigate folders with the arrow keys, pick any .md file, select an effort level, and watch the live progress tracker until the result card lands.

Point it straight at a file or folder:

skillcheck check ./SKILL.md
skillcheck ./my-skill-folder            # a folder containing a .md
skillcheck check ./SKILL.md --json      # machine-readable output
skillcheck check ./SKILL.md --output result.json

Fully headless in CI / scripts:

export SKILLCHECK_TOKEN=chk_live_...    # or OPENAI_API_KEY, etc.
skillcheck check ./SKILL.md --tasks 5 --trials 3 --json

Supported formats & agents

Skillcheck benchmarks any Markdown agent instruction, rule file, or prompt document:

Tool / Framework	File / Path Convention	Example Command
Cursor	.cursorrules, .cursor/rules/*.mdc	`skillcheck check ./.c