MCPSpec

Ship reliable MCP servers with confidence

Record sessions, generate mock servers, gate your CI pipeline, and catch Tool Poisoning — all without writing a single line of test code. The testing and reliability platform for Model Context Protocol servers.

---

Why MCPSpec?

Deterministic. Unlike LLM-based testing, MCPSpec runs are fast and repeatable. No flaky tests, no token costs. Ideal for CI.

Secure. Catch Tool Poisoning (prompt injection in tool descriptions) and Excessive Agency (destructive tools without safeguards) before they reach production.

Collaborative. Record server interactions once, share mock servers with your team. Frontend developers and CI pipelines can test against mocks — no API keys, no live dependencies.

---

Quick Start

# Install
npm install -g mcpspec

# Explore a server interactively
mcpspec inspect "npx @modelcontextprotocol/server-filesystem /tmp"

# Record a session — no test code needed
mcpspec record start "npx my-server"

# Generate a mock for your team
mcpspec mock my-recording --generate ./mocks/server.js

# Add CI gating in 30 seconds
mcpspec ci-init

Try it in 30 seconds with a pre-built community collection — no setup required:

mcpspec test examples/collections/servers/filesystem.yaml
mcpspec test examples/collections/servers/time.yaml --tag smoke

See all 70 community tests for 7 popular MCP servers.

---

Record & Mock

Record a session once. Replay it to catch regressions. Mock it for CI. No API keys required.

# 1. Record a session against your real server
mcpspec record start "npx my-server"
mcpspec> .call get_user {"id": "1"}
mcpspec> .call list_items {}
mcpspec> .save my-api

# 2. Replay against a new version — catch regressions instantly
mcpspec record replay my-api "npx my-server-v2"

# 3. Start a mock server — drop-in replacement, zero dependencies
mcpspec mock my-api

# 4. Generate a standalone .js file — commit to your repo
mcpspec mock my-api --generate ./mocks/server.js
node ./mocks/server.js

Replay output shows exactly what changed:

Replaying 3 steps...

  1/3 get_user (id=1)...       [OK] 42ms → {"name": "Alice"}
  2/3 list_items...            [CHANGED] 38ms → {"items": [...]}
  3/3 create_item (name=test)  [OK] 51ms → {"id": "abc"}

Summary: 2 matched, 1 changed, 0 added, 0 removed

Mock options:

Option	Effect
--mode match (default)	Exact input match first, then next queued response per tool
--mode sequential	Tape/cassette style — responses served in recorded order
--latency original	Simulate original response times
--latency 100	Fixed 100ms delay
--on-missing empty	Return empty instead of error for unrecorded tools
--generate <path>	Output standalone .js file (only needs @modelcontextprotocol/sdk)

Manage recordings:

mcpspec record list                    # List saved recordings
mcpspec record delete my-session       # Delete a recording

---

CI/CD Integration

ci-init generates ready-to-use pipeline configurations. Deterministic exit codes and JUnit/JSON/TAP reporters for seamless CI integration.

mcpspec ci-init                                 # Interactive wizard
mcpspec ci-init --platform github               # GitHub Actions
mcpspec ci-init --platform gitlab               # GitLab CI
mcpspec ci-init --platform shell                # Shell script
mcpspec ci-init --checks test,audit,score       # Choose checks
mcpspec ci-init --fail-on medium                # Audit severity gate
mcpspec ci-init --min-score 70                  # MCP Score threshold
mcpspec ci-init --force                         # Overwrite/replace existing

Auto-detects platform from .github/ or .gitlab-ci.yml. GitLab --force surgically replaces only the mcpspec job block, preserving other jobs.

GitHub Actions example (generated by mcpspec ci-init --platform github):

name: MCP Server Tests
on: [push, pull_request]

jobs:
  mcpspec:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '22'
      - run: npm install -g mcpspec
      - name: Run tests
        run: mcpspec test --ci --reporter junit --output results.xml
      - name: Security audit
        run: mcpspec audit "npx my-server" --fail-on high
      - name: Quality gate
        run: mcpspec score "npx my-server" --min-score 80
      - uses: mikepenz/action-junit-report@v4
        if: always()
        with:
          report_paths: results.xml

Exit codes:

Code	Meaning
0	Success
1	Test failure
2	Runtime error
3	Configuration error
4	Connection error
5	Timeout
6	Security findings above threshold
7	Validation error
130	Interrupted (Ctrl+C)

Reporters: console (default), json, junit, html, tap.

---

Security Audit

Prevent Tool Poisoning. 8 security rules covering traditional vulnerabilities and LLM-specific threats. A safety filter auto-skips destructive tools, and --dry-run previews targets before scanning.

mcpspec audit "npx my-server"                        # Passive (safe)
mcpspec audit "npx my-server" --mode active           # Active probing
mcpspec audit "npx my-server" --fail-on medium        # CI gate
mcpspec audit "npx my-server" --exclude-tools delete  # Skip tools
mcpspec audit "npx my-server" --dry-run               # Preview targets

Security rules:

Rule	Mode	What it detects
Tool Poisoning	Passive	LLM prompt injection in descriptions, hidden Unicode, cross-tool manipulation
Excessive Agency	Passive	Destructive tools without confirmation params, arbitrary code execution
Path Traversal	Passive	../../etc/passwd style directory escape attacks
Input Validation	Passive	Missing constraints (enum, pattern, min/max) on tool inputs
Info Disclosure	Passive	Leaked paths, stack traces, API keys in tool descriptions
Resource Exhaustion	Active	Unbounded loops, large allocations, recursion
Auth Bypass	Active	Missing auth checks, hardcoded credentials
Injection	Active	SQL and command injection in tool inputs

Scan modes:

• Passive (default) — 5 rules, analyzes metadata only, no tool calls. Safe for production.
• Active — All 8 rules, sends test payloads. Requires confirmation prompt.
• Aggressive — All 8 rules with more exhaustive probing. Requires confirmation prompt.

Active/aggressive modes auto-skip tools matching destructive patterns (delete_*, drop_*, destroy_*, etc.) and require explicit confirmation unless --acknowledge-risk is passed.

Each finding includes severity (info/low/medium/high/critical), description, evidence, and remediation advice.

---

Test Collections

Write tests in YAML with 10 assertion types, environments, variable extraction, tags, retries, and parallel execution.

name: Filesystem Tests
server: npx @modelcontextprotocol/server-filesystem /tmp

tests:
  - name: Read a file
    call: read_file
    with:
      path: /tmp/test.txt
    expect:
      - exists: $.content
      - type: [$.content, string]

  - name: Handle missing file
    call: read_file
    with:
      path: /tmp/nonexistent.txt
    expectError: true

Advanced features — environments, tags, retries, variable extraction, expressions:

schemaVersion: "1.0"
name: Advanced Tests

server:
  command: npx
  args: ["my-mcp-server"]
  env:
    NODE_ENV: test

environments:
  dev:
    variables:
      BASE_PATH: /tmp/dev
  staging:
    variables:
      BASE_PATH: /tmp/staging

defaultEnvironment: dev

tests:
  - id: create-data
    name: Create data
    tags: [smoke, write]
    timeout: 5000
    retries: 2
    call: create_item
    with:
      name: "test-item"
    assertions:
      - type: schema
      - type: exists
        path: $.id
      - type: latency
        maxMs: 1000
    extract:
      - name: itemId
        path: $.id

  - id: verify-data
    name: Verify created data
    tags: [smoke, read]
    call: get_item
    with:
      id: "{{itemId}}"
    assertions:
      - type: equals
        path: $.name
        value: "test-item"
      - type: expression
        expr: "response.id == itemId"

Assertion types:

Type	Description	Example
schema	Validate response structure	type: schema
equals	Exact match (deep comparison)	path: $.id, value: 123
contains	Array or string contains value	path: $.tags, value: "active"
exists	Path exists and is not null	path: $.name
matches	Regex pattern match	path: $.email, pattern: ".*@.*"
type	Type check	path: $.count, expected: number
length	Array/string length	path: $.items, operator: gt, value: 0
latency	Response time threshold	maxMs: 1000
mimeType	Content type validation	expected: "image/png"
expression	Safe expression eval	expr: "response.total > 0"

Expressions use expr-eval — comparisons, logical operators, property access, math. No arbitrary code execution.

Shorthand format for common assertions:

expect:
  - exists: $.field
  - equals: [$.id, 123]
  - contains: [$.tags, "active"]
  - matches: [$.email, ".*@.*"]

Run options:

mcpspec test ./tests.yaml              # Specific file
mcpspec test --env staging             # Switch environment
mcpspec test --tag @smoke              # Filter by tag
mcpspec test --parallel 4              # Parallel execution
mcpspec test --reporter junit --output results.xml
mcpspec test --baseline main           # Compare against baseline
mcpspec test --watch                   # Re-run on file changes
mcpspec test --ci                      # CI mode (no colors)

---

MCP Score

A 0-100 quality rating across 5 weighted categories with opinionated schema linting. Use as a CI gate or generate a badge for your README.

mcpspec score "npx my-server"
mcpspec score "npx my-server" --badge badge.svg       # Generate SVG badge
mcpspec score "npx my-server" --min-score 80          # Fail if below threshold

Scoring categories:

Category (weight)	What it measures
Documentation (25%)	Percentage of tools and resources with descriptions
Schema Quality (25%)	Property types, descriptions, required fields, constraints (enum/pattern/min/max), naming conventions
Error Handling (20%)	Structured error responses (isError: true) vs. crashes on bad input
Responsiveness (15%)	Median latency: <100ms = 100, <500ms = 80, <1s = 60, <5s = 40
Security (15%)	Findings from passive security scan: 0 = 100, <=2 = 70, <=5 = 40

Schema quality uses 6 sub-criteria: structure (20%), property types (20%), descriptions (20%), required fields (15%), constraints (15%), naming conventions (10%).

The --badge flag generates a shields.io-style SVG badge for your README.

---

More Features

Interactive Inspector

Connect to any MCP server and explore its capabilities in a live REPL.

mcpspec inspect "npx @modelcontextprotocol/server-filesystem /tmp"

Command	Description
.tools	List all available tools with descriptions
.resources	List all available resources (URIs)
.call <tool> <json>	Call a tool with JSON input
.schema <tool>	Display tool's JSON Schema input spec
.info	Show server info (name, version, capabilities)
.help	Show help
.exit	Disconnect and exit

Performance Benchmarks

Measure latency and throughput with statistical analysis across hundreds of iterations.

mcpspec bench "npx my-server"                         # 100 iterations
mcpspec bench "npx my-server" --iterations 500
mcpspec bench "npx my-server" --tool read_file
mcpspec bench "npx my-server" --warmup 10

Output includes min/max/mean/median/P95/P99 latency, standard deviation, and throughput (calls/sec). Warmup iterations (default: 5) are excluded from measurements.

Doc Generator

Auto-generate Markdown or HTML documentation from server introspection. Zero manual writing.

mcpspec docs "npx my-server"                   # Markdown to stdout
mcpspec docs "npx my-server" --format html      # HTML output
mcpspec docs "npx my-server" --output ./docs    # Write to directory

Web Dashboard

A full React UI for managing servers, running tests, viewing audit results, and more. Dark mode included.

mcpspec ui                    # Opens localhost:6274
mcpspec ui --port 8080        # Custom port
mcpspec ui --no-open          # Don't auto-open browser

10 pages: Dashboard, Servers, Collections, Runs, Inspector, Recordings, Audit, Benchmark, Docs, Score.

Baselines & Comparison

Save test runs as baselines and detect regressions between versions.

mcpspec baseline save main                      # Save current run
mcpspec baseline list                           # List all baselines
mcpspec test --baseline main                    # Compare against baseline
mcpspec compare --baseline main                 # Explicit comparison
mcpspec compare <run-id-1> <run-id-2>           # Compare two runs

Transports

MCPSpec supports 3 transport types for connecting to MCP servers:

Transport	Use case	Connection
stdio	Local processes	Spawns child process, communicates via stdin/stdout
SSE	Server-Sent Events	Connects to HTTP SSE endpoint
HTTP	Streamable HTTP	POST requests to HTTP endpoint

# stdio (default)
server:
  command: npx
  args: ["my-mcp-server"]

# SSE
server:
  transport: sse
  url: http://localhost:3000/sse

# HTTP
server:
  transport: http
  url: http://localhost:3000/mcp

---

Commands

Command	Description
mcpspec record start <server>	Record an inspector session — .call, .save, .steps
mcpspec record replay <name> <server>	Replay a recording and diff against original
mcpspec mock <recording>	Mock server from recording — --mode, --latency, --on-missing, --generate
mcpspec test [collection]	Run test collections with --env, --tag, --parallel, --reporter, --watch, --ci
mcpspec audit <server>	Security scan — --mode, --fail-on, --exclude-tools, --dry-run
mcpspec score <server>	Quality score (0-100) — --badge badge.svg, --min-score
mcpspec ci-init	Generate CI config — --platform github|gitlab|shell, --checks, --fail-on, --force
mcpspec inspect <server>	Interactive REPL — .tools, .call, .schema, .resources, .info
mcpspec bench <server>	Performance benchmark — --iterations, --tool, --args, --warmup
mcpspec docs <server>	Generate docs — --format markdown|html, --output <dir>
mcpspec compare	Compare test runs or --baseline <name>
mcpspec baseline save <name>	Save/list baselines for regression detection
mcpspec record list	List saved recordings
mcpspec record delete <name>	Delete a saved recording
mcpspec init [dir]	Scaffold project — --template minimal|standard|full
mcpspec ui	Launch web dashboard on localhost:6274

Community Collections

Pre-built test suites for popular MCP servers in examples/collections/servers/:

Collection	Server	Tests
filesystem.yaml	@modelcontextprotocol/server-filesystem	12
memory.yaml	@modelcontextprotocol/server-memory	10
everything.yaml	@modelcontextprotocol/server-everything	11
fetch.yaml	@modelcontextprotocol/server-fetch	7
time.yaml	@modelcontextprotocol/server-time	10
chrome-devtools.yaml	chrome-devtools-mcp	11
github.yaml	@modelcontextprotocol/server-github	9

70 tests covering tool discovery, read/write operations, error handling, security edge cases, and latency.

mcpspec test examples/collections/servers/filesystem.yaml
mcpspec test examples/collections/servers/time.yaml --tag smoke

Architecture

Package	Description
@mcpspec/shared	Types, Zod schemas, constants
@mcpspec/core	MCP client, test runner, assertions, security scanner (8 rules), profiler, doc generator, scorer, recording/replay, mock server
@mcpspec/cli	13 CLI commands built with Commander.js
@mcpspec/server	Hono HTTP server with REST API + WebSocket
@mcpspec/ui	React SPA — TanStack Router, TanStack Query, Tailwind, shadcn/ui

Development

git clone https://github.com/light-handle/mcpspec.git
cd mcpspec
pnpm install && pnpm build
pnpm test   # 334 tests across core + server

License

MIT