mcp-quality-gate

Quality gate for MCP servers. Like npm audit for packages, but for Model Context Protocol servers.

When an LLM connects to your MCP server, it trusts whatever you expose. Bad tool schemas mean bad tool calls. Missing descriptions mean the model guesses. 50+ tools flood the context window. Leaked environment variables expose secrets. mcp-quality-gate catches all of this in one command.

npx mcp-quality-gate validate "npx -y @modelcontextprotocol/server-filesystem /tmp"

One command. Four dimensions. 0-100 score.

What It Catches

mcp-quality-gate scores every MCP server across four dimensions:

Dimension	Weight	What It Checks	Why It Matters
Compliance	40 pts	Protocol conformance — init, tool listing, tool calls, resources, prompts, error handling	A server that doesn't follow the spec breaks every client
Quality	25 pts	Parameter descriptions, description length, deprecated tools, duplicate schemas, schema consistency	LLMs need good descriptions to make correct tool calls. 72% undocumented params = 72% guesswork
Security	20 pts	Environment variable exposure, code execution surfaces, destructive operations without warnings	Tools run with user permissions. A get-env tool leaks every secret on the machine
Efficiency	15 pts	Tool count, total schema token cost	Every tool schema eats context. 21 tools at 3000 tokens leaves less room for actual conversation

Install

npm install -g mcp-quality-gate

Requires Node.js >= 22.

Usage

# Test any stdio MCP server
mcp-quality-gate validate "npx -y @modelcontextprotocol/server-filesystem /tmp"

# Test with environment variables
mcp-quality-gate validate "npx -y @supabase/mcp-server-supabase@latest --read-only --project-ref REF" \
  --env "SUPABASE_ACCESS_TOKEN=your-token"

# JSON output for CI/CD pipelines
mcp-quality-gate validate "./my-server" --reporter json --output report.json

# Fail CI if score is below threshold
mcp-quality-gate validate "./my-server" --threshold 80

# Test HTTP/SSE servers
mcp-quality-gate validate "http://localhost:3000/mcp" --transport http

Real-World Benchmarks

Tested against official MCP reference servers (April 2026). These are real results from live server connections, not synthetic data:

Server	Score	Compliance	Quality	Efficiency	Security	What mcp-quality-gate Found
Memory	98	40/40	23/25	15/15	20/20	50% of parameters have no descriptions — LLMs have to guess argument format
Sequential Thinking	98	40/40	23/25	15/15	20/20	500+ character description — wastes context tokens on a single tool
Everything	88	40/40	23/25	15/15	10/20	get-env tool leaks environment variables. Duplicate schemas across tools
Filesystem	81	40/40	11/25	15/15	15/20	72% of params undocumented, read_file marked deprecated but still listed, duplicate schemas
Playwright	81	40/40	19/25	12/15	10/20	21 tools consuming 3000+ schema tokens, code execution surfaces, short descriptions

Servers tested: @modelcontextprotocol/server-memory, @modelcontextprotocol/server-sequential-thinking, @modelcontextprotocol/server-everything, @modelcontextprotocol/server-filesystem, @anthropic/mcp-server-playwright.

Example Output

mcp-quality-gate v0.1.0
Server: npx -y @modelcontextprotocol/server-filesystem /tmp

  lifecycle
    PASS Server reports name and version (0ms)
    PASS Server reports capabilities (0ms)
    PASS Server responds to ping (1ms)

  tools
    PASS Server lists tools without error (5ms)
    PASS Tool definitions have required fields (6ms)
    PASS Tool names follow naming convention (8ms)
    PASS Tool inputSchema has type object (4ms)
    PASS Can call a listed tool (10ms)
    PASS Calling nonexistent tool returns error (1ms)
    PASS Tool descriptions are present (8ms)

  resources
    SKIP Server lists resources without error
    SKIP Resource definitions have required fields
    SKIP Resource descriptions are present
    SKIP Can read a listed resource

  prompts
    SKIP Server lists prompts without error
    SKIP Prompt definitions have required fields
    SKIP Can get a listed prompt

  efficiency
    14 tools, ~3057 schema tokens

  quality
    Param description coverage: 28%
    Deprecated: read_file
    Duplicates: read_file, read_text_file
    CRIT 18 of 25 parameters lack descriptions (72%)
    CRIT 1 deprecated tool(s) still listed: read_file
    WARN Tools with identical schemas: read_file, read_text_file

  security
    WARN "write_file" performs destructive operations — description warns of risk

Results: 10 passed, 7 skipped (45ms)
Score: 81/100
  compliance 40/40 | quality 11/25 | efficiency 15/15 | security 15/20

Compliance Tests (17)

mcp-quality-gate connects to your server, makes real protocol calls, and verifies behavior. This is not static analysis — it's a live test suite that actually calls your tools with generated arguments.

Category	Tests	What's Verified
Lifecycle	3	Server init (name, version, capabilities), ping response
Tools	7	Tool listing, required fields, naming conventions, schema structure, live tool invocation with auto-generated arguments, error handling for nonexistent tools, description presence
Resources	4	Resource listing, required fields, descriptions, resource read
Prompts	3	Prompt listing, required fields, prompt retrieval

Tests are skipped (not failed) when a server doesn't advertise a capability. A tools-only server won't lose points for missing resources.

Full Test Reference

ID	Test	Severity
lifecycle-init-01	Server reports name and version	Critical
lifecycle-init-02	Server reports capabilities	Critical
lifecycle-init-03	Server responds to ping	High
tools-list-01	Server lists tools without error	Critical
tools-list-02	Tool definitions have required fields	Critical
tools-list-03	Tool names follow naming convention	Medium
tools-list-04	Tool inputSchema has type "object"	High
tools-call-01	Can call a listed tool	Critical
tools-call-02	Calling nonexistent tool returns error	High
tools-call-03	Tool descriptions are present	Medium
resources-list-01	Server lists resources without error	Critical
resources-list-02	Resource definitions have required fields	Critical
resources-list-03	Resource descriptions are present	Medium
resources-read-01	Can read a listed resource	Critical
prompts-list-01	Server lists prompts without error	Critical
prompts-list-02	Prompt definitions have required fields	Critical
prompts-get-01	Can get a listed prompt	Critical

Quality Analysis

Checks how well your tool schemas help LLMs understand and use your tools:

Check	What It Catches
Parameter description coverage	Parameters without descriptions — the #1 cause of incorrect LLM tool calls
Description quality (short)	Descriptions under 20 characters — too brief for LLMs to understand intent
Description quality (verbose)	Descriptions over 500 characters — wastes context tokens
Deprecated tool detection	Tools marked deprecated that are still listed — confuses tool selection
Duplicate tool detection	Tools with identical input schemas — suggests redundant or versioned tools
Required/default mismatch	Required parameters with default values — contradictory schema signals

Security Analysis

Static analysis on tool definitions to detect common security antipatterns:

Check	What It Catches
Environment variable exposure	Tools like get-env that leak secrets to the LLM
Code execution detection	Tools accepting code, script, or eval parameters — arbitrary execution surfaces
Dangerous default patterns	Destructive operations (write, delete, drop) without proper warning descriptions

Efficiency Analysis

Catches tool proliferation and schema bloat — the top causes of poor LLM performance with MCP servers:

Metric	Warning	Critical	Why
Tool count	> 20	> 50	More tools = more tokens in every request = less room for conversation
Schema tokens	> 10,000	> 30,000	Token budget is finite. Schema overhead competes with actual content

Token estimation uses chars/4 heuristic (~15% accuracy vs tiktoken for JSON schemas).

Scoring

Composite 0-100 score. Each dimension starts at its maximum and deducts for findings:

Dimension	Max	Deductions
Compliance	40	(passed / total_run) * 40
Quality	25	-5 per critical, -2 per warning
Efficiency	15	-8 per critical, -3 per warning
Security	20	-10 per critical, -5 per warning

Skipping a dimension with --skip-* flags means those points are not awarded. A server with --skip-security can score at most 80.

CLI Reference

Flag	Description	Default
-t, --transport	Transport type (stdio or http)	stdio
-r, --reporter	Output format (console or json)	console
-o, --output	Write report to file
--threshold	Minimum passing score (0-100) — exit 1 if below
--timeout	Test timeout in ms	30000
--skip	Comma-separated test IDs to skip
--only	Comma-separated test IDs to run
-e, --env	Environment variables as KEY=VAL,KEY2=VAL2
--max-tools	Critical threshold for tool count	50
--max-schema-tokens	Critical threshold for schema tokens	30000
--skip-efficiency	Skip efficiency analysis
--skip-quality	Skip quality analysis
--skip-security	Skip security analysis

CI/CD Integration

Add to your GitHub Actions workflow:

- name: Test MCP Server
  run: npx mcp-quality-gate validate "./my-server" --threshold 80 --reporter json --output mcp-quality-gate-report.json

mcp-quality-gate exits with code 1 when the score falls below --threshold, failing the CI step.

Programmatic API

import {
  createMCPClient,
  listAllTools,
  runTests,
  complianceTests,
  analyzeEfficiency,
  analyzeQuality,
  analyzeSecurity,
  ConsoleReporter,
} from "mcp-quality-gate";

const client = await createMCPClient({
  command: "node",
  args: ["./my-server.js"],
  transport: "stdio",
});

const tools = await listAllTools(client);
const efficiency = analyzeEfficiency(tools);
const quality = analyzeQuality(tools);
const security = analyzeSecurity(tools);

const result = await runTests(
  complianceTests,
  { client, timeout: 10000 },
  undefined,
  "my-server",
  efficiency,
  quality,
  security,
);

console.log(new ConsoleReporter().format(result));
await client.close();

Architecture

mcp-quality-gate
├── CLI (Commander)         → parse args, orchestrate
├── MCP Client Wrapper      → connect via stdio or HTTP, manage lifecycle
├── Compliance Tests (17)   → live protocol verification
│   ├── Lifecycle (3)       → init, capabilities, ping
│   ├── Tools (7)           → list, fields, naming, schema, call, errors, descriptions
│   ├── Resources (4)       → list, fields, descriptions, read
│   └── Prompts (3)         → list, fields, get
├── Quality Analyzer        → param descriptions, description length, deprecated, duplicates
├── Security Analyzer       → env exposure, code execution, dangerous defaults
├── Efficiency Analyzer     → tool count, schema token estimation
├── Score Calculator        → 4-dimension weighted composite (40+25+15+20=100)
└── Reporters               → console (colored), JSON (CI/CD)

Releases

mcp-quality-gate follows Semantic Versioning:

• 0.x.y — pre-1.0, API may change between minor versions
• PATCH (0.x.Y) — bug fixes, new compliance tests, doc updates
• MINOR (0.X.0) — new analyzer dimensions, new reporters, CLI flags
• MAJOR (X.0.0) — breaking API changes, scoring formula changes

How Releases Work

1. Bump version in package.json
2. Update CHANGELOG.md with the new version entry
3. Merge to main
4. CI automatically: runs lint + test + build, publishes to npm with provenance, creates a GitHub Release with tag v{version}

The prepublishOnly script runs lint && build && test as a safety gate. See CONTRIBUTING.md for full release instructions.

Roadmap

• v0.1 — Compliance tests (lifecycle, tools, resources, prompts), quality + security + efficiency analysis, 4-dimension scoring, CI/CD workflows
• v0.2 — Transport compliance tests (HTTP/SSE edge cases), response schema validation, capability refusal tests
• v0.3 — mcp-quality-gate init scaffolding, GitHub Action for CI, performance benchmarking
• v1.0 — Dynamic security testing, MCP server registry scanner, stable API

Contributing

See CONTRIBUTING.md for development setup, code standards, and how to add tests.

Security

See SECURITY.md for reporting vulnerabilities.

Code of Conduct

See CODE_OF_CONDUCT.md.

License

MIT