- Conduid
- Marketplace
- #llm-evaluation
MCP servers tagged llm-evaluation
213 live MCP servers tagged "llm-evaluation", ranked by trust score. Tags come from package metadata and repository topics, so this list covers servers across every category that work with llm-evaluation.
CLI Vs MCP
Harness comparing Claude Code performance across CLI skill, MCP server, and baseline tool surfaces. Playwright and GitHub experiments with N=5 runs.
Gutfeel
Usability testing for MCP servers. A decision model that doesn't deliberate gets a bare user mission and walks your tools for real: every path, wrong turn and dead end, mapped and…
Sec Recon Agent
Type-safe LLM vulnerability triage on Pydantic AI + MCP. Ten typed tools (NVD, OSV, KEV, EPSS, Exploit-DB, ATT&CK, SBOM, Nmap) ground a validated TriageReport with a deterministic…
Sentinelmcp
An automated red-teaming and reliability-auditing platform for AI agents - tests for prompt injection, tool hijacking and data exfiltration. Ships as an MCP server.
Checkpoint
The release gate for AI agents — run your real agent N times against stateful twins and ship only when the confidence interval says so.
Touchstone
A deterministic first-pass filter for unsupported claims in LLM output, computed without calling another LLM. Not a standalone hallucination detector. A spec plus reference implem…
Web Search Agent Evals
Evaluation harness for web-search agents: 4 agents x 2 tools matrix in isolated Docker containers, with deterministic and LLM-based grading
Vstack
AI agents fail like junior teammates, looping on bad ideas, ignoring feedback, and escalating commitment. vstack ports 34 of the most-cited organizational-behavior frameworks so y…
Longbook Rag Eval Lab
MCP-enabled evaluation and claim-grounding toolkit for long-document RAG systems - retrieval evaluation (5 methods), claim verification, a local FastAPI app, a local stdio MCP ser…
AgentRigor
Property-based reliability testing for tool-using AI agents under partial failures.
Foundation Eval Labs
A native macOS evaluation workbench for Apple's Foundation Models.
Litrag
Grounded RAG over PubMed abstracts with a built-in citation-faithfulness eval that flags hallucinated or unsupported claims. Live corpus ingestion via the PubCrawl MCP server. Lan…
Context Report
An open, signed report format for whether an agent plugin, hook, skill, or AGENTS.md actually works.
Mini Antemortem CLI
Deterministic, offline linter that catches silent traps in your prompt-eval setup — train/test leakage, judge bias, homogeneous variants — before they fake a passing score. 11 tra…
Boule
Lean, research-backed multi-LLM council with bias-aware judge controls (position-swap, verbosity-norm, stake-free judge) adapted from LLM-eval research.
Fluiq SDK
Instrument any LLM app in 2 lines. Tracing, evaluation, security, and caching for OpenAI, Anthropic, Gemini, LangChain, MCP, and more.
Awesome Verified Agents
Tools that produce evidence about what an AI agent actually did — gates, attestation, output verification, benchmarks. Inclusion bar: name the artifact a human can inspect later.
Synthetic Test Fabric
Generic engine for autonomous synthetic test loops: simulation → browser flows → scoring → feedback
Gauntlet
A CI linter for what your MCP server publishes: tool poisoning and prompt injection across every surface an agent sees, definition drift between runs, and schema rot.
Mini Omega Lock
Measure your LLM judge's noise floor — an A/B improvement smaller than the judge's own noise isn't real. Pre-flight probes for prompt-eval setups; works standalone and with omegap…
Skilltest
SkillTest: test framework and CI gate for AI agent skills (SKILL.md) — the engine behind skillemall.ai
Toolgym
Train AI agents on tools, validate real work, and publish portable mastery evidence
TracePad
LLM observability and evals in a single binary: traces, prompts, datasets, scores and review queues for LLM and agent apps. Self-hosted, SQLite inside, OpenTelemetry-native, MCP a…
Whetstone MCP
Make a cheap model perform like an expensive one on your MCP server — by changing the server, not the prompt. An Agent Skill that measures the weak/strong model gap and closes it…
Support Copilot
Customer-support AI agent: LangGraph supervisor + specialist handoffs, deterministic tools, human-in-the-loop refunds, guardrails, embedded eval gate, Langfuse traces, MCP. Fully…
Prompt Foo
Promptfoo is an open-source CLI and TypeScript/Node.js library for evaluating, red-teaming, and security-testing LLM applications, agents, and RAG pipelines. It runs deterministic…
Locestra
Local-first AI orchestration for coding agents, models, memory and tools, with a versioned routing EvalKit and optional cloud escalation.
Rewind Bench
A benchmark separating repairs that passed from repairs that held up. Bugs are injected by AST mutation; every accepted repair is re-run against a hidden second dataset
AQVEN
Python framework + local Studio for reliable LLM workflows: typed files in your repo, pre-run checks, experiments and evals, MCP server for coding agents
Consensus
A panel of frontier models debates your hardest question until it agrees, and tells you what it could not settle. Runs on your existing Claude/ChatGPT/Grok subscriptions. CLI · MC…
Ground Truth Evals
An LLM eval harness graded by computed ground truth, not an LLM judge. Worked example: poker, served to models as MCP tools — gpt-4o-mini goes from 26/75 to 75/75 with them.
ApexGraphSwarm
Local-first graph intelligence and multi-agent orchestration workbench with GraphRAG integrations, specialist swarm design, evaluation, and cost-aware delegation.
AI Redteam Notes
Pre-registered research on prompt injection through agent tool output, plus a dependency-free substrate auditor whose checks run in CI.
Sonata Labs
Clone a company into a fake Gmail, Slack and Calendar, run your agent through a simulated workday inside it, and score how much of the job it finished without a human.
Agent Reliability Toolkit
Test AI agents repeatedly, measure reliability, classify failures, and catch regressions before production.
Listing Claim Check
Checks AI-written marketplace listings against the seller's item specifics before they're published: PUBLISH or REVIEW, with the exact words behind every unbacked claim. Determini…
Smallprint Tools
Small Print's open tools: the disclosure lab, and soon the command, the MCP server and the plugin
Cadclamp
Benchmark for AI-generated parametric CAD, scored on whether the part can actually be printed
Groundwork Agents
Grounded multi-agent research assistant: LangGraph, MCP, hybrid RAG, evals, Azure/Terraform
Gemini Support Agent
Evidence-grounded IT support agent: Gemini function calling over an MCP-style tool server (KB search, ticket search, ticket drafts), with a 60-question eval harness, LLM judge, an…
Portfolio
AI / LLM engineering portfolio: approval-gated LangGraph agents, MCP, RAG and evaluations on local LLMs, built on 11+ years of automotive embedded software (CAN, AUTOSAR, UDS). ES…
Escalon
Grounded IT-support agent on Microsoft Agent Framework: hybrid search over Microsoft docs, MCP tools for Copilot Studio, Entra ID auth, ticketing, evals.
Local Code Navigator
Symbol-aware and semantic code navigation for Claude Code and Codex, plus the ablation harness that measures whether it actually helps.
ClauseGuard
RAG + LLM system for first-pass German contract compliance screening: hybrid BGB/DSGVO retrieval (RRF, cross-encoder reranking), LoRA fine-tuned DistilBERT clause classifier, evid…
Gauntlet
A CI linter for what your MCP server publishes: tool poisoning and prompt injection across every surface an agent sees, definition drift between runs, and schema rot.
Insureassist
Agentic WhatsApp customer-service assistant (Python, Claude API, MCP server) with guardrails, human handoff and eval-gated releases. Planning pack.
Rag Forge
Production-grade RAG pipelines with evaluation baked in
Groundwork
An agent that answers questions about AI engineering from a knowledge base about AI engineering, so every answer is also a worked example of the retrieval that produced it. LangGr…
Fde Guide
A field guide for FDEs and applied AI teams: value engineering and production architecture.
Grandjury
Pluralistic human evaluation infrastructure for AI in production