- Conduid
- Marketplace
- #evaluation
MCP servers tagged evaluation
171 live MCP servers tagged "evaluation", ranked by trust score. Tags come from package metadata and repository topics, so this list covers servers across every category that work with evaluation.
Codevetter
Verify AI-generated code with execution evidence — deterministic, local-first verification for coding-agent changes via a macOS app, CLI, and MCP server.
LLM Agent Harness Survey
模型决定上限，Harness 决定下限：LLM Agent 与 Harness 适配性研究综述 | A Survey on Model–Harness Adaptation (Agent = Model + Harness)
Gauntlet
A CI linter for what your MCP server publishes: tool poisoning and prompt injection across every surface an agent sees, definition drift between runs, and schema rot.
Memoturn
Open-source AI engineering platform — LLM observability, evals, metrics, prompt management, playground, and datasets. OpenTelemetry-native, self-hostable. Because "it worked when…
Locestra
Local-first AI orchestration for coding agents, models, memory and tools, with a versioned routing EvalKit and optional cloud escalation.
Autoresearch MCP Codex
Frozen Evaluation 기반 자율 반복 개선 AutoResearch MCP 에이전트 시스템 | AutoResearch MCP agent system for autonomous iteration with frozen evaluation
Shogym
Environments that meet an agent at the tool surface
Agentic Harness Engineering Workshop
Agentic-Harness-Engineering-Workshop
Multiclaw Harness
Regression harnesses for agentic AI workflows: tool-call assertions, trajectory diffs, budget checks, security fixtures, and CI gates.
Imstage
Synthetic conversations for non-commercial testing, learning and evaluation. Generic chat rendering with visible fictional-content labels.
TGS 2024042309 Develop AI Agents With OpenAI Agent Development Kit
WSQ OpenAI Agents SDK courseware with 227 visual slides and 10 self-contained Python activities covering tools, RAG, orchestration, MCP, guardrails, tracing, and evaluation.
Faultline
Failure-mode evaluation for MCP agents. Inject faults, observe trajectory, grade resilience with evidence. 5-layer grading, 18 fault types, replay.
Nocap
No evidence, no answer. 🧢 System One evidence gate for RAG & AI agents: Jev / Laya / any LLM judges before your LLM speaks. MCP server, Python SDK, CI route tests.
Rag Knowledge Base
Enterprise RAG knowledge base: ACL-filtered retrieval, backend-owned citations, fail-closed answers, evaluation gate. Spring Boot + pgvector + Flutter; local or cloud LLMs.
Ultimate Prompt Optimizer MCP
MCP server for prompt engineering: candidate prompts, LangGPT structuring, iterative optimization, promptfoo-style evals and DSPy-style train/dev search. Bring your own OpenAI-com…
Honeyrail
Open-source runtime for long-horizon, verifiable engineering work.
Gauntlet
A CI linter for what your MCP server publishes: tool poisoning and prompt injection across every surface an agent sees, definition drift between runs, and schema rot.
Finance Agent
LangGraph and MCP multi-agent prototype for structured A-share research with bounded evaluation and reflection.
Oh My Field
Field-fit agents to real work. Turn tacit know-how into reusable capabilities.
Kiboserve
Open-source Python framework to deploy AI agents via HTTP, A2A, and MCP with built-in observability
Nanohype
Template catalog and SDK for AI subsystems — agents, RAG, MCP servers, eval harnesses, infrastructure. Part of the nanohype ecosystem.
Advance Fraud Analyst
Multi-model fraud detector using Hugging Face + LangChain + MCP
AI Roadmap
AL/ML learning Gym including traditional ML, deep learning, fine-tuning, Gen AI, Agentic AI. Everything from experiments to production with evaluation and observability.
Toolkit Examples
🔧 Build safe and reliable tools for Model Context Protocol (MCP) servers to enhance AI agent interactions with external systems.
Fastevals
Give your AI agents an evaluation tool - provider-agnostic LLM eval runner with an MCP server for Claude, structured output validation, datasets, evaluators and honest cost metrics
Resultseal
Deterministic observation-integrity contracts for AI-agent tool results. HTTP 200 is not an observation.
Agent Bench
Measure whether an MCP server or skill actually improves your coding agent. A/B arms, deterministic gates, your own models. Negative results published.
Never Ask Twice
Enterprise Support MemoryAgent on Qwen Cloud that remembers customer context across sessions and proves it with a live re-ask-rate ablation.
Rag Eval MCP
Touchstone — know if you broke your RAG. An MCP server and CLI that score retrieval quality against a golden set: recall@k, MRR and hit rate, with no LLM calls.
Defense Bench
Vendor-neutral benchmark measuring how MCP security proxies/gateways DEFEND against 22+ attack vectors — crosswalked to NIST AI RMF & OWASP LLM/Agentic Top 10. CI-gated, reproduci…
Rag MCP Suite
RAG×MCPの自作スイート。検索品質を測定して判断する構成
Gq Insight MCP
MCP server for semantic search + grounded, citation-faithful answers over customer-research interviews, with a built-in eval harness. Built for Great Question.
Eval MCP Server
MCP server exposing an eval gate over all 3 primitives (tools, resources, prompts), stdio-ready.
Wasl AI
Scores whether a business is legible to AI agents on a 100-point index, then generates the MCP server that makes it legible. Deterministic scoring, cited evidence, published refus…
Makereval
MakerEval — agentic evaluations for Microsoft Copilot Studio agents: a remote MCP server plus client plugins (M365 Copilot Cowork today)
Kipimo
Swahili agent-task evaluation suite — model-agnostic seed benchmark for the East Africa coordination stack (46 tasks; golds machine-derived from authoritative sources).
Decision Memory Control Plane
Local-first control plane for evidence-linked decision memory, governed tasks, deterministic evals, and human-approved actions.
Cowork Bench
Multi-tool LLM agent benchmark: 496 end-to-end MCP tasks, deterministic evaluation, Russian localization
Think Tank Skill
🧠 Think-tank is a host-enhanced workflow skill framework that can be adapted across projects.
Awesome Agent Failures
Production failure modes for LLM agents: symptoms, reproductions, mitigations
Cloudguard
Reliability and safety harness for LLM cloud agents. Tool-selection accuracy vs. tool count, blast-radius guardrails, and prompt-injection red-teaming against a real AWS mock. Eve…
Pigeon Evals
A End-To-End RAG Pipeline that includes Evaluations, iterations, and swappable components. At its core it allows users to be able to try different embedding models and techniques.
VidWise AI
Research up to six YouTube videos with timestamped, claim-level citations.
TalentFabric AI
Five-agent certification readiness platform built on Microsoft Agent Framework, all three IQ layers (Foundry/Fabric/Work), and Microsoft Learn MCP — Agents League Hackathon.
Electripy AI
Open-source AI Application Runtime for reliable, observable, and governable production AI systems.
Compliance Watch
Contract and invoice compliance auditor built with Claude API. Hybrid RAG, tool use, MCP server, prompt evaluation, extended thinking. Full pipeline — not a chatbot.
Merakicore
Continuous learning for AI: evidence-backed corrections, scoped guidance, evaluation, and rollback.
Dew Flow Benchmark
Measure any code repository, at any commit, through any retrieval engine — with the sample sizes attached.
Hierarchy Multiple LLM With MCP
Hierarchical multi-LLM agent collaboration (Manager–Executor) with MCP; experiments + results.
CodeContextBench Dashboard
Benchmark platform for measuring how code-context tools (Sourcegraph MCP) affect coding-agent performance: adapter generation, local validation, GCP execution, and a Streamlit res…