- Conduid
- Marketplace
- #llm-evaluation
MCP servers tagged llm-evaluation
213 live MCP servers tagged "llm-evaluation", ranked by trust score. Tags come from package metadata and repository topics, so this list covers servers across every category that work with llm-evaluation.
Agent Prod
Production AI agent quality gate and risk control framework for LLMOps, agent evaluation, regression detection, gray release, audit, and observability
AI Failure Observatory
Behavioral safety, vulnerability probing, and product-risk auditing platform & MCP server for Generative AI & Large Language Models.
Omegaprompt
The overfit gate for your prompts: re-test the winning prompt on held-out examples it never tuned on, and block the ship if it doesn't generalize. Sits on top of promptfoo/DSPy. C…
Github Tool Eval
FastMCP GitHub server with confirm-gated destructive tools and a 30-case evaluation harness scoring tool selection, argument accuracy, and confirmation safety.
Hospitality Multi Agent Architecture
Multi-agent AI system for hotel operations: F&B forecasting, guest memory, environmental risk. MCP-native, FastAPI, WhatsApp delivery, evals as CI merge gates.
LLM Chess Evaluation Harness
LLM Chess Evaluation Harness ♟️
Groundtruth Geo
Deterministic, government-cited, reproducible benchmark of parcel-precise US property facts — on the exact questions frontier models fail. Benchmark + grader + MCP server.
Multi Judge Consensus
Cross-vendor multi-model review committee that gates AI/agent-generated content before it ships — consensus-based hallucination defense. Zero dependencies, local-first CLI, MCP se…
Maverik
Benchmark, compare, and cost-predict your MCP agents — JMeter for AI agents.
Ghostlab
Local end-to-end testing lab for any MCP server: coding agents role-play real users, drive your tools over multiple turns, score runs, and render/interact with MCP Apps ui:// widg…
Aletheia Paper
Aletheia: A Taxonomy and Empirical Evaluation of Behavioral Failure Patterns in Large Language Models
SQL Specialist MCP
Fine-tuned NL-to-SQL specialist served as an MCP tool, benchmarked against prompting Claude directly on accuracy, latency, and cost
Shopify Support Agent
AI customer support agent for a Shopify store. LangGraph state machine, self-built MCP server over the Admin GraphQL API, RAG with citations, layered guardrails, and an eval harne…
Multivon MCP
MCP server exposing multivon-eval + pdfhell as agent-callable tools. Drop into Claude Desktop, Cursor, Cline.
Ticketpilot
AI support-ticket triage agent that cites its sources, refuses what it shouldn't answer, and is gated by evals in CI. Kotlin · Spring Boot · Koog agent graphs · hybrid RAG on pgve…
Agent Production Patterns
Production patterns for MCP servers, LangGraph agents, tool calling, evaluation, and AI system reliability.
Data Warden
Agente NL→SQL sobre un lakehouse con cinco anillos de control: guard de allowlist sobre el AST, presupuesto de coste estimado antes de ejecutar, enmascarado por rol y auditoría. S…
Riposte
A DSL for Pokémon battle policies (Rust compiler, Python runtime, MCP server), built to measure which steering strategy best teaches a coding agent a language it has never seen. G…
Kipimo
Swahili agent-task evaluation suite — model-agnostic seed benchmark for the East Africa coordination stack (46 tasks; golds machine-derived from authoritative sources).
Openbb Workspace Bench
Benchmark for agents operating OpenBB Workspace through MCP
Truthlens
AI hallucination detector with formally verified trust scoring for LLM outputs. No API keys needed.
Grounded Legal Agent
A Gemini agent that reviews legal cases safely — real CourtListener case law only, no advice, safety-scored in Arize Phoenix.
Tracebench
RCAgentBench: a benchmark measuring how tool design and agent instructions affect LLM agents doing root cause analysis on distributed traces.
Kaos599
Profile README for Harsh Dayal (Kaos599) AI Engineer at Spectatr AI building multi-agent systems, production RAG pipelines, and LLM evaluation frameworks with LangGraph, MCP, and…
Multi Agent System A2a MCP
A production-style multi-agent system that generates context-aware briefs. Built on MCP + A2A, with persistent memory (Chroma), dynamic tool routing, self-critique, distributed tr…
Oh No My Claudecode
The coding harness that measures itself. Wraps Claude Code / Codex / OpenCode, blocks false "done" with an independent verifier, mediates side effects, writes tamper-evident recei…
Auto Ableton
MCP Server that connects your AI assistant to Ableton Live 12 via the OSC protocol. Compatible with Claude Code, Claude Desktop, ChatGPT/Codex, and Cursor.
Tech Radar Agent
Local AI-agent research copilot: hybrid retrieval (E5 + BM25 + RRF + cross-encoder), bounded ReAct agent with deterministic citation validation, FastAPI + React chat UI, and a rea…
Deep Dive Capstone
Codebase Q&A with (path:line) citations — the capstone of a from-scratch AI-engineering series. Built in 8 eval-gated stages (chat → RAG → agents → hardening → production), provid…
XOne
XOne - The Open Agentic Studio
Yashpatil582.github.io
Portfolio with 3 live AI demos: streaming RAG chat, MCP repo-explorer agent, clinical hallucination eval. Next.js 16 + pgvector + Groq.
Deeplearning.ai
Short courses from DeepLearning.AI on topics on Generative AI , new tools and skills.
Agent Eval Service
Scenario-driven framework for evaluating demographic bias in single- and multi-agent LLM workflows. Tracks bias across sequential stages, separates model non-determinism from syst…
Humanloopbench
Human-in-the-loop LLM Failure Detection & Benchmark Toolkit — Auto-flag LLM failures (hallucination, refusal, factual error) with LLM-as-Judge explanations. Confirm them in a revi…
Kg Agent Evals
Governed GraphRAG over a provenance-tracked biomedical knowledge graph, with an eval harness measuring whether the agent stays faithful to the evidence.
Product Management
A PM's operating system: turn a PRD into a full SDLC by orchestrating MetaGPT, OpenHands, Promptfoo. Built by a Senior PM for regulated AI (BFSI · Healthcare).
Kickoff Spring AI
Hands-on Spring AI tutorial, themed as a World Cup 2026 fan assistant: 16 modules from chat clients to RAG, guardrails, and evaluation.
Agent Orchestration Core
Self-hostable, multi-tenant orchestration engine for AI agents. Every run is pinned to the exact versioned definitions and policies that governed it.
Pathfinder Rag
Scraper, hybrid BM25/vector retrieval, MCP server, and a multi-hop agent client over the Pathfinder 1e ruleset. Measured end to end.
HumanWrite X MCP Server Adaptive LLM Based Content Humanization System
a MCP server for professional-grade neuro-stylistic content humanization engine designed to neutralize commercial AI detectors. By integrating advanced stylometric entropy, syntac…
Copilot Studio Simulator
Unofficial local Copilot Studio simulator: declarative agents (topics, tools, MCP, code execution, knowledge, child agents) + evals + auto-tuning, built on the pi harness SDK
Ferryman MCP
A local-first MCP host with pluggable skills, multi-provider routing, and multi-channel I/O. Kotlin. Plus a Python eval harness with rule + LLM-judge scorers over a human-authored…
Super MCP Eval
Evaluation harness for MCP servers and the agents that use them: schema compliance, tool-selection accuracy with Wilson 95% confidence intervals, DuckDB persistence, Streamlit das…
Aobench
Role-aware, permission-enforced benchmark for LLM agents that operate HPC systems — SLURM, telemetry, RBAC, docs, facility. A policy violation hard-fails the task. 88 tasks (10 ca…
AI Agent Testing
A research project to measure AI agent robustness. Contains automated testing pipelines and a benchmarking methodology developed to audit Agentic AI architectures for complex reas…
Althing
Run synthetic focus groups and user research panels using AI personas. CLI tool, Python library, any LLM.
Ulpia
The memory is yours. The model is a guest. Local-first memory and retrieval layer for AI agents: RAG with no model in the path, abstention as a verdict, MCP server, evaluation inc…
Yang Promptlab
AI security testing — find security vulnerabilities in LLM apps, chatbots, AI agents, MCP servers, and RAG systems. Mapped to OWASP (LLM, Agentic, MCP) and NIST AI RMF.
Coordination Noise Floor Protocol
Code, data & paper: How Much Coordination Gain Is Real? A Paired Noise-Floor Protocol for Multi-Agent LLM Benchmarks (KDD 2026 Workshop)
Apilot
Agentic accounts-payable automation using FastAPI: a vision LLM extracts invoice data, deterministic code makes every payment decision. LangGraph 3-way matching, duplicate & fraud…