- Conduid
- Marketplace
- #llm-evaluation
MCP servers tagged llm-evaluation
213 live MCP servers tagged "llm-evaluation", ranked by trust score. Tags come from package metadata and repository topics, so this list covers servers across every category that work with llm-evaluation.
Matterpilot
Legal AI matter platform: pluggable workflow packs (court-notice intake, contract review), matter-scoped pgvector RAG, Office add-ins, MCP server, eval harness
Auditable Multi Agent Recommender
Auditable multi-agent recommender for cloud optimization. Orchestration beats single-shot frontier models on correctness (18/18 vs 3–4/18), with a fully replayable evidence trail.…
Tournament
Benchmark LLMs on your own scenarios with a multi-judge panel. Full MCP server (tools, resources, prompts, progress) plus local GUI, CLI, and desktop app. One OpenRouter key.
Agentic Base
Run provenance, comparison validity, and the shared primitives for agentic work on research infrastructure. A base layer, not an agent framework.
Enterprise AI And Intelligent Systems
Enterprise AI engineering covering local inference, Azure OpenAI, RAG, agents, MCP, evaluation, security, and cost control.
Whichtool
Does the model actually pick the right tool from your MCP server?
ENTERPRISE AI AGENT EVALUATION PLATFORM
MCP-style AI agent evaluation platform for RAG workflows with quality gates, canary validation, regression reports, and OpenTelemetry-compatible traces.
Klarova
The trust layer between AI agents and your company's data — correct, cited, access-scoped answers, served to any agent over MCP.
Agora Backend
Multi-agent LLM debate arena with MCP evidence tools, mechanical fact-checking and blind judging. FastAPI + Bedrock.
Kakeibo
A personal finance agent built from scratch on the raw Gemini API: hand-written tool-calling loop, double-entry PostgreSQL ledger, context caching, MCP server, 46-task eval harnes…
Agentpave
A miniature agentic AI developer platform on AWS with QA baked in — one command scaffolds a governed agent with evals, guardrails, tracing, and a failing-closed CI quality gate.
Agentrig
Evidence-first regression testing for AI agents: turn production traces into reviewed, replayable cases that gate releases.
Arbiter AI
AI agent that investigates whether a new version of an AI feature is safe to ship - comparing quality, cost, latency, and reliability against evidence, then recommending Ship, Hol…
Lens Kit
Trainable 10-lens validation gate for AI-generated text. Outside-in: a separate model checks the output, fixed checks, inspectable findings, fail-closed. Apache-2.0 — own it and t…
Ukrainian LLM Eval
Reproducible Ukrainian-language evaluation of language models and agents, with optional reference tools.
Argus AI Reliability Platform
Self-healing AI reliability platform: LangGraph multi-agents + MCP tool execution + RAG-grounded Gemini evidence to detect, diagnose, and safely recover from LLM/RAG/agent failure…
Reportguard
Multi-agent QA system that checks business report numbers against the database using MCP and Gemini
Deadwax
An agentic music librarian with a deterministic validator, a repair loop, and a measured eval suite. LLM tool use, RAG, tracing and MCP.
Accura
Accura is a multi-agent AI platform designed to generate, evaluate, and refine AI responses until they meet a defined quality threshold. It uses a combination of generator and eva…
Quorel Free
Quorel is an MCP server that scores AI agent responses before they reach a user, a trust layer an agent calls mid-turn, not a dashboard someone checks after the fact.
Retirement Answer Check
Checks AI-drafted answers to US retirement-account questions (SEND or REVIEW, with an IRS or FINRA source for every flag), plus a 2026 contribution-room calculator and MCP tool.
Benchmark Search Tool MCP
Claude Code 搜索工具对比评测：5 工具 × 15 测试用例 × 52 组合，LLM-as-Judge 验证
Deepeval Agent Demo
AI Agent Evaluation POC — DeepEval with LangChain agents, RAG pipelines, multi-turn chatbots & safety evals
Dspack Gen
Generation and governance pipeline for dspack contracts: prompt compiler, model adapters, S1/S2/S3 governance gates, bounded repair loop, audit reports, and the evaluation harness…
Agentshield LLM Safety Engine
Local-first LLM safety evaluation engine for red-teaming RAG and agentic assistants with LangGraph, MCP, human review, and regression testing.
Clinical Agent Eval Demo
A deployment layer for a guardrailed clinical conversational agent: real FHIR over MCP, PHI redaction, a hash-chained audit trail, and an evaluation harness measured on held-out d…
AI Agent Evals Deepeval
Production-style reference for evaluating LLM agents, RAG pipelines & multi-turn chatbots with DeepEval — component-level evals, tool-correctness, faithfulness, bias/toxicity chec…
Toolassay
Evaluation harness for MCP tool contracts: does a model pick the right tool, finish the task, and what does it cost.
Scoreia Forge Results
Open results of La Forge (ScoreIA): AI models forge and animate a 3D knight over MCP, scored by a programmatic referee (no LLM judge).
Osanwe Public
A Markdown vault for financial analysis, run by AI agents, in which research compounds: each analysis is written back as typed, linked notes that the next one reads first. Broker…
Jobkubera Lab
AI systems engineering portfolio focused on bounded agents, AI assurance, evidence-driven automation and human-controlled execution.
Jaeger MCP Bench
A/B benchmark of MCP tool output formats for Jaeger SPM metrics. 72 randomized trials, Claude + Gemini. Built for jaegertracing/jaeger#8409.
Malmatch
말매치 Malmatch: Korean character dialogue QA as a local MCP skill pack
Enterprise Agent Reliability Lab
A production-oriented enterprise AI agent with MCP tools, Agent Skills, tracing, reliability controls, and reproducible evaluations.
Swe Gym
Reproducible RL environments for evaluating coding agents on real software-engineering tasks through MCP tools. Deterministic verification, golden reference solutions, and CI that…
Recruiting Agent
Guarded multi-turn recruiting agent: MCP server, LangChain MCP adapters, LangGraph, conversational evals with guardrail ablation, FastAPI, Docker, CI.
FourEyes
Human-gated support agent: dual MCP servers, LangGraph interrupts checkpointing before every irreversible write, three defense layers red-teamed against prompt injection.
Calma
Re-run the work, recompute the number, block the wrong one before it ships — a deterministic guardrail for AI-computed results. CLI · Claude Code hook · MCP.
Annotide
Self-hosted data annotation and labeling platform for images, video, audio, text, PDF and LLM evaluation. Your storage, your models, your cloud.
Control Finder
Ask a security question in plain English and get the NIST SP 800-53 controls that answer it, each one cited. Measured keyword, vector, and hybrid search with pgvector, cited answe…
Agentic Research Assistant
A multi-agent Research & Report Assistant orchestrated with LangGraph — runs fully keyless and never fabricates sources.
Judge MCP
MCP for MCPs — score artifacts against rubrics, explain failures and iterate until a defined quality threshold.
Tdae
Test-Driven Agent Evolution - ground-truth evaluation harness for LLM agents: accuracy scoring, per-agent metrics, regression detection. Runs offline via a mock provider.
QUIZZ
Prices get revised: an as-of agent scored on right answers and on correct refusals, behind a gate whose pass mark rises with every variant tried
Analytics Evals
Agent skills for testing analytics and text-to-SQL agents: mine golden test cases from your query history, score with pass^k and confidence intervals, and find which context files…
Lookml Agentops
Author Looker AI agents as code (agent specs -> Conversational Analytics) and diagnose why an agent's answer changed, with owners.
Nyaya Evidence Grounded Multimodal Claim Resolution Agent
Evidence-grounded multimodal claim resolution agent: LangGraph, CNN, deterministic policy engine, grounded LLM briefs, durable resume, CI-gated evals
Bedrock Agentic Rag
Self-correcting agentic RAG on AWS: LangGraph on Amazon Bedrock, S3 Vectors, event-driven Lambda ingestion, ECS Fargate + CloudFront, DynamoDB checkpoints, CDK, OIDC CI/CD gated b…
Car Safety Checker
Describe a car problem and see the official NHTSA recalls and owner complaints that match it. A measured AI model, a website that runs in the browser, and an MCP server for AI ass…
.github
Org profile and community health files for Responsible AI Labs. RAIL Score: LLM evaluation across 8 dimensions, guardrails, MCP server, red-teaming, and India DPDP / EU AI Act com…