- Conduid
- Marketplace
- #llm-evaluation
MCP servers tagged llm-evaluation
213 live MCP servers tagged "llm-evaluation", ranked by trust score. Tags come from package metadata and repository topics, so this list covers servers across every category that work with llm-evaluation.
MCP
🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓
Opik
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
Trulens
Evaluation and Tracking for LLM Experiments and AI Agents
Agentic Security
Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪
Tracely AI
Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.
Agentic AI Systems
Agentic AI explained with chickens 🐔 every pattern a runnable, CI-checked file.
Tracely AI
Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable regression tests, and block PRs that regress. Self-hostable.
Overmind
The platform for continuously improving AI agents.
Overmind
The platform for continuously improving AI agents.
Arena
mcp_arena is a production-ready Python library for building MCP (Model Context Protocol) servers with intelligent agent orchestration and domain-specific presets.
Tracely
Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.
Baba Is Eval
Claude et al. play the brilliant puzzle title "Baba is You"
decidefyi/decide
Source-backed, fail-closed policy checks for refunds, cancellations, returns, and trials. Four read-only MCP tools plus the deterministic Decision API engine.
Selfstarter
Give a coding agent a brief, not a chat, and it works on its own across sessions. Includes an example run: an open-world city built in a real game engine with no human help. In ea…
Overtime
Give a coding agent a brief, not a chat, and it works on its own across sessions. Includes an example run: an open-world city built in a real game engine with no human help. In ea…
Agentenv Framework
Creating realistic RL environments requires collaboration between researchers, engineers, and domain experts across many dimensions: artifacts, environments tools, dynamism of the…
Aaabench
A long-horizon benchmark harness: give a coding agent a real game engine, professional conditions and time, and ask it to build an open-world game. Harness only, no results.
OrcaReplay
OrcaReplay — Time travel for AI agents. Record, replay, fork, and debug any agent run with any model. Built by the OrcaRouter.ai team.
Gateway
MCP gateway & OAuth MCP servers. LangGraph agent: memory, eval/tracking, PII guard, context trimming. Docker quick start for enterprise agents.
Brain In The Fish
Score any document. Prove every claim.
Gameops Investigator
Evidence-first game operations incident investigation agent with read-only MCP tools and reproducible evaluations.
AI Reliability Copilot
Turn a production incident into a structured 9-section LLM response (severity, root cause, mitigation, postmortem). Ships with a 5-scenario regression suite + LLM-as-judge eval pi…
Invarail
The authority plane for local AI agents: a governed, self-hosted agent built so a 7–30B model on your own GPU can drive it. Measured in public.
Jev Judge MCP
Typed judgment tools for MCP agents. TypeSafe's Jev model as verify, screen, find, classify, rerank, decide, compare, extract, review, gate, and score: the model judges, policy de…
Proofagent Harness
Open-source test harness for AI agents. Stress-test production agents with adversarial multi-turn scenarios in CI
Assay
An open, reproducible trust layer for AI artifacts — skills, MCP servers, agents, and plugins. Assay inspects code, runs sandboxed behavioral audits, and generates verifiable repo…
Harness Eval
Did this change to the coding-agent harness make the code better or worse? Paired, repeated, isolated evaluation with a verdict you can trust.
server
LangWatch — independent third-party profile of a public API surface, by API Evangelist. LangWatch is an open-source LLM observability, evaluation, and AI agent testing platform. B…
Aicw AI Mentions
AICW AI Mentions (formerly AICW Rankings) is an open-source tool that helps online marketers track how often their products are mentioned in responses from popular AI chatbots.
Suede Creator Skills
25 open-source Agent Skills for Claude Code and Codex: multi-agent orchestration, A-F code review, AI evals, CI ship-gates, design, copy, SEO, iOS shipping, music rights, and cons…
Cap Evolve
Optimize any AI agent’s skills, tools/MCP, and prompts against your own evals.
Nuguard
AI red-teaming tool and LLM security framework to evaluate agentic AI applications. Tests prompt injections, handles vulnerability assessment, SBOM generation, and static analysis.
Sir Thaddeus
Private, local-first AI assistant for Windows with permissioned tools, durable memory, and evidence-driven small-model improvements.
Nasde Toolkit
CLI for benchmarks & evals of AI coding agents — on tasks you already understand, using your Claude / Codex / Gemini individual subscriptions or API keys.
Isnad
Grade every agent, scraper and model in a claim's chain — provenance, trust scoring and audit evidence for LLM pipelines
Qa Automation Portfolio
26-framework SDET monorepo: Playwright, Selenium, Cypress, Cucumber, Postman, FastAPI, Pact, AI/LLM eval (DeepEval), agentic AI (LangChain, LangGraph, DSPy, Claude), k6, flakiness…
AAE
A long-term, opinionated knowledge base for Agent Engineering — 21 categories, ≤10 curated repos each (live gh-api stats), the Evaluation Layer first, comparison tables, Mermaid d…
AI Native Engineer
Tracking my journey into AI engineering, one module at a time. Fundamentals to agents, with hands-on projects. Open to all.
Titration
Let your coding agent fix a prompt until it actually works, and prove it did. A self-hosted MCP server: cross-vendor judges, frozen baselines and memory.
Enterprise Agentic AI Platform
Multi-tenant control plane for enterprise AI agents: onboarding a new assistant is a configuration change, not a deployment. Shared runtime for retrieval, orchestration, conversat…
Relayops
Production-shaped control plane for AI support agents: scoped routing, policy broker, tool boundaries, approval queue, replay verification, audit traces, and human handoff. Protot…
Benchmark
Head-to-head benchmark comparing the official MCP to the MCP auto-created by Hintas.
Tessera
Open-source, MCP-native benchmark for whether AI agents reason reliably over fragmented enterprise knowledge — accuracy, provenance, correct refusal.
Agent Triage
Diagnose your AI agents in production. Extract policies from prompts, evaluate traces, generate diagnostic reports.
Intents
Intents is a native macOS workbench for evaluating Apple's on-device Foundation Models.
Explore AI
A platform for conversational AI with chat, RAG, tool calling, and quality evaluation. It is written in Java with Spring AI, and uses Angular on the frontend.
Atlas Agentic Rag
ATLAS: Agentic RAG as one coherent system — hybrid search + reranking + knowledge graph, tool-calling over MCP, multi-model routing, guardrails, and a real eval harness. Runs full…
Companmem
the quest for ultimate ai memory
Vacant
Accountability layer for AI agents: run the client's executable acceptance tests, gate delivery, sign every step into a verifiable receipt. Pre-registered, replicated 5x, cross-be…
Hermes Rubric
Evidence-first LLM-as-judge scoring for AI artifacts — papers, PRs, prompts, cold emails: synthesizes a rubric, collects quoted-evidence citations, scores only against that eviden…