- Conduid
- Marketplace
- #llm-as-judge
MCP servers tagged llm-as-judge
24 live MCP servers tagged "llm-as-judge", ranked by trust score. Tags come from package metadata and repository topics, so this list covers servers across every category that work with llm-as-judge.
Tracely AI
Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.
Tracely
Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.
AI Engineer Notebooks
Hands-on, framework-free Colab notebooks for the AI Engineer / Forward Deployed Engineer (FDE) skill set — model APIs, structured output, tool calling, RAG, evals-as-the-spine, ag…
Agentevals
agentevals is a framework-agnostic evaluations solution based on OpenTelemetry traces
BestTester
Production-grade Playwright + TypeScript QA framework with AI-powered testing, LLM-as-Judge evaluation, MCP server, 7 CLI agents, security fuzzing, CI/CD pipelines, Jira sync, and…
BestTester
Production-grade Playwright + TypeScript QA framework with AI-powered testing, LLM-as-Judge evaluation, MCP server, 7 CLI agents, security fuzzing, CI/CD pipelines, Jira sync, and…
Dojo.md
University for AI agents. 92 courses, 4400+ scenarios, any model via OpenRouter. Auto-training loops generate per-model SKILL.md documents. Works with Claude Code, OpenClaw, Curso…
Classifier Evals MCP Server
MCP server exposing classifier evaluation tools
Prism Verify
prism-verify — runtime LLM verifier (family-different, reasoning-stripped, multi-lens, signed Ed25519 receipts). Zero-prerequisite npx install via a verified binary launcher.
Hermes Rubric
Evidence-first LLM-as-judge scoring for AI artifacts — papers, PRs, prompts, cold emails: synthesizes a rubric, collects quoted-evidence citations, scores only against that eviden…
Mcpgauge
Self-hosted evaluation framework for MCP servers. Define YAML test suites, run agents against MCP tools, score with LLM-as-judge rubrics, and monitor results in a FastAPI dashboar…
Bayesoutputgate
Calibrated output validation with the Bayes Factor: weigh per-dimension quality scores into a pass, fail, or escalate decision on the Jeffreys scale, not an arbitrary threshold. Z…
Boule
Lean, research-backed multi-LLM council with bias-aware judge controls (position-swap, verbosity-norm, stake-free judge) adapted from LLM-eval research.
Mini Omega Lock
Measure your LLM judge's noise floor — an A/B improvement smaller than the judge's own noise isn't real. Pre-flight probes for prompt-eval setups; works standalone and with omegap…
LangValidator
Self validating LangGraph agent pipeline: every agent output passes through a checkpoint node that scores it and routes to pass, retry with feedback, or halt. Includes rule based,…
Bayesconsensus
Dawid-Skene consensus for multi-agent and multi-LLM disagreement: learn each agent's reliability and the true answer with calibrated confidence. Zero dependencies, node-free, Type…
Omegaprompt
The overfit gate for your prompts: re-test the winning prompt on held-out examples it never tuned on, and block the ship if it doesn't generalize. Sits on top of promptfoo/DSPy. C…
Clawdmarket
Autonomous agent-to-agent marketplace with live Karpathy loop self-improvement. Agents discover, hire, benchmark, and evolve programmatically. MPP/x402/MCP. No humans in the loop.
Humanloopbench
Human-in-the-loop LLM Failure Detection & Benchmark Toolkit — Auto-flag LLM failures (hallucination, refusal, factual error) with LLM-as-Judge explanations. Confirm them in a revi…
Ferryman MCP
A local-first MCP host with pluggable skills, multi-provider routing, and multi-channel I/O. Kotlin. Plus a Python eval harness with rule + LLM-judge scorers over a human-authored…
Test Bench
Open-source evaluation framework for MCP servers powered by Claude. Auto-discovers tools, generates test scenarios, runs LLM-as-judge scoring to assess correctness and safety, aud…
Outputproof
Verification infrastructure for AI agent output — assertions, LLM-as-judge scoring, audit logs, and reliability gates before generated work is trusted.
Groundlens Dev
Guardrails for LLM answers. GroundLens checks an LLM answer against its sources without a model, decides PASS, REVIEW or FAIL under your policy, and keeps a signed record.
Rubricator
The agent that turns a pile of context into a defensible comparison: one MCP tool surface, two runtimes, span-level citations and honest uncertainty. Produces comparanda analyses.