About Mcpx Eval
Mcpx Eval is an MCP server published by dylibso in the Developer Tools category: an open-ended eval framework for mcp.run tools. It has been installed 0 times through Conduid.
The repository has 22 stars and 6 forks, with the last commit a year ago. Six months or more without a commit doesn't mean the server is broken, but check the open issues (0) before depending on it in production.
Install
npx mcpx-evalThis server has no ConduID identity, so agent calls to it are not receipted. Pin the version you install and review the source before granting it credentials.
Ask AI
Ask AI about Mcpx Eval
Powered by Claude · Grounded in docs
Security checks
- ·README presentNot checked yet.
- ·License declaredNot checked yet.
- ·Tests presentNot checked yet.
- ·Dependencies pinnedNot checked yet.
- ·No dynamic code executionNot checked yet.
- !Scoped permissionsDoesn't declare a permission scope. Assume it can do anything its process can.
Releases
README
mcpx-eval
A framework for evaluating open-ended tool use across various large language models.
mcpx-eval can be used to compare the output of different LLMs with the same prompt for a given task using mcp.run tools.
This means we're not only interested in the quality of the output, but also curious about the helpfulness of various models
when presented with real world tools.
Test configs
The tests/ directory contains pre-defined evals
Installation
uv tool install mcpx-eval
Or from git:
uv tool install git+https://github.com/dylibso/mcpx-eval
Or using uvx without installation:
uvx mcpx-eval
mcp.run Setup
You will need to get an mcp.run session ID by running:
npx --yes -p @dylibso/mcpx gen-session --write
This will generate a new session and write the session ID to a configuration file that can be used
by mcpx-eval.
If you need to store the session ID in an environment variable you can run gen-session
without the --write flag:
npx --yes -p @dylibso/mcpx gen-session
which should output something like:
Login successful!
Session: kabA7w6qH58H7kKOQ5su4v3bX_CeFn4k.Y4l/s/9dQwkjv9r8t/xZFjsn2fkLzf+tkve89P1vKhQ
Then set the MCP_RUN_SESSION_ID environment variable:
$ export MCP_RUN_SESSION_ID=kabA7w6qH58H7kKOQ5su4v3bX_CeFn4k.Y4l/s/9dQwkjv9r8t/xZFjsn2fkLzf+tkve89P1vKhQ
Usage
Run an eval comparing all mcp.task runs for my-task:
mcpx-eval test --task my-task --task-run all
Only evaluate the latest task run:
mcpx-eval test --task my-task --task-run latest
Or trigger a new task run:
mcpx-eval test --task my-task --task-run new
Run an mcp.run task locally with a different set of models:
mcpx-eval test --model .. --model .. --task my-task --iter 10
Generate an HTML scoreboard for all evals:
mcpx-eval gen --html results.html --show
Test file
A test file is a TOML file containing the following fields:
name- name of the testtask- optional, the name of the mcp.run task to usetask-run- optional, one oflatest,new,allor the name/index of the task run to analyzeprompt- prompt to test, this is passed to the LLM under test, this can be left blank iftaskis setcheck- prompt for the judge, this is used to determine the quality of the test outputexpected-tools- list of tool names that might be usedignored-tools- optional, list of tools to ignore, they will not be available to the LLMimport- optional, includes fields from another test TOML filevars- optional, a dict of variables that will be used to format the prompt
README mirrored from the source repository 2 days ago. The original is authoritative.