mirroir-mcp

Give your AI eyes, hands, and a real iPhone. An MCP server that lets any AI agent see the screen, tap what it needs, and figure the rest out — through macOS iPhone Mirroring. Experimental support for macOS windows. 32 tools, any MCP client.

What's Changed

• Icon detection (YOLO CoreML) — describe_screen can now detect non-text UI elements (icons, buttons, toggles, activity rings) alongside Vision OCR text. Drop a CoreML .mlmodelc in ~/.mirroir-mcp/models/ and the server auto-detects it at startup. See Icon Detection for details.
• Startup config dump — All effective configuration values are logged at startup in a grouped two-column format. Check ~/.mirroir-mcp/mirroir.log to see exactly what settings are active.
• Component-driven exploration — The explorer matches screen regions against component definitions (.md files describing UI patterns like table rows, toggles, tab bars) instead of guessing from raw OCR. Multi-row elements (Health app summary cards) are absorbed into single tappable components. Calibrate definitions against live screens with calibrate_component.

Requirements

• macOS 15+
• iPhone connected via iPhone Mirroring

Install

/bin/bash -c "$(curl -fsSL https://mirroir.dev/get-mirroir.sh)"

or via npx:

npx -y mirroir-mcp install

or via Homebrew:

brew tap jfarcand/tap && brew install mirroir-mcp

The first time you take a screenshot, macOS will prompt for Screen Recording and Accessibility permissions. Grant both.

claude mcp add --transport stdio mirroir -- npx -y mirroir-mcp

{
  "servers": {
    "mirroir": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "mirroir-mcp"]
    }
  }
}

{
  "mcpServers": {
    "mirroir": {
      "command": "npx",
      "args": ["-y", "mirroir-mcp"]
    }
  }
}

codex mcp add mirroir -- npx -y mirroir-mcp

[mcp_servers.mirroir]
command = "npx"
args = ["-y", "mirroir-mcp"]

git clone https://github.com/jfarcand/mirroir-mcp.git
cd mirroir-mcp
./mirroir.sh

How it works

Every interaction follows the same loop: observe, reason, act. describe_screen gives the AI every text element with tap coordinates (eyes). The LLM decides what to do next (brain). tap, type_text, swipe execute the action (hands) — then it loops back to observe. No scripts, no coordinates, just intent.

Examples

Paste any of these into Claude Code, Claude Desktop, ChatGPT, Cursor, or any MCP client:

Open Messages, find my conversation with Alice, and send "running 10 min late".

Open Calendar, create a new event called "Dentist" next Tuesday at 2pm.

Open my Expo Go app, tap "LoginDemo", test the login screen with
test@example.com / password123. Screenshot after each step.

Start recording, open Settings, scroll to General > About, stop recording.

Skills

When you find yourself repeating the same agent workflow, capture it as a skill. Skills are SKILL.md files — numbered steps the AI follows, adapting to layout changes and unexpected dialogs. Steps like Tap "Email" use OCR — no hardcoded coordinates.

Place files in ~/.mirroir-mcp/skills/ (global) or <cwd>/.mirroir-mcp/skills/ (project-local).

---
version: 1
name: Commute ETA Notification
app: Waze, Messages
tags: ["workflow", "cross-app"]
---

## Steps

1. Launch **Waze**
2. Wait for "Où va-t-on ?" to appear
3. Tap "Où va-t-on ?"
4. Wait for "${DESTINATION:-Travail}" to appear
5. Tap "${DESTINATION:-Travail}"
6. Wait for "Y aller" to appear
7. Tap "Y aller"
8. Wait for "min" to appear
9. Remember: Read the commute time and ETA.
10. Press Home
11. Launch **Messages**
12. Tap "New Message"
13. Type "${RECIPIENT}" and select the contact
14. Type "On my way! ETA {eta}"
15. Press **Return**
16. Screenshot: "message_sent"

${VAR} placeholders resolve from environment variables. ${VAR:-default} for fallbacks.

Skill Marketplace

Install ready-to-use skills from jfarcand/mirroir-skills:

git clone https://github.com/jfarcand/mirroir-skills ~/.mirroir-mcp/skills

Test Runner

Run skills deterministically from the CLI — no AI in the loop. Designed for CI and regression testing.

mirroir test apps/settings/check-about
mirroir test --junit results.xml --verbose        # JUnit output
mirroir test --dry-run apps/settings/check-about    # validate without executing

Exit code 0 = all pass, 1 = any failure.

Compiled Skills

Compile a skill once to capture coordinates and timing. Replay with zero OCR.

mirroir compile apps/settings/check-about        # compile
mirroir test apps/settings/check-about            # auto-detects .compiled.json
mirroir test --no-compiled check-about            # force full OCR

AI agents auto-compile skills as a side-effect of the first MCP run. See Compiled Skills for details.

AI-Assisted Diagnosis

When a test step fails, pass --agent to get an AI diagnosis of what went wrong and suggested fixes:

mirroir test --agent gpt-5.3 apps/settings/check-about
mirroir test --agent claude-sonnet-4-6 apps/settings/check-about
mirroir test --agent ollama:llama3 apps/settings/check-about
mirroir test --agent embacle apps/settings/check-about

Built-in agents:

Custom agents can be defined as YAML profiles in ~/.mirroir-mcp/agents/.

cargo install embacle-server
embacle-server --port 3000 --provider copilot   # or: --provider claude_code
mirroir test --agent embacle my-skill

Recorder

Record interactions as a skill file:

mirroir record -o login-flow.yaml -n "Login Flow" --app "MyApp"

Generate Skill

Let an AI agent explore an app and produce SKILL.md files automatically:

Explore the Settings app and generate a skill that checks the iOS version.

Uses DFS graph traversal — tapping unvisited elements, backtracking when branches are exhausted. Duplicate screens are automatically skipped.

Component Detection

Raw OCR returns a flat list of text elements with no structure. Component definitions teach the explorer what UI patterns look like — a .md file per pattern (table rows, toggles, tab bars, summary cards). The explorer matches screen regions against these definitions to decide what to tap, what to skip, and when to backtrack.

18 iOS component definitions are included. Place custom definitions in ~/.mirroir-mcp/components/ or <cwd>/.mirroir-mcp/components/.

Test a definition against the current live screen:

Use calibrate_component with my-component.md to check how it matches.

See Component Detection for the definition format, match rules, and the detection pipeline.

Doctor

Verify your setup:

mirroir doctor
mirroir doctor --json    # machine-readable output

Configure

Set up your keyboard layout for non-US keyboards:

mirroir configure

Updating

# curl installer
/bin/bash -c "$(curl -fsSL https://mirroir.dev/get-mirroir.sh)"

# npx
npx -y mirroir-mcp install

# Homebrew
brew upgrade mirroir-mcp

# From source
git pull && swift build -c release

Uninstall

# Homebrew
brew uninstall mirroir-mcp

# From source
./uninstall-mirroir.sh

Icon Detection

By default, describe_screen uses Apple Vision OCR to detect text on screen. If a YOLO CoreML model (.mlmodelc) is present in ~/.mirroir-mcp/models/, the server auto-detects it and merges icon detection results with OCR text — giving the AI tap targets for non-text elements like buttons, toggles, and tab bar icons that text-only OCR misses.

Installing a model

Place a compiled CoreML model (.mlmodelc) in ~/.mirroir-mcp/models/. The server will find and load it automatically. You can also point to a specific path via the yoloModelPath setting or MIRROIR_YOLO_MODEL_PATH environment variable.

The yoloConfidenceThreshold setting (default: 0.3) controls the minimum detection confidence — lower values detect more elements but may introduce noise.

Configuration

Override timing defaults via settings.json:

// .mirroir-mcp/settings.json (project-local) or ~/.mirroir-mcp/settings.json (global)
{
  "keystrokeDelayUs": 20000,
  "clickHoldUs": 100000
}

Environment variables also work: MIRROIR_KEYSTROKE_DELAY_US. See TimingConstants.swift for all keys.

Documentation

Contributing

Contributions welcome. By submitting a patch, you agree to the Contributor License Agreement — your Git commit metadata serves as your electronic signature.

Why "mirroir"? — It's the old French spelling of miroir (mirror). A nod to the author's roots, not a typo.