open-computer-use

A local MCP server that gives AI agents eyes and hands on your desktop. Agents can see an app's interface, click, type, scroll, and drag — through the accessibility layer, without taking over your real mouse and keyboard. Runs entirely on your machine, on macOS, Windows, and Linux.

The tools

Nine core tools, identical across all three platforms:

Five additional window-level tools — list_windows, get_window, get_window_state, launch_app, activate_window — follow the newer window2 API and are available on Windows and macOS (Linux in progress). On macOS the window id is a CGWindowID, and the action tools accept the same optional window argument plus screenshotId for coordinate actions as on Windows.

macOS adds one display-level tool, capture_display: it captures one complete display — desktop, visible windows, menu bar, Dock, visible popovers — as a full-resolution PNG, without activating, unhiding, or focusing anything. With no display_id it captures the main display; the result's available_displays lists the other selectable IDs. It needs the Screen Recording permission, returns typed JSON errors (permission_missing, display_not_found, display_changed, capture_timeout, capture_failed, image_too_large), and its metadata maps PNG pixels to Quartz global display points — the image pixel size can be larger than the display point size on Retina displays. Unlike get_app_state/window screenshots it never restores hidden or minimized windows to obtain pixels.

Quick start

npm i -g @opensymph/open-computer-use
ocu doctor        # verify the install; macOS: prompts for permissions
ocu call list_apps

macOS 14+ needs Accessibility and Screen Recording granted once. Windows and Linux work out of the box in a signed-in desktop session (Linux desktops need AT-SPI2, which GNOME and friends ship by default).

No npm? Grab the tarball from GitHub Releases and run npm i -g <tarball> — or install it into ZCode as a plugin (below) and it will use whatever local runtime it finds.

Connect your agent

ocu install-codex-mcp      # Codex CLI & Codex App
ocu install-codex-plugin   # Codex App, plugin form
ocu install-claude-mcp     # Claude Code
ocu install-gemini-mcp     # Gemini CLI (--scope user for global)
ocu install-opencode-mcp   # opencode

Any other MCP client — add it manually:

{
  "mcpServers": {
    "open-computer-use": { "command": "open-computer-use", "args": ["mcp"] }
  }
}

ZCode

This repo ships as a ZCode plugin — one install gives you the skill and an auto-connected MCP server:

1. Install the runtime once (the plugin falls back to it when no local build exists):

   npm i -g @opensymph/open-computer-use
   

2. In ZCode, open Settings → Plugin Management → Discover and click +.

3. Add this repository — the GitHub URL opensymph/open-computer-use, or a local checkout directory.

4. Find Open Computer Use in the list and click Get.

5. Start a new session. Check Settings → MCP shows open-computer-use as connected, then just ask: "list the windows on my screen".

To remove it later: Installed tab → Open Computer Use → uninstall.

Agent skill — installable guidance that teaches agents to use these tools well:

npx skills add opensymph/open-computer-use -g -a claude-code --skill open-computer-use -y

Why this one

• Non-intrusive by design. Prefer the accessibility API over synthetic input; your real pointer, focus, and foreground app stay put unless you explicitly opt into global input.
• Three platforms, one contract. The same tool names, arguments, and results on every OS — agents don't need per-platform branching.
• A cursor you can watch. On macOS, actions drive a visible software cursor, so you can follow what the agent is doing.
• Scriptable without a client. ocu call runs any tool from your shell and prints MCP-style JSON; --calls chains sequences in one process.
• Guardrails built in. Password managers are always refused. Launching apps, stealing focus, and global input injection each sit behind an explicit environment-variable gate.
• Signed where it matters. The macOS runtime is Developer ID signed, so granted permissions survive version upgrades.

Platform status

Display-level desktop commands (all platforms)

Every runtime ships the same whole-desktop CLI commands that mirror the classic xdotool / ffmpeg desktop stack — same command names, flags, and JSON output on macOS, Windows, and Linux — handy for headless VNC desktops or full-screen observation where you want to capture or drive the entire screen rather than a single app:

open-computer-use screenshot --output shot.png   # whole-desktop PNG (base64 to stdout without --output)
open-computer-use cursor-position                # pointer x/y + desktop size (JSON, identical shape)
open-computer-use record start --output rec.mp4 --fps 60 --quality demo --polish
open-computer-use record stop --save-as demo-take   # also writes demo-take.polished.mp4 when --polish
open-computer-use record polish --input demo-take.mp4  # compositor: zoom/lens/blur/cursor/keys (or --engine ffmpeg / --ripples)
open-computer-use record discard                 # stop + delete (Cursor RecordScreen DISCARD parity)
open-computer-use record status

screenshot and cursor-position are read-only. Per-platform notes:

The Linux commands accept --display (defaults $DISPLAY, then :0; a VNC/AnyOS desktop is usually :1); Windows and macOS operate on the whole desktop and have no --display. Global synthetic input moves the real pointer/keyboard, so each platform gates it behind its own opt-in flag (default off):

OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1 open-computer-use input click --x 960 --y 600   # Linux
OPEN_COMPUTER_USE_WINDOWS_ALLOW_FOREGROUND_INPUT=1 open-computer-use.exe input type "hello"       # Windows
OPEN_COMPUTER_USE_MACOS_ALLOW_FOREGROUND_INPUT=1 open-computer-use input key ctrl+s                # macOS

These commands are CLI-only: they stay outside the shared cross-platform MCP tool surface (macOS separately exposes the full-display capture as the capture_display MCP tool described above).

Documentation

• Architecture — how the three runtimes work
• Skill references — usage, installation, troubleshooting
• Security policy and third-party notices
• Contributing

License

MIT