Wisp

Wisp is a computer-use toolkit for macOS: a daemon (wispd) that reads app windows as an indexed accessibility tree and performs UI actions with an animated agent cursor, plus a CLI (wisp) and an MCP server (wisp mcp) that any agent (Claude Code, Codex, scripts) can call. Chrome tabs are driven over the DevTools Protocol, in your own Chrome through the Wisp extension or in a separate hidden instance.

It is a from-scratch implementation of the ideas documented in DESIGN.md (how the ChatGPT/Codex desktop app's Computer Use works): indexed AX trees with diffs, window-targeted synthesized input that does not hijack the user's pointer, a glowing cursor with spring motion and click gating, settle detection, Esc to stop, and a policy file.

Install

Homebrew (recommended)

brew install --cask owo-network/brew/wisp

This installs Wisp.app and the wisp command-line tool together (both signed and notarized). Updates arrive automatically through Sparkle. Then open the app once and grant permissions:

open -a Wisp        # menu bar app; guides you through permissions on first run
wisp doctor         # shows what is still missing

Grant Accessibility (required) and Screen Recording (optional, for screenshots) to Wisp in System Settings → Privacy & Security. The menu bar item guides you: it asks for Accessibility with the system prompt, shows a hint banner, and offers "Grant Accessibility access…" and "Grant Screen Recording…" items that open the right Settings pane. Once every permission is granted the badge disappears.

Use with Claude Code

Install the plugin — one step for the skill and the session-cleanup hook:

/plugin marketplace add missuo/wisp
/plugin install wisp@wisp

It gives Claude Code a skill that teaches it when and how to drive the UI, and two hooks: one ends the Wisp session (agent cursor, banner, scratch Chrome tabs) whenever a turn finishes or you interrupt it, even if the agent forgot to; the other refuses the skill on a machine without the wisp command and hands Claude the install instructions instead. Note that ending without --app/--tab ends every Wisp session on the machine, not only the ones this agent opened.

This repository is the marketplace: .claude-plugin/marketplace.json serves the plugin directly from integrations/claude-plugin, so there is no third-party index in between and the plugin ships with the version that produced it. The skill checks that the CLI is installed and tells you how to install it if not.

• MCP server (Codex, scripts, any MCP client):

  claude mcp add wisp -- wisp mcp
  

  Over MCP the wisp_turn_end tool replaces the Stop hook: call it when the task is finished or the user interrupts.

• Skill — from a source checkout: cp -R integrations/claude-plugin/skills/wisp ~/.claude/skills/wisp.

• Stop hook — in ~/.claude/settings.json:

  {
    "hooks": {
      "Stop": [
        { "hooks": [ { "type": "command", "command": "wisp status >/dev/null 2>&1 && wisp end" } ] }
      ]
    }
  }
  

  The wisp status guard keeps the hook from starting the daemon on turns that never used Wisp (a bare wisp end would).

From source

scripts/bundle.sh            # builds release binaries, packages Wisp.app and installs ~/.local/bin/wisp
wisp doctor --request-permissions

During development you can run straight from the build tree: WISP_DAEMON=.build/debug/wispd .build/debug/wisp doctor (a terminal that already has Accessibility permission passes it on to child processes).

App icon: assets/icon/wisp-icon.svg is the source (with wisp-glyph.svg / wisp-background.svg as separate layers); assets/icon/*-1024.png are the rasterized 1024×1024 versions; assets/icon/Wisp.icon is the Icon Composer document. scripts/bundle.sh compiles the .icon with actool into Assets.car (Liquid Glass icon on macOS 26) plus a Wisp.icns fallback, or builds the .icns from the PNG when the document is absent.

Use

wisp apps                                  # running + recently used apps
wisp state --app Safari                    # indexed accessibility tree (diff on later calls)
wisp click --app Safari --el 12            # every action returns the new state diff
wisp set --app Safari --el 4 "openai.com"; wisp key --app Safari Return
wisp type --app Notes "Hello"; wisp paste --app Notes --format md "# Title"
wisp scroll --app Mail --el 22 --down --pages 2
wisp screenshot --app Preview -o shot.png; wisp click --app Preview --at 640,420   # pixels of that screenshot
wisp batch --app TextEdit <<'EOF'
{"kind":"key","key":"cmd+n"}
{"kind":"type","text":"hello"}
{"kind":"state"}
EOF
wisp end --app Safari

Chrome tabs are driven over the DevTools Protocol, in one of two places:

• Your own Chrome, through the Wisp extension (the way the Codex desktop app works). wisp chrome extension install copies the extension to ~/Library/Application Support/Wisp/chrome-extension and registers its native messaging host (a small launcher script that execs wisp native-host) for Chrome (and Brave, Edge, Chromium, Arc or Vivaldi when present); load that folder once on chrome://extensions (Developer mode, Load unpacked). From then on wisp chrome tabs lists your tabs first (tagged [user]), --tab active is the tab you are looking at, and wisp chrome new URL opens the page in your browser. Wisp attaches Chrome's debugger to a tab only while it works in it and lets go at wisp end.
• A separate Wisp Chrome with its own profile and debug port (wisp chrome launch), hidden in the background so your windows are untouched. Used automatically when no extension is connected, or on request with wisp chrome new URL --browser wisp.

wisp chrome extension install              # extension + native messaging host; then load it once on chrome://extensions
wisp chrome extension status               # what is installed, and whether the extension is connected to wispd
wisp chrome tabs                           # [user, active] your active tab, [user] the rest, then Wisp Chrome tabs
wisp chrome new https://example.com        # -> tab id (in your browser when the extension is connected)
wisp state --tab active; wisp click --tab <id> --el 5; wisp chrome eval --tab <id> "document.title"
wisp chrome upload --tab <id> [--el N] /abs/file.pdf   # fill a file input without a picker
wisp chrome dialog --tab <id> accept|dismiss [--text S] # answer an alert/confirm/prompt (state reports it as open)
wisp chrome mark --tab <id> deliverable|handoff        # keep a tab past `wisp end` (unmarked agent tabs are scratch)
wisp chrome launch                         # the separate Wisp Chrome: --remote-debugging-port (first free port from 9222),
                                           # own profile, hidden (--visible shows it; `wisp chrome show`/`hide` later)

The Wisp Chrome's port is remembered in ~/Library/Application Support/Wisp/chrome.json; wisp chrome launch reuses a running instance instead of starting another. Any Chrome window can also be controlled through accessibility like a native app: wisp state --app "Google Chrome".

The first state of an app starts with <app_specific_instructions>: built-in guidance for that app (Safari, Mail, Finder, Slack, Chrome tabs, ...) plus a browser block for anything that handles http URLs; --instructions repeats it and --no-instructions suppresses it. Drop your own Markdown at ~/.config/wisp/instructions/<bundle id>.md to override or extend a text (wisp instructions list shows every stem, wisp instructions show --app X prints what an app would receive), and set instructionsMode in the policy (wisp policy set --instructions-mode merge|replace|off) to choose whether user files merge with, replace, or switch off the built-in texts.

wisp --json … prints machine-readable JSON. wisp mcp serves every CLI capability as an MCP tool; add it to Claude Code with claude mcp add wisp -- wisp mcp. The tools mirror the commands one to one:

The model-facing guide lives in integrations/claude-plugin/skills/wisp/SKILL.md.

How it works

• Tree: AXUIElementCopyMultipleAttributeValues per element, visible-children subsets for large tables, transform passes (label association, text merging, pruning, child caps), one line per element with a stable index; diffs (~ changed, + added, - [a..b] removed) against the previous revision.
• Input: CGEvents posted with CGEventPostToPid to the process that owns the window under the pointer (sheets, menus and out-of-process Open/Save panels are separate windows and processes), tagged with a magic user-data value, carrying the target window in kCGMouseEventWindowUnderMousePointer; keys mapped through the active keyboard layout (UCKeyTranslate) and delivered to the process that owns keyboard focus; text typed as unicode key events or pasted through a restored clipboard. The real pointer never moves and the window is never raised: a synthetic app-activation event makes the target accept input while it stays in the background, so you can keep working. --activate opts into bringing an app forward for the rare one that ignores background input.
• Vision fallback: the accessibility tree is preferred, but when a window exposes no actionable elements (custom-drawn apps, canvases, games) Wisp attaches a window screenshot so the caller can look and click by pixel coordinates (--at x,y --space screenshot); --screenshot requests one on demand.
• Cursor: an overlay NSPanel at window level 102 with a glowing arrow; motion uses the spring/path constants recovered from Sky (clickAngle -44°, scoot under 196 pt, closeEnough at 99.5 % / 3.2 pt); the mouse-down is posted only when the cursor has visually arrived.
• Activity lens and banner: a small sweeping ring next to the cursor (or in the window's top-right corner while the cursor is hidden) shows whether Wisp is observing or acting; it turns gray and fades after an intervention. wisp status reports the current activity. The banner text is a policy template (wisp policy set --banner-text "✦ Wisp is controlling {app}" --banner-hint reading); --lens off hides the ring.
• Settle: AXObserver notifications plus busy flags; a quiet window of 0.3 s after at least 0.25 s, up to 5 s. A visible progress or busy indicator in the window keeps the wait going (the state is then marked not settled).
• Safety: listen-only event tap for Esc and real user input (userIntervened), policy file ~/.config/wisp/policy.json, secure fields are never read or typed into unless allowed, screen-lock check before every action and batch step plus a lock monitor that cancels an in-flight action the moment the screen locks (wisp status shows screenLocked), display kept awake while a session runs.
• Per-app approval: apps come in tiers. Password managers are denied by default (deny); the login window, the authentication agent and Wisp itself are forbidden and cannot be allowed at all; high-risk apps (System Settings, Terminal, Keychain Access, Passwords, Mail, Messages, Finder) show an on-screen prompt the first time an agent touches them, with Allow Once, Allow for This Session, Always Allow and Don't Allow (the prompt closes as Don't Allow after 120 s). wisp policy set --approval off|high-risk|all picks when to ask (all prompts for every app), --high-risk/--forbid edit the lists, wisp approvals list|clear manages stored grants (~/.config/wisp/approvals.json). --block-url HOST ... refuses wisp chrome new/goto to those hosts (subdomains included, punycode for internationalized names) and refuses actions inside a native window whose web content shows one of them.
• Chrome: Accessibility.getFullAXTree + DOMSnapshot.captureSnapshot rendered by the same engine; Input.dispatch* for actions; Page.captureScreenshot; Runtime.evaluate for wisp chrome eval. The same commands reach your own browser through the extension: its service worker attaches chrome.debugger to the tab and relays commands and events over a native messaging host (wisp native-host, a relay to the daemon socket). The host manifest points at ~/Library/Application Support/Wisp/native-host.sh, a #!/bin/sh script that execs the CLI, because Chrome cannot start a Mach-O host directly on every macOS (the process dies before it runs). Synthesized mouse events never reach Chrome's web content, which is why both paths use DevTools.

Safety and confirmations

The daemon enforces the mechanical guardrails (policy file, Esc, intervention detection, secure fields, screen lock), but deciding whether an action should happen is up to the agent. The skill therefore ships a confirmation policy in four tiers: actions the agent must hand back to the user (submitting credential changes, bypassing security walls, entering passwords), actions it must confirm right before the effect (deleting, sending, paying, changing settings or permissions), actions covered by an explicit request (a login or upload the user named), and everything else, which needs no confirmation. See the confirmation tiers.

Releases and updates

• Pushing a tag vX.Y.Z (or running the Release workflow manually) builds universal (Apple silicon + Intel) Wisp.app and wisp CLI binaries on GitHub Actions, signs them with the Developer ID certificate, notarizes and staples them, produces Wisp-X.Y.Z.zip, Wisp-X.Y.Z.dmg, wisp-cli-X.Y.Z.zip, wisp-chrome-extension-X.Y.Z.zip, SHA256SUMS.txt and a Sparkle appcast.xml, and publishes everything as a GitHub release.
• The app updates itself with Sparkle (Check for Updates… in the menu bar; automatic daily checks). The feed is https://github.com/missuo/wisp/releases/latest/download/appcast.xml; updates are signed with the EdDSA key whose public half is in assets/sparkle-public-key.txt.
• Required repository secrets: MACOS_CERTIFICATE_P12 (base64 .p12), MACOS_CERTIFICATE_PASSWORD, KEYCHAIN_PASSWORD, NOTARY_APPLE_ID, NOTARY_PASSWORD (app-specific password), NOTARY_TEAM_ID, SPARKLE_PRIVATE_KEY (exported with generate_keys -x).
• scripts/package.sh is the shared packaging step (used locally by scripts/bundle.sh with ad-hoc signing and by CI with the Developer ID); scripts/set-version.sh stamps Sources/WispCore/Version.swift.

Layout

Sources/WispCore   protocol, JSON, framing, UI tree model, transforms, renderer, diff, key parser, policy
Sources/wispd      daemon: AX snapshot, input synthesis, cursor overlay, settle, screenshots, CDP, socket server
Sources/wisp       CLI + MCP server
.claude-plugin/    marketplace.json - serves the plugin below straight from this repository
integrations/      claude-plugin: the Claude Code plugin (skill for agents + session-cleanup Stop hook)
                   chrome-extension: the Wisp extension for your own Chrome (service worker + popup)
scripts/           package.sh (build+sign), bundle.sh (local install), set-version.sh, e2e.sh (TextEdit smoke test)
.github/workflows  release.yml (sign, notarize, Sparkle appcast, GitHub release)
assets/            app icon sources and the Sparkle public key

swift test runs the core unit tests; scripts/e2e.sh exercises the daemon against TextEdit (WISP_E2E_CHROME=1 adds the Wisp Chrome, WISP_E2E_EXTENSION=1 the extension in your browser).

License

Wisp is licensed under the PolyForm Noncommercial License 1.0.0: free to use for any noncommercial purpose. See the license for the definition of noncommercial and for personal-use and noncommercial-organization terms. For a commercial license, contact the maintainer.