Computer Use AI SDK

• We've built an MCP server that controls computer

• You've heard of OpenAI's operator, you've heard of Claude's computer use. Now the open source alternative: Computer Use SDK from screenpipe.

• It's native on macOS—no virtual machine bs, no guardrails. Use it with any app or website however you want.

• No pixel-based bs—it relies on underlying desktop-rendered elements, making it much faster and far more reliable than pixel-based vision models.

• You can now build your own agents getting started with our simple Hello World Template using our MCP server and client.

• There are tools that our MCP Server provides out of the box:

  • Launch apps
  • Read content
  • Click
  • Enter text
  • Press keys

• These will be computational primitives to allow the AI to control your computer and do your tasks for you. What will you build? Come check us out at https://screenpi.pe

Demos

agent sending a message

https://github.com/user-attachments/assets/f8687500-9a8c-4a96-81b6-77562feff093

get latest whatsapp messages

open arc browser

Get started

git clone https://github.com/m13v/computer-use-ai-sdk.git
cd MCP-server-client-computer-use-ai-sdk

# Install Rust (if not already installed)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# Install Node.js and npm (if not already installed)
# Visit https://nodejs.org/ or use nvm

# run backend server
cd mcp-server-os-level
cargo run --bin server
# keep it running

Option 1: CLI Interface

# run CLI interface client in a new terminal  (good for debugging)
cd mcp-client-cli-interface
npm install  # install dependencies first

# Set your Anthropic API key as an environment variable
export ANTHROPIC_API_KEY=sk-ant-xxxx  # Replace with your actual Anthropic API key
# For Windows, use: set ANTHROPIC_API_KEY=sk-ant-xxxx
# For permanent setup, add to your shell profile (.bashrc, .zshrc, etc.)

npx tsx main.ts

Option 2: Web app Interface

# run CLI interface client in a new terminal  (good for debugging)
cd mcp-client-nextjs
npm install  # install dependencies first

# Set API key via command line
echo "ANTHROPIC_API_KEY=sk-ant-XXXXXXXX" > .env  # replace XXXXXXXX with your actual key
# Or append if you want to keep other env variables
# echo "ANTHROPIC_API_KEY=sk-ant-XXXXXXXX" >> .env

npm run dev
# go to provided localhost web page

What do I do with it?

• Build custom worfklows of agents to performs various actions
• Build custom UI to make it easy for users to automate their computer work
• Save workflow and run in cron
• Combine with other MCP servers to do something cool, e.g.: fill out a google sheet based on the history of people i talk to throughout the day

Request features and endpoints in github issues

https://github.com/m13v/computer-use-ai-sdk/issues/new/choose