SpecialAgent

Connect AI to Unreal Engine 5

Full Python API access • 71+ level design tools • Visual feedback loop

What is SpecialAgent?

SpecialAgent bridges AI assistants and Unreal Engine 5 through the Model Context Protocol (MCP). Connect Claude, GPT, or any MCP-compatible LLM directly to your editor and control it through natural language.

At its core, SpecialAgent provides unrestricted Python execution with full access to UE5's Python API—meaning your AI assistant can do anything the editor can do. On top of that foundation, 71+ purpose-built tools handle common level design tasks without writing a single line of code.

Native HTTP/SSE transport. No external bridges or dependencies.

Features

Two Layers of Power

Full Python Access

Execute arbitrary Python with complete unreal module access. Your AI assistant can:

• Import and process assets
• Create and modify Blueprints
• Generate materials and textures
• Automate project configuration
• Build custom editor utilities
• Run validation and QA checks
• Anything the UE5 Python API supports

This is the unlimited foundation. If you can script it, AI can do it.

Level Design Toolkit

71+ specialized tools for world-building workflows:

Visual Feedback Loop

Capture viewport screenshots and return them to vision-enabled LLMs. Your AI assistant can see what it built, evaluate the results, and refine its approach.

Describe intent → Execute → Screenshot → AI analyzes → Iterate

Installation

Requirements

• Unreal Engine 5.6 or later
• Windows, Mac, or Linux
• MCP-compatible client (Cursor, Claude Desktop, etc.)

Setup

1. Clone or download this repository into your project's Plugins folder:

   YourProject/
   └── Plugins/
       └── SpecialAgent/
   

2. Regenerate project files (right-click .uproject → Generate Visual Studio/Xcode project files)

3. Build and launch your project

4. Enable the plugin in Edit → Plugins → Search "SpecialAgent"

5. Restart the editor

Quick Start

1. Verify the Server

Once the editor launches, check the Output Log for:

LogSpecialAgent: MCP Server started on port 8767

Or test with curl:

curl http://localhost:8767/health

2. Configure Your MCP Client

Add SpecialAgent to your MCP client configuration:

{
  "mcpServers": {
    "SpecialAgent": {
      "url": "http://localhost:8767/sse",
      "transport": "sse"
    }
  }
}

3. Connect and Build

Your AI assistant now has access to:

• Python execution with full UE5 API
• 71+ level design tools
• Viewport screenshot capture
• Editor utilities (save, undo, redo)

Service Categories

Example Workflows

Populate a Forest (via Tools)

1. assets/search → Find tree and rock assets
2. world/scatter_in_area → Place 500 trees with randomization
3. foliage/paint_in_area → Add grass and ground cover
4. screenshot/capture → Get visual for AI analysis
5. Iterate based on feedback

Configuration

Edit Config/DefaultSpecialAgent.ini to customize:

[/Script/SpecialAgent.SpecialAgentSettings]
; Server port (change if 8767 is in use)
ServerPort=8767

; Auto-start server when editor launches
bAutoStart=true

; Enable verbose logging
bVerboseLogging=false

Architecture

┌─────────────────────────────────────────┐
│        MCP Client (Claude, etc.)        │
└──────────────┬──────────────────────────┘
               │ HTTP/SSE + JSON-RPC 2.0
┌──────────────▼──────────────────────────┐
│       SpecialAgent MCP Server           │
│                                         │
│  ┌─────────────────────────────────┐    │
│  │   Python Service (Primary)      │    │
│  │   Full unreal module access     │    │
│  └─────────────────────────────────┘    │
│                                         │
│  ┌─────────────────────────────────┐    │
│  │   14 Services (71+ Tools)       │    │
│  │   Level design & utilities      │    │
│  └─────────────────────────────────┘    │
│                                         │
│  ┌─────────────────────────────────┐    │
│  │   Game Thread Dispatcher        │    │
│  │   Thread-safe API access        │    │
│  └─────────────────────────────────┘    │
└──────────────┬──────────────────────────┘
               │
┌──────────────▼──────────────────────────┐
│        Unreal Engine 5 Editor           │
└─────────────────────────────────────────┘

Documentation

Design Philosophy

The 71+ tools exist for convenience and discoverability. Python execution is the real power.

When your AI assistant sees world/place_in_circle, it learns circular placement is possible. But for custom logic—density falloff, terrain-aware positioning, asset variation based on rules—it writes Python.

Both layers work together: quick tools for common tasks, unlimited scripting for everything else.

Troubleshooting

Server Won't Start

• Check if port 8767 is in use: netstat -an | grep 8767
• Change port in DefaultSpecialAgent.ini
• Verify plugin is enabled in Edit → Plugins

Connection Refused

• Ensure Unreal Editor is running
• Check Output Log for server startup messages
• Verify firewall isn't blocking localhost

Tools Not Appearing

• Call tools/list to verify registration
• Check for errors in Output Log
• Restart the editor

Client not connecting

• Some IDEs like Cursor may need to be started after your Unreal Engine editor as the connection attempt only occurs on startup.

Technical Details

Dependencies

• PythonScriptPlugin (included with UE5)
• EditorScriptingUtilities (included with UE5)

Contributing

Contributions are welcome! Please read the architecture documentation before submitting PRs.

License

MIT License - See LICENSE file for details.

Give your AI assistant the keys to Unreal Engine.