Autoware CLAW

ROSClaw Southbound Driver for Autoware — bridges LLM agents to Autoware via the Model Context Protocol (MCP).

Autoware CLAW lets an LLM agent query vehicle state, resolve GPS coordinates to lane-aligned goals, and send control commands to Autoware — all through a structured MCP tool interface. The agent runs inside an isolated NemoClaw sandbox with GPU-accelerated inference via Ollama, while an RViz panel provides an operator chat UI for oversight.

Demo

Architecture

System Overview

┌───────────────────────────────────────────────────────────────────────┐
│  Host Machine (ROS 2 Humble)                                          │
│                                                                       │
│  ┌─────────────────────┐          ┌────────────────────────────────┐ │
│  │  Autoware Planning   │          │  Docker Containers             │ │
│  │  Simulator           │          │                                │ │
│  │                      │          │  ┌──────────────────────────┐  │ │
│  │  - Localization       │          │  │  NemoClaw (LLM Agent)    │  │ │
│  │  - Perception         │          │  │  - OpenClaw sandbox      │  │ │
│  │  - Planning           │          │  │  - Web dashboard :18789  │  │ │
│  │  - Control            │          │  │  - Brave web search      │  │ │
│  │  - Vehicle Simulator  │          │  └───────────┬──────────────┘  │ │
│  └──────────┬───────────┘          │              │ internal-net     │ │
│             │                      │  ┌───────────▼──────────────┐  │ │
│             │ ROS 2 topics         │  │  Ollama (LLM Inference)  │  │ │
│             │                      │  │  - gemma4 model          │  │ │
│  ┌──────────▼───────────┐          │  │  - GPU accelerated       │  │ │
│  │  Autoware MCP Server │          │  └──────────────────────────┘  │ │
│  │  (ROS 2 Node)        │◄─── SSE/JSON-RPC ──►                     │ │
│  │  :8765               │          └────────────────────────────────┘ │
│  └──────────────────────┘                                             │
│                                                                       │
│  ┌───────────────────────────────────────────────────────────────┐   │
│  │  RViz2 + Autoware CLAW Chat Panel                             │   │
│  │  - Operator oversight & chat interface                        │   │
│  │  - Pre-configured layout with vehicle visualization           │   │
│  └───────────────────────────────────────────────────────────────┘   │
└───────────────────────────────────────────────────────────────────────┘

Three-Layer Design

1. MCP Server (host, ROS 2 node)

   • Subscribes to Autoware ROS 2 topics and caches vehicle state with thread-safe locking
   • Exposes 17 MCP tools via SSE transport on 127.0.0.1:8765
   • Publishes control commands, gear, turn indicators, hazard lights, and heartbeat
   • Resolves GPS coordinates to lane-aligned goals using Lanelet2 + MGRSProjector

2. NemoClaw + Ollama (Docker containers)

   • NemoClaw: NVIDIA's LLM agent sandbox with Autoware MCP tools pre-configured
   • Ollama: Local LLM inference server running gemma4 with GPU acceleration
   • Network isolation: internal-net (Ollama, no internet) + egress-net (web search, MCP)
   • L7 network policy restricts MCP access to /sse, /messages/**, /health

3. RViz Chat Panel (C++ / Qt)

   • Operator-facing chat UI embedded in RViz
   • Connects to NemoClaw gateway with health monitoring
   • Dark theme, monospace font, persistent URL configuration

Data Flow

Operator types in RViz Chat Panel
        │
        ▼
NemoClaw receives message, reasons using gemma4 (Ollama)
        │
        ▼
NemoClaw calls MCP tools (e.g., autoware_get_vehicle_state)
        │
        ▼
MCP Server reads cached ROS 2 state / publishes commands
        │
        ▼
Autoware executes (planning, control, vehicle simulation)
        │
        ▼
Response flows back: Autoware → MCP → NemoClaw → RViz Chat

ROS 2 Topic Interface

Subscriptions (state caching):

Publications (commands):

Packages

MCP Tools (17 tools)

Display Tools (read-only)

Coordinate Resolution Tools

Navigation Commands

Direct Vehicle Control

Heartbeat Control

Prerequisites

• ROS 2 Humble with Autoware workspace built
• Docker with NVIDIA Container Toolkit (for GPU inference)
• NVIDIA GPU (required for Ollama + gemma4 inference)
• Python packages: rosclaw, mcp, uvicorn, starlette (installed via pip)
• Brave API key (optional, for NemoClaw web search): set BRAVE_API_KEY environment variable

For a detailed step-by-step guide on integrating autoware_claw into an Autoware workspace, see Building with Autoware.

Build

# Build autoware_claw and its RViz plugins
colcon build --packages-select autoware_claw autoware_claw_rviz_plugins
source install/setup.bash

During the first build, Docker images are built and the gemma4 model (~5 GB) is pulled into a persistent volume. Subsequent builds skip the download.

Launch Commands

Full Integration (recommended)

Launches Autoware planning simulator + MCP server + NemoClaw/Ollama Docker containers + RViz with chat panel:

ros2 launch autoware_claw planning_simulator.launch.xml \
  map_path:=/path/to/your/map

Example with a specific map:

ros2 launch autoware_claw planning_simulator.launch.xml \
  map_path:=$HOME/Downloads/shinjyuku/Shinjuku-Map/map

MCP Server Only

If you want to run only the MCP server against an already-running Autoware instance:

ros2 launch autoware_claw mcp_server.launch.xml

With custom configuration:

ros2 launch autoware_claw mcp_server.launch.xml \
  host:=0.0.0.0 \
  port:=9000 \
  config_file:=/path/to/custom_config.yaml

Launch Arguments

Planning Simulator Arguments

Autoware CLAW Arguments

Perception Arguments

Common Launch Patterns

# Without NemoClaw Docker (use external LLM agent)
ros2 launch autoware_claw planning_simulator.launch.xml \
  map_path:=/path/to/map \
  launch_nemoclaw_docker:=false

# Without perception (lighter resource usage)
ros2 launch autoware_claw planning_simulator.launch.xml \
  map_path:=/path/to/map \
  perception/enable_object_recognition:=false \
  perception/enable_detection_failure:=false

# Custom MCP port
ros2 launch autoware_claw planning_simulator.launch.xml \
  map_path:=/path/to/map \
  mcp_port:=9000

# With Brave web search enabled
BRAVE_API_KEY=your-key-here ros2 launch autoware_claw planning_simulator.launch.xml \
  map_path:=/path/to/map

Configuration

MCP Server (config/mcp_server.param.yaml)

/**:
  ros__parameters:
    # MCP transport
    transport: "sse"        # "sse" (HTTP) or "stdio" (stdin/stdout)
    host: "127.0.0.1"       # Bind address (use 0.0.0.0 for Docker access)
    port: 8765              # MCP server port

    # Heartbeat for vehicle_cmd_gate external input
    heartbeat_rate_hz: 20.0

    # Vehicle parameters
    wheelbase_m: 2.79       # Wheelbase for bicycle model (twist -> steering conversion)
    max_steering_rad: 1.0   # Max steering angle limit
    max_velocity_mps: 25.0  # Max velocity limit

    # Lanelet2 map (for coordinate resolver - GPS to lane alignment)
    map_path: ""            # Path to lanelet2_map.osm file
    map_origin_lat: 0.0     # Map origin latitude
    map_origin_lon: 0.0     # Map origin longitude
    map_origin_alt: 0.0     # Map origin altitude

Docker Services (docker-compose.yml)

Network Security

NemoClaw runs inside Docker with layered network isolation:

┌─────────────────────────────────────────────┐
│  Docker                                      │
│                                              │
│  ┌──────────┐   internal-net   ┌──────────┐ │
│  │ NemoClaw │◄────────────────►│  Ollama  │ │
│  │          │   (no internet)  │  (GPU)   │ │
│  └────┬─────┘                  └──────────┘ │
│       │                                      │
│       │ egress-net (L7 policy restricted)    │
│       │                                      │
└───────┼──────────────────────────────────────┘
        │
        ▼ host.docker.internal:8765
   MCP Server (host)

• internal-net (bridge, internal: true) — Ollama only. No external access.
• egress-net (bridge) — NemoClaw can reach:
  • Host MCP server via host.docker.internal:8765
  • Brave Search API (if BRAVE_API_KEY is set)
• L7 network policy (autoware-mcp-policy.yaml) restricts MCP endpoints to:
  • GET /sse — SSE stream connection
  • POST /messages/** — JSON-RPC tool calls
  • GET /health — Health check
• NemoClaw dashboard binds to 127.0.0.1:18789 only (not exposed to network)
• Config lockdown — openclaw.json is owned by root with 444 permissions. Gateway auth token is generated at build time.

Interaction

After launching, you can interact with the LLM agent through:

1. RViz Chat Panel — Type messages directly in the chat panel embedded in RViz
2. NemoClaw Dashboard — Open http://localhost:18789 in a browser

The agent can autonomously:

• Query vehicle state (autoware_get_vehicle_state)
• Check surrounding traffic (autoware_get_surrounding_objects)
• Find a route to a GPS location (autoware_resolve_goal -> autoware_set_goal)
• Engage/disengage autonomous driving (autoware_engage)
• Send direct vehicle controls in EXTERNAL mode (autoware_set_gate_mode -> autoware_send_control)
• Trigger emergency stop (autoware_emergency_stop)

Project Structure

autoware_claw/
├── autoware_claw/                  # Python ROS 2 package
│   ├── autoware_claw/              # Python modules
│   │   ├── autoware_mcp_server.py  # MCP tool registration, handlers, SSE/stdio server
│   │   ├── autoware_ros_node.py    # ROS 2 subscriptions, publishers, state cache
│   │   ├── coordinate_resolver.py  # Lanelet2 GPS-to-lane mapping via MGRSProjector
│   │   ├── topic_adapters.py       # ROS message builders (Twist->Control, etc.)
│   │   └── types.py                # Dataclasses: VehicleState, DetectedObject, etc.
│   ├── config/
│   │   └── mcp_server.param.yaml   # MCP server configuration
│   ├── docker/
│   │   ├── nemoclaw.Dockerfile     # Multi-stage: Node.js builder + sandbox runtime
│   │   └── autoware-overlay/
│   │       ├── inject-mcp-config.py       # Injects MCP endpoint into openclaw.json
│   │       └── autoware-mcp-policy.yaml   # L7 network policy for sandbox
│   ├── docker-compose.yml          # Ollama + NemoClaw service definitions
│   ├── launch/
│   │   ├── mcp_server.launch.xml           # Standalone MCP server launch
│   │   └── planning_simulator.launch.xml   # Full integration launch
│   ├── CMakeLists.txt
│   ├── package.xml
│   ├── setup.py
│   └── setup.cfg
│
└── autoware_claw_rviz_plugins/     # C++ RViz2 plugin package
    ├── src/
    │   ├── autoware_claw_panel.hpp # Qt panel: URL input, status, chat display, message input
    │   └── autoware_claw_panel.cpp # Network communication, health checks, message handling
    ├── rviz/
    │   └── autoware_claw.rviz      # Pre-configured RViz layout with Claw panel
    ├── CMakeLists.txt
    └── package.xml

Troubleshooting

autoware_shape_estimation not found

If perception packages are not built, disable them:

ros2 launch autoware_claw planning_simulator.launch.xml \
  map_path:=/path/to/map \
  perception/enable_object_recognition:=false \
  perception/enable_detection_failure:=false

Docker GPU not available

Ensure NVIDIA Container Toolkit is installed:

nvidia-smi                           # GPU driver check
docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi  # Docker GPU check

NemoClaw not connecting to MCP server

Verify the MCP server is running and healthy:

curl http://localhost:8765/health
# Expected: {"status": "ok", "connected": true}

gemma4 model download slow

The model (~5 GB) is cached in a Docker volume. If the initial pull is slow, you can pre-download it:

docker compose -f $(ros2 pkg prefix autoware_claw)/share/autoware_claw/docker/docker-compose.yml \
  run --rm ollama-pull

License

Apache-2.0