ATR - Agentic Test Runner

ATR is an AI-powered test runner that automatically analyzes failures and runs browser-based behavior tests using natural language.

Features

• AI Failure Analysis: Run any command and get intelligent analysis when it fails
• Browser Behavior Testing: Write tests in natural language, let AI execute them
• Multiple LLM Backends: Supports Google Gemini API, Vertex AI, and CLI tools (Claude, Gemini)
• CLI Backend Support: Use Claude CLI or Gemini CLI as backends - no API keys needed
• Cross-platform Desktop Control: Mouse, keyboard, screen capture, window/app management on Linux (X11), macOS, and Windows via atr computer, with multi-monitor support and an in-process LLM agent (atr computer ask)
• MCP Server: Expose browser AND desktop tools to any MCP-compatible client
• Cross-Platform: Works on Linux, macOS, and Windows
• Extensible: Tool-based architecture for custom extensions

Quick Start

Install

macOS / Linux:

curl -fsSL https://raw.githubusercontent.com/imyousuf/agentic-test-runner/main/install.sh | sh

Windows (PowerShell):

irm https://raw.githubusercontent.com/imyousuf/agentic-test-runner/main/install.ps1 | iex

From source:

go install github.com/imyousuf/agentic-test-runner/cmd/atr@latest

Or download from Releases.

Configure

# Option 1: Using Claude or Gemini CLI (no API key needed)
# If you have claude or gemini CLI installed, ATR auto-detects them
atr config init  # Creates config with detected CLI as default backend

# Option 2: Using Gemini API
export GEMINI_API_KEY="your-api-key"

# Option 3: Using Vertex AI
gcloud auth application-default login
export GOOGLE_CLOUD_PROJECT="your-project"

Run

Analyze a failing command:

atr run --cmd "go test ./..."

When the command fails, ATR's AI agent will:

• Analyze the failure output
• Run diagnostic commands
• Read relevant source files
• Provide actionable recommendations

Run a behavior test:

# Create a test file
cat > login.test.txt << 'EOF'
Test: User can log in

Steps:
1. Navigate to http://localhost:3000/login
2. Enter "user@example.com" in email field
3. Enter "password" in password field
4. Click "Sign In" button
5. Verify URL contains /dashboard
EOF

# Run the test
atr run --behavior login.test.txt

Example Output

Executing: go test ./...
Directory: /path/to/project

--- FAIL: TestUserAuth (0.05s)
    auth_test.go:42: expected 200, got 401

✗ Command failed (exit code: 1, duration: 2.3s)

Analyzing failure with AI agent...
Using model: gemini-2.0-flash-exp (gemini-api)

======================================================================
ANALYSIS RESULTS
======================================================================

Status: FAILURE

Summary:
  TestUserAuth fails because the auth middleware expects a JWT token,
  but the test doesn't provide one in the request headers.

Root Cause:
  Line 38 in auth_test.go creates a request without Authorization header.
  The auth middleware (middleware/auth.go:15) rejects it with 401.

Recommendations:
  1. Add mock JWT token to test request
  2. Or bypass auth middleware in test setup
  3. Check if middleware was recently added to the route

Files Examined:
  - auth_test.go
  - middleware/auth.go
  - routes/api.go

Documentation

• Getting Started - First steps with ATR
• Installation - All installation methods
• Configuration - Config file and authentication setup
• CLI Reference - All commands and flags
• Behavior Testing - Write browser tests in natural language
• Browser Server - HTTP server for programmatic browser control
• MCP Server - MCP protocol server for CLI tool integration
• Architecture - How ATR works internally
• llms.txt - Quick reference for AI agents

Claude Code Integration

ATR includes Claude Code skills for seamless AI-assisted browser automation. Install the skills to enable natural language control of ATR within Claude Code.

Installation

1. Add the ATR marketplace:

   /plugin marketplace add imyousuf/agentic-test-runner
   

2. Install the skills plugin:

   /plugin install atr-skills@atr-marketplace
   

Available Skills

Skill	Description
atr-browser	Control browser via ATR server (navigate, click, fill, screenshot, browser ask)
atr-computer	Cross-platform desktop control (mouse, keyboard, screen, windows, apps, multi-monitor) plus computer ask — an in-process LLM agent that takes a natural-language instruction and drives the desktop end-to-end
atr-analyze	Run tests with AI analysis (default for test suites - keeps context clean)
atr-behavior	Run natural language browser tests

The atr-browser and atr-computer skills cross-reference each other and can be loaded together when a workflow spans both (e.g. drag a file from the file manager into a browser drop zone).

Usage Examples

Once installed, Claude Code automatically uses these skills when relevant:

• "Navigate to google.com and take a screenshot"
• "Run the pytest tests" (uses atr-analyze for clean output)
• "Run the behavior tests in tests/e2e/"
• "Open the GNOME calculator and tell me what 17×23 is" (computer ask)

MCP Server

ATR can run as an MCP (Model Context Protocol) server, exposing browser automation tools to any MCP-compatible client like Claude CLI or Gemini CLI.

# Start the MCP server
atr mcp serve

Integration with Claude CLI

# Inline configuration
claude -p "Navigate to example.com and take a screenshot" \
  --mcp-config '{"mcpServers":{"atr-browser":{"command":"atr","args":["mcp","serve"]}}}' \
  --allowedTools "mcp__atr-browser__*"

# Or add to ~/.claude.json

Integration with Gemini CLI

Add to your project's .gemini/settings.json:

{
  "mcpServers": {
    "atr-browser": {
      "command": "atr",
      "args": ["mcp", "serve"],
      "trust": true
    }
  }
}

See MCP Server Documentation for the full list of browser and desktop tools exposed via MCP.

Desktop Control (atr computer)

ATR also drives the desktop directly — mouse, keyboard, screen capture, window/app management — for tests and automation that go beyond the browser.

# Start the daemon (default: 3-second countdown before each action; Ctrl+C to abort)
atr computer start

# Low-level primitives
atr computer screenshot --output /tmp/desktop.png
atr computer click --display 0 800 50
atr computer type "hello world"
atr computer window list

# High-level: ask an agent to do it for you
atr computer ask "open xclock and tell me what time it shows"
atr computer ask --max-steps 30 "open the GNOME calculator and compute 17 * 23"

atr computer stop

Default LLM models for atr computer ask (and atr browser ask):

Tier	Model
flash	gemini-3.1-flash-preview
pro	gemini-3.2-pro-preview

Switch with atr --model pro computer ask "...". Backend claude-cli uses the Claude CLI subprocess via MCP and ignores the model flag.

Multi-monitor: bounds reported by atr computer displays and window list are in root coordinates (the bounding box of all monitors at origin). Use --display N to pass display-local pixels for clicks. Linux is X11-only in v1; Wayland is tracked as future work.

See CLI Reference — atr computer and Computer Server REST API for full details.

Configuration

ATR can be configured via ~/.atr/config.yaml:

# CLI backend (no API key needed - uses installed CLI tools)
backend: claude-cli  # or gemini-cli

# Or API backends:
backend: gemini-api  # or vertex-ai
model: flash         # or pro

gemini:
  api_key: "your-key"

# Or for Vertex AI:
vertex:
  project: your-project
  location: us-central1

See Configuration Guide for all options including CLI backends, Vertex AI authentication methods (ADC, service account, workload identity), and MCP server configuration.

Requirements

• Go 1.25+ (for building from source)
• One of the following LLM backends:
  • Claude CLI or Gemini CLI (recommended - no API key needed)
  • Google Gemini API key
  • Google Cloud project with Vertex AI

Linux runtime dependencies (for atr computer)

The desktop computer-use feature uses robotgo which requires X11 development headers and a few system libraries. On Debian/Ubuntu install them with:

make install-deps-linux
# or, manually:
sudo apt-get install -y \
  libxtst-dev libxss-dev libpng-dev \
  libxkbcommon-dev libx11-dev xclip xsel

Optional GUI overlay packages:

• zenity — abortable progress dialog for the per-action countdown (recommended). Install with sudo apt-get install -y zenity.
• libnotify-bin — fallback when zenity is not present; provides notify-send for visual-only notifications. Install with sudo apt-get install -y libnotify-bin.

If neither is available the daemon falls back to terminal-only countdown — no functionality is lost, but Claude-Code/MCP users won't see a visible interrupt.

Note: Linux support for atr computer is X11 only in v1. Wayland is tracked as future work.

Cross-platform builds: robotgo and webview-style libraries depend on platform-native CGo, so cross-compiling from a single host is no longer supported. Each release artifact is built on its own platform's CI runner (ubuntu-latest, macos-latest, windows-latest). For local development, make build produces a binary for your current platform.

Simple Testing

To test the browser functionality just using this command should give you some idea -

atr browser stop; atr browser start && atr browser navigate "https://optimizely.com" && atr browser screenshot --full --file && atr browser ask "what is the title of the page?" && atr browser stop && atr run --behavior=<PATH_TO_GOPATH>/src/github.com/imyousuf/agentic-test-runner/examples/behavior/simple.test.txt && echo "test passed"

Contributing

Contributions are welcome! Please feel free to submit issues and pull requests.

License

Apache License 2.0 - see LICENSE for details.