frontend-visualqa

Gives coding agents eyes for frontend work — visual QA and verification powered by Yutori Navigator.

What it does

• Verifies explicit visual claims against a running localhost frontend
• Captures screenshots for quick visual inspection
• Reuses browser sessions across MCP tool calls for multi-step debugging
• Works as a CLI (frontend-visualqa verify), MCP server (frontend-visualqa serve), or agent skill (/frontend-visualqa)

Does not start your dev server. If the URL is unreachable, claims return not_testable.

Why visualqa?

Playwright MCP can click, type, and assert against the DOM — but it cannot see the page. It can run cleanly on the wrong page, assert modal.isVisible() on a modal rendered off-screen, or miss a layout that broke on mobile.

Navigator is an AI agent powered by a computer-use model (n1.5) that perceives a webpage via a screenshot and predicts actions to take in a browser. Two capabilities matter here:

• Self-correcting navigation — Point the agent at the product catalog instead of a specific product page and Navigator recognizes the wrong page, clicks through to the right one, and reports trace.wrong_page_recovered: true. Playwright MCP would run assertions on the wrong page and silently pass — garbage in, garbage out.

  Navigator lands on the product catalog

  →

  Navigated to the correct product page

• Rich visual evaluation — On the cart page, both items show sale prices ($149.99 and $79.99) but Navigator caught that the subtotal of $279.98 uses the original prices — the discount was never applied. On the API dashboard, the quota label reads "100%" but the progress bar is visibly only two-thirds full. Playwright MCP would pass both — the DOM text is consistent and the progress bar width is just a CSS value.

  Navigator catches the discount-not-applied bug

  Label says 100% but the bar is only about two-thirds full

Install

Prerequisites

Install uv if you don't already have it:

curl -LsSf https://astral.sh/uv/install.sh | sh

Quick install (recommended)

1. Install CLIs:

   uv tool install frontend-visualqa \
     --with-executables-from yutori \
     --with-executables-from playwright
   playwright install chromium
   

   This installs the frontend-visualqa, yutori, and playwright CLIs and downloads the Chromium browser binary.

2. Log into Yutori API:

   yutori auth login
   

   This opens your browser to save your Yutori API key to ~/.yutori/config.json.

   Or, manually add your API key

   Go to platform.yutori.com and add your key to the config file:

   mkdir -p ~/.yutori
   cat > ~/.yutori/config.json << 'EOF'
   {"api_key": "yt-your-api-key"}
   EOF
   

3. Register the MCP server using add-mcp (works with all clients):

   npx add-mcp -g -n frontend-visualqa "frontend-visualqa serve"
   

   Pick the clients you want to configure.

4. Install skills using skills.sh:

   npx skills add yutori-ai/frontend-visualqa -g
   

   -g installs at user scope. Omit -g for project-local install.

   Invoke the skill with /frontend-visualqa in Claude Code and $frontend-visualqa in Codex CLI. Agents can also auto-invoke.

5. Restart your agent (Codex, Claude Code, etc) so the installs are picked up.

   To uninstall later:

   uv tool uninstall frontend-visualqa
   npx skills remove -g frontend-visualqa
   

   add-mcp has no remove command. Delete the frontend-visualqa entry from your client's MCP config (e.g. ~/.mcp.json).

Manual per-client setup

/plugin marketplace add yutori-ai/frontend-visualqa
/plugin install frontend-visualqa@frontend-visualqa-plugins

claude mcp add --scope user frontend-visualqa -- frontend-visualqa serve

codex mcp add frontend-visualqa -- frontend-visualqa serve

$skill-installer install https://github.com/yutori-ai/frontend-visualqa/tree/main/.agents/skills/frontend-visualqa

uv sync
uv run playwright install chromium

Uninstall

/plugin uninstall frontend-visualqa@frontend-visualqa-plugins -s user

rm -rf ~/.agents/skills/frontend-visualqa

Examples

The repo includes demo pages you can use immediately — no dev server required:

# From the repo root, serve the included demo pages
cd /path/to/frontend-visualqa
lsof -ti:8000 | xargs kill 2>/dev/null; python3 -m http.server 8000 -d examples &

Self-correcting navigation — start on the wrong page and watch Navigator find its way:

# Navigator lands on the product catalog, clicks through to find the product detail page
# After each evidence screenshot, the Yutori overlay replays the last action
frontend-visualqa verify http://localhost:8000/ecommerce_store.html \
  --headed \
  --claims 'The product detail page shows Wireless Headphones Pro priced at $149.99'

Catching regressions — mix passing and failing claims:

frontend-visualqa verify http://localhost:8000/analytics_dashboard.html \
  --headed \
  --claims \
  'The API status indicator shows Active' \
  'The monthly quota progress bar is completely filled'
# → first claim passes, second fails (label says 100% but bar is ~65% full)

Catching pricing bugs — verify that discounts are actually applied:

frontend-visualqa verify 'http://localhost:8000/ecommerce_store.html#/cart' \
  --headed \
  --claims 'The displayed cart subtotal equals the sum of the visible sale prices'
# → fails: Navigator sees the sale prices sum to $229.98 while the displayed subtotal is $279.98

Autonomous form filling — Navigator fills a multi-step form, picks a date, and catches a timezone bug:

frontend-visualqa verify 'http://localhost:8000/booking_form.html' \
  --headed \
  --max-steps-per-claim 25 \
  --claims 'The date on the confirmation page matches the date selected on the calendar' \
  --navigation-hint "Fill out the form with example data (grayed text is showing example format, not filled out values)"
# → fails: Navigator fills the form, picks a date, books the slot, and catches the off-by-one on the confirmation page

--navigation-hint gives Navigator context it can't infer from pixels alone. Here, the booking form shows placeholder text like "John Doe" and "555-0123" — the model can mistake these for already-filled values and skip the form. The hint tells it that grayed text is placeholder format, not real data, so it fills every field correctly.

Login flow with visual bug detection — when only the first claim needs setup, put the hint next to that claim in a claims file (examples/login_flow_claims.md). The runner reuses the logged-in session for later claims, and only the claim that carries the hint gets the login guidance:

frontend-visualqa verify http://localhost:8000/yutori_login.html \
  --headed \
  --no-reset-between-claims \
  --max-steps-per-claim 20 \
  --claims-file examples/login_flow_claims.md
# → first two claims pass, third fails: label says "100% used" but the progress bar is ~40% filled

Use against your own frontend the same way — just swap the URL:

frontend-visualqa screenshot http://localhost:3000
frontend-visualqa verify http://localhost:3000/dashboard \
  --claims 'The revenue chart is visible without scrolling'

frontend-visualqa verify http://localhost:8000/ecommerce_store.html \
  --headed \
  --claims 'The cart badge shows 3 items' \
  --navigation-hint "Click 'Add to Cart' on the Mechanical Keyboard K7 product card."

frontend-visualqa verify http://localhost:8000/analytics_dashboard.html \
  --headed \
  --claims 'The /api/v1/webhooks endpoint returned a 200 OK status'
# → fails: Navigator scrolls to the request table and finds a 500 Error

frontend-visualqa verify http://localhost:8000/yutori_login.html \
  --headed \
  --claims 'The email field shows "Please enter a valid email address" after submitting the empty form' \
  --navigation-hint 'The grayed text in the fields is placeholder, not real input. Click the Continue button immediately without typing anything.'

See examples/CLAIMS.md for the full list of pages, claims, and expected results.

MCP tools

Recommended agent workflow

1. Ensure the local frontend is running
2. If the page is auth-gated, call manage_browser with action="login" and the login page URL to open a persistent headed browser the user can sign into
3. take_screenshot to confirm page state
4. Write 1–5 concrete visual claims
5. verify_visual_claims
6. Fix code, rerun claims until they pass

CLI reference

frontend-visualqa <command> [options]

frontend-visualqa verify <url> --claims 'claim1' 'claim2' [options]
# or
frontend-visualqa verify <url> --claims-file claims.md [options]

Progress is written to stderr as each claim starts and completes. Final JSON still goes to stdout.

Markdown claims file example:

# Dashboard checks

- The page title reads "Analytics Dashboard"
- The API status indicator shows Active

Per-claim metadata can live under an individual claim bullet:

- After logging in, the dashboard shows "Welcome back, Developer"
  - navigation_hint: Type "test@yutori.com" in the email field, type "password123" in the password field, then click Continue.
- The API Calls Today stat card shows the value 1,247

frontend-visualqa verify http://localhost:8000/analytics_dashboard.html \
  --claims-file examples/login_flow_claims.md \
  --reporter native \
  --reporter markdown

This file-based input path is CLI-only; the MCP tool still takes an explicit list[str] of claims. For MCP and skill-driven workflows, make separate warm-session verify_visual_claims calls when the setup differs between claims instead of expecting one navigation hint to apply selectively within a batch.

Browser modes and visualization

Persistent mode uses one shared Playwright profile-backed context and supports one named session at a time. Use --run-name if you only want to tag CI output, and use ephemeral mode if you need multiple simultaneous named sessions.

# 1. One-time login — opens a headed browser, log in, press Enter to save
frontend-visualqa login http://localhost:3000/login

# 2. Subsequent runs reuse the saved session
frontend-visualqa verify http://localhost:3000/dashboard \
  --browser-mode persistent \
  --run-name dashboard-auth \
  --claims 'The user avatar is visible in the header'

frontend-visualqa login http://localhost:3000/login \
  --user-data-dir /tmp/my-project-profile

frontend-visualqa verify http://localhost:3000/dashboard \
  --browser-mode persistent \
  --user-data-dir /tmp/my-project-profile \
  --claims 'The dashboard loads without a login redirect'

frontend-visualqa verify http://localhost:3000 \
  --headed --no-visualize \
  --claims 'The API status indicator shows Active'

Writing good claims

Claims should be observable, scoped, and provable from pixels. Prefer direct, falsifiable wording over broad interpretations like "is correct."

If a claim requires interaction first, use --navigation-hint instead of encoding steps in the claim text.

Result statuses

For the CLI, frontend-visualqa verify exits 0 only when every claim passes. It exits 1 if any claim is failed, inconclusive, or not_testable. Usage errors still exit with argparse's standard 2.

Reporters

Output format for persisted artifacts. Does not affect CLI stdout or MCP tool responses (always native JSON).

When a Markdown claims file is provided, the markdown reporter preserves the original non-claim lines and annotates only the claim bullets so the report can be fed back in as input.

Each claim result contains:

• finding — the verdict explanation (what was observed)
• proof — the decisive artifact paths, step number, and a compact extracted-text preview
• page — URL and viewport where the claim was evaluated
• trace — the execution trace: actions taken, rich events, screenshot paths, and the saved trace path

{
  "claim": "The monthly quota progress bar is completely filled",
  "status": "failed",
  "finding": "The quota label reads '100%' and '12,500 / 12,500 requests used', but the progress bar is visually only about 65% filled — the bar and the label disagree.",
  "proof": {
    "screenshot_path": "artifacts/run-.../claim-02/step-04.webp",
    "step": 4,
    "after_action": "scroll([640, 720], direction=down, amount=1)",
    "text": null,
    "text_path": null
  },
  "page": {
    "url": "http://localhost:8000/analytics_dashboard.html",
    "viewport": { "width": 1280, "height": 800, "device_scale_factor": 1.0 }
  },
  "trace": {
    "steps_taken": 4,
    "wrong_page_recovered": false,
    "screenshot_paths": ["..."],
    "actions": ["..."],
    "trace_path": "artifacts/run-.../claim-02/trace.json"
  }
}

frontend-visualqa verify http://localhost:3000 \
  --claims 'The checkout total matches the sum of line items' \
  --reporter native --reporter ctrf

CI / GitHub Actions

The repo includes a GitHub Actions workflow (.github/workflows/visualqa.yml) that runs visual QA checks on pull requests targeting main (and supports manual dispatch via workflow_dispatch). Use it as a template for adding frontend-visualqa to your own CI pipeline.

What the workflow does

1. Installs frontend-visualqa via uv tool install and downloads Playwright's Chromium via playwright install chromium --with-deps
2. Serves the example pages with Python's built-in HTTP server
3. Runs visual claims against the login page — element checks, form validation, post-login dashboard
4. Verifies that known visual bugs are caught (progress bar mismatch)
5. Uploads screenshot artifacts and CTRF reports for inspection

Setting up in your own repo

1. Add your Yutori API key as a secret. Go to your repo's Settings > Secrets and variables > Actions and add YUTORI_TESTING_API_KEY with your Yutori API key.

2. Copy the workflow. Adapt .github/workflows/visualqa.yml to your project — replace python3 -m http.server with your dev server start command and wait-on or a sleep until it's ready:

   - name: Start dev server
     run: |
       npm start &
       npx wait-on http://localhost:3000 --timeout 60000
   

3. Write claims for your pages. Each frontend-visualqa verify step tests a set of visual claims against a URL:

   - name: Visual QA — Dashboard
     run: |
       set -o pipefail
       frontend-visualqa verify http://localhost:3000/dashboard \
         --claims \
         'The revenue chart is visible without scrolling' \
         'The sidebar shows 5 navigation items' \
         --reporter native --reporter ctrf | tee visualqa-dashboard.json
   

4. Upload artifacts so screenshots and reports are available even when tests fail:

   - name: Upload visual QA artifacts
     if: always()
     uses: actions/upload-artifact@v6
     with:
       name: visualqa-results
       path: |
         artifacts/
         visualqa-*.json
   

Testing claims that should fail

To verify that frontend-visualqa catches known bugs, capture the exit code and validate the output contains a real failure (not a crash):

- name: Visual QA — Catch known bug
  run: |
    set -o pipefail
    exit_code=0
    frontend-visualqa verify http://localhost:3000 \
      --claims 'The progress bar matches the displayed percentage' \
      --reporter native --reporter ctrf | tee visualqa-bug.json \
      || exit_code=$?

    if [ "$exit_code" -eq 0 ]; then
      echo "UNEXPECTED: claim passed" && exit 1
    fi

    # Verify tool produced a real failed claim, not just a crash
    if [ ! -s visualqa-bug.json ]; then
      echo "ERROR: no output — tool may have crashed" && exit 1
    fi

    if ! python3 -c "
    import json, sys
    data = json.load(open('visualqa-bug.json'))
    results = data.get('results', [])
    if not results or not any(r.get('status') == 'failed' for r in results):
        print('ERROR: output has no failed claim'); sys.exit(1)
    "; then exit 1; fi

    echo "Expected failure: visual bug detected"

Development

uv sync
uv run playwright install chromium
uv run frontend-visualqa --help

Editable install:

uv pip install -e .