vggt-mps

Port of Facebook Research's VGGT (Visual Geometry Grounded Transformer) to Apple Silicon via PyTorch's MPS backend. Takes single or multi-view images and produces depth maps, camera poses, and 3D point clouds.

What it produces

Given N input images, VGGT predicts:

Architecture

The upstream VGGT model is a 1B-parameter transformer trained on multi-view geometry tasks. This repo wraps it with:

• MPS device detection and dtype handling (float32 for Metal compatibility)
• A sparse attention module (vggt_sparse_attention.py) that patches the model at runtime for O(n) memory scaling instead of O(n^2)
• A unified CLI (vggt command with subcommands)
• A Gradio web interface
• An MCP server for Claude Desktop integration

vggt-mps/
  src/
    vggt_core.py                # Core VGGT processing
    vggt_sparse_attention.py    # Runtime sparse attention patch
    config.py                   # Centralized configuration
    visualization.py            # 3D visualization
    commands/                   # CLI subcommands (demo, reconstruct, test, benchmark, web)
    utils/                      # Model loader, image utils, export
  tests/                        # MPS, sparse attention, integration tests
  repo/vggt/                    # Vendored upstream VGGT source

Sparse attention

The sparse attention module replaces standard O(n^2) cross-view attention with a covisibility-masked variant. No retraining required -- it patches the loaded model at runtime.

Output difference vs. standard attention: reported as 0.000000 in tests. In practice this means numerically identical within float32 precision.

Requirements

• Apple Silicon Mac (M1, M2, or M3)
• 8 GB+ RAM
• 6 GB disk for model weights
• Python 3.10+

Install

From source (recommended)

git clone https://github.com/jmanhype/vggt-mps.git
cd vggt-mps
pip install -e .

Or with uv:

make install   # uses uv pip install -e .

Download model weights

vggt download
# or: python main.py download

The model downloads from Hugging Face (~5 GB).

Usage

CLI

vggt demo                              # run with sample images
vggt demo --kitchen --images 4         # kitchen dataset, 4 views
vggt reconstruct data/*.jpg            # your own images
vggt reconstruct --sparse data/*.jpg   # sparse attention for large sets
vggt reconstruct --export ply data/*.jpg
vggt web                               # launch Gradio UI
vggt web --port 8080 --share           # public link
vggt test --suite all                  # run test suite
vggt benchmark --compare               # performance comparison

Python

from src.vggt_sparse_attention import make_vggt_sparse

# Patch any loaded VGGT model for sparse attention
sparse_model = make_vggt_sparse(model, device="mps")
output = sparse_model(images)

MCP server (Claude Desktop)

Add to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "vggt-agent": {
      "command": "uv",
      "args": [
        "run", "--python", "/path/to/vggt-mps/vggt-env/bin/python",
        "--with", "fastmcp", "fastmcp", "run",
        "/path/to/vggt-mps/src/vggt_mps_mcp.py"
      ]
    }
  }
}

Available MCP tools: vggt_quick_start_inference, vggt_extract_video_frames, vggt_process_images, vggt_create_3d_scene, vggt_reconstruct_3d_scene, vggt_visualize_reconstruction.

Dependencies

Limitations

• Runs on Apple Silicon only. No CUDA path in this repo (use upstream VGGT for that).
• Uses float32 exclusively; MPS does not support float16 autocast for this model.
• The vggt download command pulls ~5 GB over the network with no resume support.
• Not published to PyPI yet. Install from source.
• Sparse attention memory numbers in the table above are asymptotic ratios, not measured byte counts.
• The vendored repo/vggt/ tree is a snapshot and may drift from upstream.

References

• VGGT paper
• facebook/vggt (upstream)
• Hugging Face model

Contributing

Development branch is develop. See CONTRIBUTING.md.

License

MIT