Multi Agent Federal Form Automation System
No description available
Ask AI about Multi Agent Federal Form Automation System
Powered by Claude Β· Grounded in docs
I know everything about Multi Agent Federal Form Automation System. Ask me about installation, configuration, usage, or troubleshooting.
0/500
Reviews
Documentation
Multi-Agent Federal Form Automation System
Vision-guided discovery + Contract-first execution = Voice-accessible government services
The multi-agent federal form automation system uses FederalScout to visually discover wizard structures and FederalRunner to execute them atomically. Together, these specialized agents transform government calculatorsβfrom FSA Student Aid Estimators to Social Security retirement calculatorsβinto voice-accessible tools.
πΊ Watch the System in Action
βΆοΈ Watch on YouTube: Complete Technical Walkthrough
See live discovery sessions, the contract-first pattern in action, and voice-to-result demos
π― The Problem
Every year, millions of Americans struggle with government form wizards:
- FSA Student Aid Estimator β 6 pages, 47 fields, 15 minutes
- Social Security Calculator β 3 pages, complex eligibility rules
- IRS Tax Withholding β 5 pages, conditional logic
- Federal Loan Simulator β 4 pages, repetitive data entry
What if you could just describe your situation and get results?
π‘ The Solution
A multi-agent system that combines AI vision with browser automation to transform government calculators into voice-accessible tools.
System Architecture

Two Specialized Agents
π FederalScout (Discovery Agent)
- Runs locally via Claude Desktop
- Uses Claude Vision to discover form structures
- Generates contract-first artifacts (Wizard Structure + User Data Schema)
- One-time setup per form
π FederalRunner (Execution Agent)
- Deploys to Google Cloud Run
- Reads schemas to collect user data naturally
- Executes forms atomically (8 seconds)
- Production-ready for daily use
ποΈ Contract-First Pattern Innovation
Traditional approach:
Discover form β Hardcode mappings β Maintain code per form
Our approach:
Discovery generates contract β Claude reads schema β Universal execution
The Contract (JSON Schema)
FederalScout discovers forms and generates TWO artifacts:
- Wizard Structure - Playwright execution instructions
- User Data Schema - THE CONTRACT defining required inputs
Result: FederalRunner doesn't need field mappers. Claude reads the schema and collects data naturally!
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "https://formflow.io/schemas/fsa-student-aid-estimator.json",
"title": "FSA Student Aid Estimator - User Data Schema",
"type": "object",
"required": ["birth_month", "birth_day", "birth_year", "marital_status"],
"properties": {
"birth_month": {
"type": "string",
"pattern": "^(0[1-9]|1[0-2])$",
"description": "Student's birth month (01-12)"
}
}
}
Claude reads this schema β Asks user for birth_month β Validates format β FederalRunner maps to selector β Executes!
No hardcoded field mappers. Universal design. Works with ANY form.
π Key Innovations
1. Vision-Guided Discovery
"When AI can see screenshots, you don't need to hardcode selectors."
Traditional automation:
- Manual DOM inspection
- Brittle CSS selectors
- Breaks on updates
FederalScout approach:
- Claude Vision analyzes screenshots
- Self-correcting interaction loops
- Adapts to form changes
- 99%+ accuracy
2. Contract-First Pattern
"The schema is the contract between discovery and execution."
Discovery generates schema β Claude collects data by reading schema β Execution validates and runs
Benefits:
- No field_mapper.py needed
- Universal tools work with ANY wizard
- Type-safe data collection
- Automatic validation
3. Atomic Execution
"Every execution is reproducible and traceable."
Pattern: Launch β Fill β Extract β Close (8 seconds)
- No session state between executions
- Complete screenshot audit trail
- Cloud Run compatible
- 100% reproducible
4. Conversation Size Optimization
"Strategic tool design prevents token limits."
Critical learnings:
- MCP ImageContent format β 50-70% size reduction
- Batch operations β 83% fewer tool calls
- Incremental saves β Zero data loss
- Screenshot optimization β 115KB β 50KB
Result: Handles 10-15+ page wizards within Claude Desktop limits
π οΈ Tech Stack
Core Technologies
- Python 3.11+ - Backend language
- Playwright - Browser automation
- MCP Protocol - Model Context Protocol for tool integration
- Pydantic - Data validation and schemas
- FastAPI - HTTP server for FederalRunner
- Auth0 + OAuth 2.1 - Cloud authentication
Deployment
- FederalScout: Local MCP (stdio) β Claude Desktop
- FederalRunner: Remote MCP (HTTP) β Google Cloud Run
- Storage: JSON files (git-tracked, version-controlled)
π¦ Quick Start
Prerequisites
- Python 3.11+
- Claude Desktop (for discovery)
- Google Cloud account (for execution deployment)
1. Install FederalScout (Discovery Agent)
# Clone repository
git clone https://github.com/georgevetticaden/multi-agent-federal-form-automation-system.git
cd multi-agent-federal-form-automation-system
# Install FederalScout
cd mcp-servers/federalscout-mcp
python3.11 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
# Install Playwright browsers
# WebKit: Default, works in headless mode with government sites
# Chromium: Required for demo mode (connecting to existing browser)
playwright install webkit chromium
# Optional: Install Firefox for additional browser testing
# playwright install firefox
2. Configure Claude Desktop
Edit: ~/Library/Application Support/Claude/claude_desktop_config.json
{
"mcpServers": {
"federalscout": {
"command": "/path/to/multi-agent-federal-form-automation-system/mcp-servers/federalscout-mcp/venv/bin/python",
"args": ["/path/to/multi-agent-federal-form-automation-system/mcp-servers/federalscout-mcp/src/server.py"],
"env": {
"FEDERALSCOUT_HEADLESS": "false",
"FEDERALSCOUT_BROWSER_TYPE": "webkit",
"FEDERALSCOUT_WIZARDS_DIR": "/path/to/multi-agent-federal-form-automation-system/wizards"
}
}
}
}
3. Discover Your First Form
- Restart Claude Desktop
- Start conversation:
"Discover the FSA Student Aid Estimator at https://studentaid.gov/aid-estimator/" - Claude uses FederalScout to map the wizard
- Both artifacts saved to
wizards/directory
4. Deploy FederalRunner (Execution Agent)
cd mcp-servers/federalrunner-mcp
# Configure Auth0 (see docs/execution/AUTH0_SETUP.md)
cp .env.example .env
# Edit .env with Auth0 credentials
# Deploy to Cloud Run
./scripts/deploy-to-cloud-run.sh
Detailed guides:
π Project Structure
multi-agent-federal-form-automation-system/
βββ mcp-servers/
β βββ federalscout-mcp/ # Discovery agent (local MCP)
β β βββ src/
β β β βββ server.py # MCP stdio server
β β β βββ discovery_tools.py # 7 discovery tools
β β β βββ playwright_client.py
β β β βββ models.py # Pydantic models
β β βββ tests/
β β
β βββ federalrunner-mcp/ # Execution agent (HTTP MCP)
β βββ src/
β β βββ server.py # FastAPI HTTP server
β β βββ auth.py # OAuth 2.1
β β βββ execution_tools.py # 3 execution tools
β β βββ schema_validator.py
β βββ Dockerfile # Cloud Run container
β
βββ wizards/ # Discovered wizard data (SHARED)
β βββ wizard-data/ # Wizard structures (Playwright)
β β βββ fsa-student-aid-estimator.json
β βββ wizard-schemas/ # User data schemas (THE CONTRACT)
β βββ fsa-student-aid-estimator-schema.json
β
βββ schemas/ # Universal validation schemas
β βββ wizard-structure-v1.schema.json
β
βββ agents/ # Agent instruction prompts
β βββ federalscout-instructions.md
β βββ federalrunner-instructions.md
β
βββ docs/ # Documentation
β βββ images/
β β βββ federal-form-architecture.png
β βββ discovery/
β β βββ CLAUDE_DESKTOP_SETUP.md
β β βββ OPTIMIZATIONS.md
β βββ execution/
β βββ DEPLOYMENT_GUIDE.md
β βββ AUTH0_SETUP.md
β
βββ requirements/ # Technical specifications
β βββ shared/
β β βββ CONTRACT_FIRST_FORM_AUTOMATION.md
β β βββ WIZARD_STRUCTURE_SCHEMA.md
β βββ discovery/
β β βββ DISCOVERY_REQUIREMENTS.md
β βββ execution/
β βββ EXECUTION_REQUIREMENTS.md
β
βββ CLAUDE.md # Implementation guide for AI
βββ README.md # This file
π§ͺ Testing
FederalScout Discovery Tests
cd mcp-servers/federalscout-mcp
source venv/bin/activate
# Run all tests
pytest tests/ -v
# Visual debugging (browser visible)
FEDERALSCOUT_HEADLESS=false pytest tests/ -v
FederalRunner Execution Tests
cd mcp-servers/federalrunner-mcp
# Start local server
python src/server.py
# In another terminal
pytest tests/remote/ -v
π Real-World Applications
Government Forms Supported
Federal:
- β FSA Student Aid Estimator (7 pages, 17 fields) - Fully discovered & executable
- β Federal Student Loan Simulator (6 pages, repeatable fields) - Fully discovered & executable
- π Social Security Quick Calculator (3 pages) - Planned
- π IRS Tax Withholding Estimator (5 pages) - Planned
Potential Extensions:
- Medicare Eligibility Calculator
- SNAP Benefits Estimator
- Housing Assistance Applications
- Small Business Loan Calculators
- Veteran Benefits Estimators
Beyond Government
This pattern works for any multi-page web form lacking APIs:
Healthcare: Insurance quotes, prior authorization, patient assistance Finance: Loan applications, refinancing, credit cards Education: Scholarship applications, financial aid calculators
Anywhere humans click through wizards, AI can automate.
π Documentation
Discovery Phase
- FederalScout README - Detailed tool documentation
- Claude Desktop Setup - Configuration guide
- Optimizations - Conversation size management
- Agent Instructions - Discovery workflow
Execution Phase
- FederalRunner README - Execution tools
- Deployment Guide - Cloud Run deployment
- Test Instructions - Testing guide
- Auth0 Implementation - OAuth configuration
- Auth0 Concepts - OAuth 2.1 concepts
Technical Specifications
- Contract-First Pattern - Core pattern
- Wizard Structure Schema - JSON format
- MCP Tool Specifications - Tool contracts
- Discovery Requirements - FederalScout specs
- FastAPI Server Requirements - Server implementation
- Auth0 Requirements - OAuth configuration
- Deployment Requirements - Cloud Run deployment
Implementation Guide
- CLAUDE.md - Complete implementation roadmap for AI assistants
π License
MIT License - See LICENSE for details.
Key points:
- β Free for personal, academic, and commercial use
- β Modify and distribute as needed
- β No warranty provided
- β οΈ Production use with real PII requires compliance review

