HomeAssist

Voice Controlled Personal AI infrastructure. Always listening. Low latency. Actually useful.

HomeAssist V3 is a voice-first assistant tailored to my life, workflows, and data—modular, agentic, and built to feel like a real companion/mentor (not a reactive chatbot). It doesn't just answer: it proactively surfaces what matters, delivers briefings, and nudges me at the right moment.

V3 improvements: Dramatically faster response times and proactive jump-in briefings that speak immediately on wake word detection.

Say the wake word and speak naturally—HomeAssist handles lights, music, calendar, questions, and more. The first steps on the road to building a real-life Jarvis.

What It Does

HomeAssist is a hands-free voice interface that:

• Listens continuously for a customizable wake word
• Transcribes speech in real-time with streaming ASR
• Thinks with LLMs to understand requests and generate responses
• Speaks responses via neural text-to-speech
• Controls smart home devices through an extensible tool system
• Remembers you across sessions with persistent and semantic memory

You can interrupt it mid-sentence (barge-in), ask multi-step questions, and have natural back-and-forth conversations.

Capabilities

Voice Interaction

Smart Home & Services

Memory & Context

Tool Chaining

Multi-step requests are handled automatically:

"Find that song we talked about and text me the link"

The assistant recognizes this needs multiple tools (search → SMS), executes them in sequence, and confirms completion.

Architecture

Design Philosophy

1. Modular providers — Each component (transcription, TTS, wake word, LLM) is swappable via configuration
2. State machine coordination — Audio components are orchestrated through explicit state transitions to prevent conflicts
3. Process isolation — Wake word + fast termination detection run in separate processes to prevent model corruption
4. Tool abstraction — Smart home control via MCP (Model Context Protocol) for clean separation

System Flow

┌─────────────────┐     Wake word      ┌─────────────────┐
│  Microphone     │───────────────────►│  Transcription  │
│  (always on)    │                    │  (streaming)    │
└─────────────────┘                    └────────┬────────┘
                                                │
                                                ▼
┌─────────────────┐     Response       ┌─────────────────┐
│  TTS Speaker    │◄───────────────────│  LLM + Tools    │
│  (neural voice) │                    │  (orchestrator) │
└─────────────────┘                    └────────┬────────┘
                                                │
                                                ▼
                                       ┌─────────────────┐
                                       │  MCP Server     │
                                       │  (tool actions) │
                                       └─────────────────┘

Component Overview

LLM Strategy

Tool Calling Subagent API

The Tool-Calling Mini inference server (https://inference.stuart-labs.com) exposes a fine-tuned Qwen3 model specialized in tool-call routing for the same 12 tools used by HomeAssist. It accepts natural-language messages via POST /v1/chat/completions and returns structured tool_calls with extracted arguments — handling tool selection and parameter extraction on-device rather than through OpenAI.

This provides an alternative delegation path for the tool subagent (tool_subagent_model in config), where tool-call inference can be routed to the local model instead of gpt-4o-mini. Auth uses rotating HMAC-SHA256 API keys (5-minute TTL) from a shared refresh token.

Full API spec: TOOL_CALLING_MINI_API.md

Project Structure

HomeAssist/
├── assistant_framework/       # Core voice assistant
│   ├── orchestrator.py     # Main coordination logic
│   ├── config.py              # All configuration
│   ├── providers/             # Pluggable implementations
│   │   ├── transcription/  # Speech-to-text
│   │   ├── response/          # LLM responses
│   │   ├── tts/               # Text-to-speech
│   │   ├── wakeword/       # Wake word detection
│   │   ├── termination/       # Fast termination detection ("over out")
│   │   └── context/           # Conversation history
│   ├── utils/                 # Shared utilities
│   │   ├── state_machine.py   # Audio lifecycle management
│   │   ├── barge_in.py        # Interrupt detection
│   │   ├── briefing_manager.py
│   │   └── briefing_processor.py
│   └── interfaces/            # Abstract base classes
│       ├── termination.py     # TerminationInterface
│
├── mcp_server/                # Tool server
│   ├── server.py              # MCP entry point
│   ├── tools/                 # Tool implementations
│   │   ├── kasa_lighting.py
│   │   ├── spotify.py
│   │   ├── calendar.py
│   │   ├── weather.py
│   │   └── ...
│   └── tools_config.py        # Enable/disable tools
│
├── scripts/scheduled/         # Background jobs
│   ├── email_summarizer/      # Email digest pipeline
│   ├── news_summary/          # News summary pipeline
│   └── calendar_briefing/     # Calendar reminder announcements
│
├── audio_data/                # Model files
│   ├── wake_word_models/
│   └── piper_models/
│
└── state_management/          # Runtime state
    ├── persistent_memory.json
    └── conversation_summary.json

Quick Start

1. Configure environment

cp env.example .env
# Edit .env with your API keys

Required keys:

• OPENAI_API_KEY — LLM responses
• ASSEMBLYAI_API_KEY — Transcription
• SUPABASE_URL + SUPABASE_KEY — Memory storage

2. Install dependencies

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

3. Run

python3 -m assistant_framework.main continuous

4. Interact

1. Say "Hey Honey" (or your configured wake word)
2. Speak your request
3. Say "send it" or wait for auto-send
4. Listen to the response (interrupt anytime by speaking)
5. Say "over out" to end immediately (including during speech, if fast termination is enabled)

Configuration

All settings are in assistant_framework/config.py. Key sections:

See SETUP.md for detailed configuration reference.

Extending

Adding a new tool

1. Create mcp_server/tools/your_tool.py implementing the tool interface
2. Register in mcp_server/tools_config.py
3. The assistant automatically discovers and can use the tool

Adding a new provider

1. Implement the appropriate interface in assistant_framework/interfaces/
2. Create provider class in assistant_framework/providers/
3. Register in assistant_framework/factory.py
4. Select via config

Custom wake word

1. Train a model using OpenWakeWord
2. Place .onnx file in audio_data/wake_word_models/
3. Update WAKEWORD_CONFIG in config

Requirements

• Python 3.10+
• macOS or Linux with audio support
• Microphone and speakers
• API keys for cloud services (OpenAI, AssemblyAI, etc.)

License

MIT License — See LICENSE file for details.