AnyLoom: AnythingLLM Local AI Agentic Stack

A fully local, multi-agent AI system that gives you ChatGPT-level intelligence with complete privacy and control over your data.

Now with Docker! One command starts the entire stack. Zero manual setup.

💡 What Can You Do With This?

Run a production-grade AI assistant stack entirely on your hardware:

• 🔒 100% private — No data leaves your machine. No API keys. No subscriptions.
• 🧠 Advanced reasoning — Qwen3-30B MoE (30.5B params, 3.3B active) with hybrid thinking mode
• 📚 Hybrid RAG search — Finds YOUR information better than pure vector search (dense + sparse retrieval)
• 🤖 Multi-agent swarm — DyTopo coordination routes complex tasks to specialized agents that collaborate, with optional RAG context pre-fetch for domain grounding
• 🛡️ Reliability hardening — Checkpoint crash recovery, deny-first policy enforcement, deterministic output verification, stalemate detection
• 🛠️ 8 MCP servers — Memory knowledge graph, web search, file operations, sequential thinking, RAG search, multi-agent swarm, system diagnostics
• 🐋 Docker-first architecture — One command to start/stop everything. Auto-restart. Zero networking hassles.
• 💬 AnythingLLM UI — Clean interface for chat, document Q&A, and workspace management

Ideal for:

• Engineers who need AI assistance with proprietary codebases
• Researchers handling sensitive documents (legal, medical, financial)
• Privacy-conscious users who want ChatGPT-level capability without cloud dependency
• Developers building custom AI workflows with persistent memory and multi-agent collaboration

Why AnyLoom vs Cloud AI or Single-LLM Setups?

The bottom line: If you need ChatGPT-level capability for sensitive work, AnyLoom gives you near the same intelligence without the privacy trade-offs or subscription costs.

🌐 How It Works

AnyLoom runs as a Docker Compose stack with these services:

• Qdrant (port 6333) — Vector database for hybrid dense+sparse RAG
• llama.cpp LLM (port 8008) — GPU-accelerated inference with 131K context (Qwen3-30B-A3B)
• llama.cpp Embedding (port 8009) — BGE-M3 embedding server for AnythingLLM (1024-dim dense vectors)
• AnythingLLM (port 3001) — Web UI for chat and document management
• DyTopo swarm (Python, runs natively) — Multi-agent orchestration with stigmergic trace-aware routing
• Health Monitor (Python sidecar) — Deterministic health checks with auto-restart and crash window protection
• 8 MCP servers — RAG search, DyTopo swarm, memory graph, web search, file ops, system diagnostics, and more

Everything starts with one command. Docker handles networking, GPU access, auto-restart, and data persistence.

The entire RAG-prompt set fits comfortably inside the token limit. Context length is configurable (default 131K). Q4_K_M model weights are ~18.6 GiB, leaving ample room for KV cache on 32GB GPUs. See docs/llm-engine.md for VRAM budget details.

✅ Runs on a single GPU (requires 32GB+ VRAM; optimized for RTX 5090)

🛠️ Prerequisites

All you need:

Docker handles everything: Qdrant, llama.cpp (LLM + Embedding), and AnythingLLM run as containers. No manual WSL setup or service management!

🚀 Quickstart

1. Clone and Download Model

git clone <repo-url>
cd AnyLoom

# Download models
mkdir -p models
pip install huggingface_hub

# LLM model — Qwen3-30B-A3B Q4_K_M (~18.6 GB, GPU)
huggingface-cli download Qwen/Qwen3-30B-A3B-Instruct-2507-GGUF \
  Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf \
  --local-dir models

# Embedding model — BGE-M3 Q8_0 (~605 MB, GPU)
huggingface-cli download ggml-org/bge-m3-Q8_0-GGUF \
  bge-m3-q8_0.gguf \
  --local-dir models

Already have the LLM GGUF? Symlink instead of re-downloading: ln -s ~/.lmstudio/models/lmstudio-community/Qwen3-30B-A3B-Instruct-2507-GGUF/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf models/

2. Start the Docker Stack

# One command starts everything (creates volumes, checks model, waits for health)
bash scripts/docker_start.sh

# Or manually (must create volumes first)
docker volume create anyloom_qdrant_storage
docker volume create anyloom_anythingllm_storage
docker volume create anyloom_anythingllm_hotdir
docker compose up -d

Startup takes ~2 minutes while llama.cpp loads the model into GPU VRAM. First query may take an additional 1-2 minutes as the prompt cache warms up.

3. Configure AnythingLLM

1. Open http://localhost:3001 and complete the initial setup wizard (password, preferences). The API is locked until this is done.
2. Then run the automated configuration:

python scripts/configure_anythingllm.py

This configures AnythingLLM system defaults (LLM provider, max tokens, BGE-M3 embedding, vector DB, chunk size/overlap, default system prompt), creates an AnyLoom workspace, uploads and embeds the RAG reference documents from rag-docs/anythingllm/ into the workspace's vector store, pushes tuned workspace settings, and runs a smoke test. Re-running the script is safe — it skips documents that are already uploaded and embedded.

4. Access Services

• AnythingLLM UI: http://localhost:3001
• llama.cpp LLM API: http://localhost:8008/v1/models
• llama.cpp Embedding API: http://localhost:8009/v1/embeddings
• Qdrant Dashboard: http://localhost:6333/dashboard

5. Run Benchmarks (Optional)

# Install Python dependencies first
pip install -r requirements-dytopo.txt

# Test the full stack (all 6 phases)
ANYTHINGLLM_API_KEY=your-key python scripts/benchmarks/bench_run_all.py

# Or test just llama.cpp directly (no AnythingLLM needed)
ANYTHINGLLM_API_KEY=your-key python scripts/benchmarks/bench_phase5_llm.py

Phase 5 validates llama.cpp directly — fabrication guards, tool boundary awareness, and depth calibration. Current score: 15/20 (75%) with perfect marks on fabrication guards, adversarial resistance, cross-workspace parity, depth stability, and LLM direct validation. See benchmark results for full scores.

🔧 Management Commands

# View logs
bash scripts/docker_logs.sh llm           # llama.cpp only
bash scripts/docker_logs.sh anythingllm  # AnythingLLM only
docker compose logs -f                    # All services

# Stop services
bash scripts/docker_stop.sh
# Or: docker compose down

# Restart a specific service
docker compose restart llm

# Check status
docker compose ps

# Start the health monitor sidecar (optional, runs alongside Docker)
python scripts/health_monitor.py

# Remove everything including data (⚠️ DESTRUCTIVE)
docker compose down -v

📚 Documentation

Start here: INSTALL.md — Docker-based installation guide (repo root)

Reference documentation in docs/:

DyTopo Package (src/dytopo/)

Supporting Services

🔄 Data & Persistence

• Docker Volumes (persist across restarts):

  • anyloom_qdrant_storage — Vector database
  • anyloom_anythingllm_storage — AnythingLLM workspaces
  • anyloom_anythingllm_hotdir — AnythingLLM document collector

• Host Bind Mount:

  • ./models/ — GGUF model files (~19.2 GB total). LLM model (~18.6 GB) + embedding model (~605 MB). Place both files here before starting.

• Filesystem Access: All configuration files and Python scripts are local

• Model Updates: Replace the GGUF file in ./models/ and restart: docker compose restart llm

• RAG Re-indexing: Re-run python scripts/configure_anythingllm.py (idempotent) or re-embed documents via AnythingLLM UI

# View volumes
docker volume ls | grep anyloom

# Backup a volume
docker run --rm -v anyloom_qdrant_storage:/data -v $(pwd):/backup ubuntu tar czf /backup/qdrant_backup.tar.gz /data

# Remove all data (⚠️ DESTRUCTIVE)
docker compose down -v

✅ You're now running a next-gen, fully local AI agentic stack. Start creating, querying, and orchestrating with AnyLoom today.