AnyLoom: AnythingLLM Local AI Agentic Stack

A fully local, multi-agent AI system that gives you ChatGPT-level intelligence with complete privacy and control over your data.

Now with Docker! One command starts the entire stack. Zero manual setup.

---

💡 What Can You Do With This?

Run a production-grade AI assistant stack entirely on your hardware:

• 🔒 100% private — No data leaves your machine. No API keys. No subscriptions.
• 🧠 Advanced reasoning — Qwen3-30B MoE (30.5B params, 3.3B active) with hybrid thinking mode
• 📚 Hybrid RAG search — Finds YOUR information better than pure vector search (dense + sparse retrieval)
• 🤖 Multi-agent swarm — DyTopo coordination routes complex tasks to specialized agents that collaborate, with optional RAG context pre-fetch for domain grounding
• 🛡️ Reliability hardening — Checkpoint crash recovery, deny-first policy enforcement, deterministic output verification, stalemate detection
• 🛠️ 8 MCP servers — Memory knowledge graph, web search, file operations, sequential thinking, RAG search, multi-agent swarm, system diagnostics
• 🐋 Docker-first architecture — One command to start/stop everything. Auto-restart. Zero networking hassles.
• 💬 AnythingLLM UI — Clean interface for chat, document Q&A, and workspace management

Ideal for:

• Engineers who need AI assistance with proprietary codebases
• Researchers handling sensitive documents (legal, medical, financial)
• Privacy-conscious users who want ChatGPT-level capability without cloud dependency
• Developers building custom AI workflows with persistent memory and multi-agent collaboration

---

Why AnyLoom vs Cloud AI or Single-LLM Setups?

	AnyLoom	Cloud AI (ChatGPT, Claude)	Single Local LLM
Privacy	✅ 100% local, zero telemetry	❌ Your data trains their models	✅ Local
Cost	✅ One-time hardware investment	❌ $20-200/month subscription	✅ Free after setup
Retrieval Quality	✅ Hybrid dense+sparse RAG	⚠️ Dense-only embeddings	⚠️ Basic or no RAG
Multi-Agent Swarm	✅ DyTopo routing, 3-5 agents	❌ Single model per request	❌ Single model
Persistent Memory	✅ MCP knowledge graph across sessions	⚠️ Limited to conversation	❌ No cross-session memory
Tool Ecosystem	✅ 8 MCP servers (RAG, swarm, web, memory, files, diagnostics)	⚠️ Limited, cloud-gated	❌ Manual integration
Context Window	✅ 131K tokens (configurable)	⚠️ 128K (expensive tiers)	⚠️ Varies by model
Offline Use	✅ Fully functional	❌ Requires internet	✅ Fully functional

The bottom line: If you need ChatGPT-level capability for sensitive work, AnyLoom gives you near the same intelligence without the privacy trade-offs or subscription costs.

---

🌐 How It Works

AnyLoom runs as a Docker Compose stack with these services:

• Qdrant (port 6333) — Vector database for hybrid dense+sparse RAG
• llama.cpp LLM (port 8008) — GPU-accelerated inference with 131K context (Qwen3-30B-A3B)
• llama.cpp Embedding (port 8009) — BGE-M3 embedding server for AnythingLLM (1024-dim dense vectors)
• AnythingLLM (port 3001) — Web UI for chat and document management
• DyTopo swarm (Python, runs natively) — Multi-agent orchestration with stigmergic trace-aware routing
• Health Monitor (Python sidecar) — Deterministic health checks with auto-restart and crash window protection
• 8 MCP servers — RAG search, DyTopo swarm, memory graph, web search, file ops, system diagnostics, and more

Everything starts with one command. Docker handles networking, GPU access, auto-restart, and data persistence.

Component	Tokens
Total Token Budget	131K
System prompt	~2K
MCP tool definitions (6 AnythingLLM + 2 agent)	~3K
RAG snippets (16 × ~500 tokens)	~8K
Chat history (30 messages)	~12K
Overhead Subtotal:	~25K
Remaining for chat	~106K

The entire RAG-prompt set fits comfortably inside the token limit. Context length is configurable (default 131K). Q4_K_M model weights are ~18.6 GiB, leaving ample room for KV cache on 32GB GPUs. See docs/llm-engine.md for VRAM budget details.

✅ Runs on a single GPU (requires 32GB+ VRAM; optimized for RTX 5090)

---

🛠️ Prerequisites

All you need:

Component	Requirement
Docker Desktop	v24.0+ with WSL2 integration and GPU support enabled
NVIDIA GPU	RTX 4090/5090 or similar (32GB VRAM recommended for full 131K context. 24GB GPUs can run with reduced context.)
NVIDIA Driver	535+ (for CUDA 12 support)
Python	3.10+ (for benchmarks and DyTopo scripts)
Disk Space	~100GB for models and data

Docker handles everything: Qdrant, llama.cpp (LLM + Embedding), and AnythingLLM run as containers. No manual WSL setup or service management!

---

🚀 Quickstart

1. Clone and Download Model

git clone <repo-url>
cd AnyLoom

# Download models
mkdir -p models
pip install huggingface_hub

# LLM model — Qwen3-30B-A3B Q4_K_M (~18.6 GB, GPU)
huggingface-cli download Qwen/Qwen3-30B-A3B-Instruct-2507-GGUF \
  Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf \
  --local-dir models

# Embedding model — BGE-M3 Q8_0 (~605 MB, GPU)
huggingface-cli download ggml-org/bge-m3-Q8_0-GGUF \
  bge-m3-q8_0.gguf \
  --local-dir models

Already have the LLM GGUF? Symlink instead of re-downloading: ln -s ~/.lmstudio/models/lmstudio-community/Qwen3-30B-A3B-Instruct-2507-GGUF/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf models/

2. Start the Docker Stack

# One command starts everything (creates volumes, checks model, waits for health)
bash scripts/docker_start.sh

# Or manually (must create volumes first)
docker volume create anyloom_qdrant_storage
docker volume create anyloom_anythingllm_storage
docker volume create anyloom_anythingllm_hotdir
docker compose up -d

Startup takes ~2 minutes while llama.cpp loads the model into GPU VRAM. First query may take an additional 1-2 minutes as the prompt cache warms up.

3. Configure AnythingLLM

1. Open http://localhost:3001 and complete the initial setup wizard (password, preferences). The API is locked until this is done.
2. Then run the automated configuration:

python scripts/configure_anythingllm.py

This configures AnythingLLM system defaults (LLM provider, max tokens, BGE-M3 embedding, vector DB, chunk size/overlap, default system prompt), creates an AnyLoom workspace, uploads and embeds the RAG reference documents from rag-docs/anythingllm/ into the workspace's vector store, pushes tuned workspace settings, and runs a smoke test. Re-running the script is safe — it skips documents that are already uploaded and embedded.

4. Access Services

• AnythingLLM UI: http://localhost:3001
• llama.cpp LLM API: http://localhost:8008/v1/models
• llama.cpp Embedding API: http://localhost:8009/v1/embeddings
• Qdrant Dashboard: http://localhost:6333/dashboard

5. Run Benchmarks (Optional)

# Install Python dependencies first
pip install -r requirements-dytopo.txt

# Test the full stack (all 6 phases)
ANYTHINGLLM_API_KEY=your-key python scripts/benchmarks/bench_run_all.py

# Or test just llama.cpp directly (no AnythingLLM needed)
ANYTHINGLLM_API_KEY=your-key python scripts/benchmarks/bench_phase5_llm.py

Phase 5 validates llama.cpp directly — fabrication guards, tool boundary awareness, and depth calibration. Current score: 15/20 (75%) with perfect marks on fabrication guards, adversarial resistance, cross-workspace parity, depth stability, and LLM direct validation. See benchmark results for full scores.

---

🔧 Management Commands

# View logs
bash scripts/docker_logs.sh llm           # llama.cpp only
bash scripts/docker_logs.sh anythingllm  # AnythingLLM only
docker compose logs -f                    # All services

# Stop services
bash scripts/docker_stop.sh
# Or: docker compose down

# Restart a specific service
docker compose restart llm

# Check status
docker compose ps

# Start the health monitor sidecar (optional, runs alongside Docker)
python scripts/health_monitor.py

# Remove everything including data (⚠️ DESTRUCTIVE)
docker compose down -v

---

📚 Documentation

Start here: INSTALL.md — Docker-based installation guide (repo root)

Reference documentation in docs/:

Document	Contents
architecture.md	System topology, VRAM budget, port assignments
llm-engine.md	llama.cpp Docker container config, GPU settings, troubleshooting
qwen3-model.md	Qwen3-30B-A3B MoE architecture, quantization, sampling
bge-m3-embedding.md	BGE-M3 embedding architecture (ONNX INT8 CPU for MCP RAG + llama.cpp GGUF for AnythingLLM, 1024-dim dense vectors)
qdrant-topology.md	Qdrant Docker container, collection schema, sync
qdrant-servers.md	MCP server inventory, tool definitions, token budget
dytopo-swarm.md	DyTopo multi-agent routing, package architecture, domains, lifecycle
anythingllm-settings.md	AnythingLLM Docker container, provider config, workspace setup
benchmark-results-showcase.md	Benchmark results across all rounds

DyTopo Package (src/dytopo/)

Module	Purpose
models.py	Pydantic v2 data models (AgentState, SwarmTask with RAG context field, SwarmMetrics, etc.)
config.py	YAML configuration loader with defaults (dytopo_config.yaml)
agents.py	System prompts, JSON schemas, domain rosters
router.py	MiniLM-L6-v2 embedding, cosine similarity, threshold, degree cap, intent embedding enrichment, descriptor separation validation, optional HyDE (Hypothetical Document Embeddings) for manifold-projected routing
stigmergic_router.py	Trace-aware topology: Qdrant-persisted swarm traces, time-decayed boost matrix
graph.py	NetworkX DAG construction, cycle breaking, topological sort
orchestrator.py	Main swarm loop with singleton inference client, Aegean termination, memory persistence; integrates checkpoint, policy, verifier, stalemate modules via guarded imports
governance.py	Convergence detection, stalling detection, re-delegation, Aegean consensus voting, stalemate detection with generalist fallback
checkpoint.py	CheckpointManager for crash recovery — atomic writes, Pydantic v2 serialization, resume from last good checkpoint
policy.py	PolicyEnforcer (PCAS-Lite) — deny-first tool-call policy enforcement with path traversal prevention
verifier.py	OutputVerifier for deterministic output verification — syntax check, schema validation, no LLM
audit.py	JSONL audit logging to ~/dytopo-logs/{task_id}/
health/checker.py	Pre-run health probes for LLM, Qdrant, AnythingLLM, GPU
memory/writer.py	Post-run swarm result persistence to structured storage

Supporting Services

Component	Purpose
src/mcp_servers/system_status_mcp.py	FastMCP server: 6 diagnostic tools (service_health, qdrant_collections, gpu_status, llm_slots, docker_status, stack_config)
scripts/health_monitor.py	Standalone sidecar: periodic health checks, auto-restart via docker restart, crash window protection (3 attempts/15min), JSONL logging
scripts/visualize_trace.py	CLI trace visualizer: generates Mermaid flowcharts and self-contained HTML timelines from audit.jsonl logs, with loop/stall detection

---

🔄 Data & Persistence

• Docker Volumes (persist across restarts):

  • anyloom_qdrant_storage — Vector database
  • anyloom_anythingllm_storage — AnythingLLM workspaces
  • anyloom_anythingllm_hotdir — AnythingLLM document collector

• Host Bind Mount:

  • ./models/ — GGUF model files (~19.2 GB total). LLM model (~18.6 GB) + embedding model (~605 MB). Place both files here before starting.

• Filesystem Access: All configuration files and Python scripts are local

• Model Updates: Replace the GGUF file in ./models/ and restart: docker compose restart llm

• RAG Re-indexing: Re-run python scripts/configure_anythingllm.py (idempotent) or re-embed documents via AnythingLLM UI

# View volumes
docker volume ls | grep anyloom

# Backup a volume
docker run --rm -v anyloom_qdrant_storage:/data -v $(pwd):/backup ubuntu tar czf /backup/qdrant_backup.tar.gz /data

# Remove all data (⚠️ DESTRUCTIVE)
docker compose down -v

---

✅ You're now running a next-gen, fully local AI agentic stack. Start creating, querying, and orchestrating with AnyLoom today.