Jonathan Mohan

For context, desktop computers were so slow just a 3 decades ago that searching for files was not practical. We are now very near running models that are almost as good as hosted ai locally. Including the multi modals too. Setting this up: # Daily driver — text + vision + coding, 35B MoE / 3B active, ~17GB Q4 # Vision is built-in (no separate VL model needed) ollama pull qwen3.6:35b-a3b # RAG embeddings — exceeds OpenAI, MTEB 70.58, ~5GB Q4 ollama pull qwen3-embedding:8b # Local AI Setup Guide Hardware: M5 Max 128GB MacBook Pro + DGX Spark (optional) ## 1. Install ```bash # Ollama (inference engine) brew install ollama echo 'export OLLAMA_USE_MLX=1' >> ~/.zshrc source ~/.zshrc ollama serve & # Open WebUI (frontend — vision, search, RAG, tools, code execution) docker run -d -p 3000:8080 --name open-webui \ -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \ -v open-webui:/app/backend/data \ ghcr.io/open-webui/open-webu… # SearXNG (private web search) docker run -d -p 8080:8080 --name searxng searxng/searxng ``` Enable JSON output in SearXNG: edit `settings.yml`, add `formats: - json` under `search:`. In Open WebUI Admin → Web Search: enable, set engine to `searxng`, URL to `http://host.docker.internal:8080/search?q=<query>`. ## 2. Pull Models ```bash # Daily driver — text + vision + coding, 35B MoE / 3B active, ~17GB Q4 # Vision is built-in (no separate VL model needed) ollama pull qwen3.6:35b-a3b # RAG embeddings — exceeds OpenAI, MTEB 70.58, ~5GB Q4 ollama pull qwen3-embedding:8b # Tool calling router — tiny, fast, 0.960 agent score ollama pull qwen3:1.7b ``` Optional: ```bash # Dedicated coding model (if you want a separate code-focused model) ollama pull qwen3-coder:30b # Separate vision specialist (if you need deeper image analysis than qwen3.6 provides) ollama pull qwen3-vl:8b ``` ## 3. RAG (document Q&A) Open WebUI has built-in RAG. In Admin → Documents: - Embedding model: `qwen3-embedding:8b` - Chunk size: 1024, overlap: 128 - Upload PDFs/docs directly in chat with `#` prefix For heavy document work, install Marker for PDF parsing: ```bash pip install marker-pdf marker_single input.pdf output/ ``` ## 4. Code Execution Open WebUI has built-in sandboxed code execution. Enable in Admin → Settings. For MCP-based sandbox (works with any MCP client): ```bash git clone github.com/Automata-Labs-tea… cd code-sandbox-mcp && npm install ``` ## 5. DGX Spark (if owned) ```bash # Point Open WebUI at Spark's Ollama docker run -d -p 3000:8080 \ -e OLLAMA_BASE_URL=http://SPARK_IP:11434 \ ghcr.io/open-webui/open-webu… # Or use TensorRT-LLM for max performance on Spark # (2.5x faster than Ollama on supported models) ``` Run heavy models on Spark, light models on Mac. Open WebUI can connect to multiple Ollama instances. ## 6. What You Get vs Cloud | Feature | Local | vs GPT/Claude | |---------|-------|:---:| | Chat | Qwen3.6-35B-A3B | 85-90% | | Images | Qwen3-VL-8B | 90-95% (beats GPT-4o on docs) | | Web search | SearXNG | 90% | | Documents | RAG + Marker | 70-80% | | Tool calling | Qwen3 | 90% | | Code exec | Sandbox | 85% | | Privacy | 100% local | — | | Cost after hardware | $0 | — | ## 7. When to Use Cloud Keep Claude Code + Opus 4.6 for: - Complex multi-file refactoring - Hard debugging requiring deep reasoning - Novel architectural decisions - Long multi-turn conversations (200K+ context) Everything else runs locally