The Local AI Self-Hosted Stack

Install Ollama from ollama.com — it runs as a local server that manages model downloads and inference. Run `ollama pull llama3.2` for a capable model that runs on most hardware. For 8GB+ VRAM, try `ollama pull mistral` or `ollama pull qwen2.5:14b`. Verify with `ollama run llama3.2` in your terminal before moving on.

Run Open WebUI via Docker: `docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway ghcr.io/open-webui/open-webui:main`. It auto-detects your local Ollama instance. Visit localhost:3000 and create a local account — you now have a ChatGPT-quality interface running entirely offline with conversation history, model switching, and file uploads.

Install Jan from jan.ai for a native desktop app to browse, download, and run models without the command line. Use the model hub to try quantized Llama 3, Mistral, and Phi-3 variants before committing to a model for production. Jan runs its own local server on port 1337 and complements Ollama for model experimentation.

Sign up for a free Groq API key at console.groq.com — the free tier allows 14,400 tokens/min. In Open WebUI go to Settings → Connections and add Groq as an OpenAI-compatible endpoint using https://api.groq.com/openai/v1 and your key. You can now switch to llama-3.1-70b-versatile when local inference is too slow or you need higher output quality.

In Open WebUI, create two presets: one routing to local Ollama models (offline-safe, zero cost) and one to Groq (fast cloud). Test both with the same prompt and compare quality vs latency. Use local for drafts and iteration, Groq for final outputs where quality matters. You now have a fully private AI stack with a paid-quality escape hatch.