1 Why OMLX? — The Local LLM Journey
Samuel opens by reflecting on the extensive testing he's done running local LLMs — from Turbo Quant and GGML vs MLX formats, to M5 vs M1 hardware with varying RAM configurations. After all that experimentation, he's arrived at a definitive answer for June 2026: OMLX is the cleanest, most refined way to run open-source local LLMs on a Mac.
He emphasizes this isn't just another tool comparison — it's where he's landed after months of real-world use across multiple projects. The video covers setup, rationale, and integration with coding agents like Claude Code, Pi, Open Code, and Hermes.
2 The Special Trick — SSD-Backed KV Cache
The feature that sets OMLX apart from every other local inference solution is its persistent SSD-backed KV cache. Unlike standard inference servers that lose all cache on restart, OMLX stores cache blocks on disk in safe tensor format — the same format used for model weights.
Two-Tier Cache Architecture
- Hot blocks stay in RAM — actively used context windows for maximum speed
- Cold blocks go to SSD — using an LRU (Least Recently Used) eviction policy, older cache blocks are pushed to disk rather than deleted
- Prefix restoration — previously seen prompt prefixes are restored across requests and even server restarts, never recomputed
This means cold starts are dramatically faster. When you restart OMLX or your Mac, the model picks up where it left off — cached context from previous sessions is instantly available without reprocessing.
3 Where Ollama & LM Studio Fall Short
Samuel explains why he moved away from the two most popular Mac inference tools:
Ollama
- Slowly rolling out MLX support, but as of the time of recording, only had one MLX model (Qwen 3.5)
- Limited model selection compared to the MLX community's extensive catalog on Hugging Face
LM Studio
- Has a nice UI for downloading and targeting MLX models specifically — great for beginners
- However, it's become very bloated over time
- Running a heavy GUI application consumes RAM that should be reserved for the LLMs themselves
- Default context size is only ~4,000 tokens, which is restrictive for agentic workloads
4 Installing OMLX
Installation is straightforward — OMLX releases everything on GitHub with pre-releases and stable builds:
- Navigate to the OMLX website or the GitHub releases page
- Download the DMG file for the latest stable build (or build from the zip yourself)
- After installation, a menu bar icon appears at the top of the screen
- From the menu, you can open Settings or the Web Dashboard
Samuel notes that the Settings panel tends to be cleaner than the web dashboard for configuration, though both are available.
5 UI Walkthrough & Downloading Models
Models download directly from Hugging Face (or ModelScope). The workflow:
- Search for your desired model on Hugging Face — e.g.,
Gemma 4 12B - Filter for MLX variants to find quantized versions optimized for Apple Silicon
- Copy the Hugging Face URL and paste it into OMLX's settings panel
- Hit download — OMLX handles the rest
Recommended Models for Agentic Work
For coding and agentic tasks, Samuel specifically recommends Mixture of Experts (MoE) models. He demonstrates downloading the Qwen 3.6 MoE at 8-bit quantization from the MLX Community on Hugging Face — a model well-suited for coding tasks that fits comfortably in 128 GB RAM.
You can also quantize your own models through OMLX, though he doesn't cover quantization in this episode.
6 Serving Your Local Model
Once a model is downloaded and loaded, OMLX exposes an OpenAI-compatible API server with multiple endpoints. The web dashboard provides:
- Endpoint URLs — ready to paste into any tool that supports OpenAI-compatible APIs
- Quick start snippets — pre-built configuration for Claude Code, Pi, Open Code, and other harnesses
- Tailscale network endpoints — if you're running Tailscale, OMLX exposes endpoints for remote access, letting you use local models from anywhere on your network
Samuel demonstrates launching Claude Code with the local OMLX model, showing the Activity Monitor as the GPU starts processing requests. He notes the terminal tool he uses is Warp.
7 Seeing the Cache in Action
The web dashboard provides real-time visibility into the cache system:
- Token speed — tokens per second with and without cache hits
- Cached token count — how many tokens are currently cached
- Efficiency gains — percentage improvement from cache vs. cold computation
- Cache size on disk — in Samuel's demo, the cache grew to 1.5 GB after just a brief coding session
The cache can be cleared manually when needed. Samuel also highlights the RAM impact: even though the Qwen 3.6 MoE model is ~36 GB, with context and cache, the actual RAM usage climbed to around 80 GB — a critical consideration for hardware planning.
8 Context Size & Model Parameters
OMLX allows fine-tuning model parameters through the settings panel. Samuel explores the context size configuration:
- Qwen 3.6 MoE supports up to 262,144 tokens of context
- Contrast with LM Studio's default of ~4,000 tokens — a massive difference for agentic workloads
- Changing context size requires a model reload — plan accordingly
His practical advice: start with the default context and use the model until the context window runs out. Thanks to SSD-backed caching, you won't lose the performance gains from previous work — cached prefixes persist through reloads.
9 Integrating with Agent Harnesses
Samuel shares his post-recording realization: while he demoed Claude Code, he actually prefers running local models through Open Code. The reason? Claude Code is notorious for context blowing — burning through context windows quickly — and with limited local hardware, preserving context is essential.
Setting Up Providers
Integration with any coding agent follows the same pattern:
- Create a new provider in your agent's configuration pointing to OMLX's base URL
- Set the API key (configured during OMLX setup)
- Select the model name from OMLX's loaded models
He demonstrates this with Pi, creating a new provider called "OMLX", linking it to the local server URL, and selecting the Qwen model — instantly running a fully local agentic coding session.
Network Access via Tailscale
For users with Tailscale, OMLX's endpoints are accessible across your entire mesh network — meaning you can run the model on a powerful Mac and access it from any device in your home or beyond.
🎯 Key Takeaways
🔑 Key Takeaways
- OMLX is the definitive Mac LLM server — built on top of MLX LM, it adds a server layer, web dashboard, and persistent caching without the bloat of LM Studio or the limited MLX support of Ollama
- SSD-backed KV caching is the killer feature — cache blocks persist to disk in safe tensor format, surviving server restarts and eliminating redundant recomputation of previously seen prefixes
- Two-tier cache architecture — hot blocks stay in RAM for speed, cold blocks evict to SSD using LRU policy, balancing performance with memory constraints
- RAM planning must account for context — a 36 GB model can consume 80 GB of RAM with active context and cache; plan hardware accordingly
- MoE models are ideal for agentic coding — Mixture of Experts architectures like Qwen 3.6 MoE offer strong coding performance while fitting in consumer hardware
- Ollama's MLX support is still limited — only one MLX model was available at the time of recording, making it impractical for MLX-first workflows
- LM Studio has become too bloated — its GUI consumes RAM that should be reserved for the LLMs themselves; OMLX is leaner and more resource-efficient
- Open Code over Claude Code for local models — Claude Code's aggressive context usage is wasteful on limited local hardware; Open Code preserves context better
- Tailscale integration enables remote access — OMLX exposes network endpoints that work across your Tailscale mesh, letting you serve models to any device
- Context size matters for agents — Qwen 3.6 MoE supports 262K tokens vs. LM Studio's default 4K; large context windows are critical for agentic coding workflows
🔗 Resources & Links
- OMLX — official site for downloading the OMLX local LLM server for Mac
- Code Examples — configuration files and setup snippets referenced in the video