What happens to the AI industry when a 744-billion-parameter model runs on a $2,000 desktop? Colibrì, an open-source Mixture-of-Experts runtime written in pure C with zero dependencies, just answered that question. The project, which hit nearly 16,000 GitHub stars in its first three weeks, can load and run GLM-5.2 (a 744B-parameter MoE model from Z.AI) on consumer machines with as little as 25 GB of RAM. It achieves this by streaming expert weights from disk on demand instead of holding the entire model in memory. The result is the first practical way to run a frontier-scale model on hardware that costs less than a single H100 GPU fan.
For solo founders and small teams priced out of cloud GPU rentals, Colibrì changes the math of AI experimentation. Instead of paying thousands per month to access frontier models through APIs, you can run a genuinely large model locally on a moderately spec'd workstation. The implications for prototyping, privacy-sensitive workloads, and latency-sensitive applications are significant.
How Colibrì Squeezes a 744B Model Into 25GB of RAM
The key insight is that Mixture-of-Experts models activate only a fraction of their total parameters per token. GLM-5.2 has 744 billion total parameters spread across 75 MoE layers with 256 experts each (19,456 experts in total). But per token, only approximately 40 billion parameters are active, and only about 11 billion of those change from token to token (the routed experts). Colibrì exploits this sparsity ruthlessly: the dense parts (attention, shared experts, embeddings) stay resident in RAM at int4 precision using about 9.9 GB, while the 19,456 routed experts live on disk (approximately 370 GB in the int4 container) and are loaded on demand with a per-layer LRU cache.
The engine treats VRAM, RAM, and disk as a single managed memory hierarchy. When fast memory is insufficient, speed drops but quality does not. The default policy never silently changes model precision or router semantics. A key engineering choice: the entire runtime is a single C file with small headers, no Python at runtime, and optional CUDA support for GPU expert tiers. Build tools need only GCC with OpenMP, and the setup script self-tests in under a minute.
Measured Performance: Real Numbers From Real Hardware
The performance data, documented transparently in the repository, tells a story of tradeoffs rather than magic. On the developer's own machine (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX measured at roughly 1 GB/s), cold decode runs at 0.05 to 0.1 tokens per second. That is slow but functional: a 744B model answering correctly on a machine that costs less than one server GPU. With a warm cache, pinned hot experts, and MTP speculative decoding enabled, useful response latency drops considerably.
Community benchmarks from real machines paint a more promising picture. An Apple M5 Max with 128 GB unified memory running the Metal backend achieves 1.83 tokens per second with a 39.7 GB warm pin and 66% expert hit rate. A Ryzen 9 9950X with a Samsung 9100 Pro PCIe 5.0 NVMe hits 0.28 tokens per second, with the bottleneck flipping from disk to matmul (57% compute-bound). The fastest datapoint so far: a dedicated high-memory host with six RTX 5090s and 251 GB RAM sustains 6.00 tokens per second with 100% expert hit rate and zero disk wait. Crucially, the engine automatically sizes its expert cache to available RAM. Machines with more memory get proportionally more cache, which directly translates to higher hit rates and faster generation.
What Makes Colibrì Different From Other Model Runtimes
The comparison that matters is llama.cpp, the dominant open-source model runtime. Llama.cpp is mature, broadly supported, and powers everything from Ollama to local AI apps. But it is designed for dense models that fit entirely in RAM, and its MoE support is still maturing. Colibrì is purpose-built for the streaming MoE workload: it assumes the model will never fully fit in any fast tier and optimizes for that case. Features that distinguish it include native MTP speculative decoding (GLM-5.2's own multi-token-prediction head drafts tokens at 2.2 to 2.8 tokens per forward, with 39-59% acceptance), grammar-forced drafts for constrained JSON output, a web dashboard with live metrics and a Brain visualization showing all 19,456 experts as a living cortex, and crash-safe KV-cache persistence that lets conversations reopen across engine restarts without re-prefilling.
For developers who want to try it, the quick start is straightforward. After cloning the repo, run cd c && ./setup.sh to build and self-test. The model download requires approximately 400 GB of free disk for the int4 container, available on Hugging Face. A single command handles the entire process: ./coli convert --model /path/to/model downloads GLM-5.2 FP8 shard by shard (never needing the full 756 GB at once) and converts to the int4 container. Then COLI_MODEL=/path/to/model ./coli chat starts the interactive chat interface. For an OpenAI-compatible API, ./coli serve exposes standard chat completions endpoints on localhost.
Who This Is For
Colibrì is for developers and founders who need to experiment with frontier-scale models but cannot justify the cloud GPU spend. If you are prototyping an AI product that requires understanding the behavior of a genuinely large model rather than a distilled or quantized variant, Colibrì gives you that capability on a high-end consumer desktop. It is also well-suited for privacy-sensitive workloads where data cannot leave the machine, and for teams building specialized AI tools that need local inference without API latency or rate limits. The project is Apache 2.0 licensed, and the developer actively welcomes community benchmark contributions. The more datapoints from real hardware, the faster the engine improves for everyone.
