A 744-billion-parameter AI model that needs only 25 GB of RAM and zero GPU horsepower. That is the promise of Colibri, the open-source inference engine that hit 15,900+ GitHub stars in its first week and is challenging the core assumption that running frontier AI requires datacenter hardware. Built entirely in pure C with no dependencies, Colibri streams the 370 GB GLM-5.2 model from disk and activates only the experts it needs for each token. The result is a working 744B-parameter model that answers queries on machines costing less than a single H100 GPU fan.
For founders building AI products, this changes the economics of inference. If a frontier-class model runs on a $1,000 desktop, the hardware barrier for local AI collapses. Cloud inference costs, GPU scarcity, and vendor lock-in all become optional constraints rather than structural ones.
How Colibri Squeezes a 744B Model Into Consumer Hardware
The trick is Mixture-of-Experts architecture. GLM-5.2 activates roughly 40 billion parameters per token, and only about 11 billion of those change between tokens (the routed experts). Colibri keeps the dense part of the model attention, shared experts, and embeddings resident in RAM at int4 precision, consuming about 9.9 GB. The 19,456 routed experts live on disk at roughly 370 GB total and stream in on demand. Each expert is only 19 MB at int4, so the engine can prefetch and cache them using a per-layer LRU system, an optional pinned hot-store, and the OS page cache as a free second layer.
The engine supports multi-token prediction (MTP) speculative decoding, which drafts multiple tokens in one forward pass and achieves 2.2 to 2.8 tokens per forward on a warm cache. It also includes grammar-forced speculative drafting for constrained JSON and structured output workloads, where the grammar itself acts as a draft source. On cold starts, disk I/O is the bottleneck the engine reads roughly 11 GB per token across 75 MoE layers. But with warm caches, pinned hot experts, and MTP, practical response latency drops considerably.
Installation requires no Python at runtime and no GPU. The quick start is a single setup script:
cd c && ./setup.sh && ./coli convert --model /nvme/glm52_i4 && COLI_MODEL=/nvme/glm52_i4 ./coli chat
The converter downloads GLM-5.2 in FP8 shards and requantizes them to the int4 container without ever holding the full 756 GB FP8 checkpoint on disk at once. An OpenAI-compatible API server is built in: coli serve keeps one model process loaded and exposes /v1/chat/completions with streaming support.
Why This Matters for AI Builders
Colibri is not just a technical curiosity. It represents a structural shift in who can run frontier AI. Before Colibri, running a 744B model required multiple H100s or A100s hardware that costs tens of thousands of dollars and is often inaccessible to individual developers. Colibri runs on a consumer desktop with 25 GB of RAM and an NVMe drive.
The implications for the AI ecosystem are significant. First, inference pricing will face downward pressure as local alternatives become viable. If solo founders can run GLM-5.2 locally for the cost of electricity, cloud API providers lose their pricing power for high-volume inference workloads. Second, the privacy argument for local AI gets stronger no data leaves the machine, which matters for regulated industries like healthcare, finance, and legal. Third, Colibri validates the Mixture-of-Experts approach as the architectural path to efficient frontier models sparse activation means massive model capacity at a fraction of the compute cost.
Honest Limitations You Need to Know
The README is refreshingly transparent about performance. On a WSL2 setup with 12 cores, 25 GB RAM, and NVMe storage, cold decode runs at 0.05 to 0.1 tokens per second. That is slow enough to be impractical for interactive use without caching. With warm caches and MTP speculation enabled, speed improves to 2.2 to 2.8 tokens per forward, which is usable for chat. On more capable hardware six RTX 5090s with full expert residency Colibri reaches 6.84 tokens per second, competitive with cloud-hosted models.
The engine also supports optional GPU backends for CUDA (NVIDIA) and Metal (Apple Silicon). On an M4 Max with 128 GB unified memory, Metal acceleration pushes decode from 0.30 to 0.42 tokens per second a modest gain due to the I/O bottleneck, but it demonstrates the flexibility of the architecture. The critical insight is that Colibri treats VRAM, RAM, and storage as one managed memory hierarchy. More fast memory means faster inference, but the model works at any tier.
Who This Is For
Colibri is for three types of builders. Privacy-conscious founders who cannot send proprietary code or data to cloud APIs will find local inference indispensable. Developers building AI-powered applications in regions where GPU access is limited or expensive will benefit from running state-of-the-art models on commodity hardware. And researchers working on fine-tuning or prompt engineering at scale will appreciate eliminating per-token cloud costs.
The project is also valuable for teams evaluating the GLM-5.2 architecture itself. Because the engine is a single C file with small headers and zero dependencies, it is auditable, forkable, and embeddable. The tokenizer, the attention mechanism, and the MoE router are all visible in plain C code, not hidden behind layers of Python framework abstractions. For anyone who wants to understand how modern MoE models actually work at the kernel level, Colibri is the best open reference implementation available.
Quick start summary: clone the repo, run cd c && ./setup.sh to build the engine, download the pre-converted int4 model from Hugging Face (mirror with int8 MTP heads for working speculative decoding), and run COLI_MODEL=/path/to/model ./coli chat. The web dashboard at ./coli web provides live metrics on token speed, cache hit rates, and expert tier distribution. For production use, coli serve exposes an OpenAI-compatible API with queue management, multiple KV slots, and crash-safe conversation persistence.
