Twenty three thousand, two hundred and fifty six GitHub stars. Two thousand six hundred and fifty two forks. A single Python library that lets you load DeepSeek-V3, a 671-billion-parameter model, on the same GPU hardware sitting inside a mid-range gaming PC. That is the scope of AirLLM v3.0, released in June 2026 by creator Gavin Li, and it is reshaping the conversation around who gets to run frontier AI models.
The premise is simple but the engineering is not. AirLLM never holds more than one layer of a model on the GPU at any moment. It splits the model into layer shards, loads them one at a time, computes, discards, and moves to the next. The result is that VRAM requirements depend on the size of a single layer, not the size of the full model. A 671B parameter model like DeepSeek-V3 fits in roughly 12GB of memory. Qwen3-235B, a 235-billion parameter mixture-of-experts model, runs in approximately 3GB. A standard 70B class model like Llama 3.x fits on a 4GB GPU card. No quantization, no distillation, no pruning. Just raw full-precision inference on hardware that millions of developers already own.
What AirLLM v3.0 Brings to the Table
Version 3.0 is the most significant update since AirLLM's initial release in June 2023. The headline feature is FP8 model support, which enables the library to work with the latest generation of models that ship FP8 weights natively. This is particularly relevant for DeepSeek-V3, which uses FP8 in its architecture, and Qwen3's latest variants. The update also introduces overlapping prefetching, which overlaps the model loading and compute phases for roughly 10 percent speed improvement. Combined with the existing block-wise quantization compression (8-bit or 4-bit) that can speed inference up to 3x with minimal accuracy loss, AirLLM now offers a broad spectrum of performance options depending on the user's hardware constraints.
The v3.0 release also broadens model compatibility. AirLLM now works out of the box with virtually every popular open-weight model family through a single AutoModel API. That includes Llama 2, 3, 3.1, 3.3, and 4; Qwen 1, 2, 2.5, and 3 (including MoE and FP8 variants); DeepSeek V2, V3, and R1; Mistral and Mixtral; Phi, Gemma, ChatGLM, Baichuan, InternLM, and Yi. The developer simply passes the Hugging Face model ID to AutoModel.from_pretrained() and the library handles everything else.
How AirLLM Compares to Alternatives
The landscape for running large models on consumer hardware has several players, each with a different approach. Ollama focuses on ease of use with a Docker-like interface but typically requires quantized models to fit on consumer GPUs. llama.cpp uses CPU-optimized inference with GPU offloading, trading speed for broad hardware compatibility. vLLM excels at serving multiple concurrent requests with PagedAttention but demands significant GPU memory for large models. Hugging Face's Text Generation Inference (TGI) is production-grade but designed for server environments, not single-GPU setups.
AirLLM's differentiator is that it does not require quantization to run a 70B model on a 4GB card. Where other solutions would need 4-bit or 8-bit quantization (with attendant accuracy tradeoffs), AirLLM keeps the model at full precision by sharding across layers and offloading to system memory. This layer-sharding approach means slower per-token generation compared to solutions that keep the full model on GPU, but it opens the door to models that simply would not fit otherwise. For batch inference, experimentation, and development work, the tradeoff is often acceptable.
The practical implications are significant for the AirLLM table of supported hardware. A Qwen3-8B class model runs on just 1-2GB of VRAM. A 30-47B MoE model like Qwen3-30B or Mixtral runs on 1-3GB. The aforementioned Qwen3-235B runs on 3GB. Llama 3.1 405B runs on approximately 8GB. And DeepSeek-V3 at 671B parameters runs on approximately 12GB. A standard NVIDIA RTX 4070 with 12GB of VRAM, a card that retails for under $600, is enough to run the largest open-weight model currently available.
Getting Started with AirLLM
Installation is a single pip command. From there, the developer imports AutoModel and passes any supported Hugging Face model ID. Here is the complete workflow:
pip install airllmfrom airllm import AutoModel
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")
# Or for the largest models:
# model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3") # 671B
# model = AutoModel.from_pretrained("Qwen/Qwen3-235B-A22B") # 235B
input_text = ["What is the capital of the United States?"]
input_tokens = model.tokenizer(input_text, return_tensors="pt", max_length=128)
generation_output = model.generate(input_tokens["input_ids"].cuda(), max_new_tokens=20)
print(model.tokenizer.decode(generation_output.sequences[0]))MacOS is also supported, including Apple Silicon, enabling 70B models to run on high-end MacBooks. The library handles the model splitting automatically, saving the sharded version to disk on first run for faster subsequent loads. For gated models like Meta's Llama variants, the developer passes a Hugging Face token via the hf_token parameter.
Who This Is For
AirLLM v3.0 is built for developers and researchers who want to run state-of-the-art open-weight models on hardware they already own rather than renting cloud GPU instances at $2 to $30 per hour. It is particularly well suited for three groups: solo founders building local-first AI applications who need privacy and zero API costs; researchers experimenting with model behavior who need to iterate rapidly without cloud provisioning; and developers in regions where cloud GPU access is expensive or unreliable. The tradeoff is slower token generation compared to full-GPU inference, but for offline development, batch processing, and experimentation, the ability to run a 671B model on a $600 GPU card changes the economics of open-weight AI entirely.
