AirLLM is a Python library that runs arbitrarily large LLMs on a single consumer GPU by streaming model layers — and, for MoE models, individual experts — from disk through VRAM rather than keeping the full model resident. It achieves this without quantization, distillation, or pruning by default, though optional 4-bit/8-bit block-wise compression is available via bitsandbytes. The project was extracted from a broader repository called Anima in late 2023; the Python package is airllm and its source lives under air_llm/, while dormant Anima-era directories (rlhf/, anima_100k/, training/) remain in the repo as historical artifacts. Since v3.0.0, streaming is driven by PyTorch forward hooks registered on Transformers' own forward/generate, which is why most modern architectures work through the generic AirLLMBaseModel without per-family subclasses.

The Core runtime section is the heart of the docs — start with Overview and dispatch for the big picture, then AirLLMBaseModel, Layer streaming internals, On-disk splitting and persistence, and Compression for how weights move from disk to GPU and back. The Supported models section documents per-family subclasses — Qwen family, Mixtral and Mistral, ChatGLM, InternLM and Baichuan, and Kimi K3 — each covering the layer-name overrides and quirks needed for that architecture. The Installation and setup section covers Installation and dependencies, the macOS MLX backend for Apple Silicon, Upgrading notes for version bumps, and the Anima legacy background. The Testing section describes the splitter tests (Splitter tests), the Compression test, and the manual GPU streaming test harness; the Release process section documents how versions are cut and published to PyPI.

If you want to understand how AirLLM fits so much model into so little VRAM, read Overview and dispatch and then Layer streaming internals. If you are adding support for a new model architecture, read AirLLMBaseModel first and then one of the existing subclasses in Supported models — Qwen family or Kimi K3 are the most instructive examples. If you are installing AirLLM or debugging an import error, go straight to Installation and dependencies; macOS users should also read macOS MLX backend. If you are contributing changes or cutting a release, read Testing for how to exercise the splitter and streaming paths, and Release process for the tag-and-publish workflow.