AirLLMKimiK3 adapts streaming inference for K3's decoder architecture by pointing layer references through a language_model prefix and streaming its 896-expert MoE layers per-expert rather than per-layer, keeping fixed modules like vision and projection in resident memory. K3's MXFP4 quantization and dense expert structure demand specific dependencies: flash-attn, CUDA 12, and transformers 4.56.x, with compressed-tensors support for packed weight expansion.
AirLLMKimiK3, implemented in air_llm/airllm/airllm_kimi_k3.py, overrides set_layer_names_dict to point embedding, layers, norm, and lm_head under the language_model prefix, because K3's decoder lives one level deeper than standard *ForCausalLM models.[1] Four modules — output_attn_res_norm, output_attn_res_proj, mm_projector, and vision_tower — are listed under the resident key, meaning they are loaded once and kept in memory rather than streamed, because together they are well under 1 GB; this also ensures they are split out during on-disk splitting even though they are not decoder layers.[1][2] K3 uses per-expert streaming (keyed by expert_prefix) rather than per-layer streaming: each layer holds 896 experts but a token needs only ~1 GB of them, so streaming by expert avoids loading the full ~55 GB layer.[1] Kimi K3 uses MXFP4 weights that cross PCIe packed and expand on the GPU, moving 4× less data than full-precision weights.[3]
Kimi K3 (2.8T) requires three steps beyond a standard pip install airllm: running pip install compressed-tensors flash-attn (K3's model code mandates flash attention regardless of what you request), using a CUDA 12 build of torch (no prebuilt flash-attn wheel exists for CUDA 13), and pinning transformers to 4.56.x (K3's remote code does not load on 5.x) — see Installation and dependencies for general dependency guidance.[4]
Sources