Block-wise quantization (4-bit or 8-bit) is enabled by passing compression='4bit' or compression='8bit' to AutoModel.from_pretrained; it requires bitsandbytes and airllm ≥ 2.0.0 — see Installation and dependencies for setup details.[1] AirLLM's block-wise compression quantizes weights only — not activations — because the bottleneck is disk loading rather than matrix-multiply throughput, which makes accuracy loss easier to control.[1]
split_and_save_layers in air_llm/airllm/utils.py raises an AssertionError if a compression argument is passed without bitsandbytes installed.[2] When compression is active, split_and_save_layers appends the compression type to the shard directory name (e.g., splitted_model.4bit), keeping compressed and uncompressed shards in separate directories.[2] check_space in air_llm/airllm/utils.py adjusts the estimated model size before the disk-space check: 4bit compression scales the raw byte count by 1/0.2813 (~3.55×), while 8bit halves it.[2]
Sources