Intel’s research team released BITCOS, a new storage layout for ternary‑quantized large language models that records a zero‑presence bitmap and a compact sign stream. By moving the zero‑check out of the weight value itself, the format drops the average storage cost from roughly 1.6 bits per weight to 1.485 bits, and the smaller memory footprint translates into decoding speed gains of up to 27 % on supported Intel CPUs and GPUs.
How BITCOS Achieves Ternary Weight Compression
Traditional ternary models use three possible values (‑1, 0, +1) and are usually packed by fitting five trits into an eight‑bit byte, which averages about 1.6 bits per weight. BITCOS ("BITmap and COmpacted Signs") splits the weight array into two parallel streams. The first stream stores a single 1 for every non‑zero weight and a 0 for zeros; the second stream stores a sign bit only for the non‑zero entries. The per‑weight cost becomes 2 − z bits, where z is the zero fraction. Across 29 checkpoints from seven ternary model families, zero fractions ranged from 29.7 % to 51.5 %; any model with more than 37.5 % zeros (26 of the 29 checkpoints) already beats five‑trit packing.
Decoding Performance on Intel CPUs and GPUs
Because BITCOS changes only the on‑disk layout, the original ‑1/0/+1 values are reconstructed at runtime without any retraining or accuracy loss. Intel supplied dedicated unpacking kernels for AVX‑512 and AVX2 instruction sets and for Xe2 GPUs. On AVX‑512 hardware the kernel uses the presence bitmap as a mask and the pdep instruction to scatter sign bits into their original positions. Xe2 GPUs lack a direct equivalent, so the implementation falls back to a 2 KB lookup table.
- 64‑core Xeon server: 10 %–18 % faster decoding compared with a fixed 2‑bit kernel.
- 24‑core Core Ultra 9: 2 %–15 % faster.
- Integrated Arc 140V: 9 %–22 % faster.
- Discrete Arc Pro B70: 2 %–27 % faster.
On an eight‑core Lunar Lake CPU, the fixed 2‑bit kernel outperformed BITCOS for every model because the system’s memory bandwidth made unpacking the bottleneck. The reported numbers reflect token‑by‑token decoding at batch size 1 after the model has been loaded; they do not include cold‑start latency reductions.
Operational Limits and Compatibility Questions
The evaluation used only Intel silicon; no Nvidia, AMD, or Arm platforms were tested. All benchmarks were run on a single batch and on a limited set of seven models, so results may differ for larger batches, different model families, or non‑Intel hardware. The underlying paper has not been peer‑reviewed, which suggests that independent validation is still pending. Practitioners should also note that BITCOS requires custom unpacking kernels; integrating it into existing inference stacks will involve building or linking the appropriate AVX‑512, AVX2, or Xe2 kernel.
Related CloudNinjas coverage: AI engineering.
What This Means For Practitioners
For teams that already deploy ternary‑quantized LLMs on Intel CPUs or Xe2 GPUs, BITCOS offers a low‑effort way to shave memory traffic and gain modest latency improvements without changing model accuracy. The format is most beneficial when the model’s zero fraction exceeds roughly 38 %, a condition common in post‑training quantization pipelines. Adoption steps include:
- Profile the zero‑weight ratio of your ternary checkpoints; if it is above the threshold, BITCOS will likely reduce storage size.
- Integrate the provided unpacking kernels into your inference service, ensuring the target hardware supports the required instruction set.
- Run end‑to‑end latency tests with your typical batch sizes, because the reported gains apply to batch‑size‑1 decoding and may shrink with larger batches.
- Validate the approach on any non‑Intel hardware you plan to use, or fall back to the standard five‑trit packing if compatibility cannot be guaranteed.
Overall, BITCOS demonstrates that storage‑format engineering can still deliver measurable performance benefits for quantized models, especially in memory‑bound inference scenarios.
