TorchAO: Elevating Low-Bit Inference on Arm CPUs
TorchAO is revolutionizing the way we implement high-performance low-bit linear and embedding operators specifically targeted for Arm CPUs. In our latest update, we’re thrilled to announce three pivotal advancements: dynamic kernel selection, integration with Arm’s KleidiAI library, and support for quantized tied embeddings. These enhancements are crafted to supercharge performance while broadening the scope for low-bit inference in PyTorch, particularly with ExecuTorch, PyTorch’s innovative solution for efficient on-device execution.
For example, leveraging KleidiAI kernels has led to an astonishing improvement—over 2x enhancement in prefill performance on a 4-bit quantized Llama1B model running on an M1 Mac, achieving speeds of 373 tokens per second!
Dynamic Kernel Selection
One of the standout features introduced in TorchAO is dynamic kernel selection. Our low-bit operators now adeptly select the most suitable kernel based on:
- The format of packed weights
- CPU capabilities, such as has_arm_neon_dot and has_arm_i8mm
- The dimensions of the activation tensor
This dynamic dispatch mechanism ensures optimized execution tailored to the unique characteristics of both the hardware and workload.
How It Works
When working with quantized model weights, these can be packed into a format aimed at optimizing a specific linear kernel. Each packed weight contains headers that detail its format. Upon invoking a linear operator with these packed weights, we assess the weight format in conjunction with current CPU capabilities. This examination allows us to identify a suite of compatible kernels for these weights, with function pointers cached in a registration table.
More Read
For instance, weights packed in format1 might be compatible with both GEMV and GEMM kernels (such as gemv_kernel1 and gemm_kernel1), while weights structured in format2 may only work with a GEMV kernel (like gemv_kernel2). This registration table then leverages this information to deliver swift, optimized execution.
Active Kernels
Curious about which kernels are currently being utilized? Set TORCH_CPP_LOG_LEVEL=INFO to gain comprehensive insights into active kernels.
KleidiAI Integration
Our collaboration with KleidiAI—an open-source library from Arm filled with highly optimized microkernels—has taken the performance of TorchAO to unparalleled heights. The integration allows KleidiAI kernels to be registered and used automatically where feasible (like 8-bit dynamic activation with 4-bit weights). In the event of coverage gaps—such as with non-4-bit weights or tied embedding layers—our in-house GEMV neondot kernels automatically serve as a reliable fallback, no additional configuration needed.
This hybrid approach effectively merges superior performance delivered by KleidiAI with the robust operator support from TorchAO’s kernels. The result? A remarkable boost in prefill performance that surpasses 2x speed improvement, especially with ExecuTorch recording rates above 373 tokens per second on an M1 Mac.
Quantized Tied Embedding and lm_head Kernels
Tied embeddings are crucial in compact Large Language Models (LLMs), including models like LLaMA 3.2 and Phi-4 Mini. In this context, the weights for the embedding layer are shared with the weights of the final linear layer—referred to as lm_head:
With recent models boasting vocabulary sizes exceeding 100,000 tokens, these layers may occupy a substantial fraction of the total model size. Unfortunately, on mobile devices, this often leads to weight duplication due to the necessity of packing weights into different formats for linear and embedding kernels, which complicates the sharing of weights without degrading performance.
To combat this challenge, we have engineered efficient quantized kernels for both embedding and linear operations that utilize a unified weight format. This innovation is available through the SharedEmbeddingQuantizer, allowing the same quantized weights to function for both the input embedding and the output lm_head. This strategic reduction minimizes model size without compromising performance. The quantizer supports a diverse array of configurations, such as:
- 8-bit dynamic activations
- x-bit weights, where x ranges from 1 to 8
Try It Out and Contribute!
All these exciting enhancements are readily accessible through TorchAO’s quantization APIs and have been seamlessly integrated into ExecuTorch for shining efficiency in deploying quantized models on mobile and edge devices.
We warmly invite you to experiment with these updates, share your feedback, and actively contribute to this transformative journey! For those intrigued by low-bit quantization and mobile deployment, we encourage you to connect with the ExecuTorch community on both Discord and GitHub. Your input and participation are invaluable in shaping the future of efficient model deployment!
Inspired by: Source
