Gemma 3n: Transforming Mobile AI with Innovative Techniques

Launched in early preview last May, Gemma 3n is now officially available, marking a significant leap in mobile-first, on-device AI applications. The latest iteration introduces new techniques designed to boost both efficiency and performance, setting a new standard for the capabilities of mobile AI.

Revolutionary Per-Layer Embeddings (PLE)

One of the standout features of Gemma 3n is the use of Per-Layer Embeddings (PLE). This innovative technique reduces the RAM required to run a model while preserving the total parameter count. Essentially, it operates by loading only the core transformer weights into accelerated memory, typically VRAM, while keeping the additional parameters on the CPU. For example, the 5-billion-parameter version of the model only requires 2 billion parameters to be loaded into the accelerator; similarly, the 8-billion variant needs only 4 billion. This allows for greater efficiency without sacrificing performance.

Introducing MatFormer Technology

Another exciting advancement in Gemma 3n is the MatFormer (Matryoshka Transformer) technology. This allows for the nesting of transformers, enabling a larger model (e.g., one with 4 billion parameters) to contain a smaller version of itself (e.g., with only 2 billion parameters). Google’s elastic inference offers developers the flexibility to choose between the full model and its faster, yet fully-functional sub-model, enhancing the efficiency of mobile applications. Additionally, the MatFormer technology supports a Mix-n-Match method, allowing developers to adjust parameters and create custom model sizes by altering specific dimensions of the hidden layers.

This technique enables precise adjustments to the E4B model’s parameters, allowing selective skipping of some layers and the ability to modify hidden dimensions from 8192 to 16384.

Dynamic Inference with Elastic Support

Looking ahead, Gemma 3n is set to fully support elastic inference, facilitating dynamic switching between the full model and its smaller sub-model in real-time. This adaptability can greatly enhance user experiences based on real-time demands and device capabilities, ensuring optimal performance no matter the situation.

Kv Cache Sharing: A Speedy Advance

In efforts to further amplify inference speed, Gemma 3n has incorporated KV cache sharing. This feature targets the critical time-to-first-token metric essential for streaming applications. With this new technique, keys and values from the middle layers of the model are directly shared with the upper layers. Google reports a significant boost of 2x improvement in prefill performance compared to Gemma 3 4B, showcasing how technology is evolving to meet user expectations.

Native Multimodal Capabilities

Gemma 3n extends its prowess by introducing native multimodal features, thanks to integrated audio and video encoders. The audio encoder facilitates on-device automatic speech recognition and translation, making it a powerful tool for diverse applications. The encoder is designed to generate a token every 160ms of audio, translating to approximately 6 tokens per second, effectively integrating audio context into the language model.

Strong results have been recorded in translating between English and languages like Spanish, French, Italian, and Portuguese. Despite the capability to handle long audio clips, initial limitations constrain processing to 30-second clips at launch.

Enhanced Visual Processing

Gemma 3n is also equipped with an impressive capacity for visual processing, supporting resolutions of 256×256, 512×512, and 768×768 pixels. The platform can process up to 60 frames per second on devices like the Google Pixel. In stark contrast to its predecessor, Gemma 3, it delivers a remarkable 13x speedup with quantization and a 6.5x increase without it, while also boasting a memory footprint that is four times smaller.

Gemma 3n represents a significant evolution in mobile AI technology, offering a suite of features designed to meet the growing demands of developers and users alike. With its enhanced efficiency, flexibility, and processing capabilities, Gemma 3n is set to redefine the landscape of on-device AI applications, making it an indispensable tool in the tech arsenal.