Unlocking Local AI Inference with Cactus: A Deep Dive into the Game-Changing SDK
Cactus, an innovative startup backed by Y Combinator, is revolutionizing local AI inference for mobile phones, wearables, and a variety of low-power devices. By leveraging cross-platform, energy-efficient kernels and a robust native runtime, Cactus delivers an impressive sub-50ms time-to-first-token for on-device inference. This not only eliminates network latency but also prioritizes user privacy, making it a compelling choice for developers and businesses alike.
- Introduction to Cactus SDK v1
- Developer-Focused Features
- A Broader Approach to On-Device AI
- Built-In Versioning and Over-the-Air Updates
- Cloud Fallback for Essential Functions
- Comprehensive Overhaul in Version v1
- Advanced Features Beyond LLM Inference
- Performance Benchmarks and Real-World Applications
- Exploring Cactus through Chat Apps
- Accessibility and Open Source
Introduction to Cactus SDK v1
The newly launched version v1 of the Cactus SDK is currently in beta. It is specifically designed to enhance performance on lower-end hardware while also providing an optional cloud fallback aimed at increasing reliability. This dual approach allows developers to maintain a high level of service, even in circumstances where local capabilities may not be sufficient.
Developer-Focused Features
Cactus empowers developers to seamlessly deploy AI models within any application using its SDK, offering native bindings for popular frameworks like React Native, Flutter, and Kotlin Multiplatform. Although native support for Swift is still evolving, iOS developers can still utilize Kotlin Multiplatform bindings to integrate Cactus functionality into their Swift apps. This flexibility makes it easier for developers to bring sophisticated AI features to their applications without getting bogged down by platform limitations.
A Broader Approach to On-Device AI
Unlike the platform-specific solutions provided by Apple and Google—such as Apple’s Foundation frameworks and Google’s AI Edge—Cactus adopts a more universal approach to on-device AI inference. This flexibility allows developers to work with a diverse range of models, including Qwen, Gemma, Llama, DeepSeek, Phi, Mistral, and many others. To maximize performance, Cactus supports various quantization levels, from FP32 down to 2-bit, making it more adaptable to the constraints of lower-end hardware.
Built-In Versioning and Over-the-Air Updates
One of the standout features of the Cactus SDK is its built-in model versioning and over-the-air updates. Developers can push new model versions to users without the need for app updates, streamlining the deployment process. The SDK manages downloading, caching, and switching between model versions in the background, creating a seamless experience for end-users.
More Read
Cloud Fallback for Essential Functions
For complex or large-context tasks, Cactus offers a fallback to cloud-based models, ensuring high availability and robust performance. This feature addresses user feedback from the earlier v0 version, allowing developers to secure guaranteed response times for critical user-facing functions, enhancing the overall user experience.
Comprehensive Overhaul in Version v1
Cactus has made significant updates in version v1, including a complete overhaul of its inference engine. The transition from GGUF to a proprietary format has introduced optimized ARM-CPU kernels, resulting in better performance across all supported devices. Moreover, the SDKs have been redesigned to enhance API consistency across different programming languages, while still maintaining backward compatibility wherever possible. Enhanced telemetry and monitoring systems provide granular insights into model performance, usage patterns, and potential areas for optimization, giving developers the data they need to make informed decisions.
Advanced Features Beyond LLM Inference
Version v1 also marks the inclusion of advanced features beyond large language model (LLM) inference. Cactus now supports tool calling and voice transcription, expanding its utility for developers. The Flutter SDK additionally introduces RAG fine-tuning, while the React Native variant includes image embedding functionalities. These capabilities enrich the development experience, bringing sophisticated AI tools closer to a wider range of applications.
Performance Benchmarks and Real-World Applications
To illustrate Cactus’ impressive performance across various devices, the company has released benchmarks using the LFM2-VL-450m and Whisper-Small models. These benchmarks evaluate critical metrics such as tokens per second and prefill/decode times. Notable results show that a Mac M4 Pro reaches 173 tok/s, an iPhone 17 Pro achieves 136 tok/s, a Galaxy S25 Ultra gets 91 tok/s, and a Raspberry Pi 5 registers 24 tok/s—demonstrating the versatility and adaptability of Cactus across platforms.
For context, the smallest model, gemma-3-270m-it, has a compact footprint of 172 MB and focuses solely on completion tasks. In contrast, more advanced models like Qwen3-1.7B require more substantial resources, with a footprint of 1,161 MB while also supporting completion, tool calling, and speech functionalities.
Exploring Cactus through Chat Apps
Cactus has also launched chat applications for both iOS and Android that allow the use of various LLMs. This enables developers to benchmark latency and throughput when accessing these models through the Cactus SDK, providing a tangible way to evaluate performance.
Accessibility and Open Source
Cactus is committed to fostering innovation and learning by offering its SDK as an open-source solution free for students, educators, non-profits, and small businesses. Developers can easily clone it from GitHub, encouraging widespread experimentation and collaboration within the developer community.
The integration of Cactus’s SDK into various applications signals a significant shift in how on-device AI can be implemented effectively and efficiently, paving the way for enhanced user experiences across a multitude of platforms.
Inspired by: Source
