Unveiling Apple’s Core AI Framework: A Game Changer for On-Device Machine Learning

At the recently concluded WWDC 26, Apple unveiled its Core AI framework, the much-anticipated successor to Core ML. This revolutionary framework is specifically designed to enable developers to deploy large language models and generative AI entirely on Apple devices, including the iPhone, iPad, Mac, and Apple Vision Pro. With compatibility for both custom-converted PyTorch models and pre-optimized open-source models, Core AI promises to reshape the way developers leverage artificial intelligence.

The Unified Architecture of Core AI

Core AI brings forth a unified architecture that facilitates the deployment of a wide variety of models. From compact, 3 billion-parameter vision models to expansive 70 billion-parameter reasoning models, developers can seamlessly integrate powerful AI capabilities into their applications. Apple emphasizes that this framework is rooted in its commitment to user data privacy, negating the need for server dependencies and eliminating per-token cloud costs.

As the backbone of Apple Intelligence, Core AI is built exclusively for Apple Silicon, ensuring optimal performance and security. Developers can create what Apple refers to as “custom intelligence”—tailored solutions that harness the potential of advanced machine learning techniques.

Core AI’s Impressive Capabilities

One of the standout features of the Core AI framework is its unified hardware access. This advantage allows workloads to execute seamlessly across the CPU, GPU, and Neural Engine, all managed through a single API. Additionally, Core AI introduces a memory-safe Swift API that guarantees zero-copy data paths and meticulous control over inference memory. The ahead-of-time (AOT) compilation ensures that developers enjoy near-instant load times by shifting workload off the user’s device.

Transitioning from PyTorch to Core AI

For developers looking to leverage their existing PyTorch models, Core AI offers an accessible conversion path. Utilizing the Core AI PyTorch, you can export a PyTorch model as a torch.export.ExportedProgram and convert it to a Core AI AIProgram with simple commands like TorchConverter().add_exported_program(ep).to_coreai(). This pushes innovation further, allowing developers to author brand-new Core AI models from a PyTorch base using built-in composite operations, including attention, RoPE embeddings, RMSNorm, and gather-matmul.

Moreover, creating custom Metal kernels for lower-level optimization is also within reach, providing flexibility and control over model execution that many developers will find invaluable.

Model Compression for Optimal Performance

Another critical aspect of deploying AI models on Apple’s framework is model compression. This process employs optimization techniques such as quantization and palettization, which help align with the execution patterns of the Core AI runtime. Adequate compression results in reduced memory footprint, lower inference latency, and minimized power consumption—all while optimizing performance.

Model compression can help reduce the model’s memory footprint (disk size and at runtime), decrease inference latency, reduce power consumption, or optimize all aspects simultaneously.

Specialization for Enhanced Performance

When it comes to running an AIModel, the framework automatically specializes based on the current hardware and OS version during the model’s initial load into the model cache. While this may result in longer initial load times, subsequent uses of the model are significantly faster once cached. Developers can customize how and when specialization occurs using SpecializationOptions, and they can manage the cache for models using the AICacheModel class.

Core AI vs. Core ML vs. MLX Swift

With the introduction of Core AI, Apple now supports three distinct frameworks for running machine learning and AI applications on its platforms: Core ML, Core AI, and MLX Swift. According to community feedback, Core ML is optimal for traditional, non-neural machine learning tasks such as decision trees or tabular feature engineering. In contrast, Core AI is geared towards neural networks and transformers, while MLX is intended for working with custom model weights albeit potentially with varying performance levels.

As developers explore Core AI, its long-term utility will likely hinge on the continued growth of both official and community support, highlighting an exciting time for innovation within Apple’s ecosystem.