Lyft’s Innovative Shift: The Hybrid Architecture of LyftLearn

In a bold move to optimize its machine learning operations, Lyft has rearchitected its machine learning platform, LyftLearn, into a hybrid system. This strategic decision to migrate offline workloads to AWS SageMaker while retaining Kubernetes for online model serving illustrates a pragmatic approach that balances operational complexity and control. Let’s dive deeper into this transformation and uncover the rationale behind this shift.

The Shift to AWS SageMaker

Lyft’s engineering team made a decisive move by migrating LyftLearn Compute, the component responsible for training and batch processing, to AWS SageMaker. This transition was fueled by the desire to eliminate a range of operational challenges that had previously bogged down their engineering resources. Background watcher services, cluster autoscaling issues, and the complexities of managing eventually-consistent states had consumed significant engineering effort. With SageMaker’s managed services, Lyft aimed to free up resources for more valuable contributions to their machine learning capabilities.

Maintaining Real-time Performance with Kubernetes

While Lyft fully embraced managed services for batch workloads, they chose to keep their LyftLearn Serving component on Kubernetes. This decision stemmed from an established infrastructure that had already proven effective in delivering the necessary performance. The deep integration with internal tooling was seen as a vital advantage that warranted maintaining a custom-built serving architecture.

The diagram below illustrates the hybrid architecture implemented by Lyft:

LyftLearn Hybrid High-Level Architecture (source)

Operational Complexity and the Need for Streamlined Processes

Initially constructed entirely on Kubernetes, LyftLearn experienced a steep increase in operational complexity as more machine learning capabilities were added. Custom orchestration logic was needed for each new capability, and synchronizing the state of Kubernetes with the platform’s database became increasingly cumbersome. This scenario led to the implementation of multiple watcher services to tackle out-of-order events and monitor container status transitions, which further complicated the operational landscape.

Leveraging AWS for Offline Workloads

The decision to utilize AWS SageMaker for offline workloads effectively addressed several pain points. For instance, the traditional watcher architecture was replaced with event-driven state management utilizing AWS EventBridge and SQS. This transition allowed for on-demand provisioning that eliminated idle cluster capacity costs and significantly simplified the infrastructure. However, a key challenge was ensuring complete compatibility with the existing machine learning codebase through this migration.

Creating Cross-Platform Compatibility

To ensure that existing systems remained functional during the transition, Lyft built cross-platform Docker images that could replicate the Kubernetes runtime environment within SageMaker. This approach seamlessly handled credential injection, metrics collection, and configuration management. Furthermore, for latency-sensitive workloads—those requiring retraining every 15 minutes—Lyft adopted Seekable OCI (SOCI) indexes alongside SageMaker warm pools. This allowed them to achieve startup times comparable to their Kubernetes setup.

Tackling Networking Challenges with Spark

One of the most intricate challenges involved managing Spark’s bidirectional communication requirements, particularly between SageMaker Studio and EKS clusters. Default networking configurations in SageMaker blocked inbound connections, which are critical for Spark executors to communicate with notebook drivers. To resolve this, Lyft collaborated with AWS to enable custom networking configurations in their Studio domains, ensuring smooth performance throughout the process.

Seamless Migration with Minimal Configuration Changes

The entire migration process was meticulously executed repository by repository, allowing both infrastructures to run in parallel. This strategy required minimal configuration changes and introduced a compatibility layer ensuring that the same Docker image could serve models regardless of whether they were in SageMaker or Kubernetes. This approach has not only reduced infrastructure-related incidents but has also helped free up engineering capacity for further platform enhancements.

The Philosophy Behind the Transition

Yaroslav Yatsiuk, one of the key figures behind this transformation, encapsulated the essence of this strategic move with profound insights: “The best platform engineering isn’t about the technology stack you run—it’s about the complexity you hide and the velocity you unlock.” This philosophy underscores Lyft’s mission to mitigate operational challenges while enhancing its machine learning capabilities.