Building the Future of E2E Observability at Netflix: Insights from QCon London 2026
At QCon London 2026, Netflix engineers Prasanna Vijayanathan and Renzo Sanchez-Silva shed light on their groundbreaking work, “Ontology‐Driven Observability: Building the E2E Knowledge Graph at Netflix Scale.” Their presentation explored the design and implementation of an innovative end-to-end (E2E) knowledge graph, modeling the intricate Netflix user experience through a web of user interactions, devices, services, and infrastructure.
Understanding End-to-End (E2E) Observability
End-to-End (E2E) Observability is a crucial concept for monitoring complex systems. It involves capturing and analyzing the entire state of a system, from the frontend user experience to the backend services and underlying cloud infrastructure. This comprehensive grasp allows companies like Netflix to monitor, understand, and debug their platforms efficiently.
Vijayanathan captivated the audience by encouraging them to envision a system capable of immediate issue detection, prioritized incident triage, root cause identification, and proactive prediction of potential problems. Such a system offers invaluable support in maintaining a seamless user experience, which is at the heart of Netflix’s operations.
Real-world Challenges in Observability
In their recent incident response, Netflix faced a time-consuming challenge: it took four hours from the initial alert to resolution. Multiple teams—more than 30 engineers—were involved in triaging, debugging, and identifying the root causes of the initial alert and three subsequent incidents. This highlighted the complexities inherent in achieving robust E2E observability.
Common obstacles include numerous siloed data sources, disjointed alert systems, complex troubleshooting processes, and inadequate detection methods. These challenges can lead to inefficient incident management and prolonged downtime.
More Read
The Power of Connectedness in Data
At Netflix, the idea of Connectedness plays a critical role in overcoming these challenges. By bridging gaps and eliminating silos, Netflix aims to create a more cohesive environment for data. Connected data in E2E observability enables:
- Enrichment for a single source of truth
- Minimization of duplicated efforts
- Effective triage and troubleshooting processes
- Enhanced accuracy in diagnostics
Introducing the MELT Layer
To facilitate this connectedness, Vijayanathan introduced the MELT Layer (Metrics, Events, Logs, Traces). This unified observability layer integrates data from users, devices, and services, significantly improving incident resolution times. The MELT Layer acts as a backbone for monitoring comprehensive system states.
Ontology: The Foundation of Knowledge Representation
Sanchez-Silva took the stage to explore the concept of Ontology—the formal specification of types, properties, and relationships within a system. Ontology is not merely about data; it’s about the relationships that bind different data points together.
The Ontology data structure utilizes The Triple concept, which is a tuple consisting of a Subject, Predicate, and Object. This structure allows for the definition and querying of specific facts within a knowledge graph.
For instance:
java
api-gateway | rdf:type | ops:Application
api-gateway | ops:ownedBy | “Team Bedrock”
INC-5377 | rdf:type | ops:Incident
INC-5377 | ops:affects | api-gateway
These triples help create a clear map of relationships between various entities within Netflix’s ecosystem.
Operational Namespaces and Their Importance
Sanchez-Silva detailed how Netflix employs 12 Operational Namespaces that connect various elements of its infrastructure, including Slack, Alerts, Metrics, Logs, and E2E. While incident knowledge can often be scattered across these namespaces, the ontology serves to impose order by capturing, structuring, and preserving a machine-readable triple data structure.
The Knowledge Flywheel: Driving Adaptability
A key element of this system is the Knowledge Flywheel, which enhances resilience through a three-state cycle: Observer, Enrich, and Infer. Each rotation captures and encodes knowledge, which drives smarter subsequent rotations.
Using an AI co-developer named Claude, two flywheels can effectively “spin together” within one system. The first flywheel focuses on knowledge, while the second addresses code development through a git worktree.
Illustrative Example of the Flywheel
Flywheel 1: Knowledge
- Slack → Enrich → Infer → Adapt
Flywheel 2: Code (git worktree)
- Worktree → Claude → PR → Review → Merge
This collaborative process allows Claude to propose pull requests (PR), which can be reviewed and merged by humans, ensuring a blend of automated efficiency and human oversight.
Future Outlook for Netflix’s E2E Observability
Looking forward, Netflix aims to push the boundaries of its observability efforts. The company plans to automate root cause analyses and develop auto-remediation strategies, paving the way for a self-healing infrastructure. The implementation of ontology-driven observability at Netflix not only enhances their operational capabilities but also elevates the user experience to new heights, affirming their commitment to innovation in technology and service delivery.
By investing in these advanced observability solutions and breaking down barriers, Netflix is setting a standard for how complex systems can be managed and optimized in real-time.
Inspired by: Source
