Revolutionizing Incident Management: The Introduction of Expedia Group’s Service Telemetry Analyzer (STAR)
Expedia Group is making waves in the tech world with the launch of its cutting-edge Service Telemetry Analyzer (STAR), an AI-assisted observability platform that significantly boosts engineers’ ability to investigate production incidents. This innovative solution analyzes service telemetry and generates structured root cause assessments, all aimed at speeding up the incident investigation process.
Understanding STAR: A New Era in Observability
The primary goal of STAR is to minimize two critical metrics: Time to Know (TTK) and Time to Recover (TTR). By streamlining the investigation process, STAR enables engineers to quickly identify the source of service degradation, enhancing operational efficiency. Unlike autonomous AI agents, STAR follows a deterministic workflow. This means it meticulously collects telemetry data, analyzes it using domain-specific prompts, consolidates intermediate findings, and delivers a comprehensive final report containing potential root causes and actionable next steps.
“Our objective with this service was to minimize the time to know (TTK) and time to recover (TTR),” the engineering team candidly expresses.
The Architecture Behind STAR
STAR is built as a FastAPI application that integrates seamlessly with Datadog, allowing it to retrieve vital service metrics. Additionally, it operates through an internal generative AI gateway that manages authentication and access to various Large Language Model (LLM) providers. The platform employs prompt chaining, enabling the execution of multiple specialized analyses before producing a unified diagnosis. However, it is important to note that the current version does not leverage advanced capabilities like function calling, retrieval-augmented generation (RAG), or autonomous tool usage. Instead, it sticks to predefined workflows that ensure consistent analyses and recommendations.
STAR Architecture (Source: Expedia Blog Post)
Analyzing Telemetry: Metrics that Matter
STAR’s design emphasizes standardized infrastructure telemetry obtained from Kubernetes-based services and Java Virtual Machine (JVM) applications. By focusing on metrics such as request throughput, latency, HTTP, gRPC, GraphQL error rates, CPU and memory utilization, container restart events, and various Kubernetes probe statuses, STAR provides a holistic view across multiple services developed in different programming languages. This approach eliminates silos and fosters a more cohesive understanding of service health.
More Read
Asynchronous Architecture for Enhanced Performance
To enhance its performance capabilities, Expedia transitioned STAR from a FastAPI background task system to a more scalable Celery-based asynchronous architecture, with Redis serving as both the message broker and result backend. The engineering team pointed out that the majority of the processing workload consists of I/O-bound operations involving telemetry retrieval and LLM requests. This asynchronous model allows STAR to process multiple analysis tasks concurrently while still adhering to rate limits imposed by Datadog and the internal AI gateway.
Practical Applications: Supporting Engineers in Real Time
STAR has proved invaluable in various use cases, including production incident investigations, post-incident analyses, Kubernetes troubleshooting, and JVM memory diagnostics. Engineers utilize the platform as an assistive tool rather than an autonomous system, allowing them to review and validate the generated findings before taking action based on the recommendations. This human-centric approach ensures that decision-making remains under human control, fostering accountability in critical operations.
Future Enhancements: Expanding STAR’s Capabilities
Looking ahead, the team is actively working on enhancing STAR’s functionalities. Upcoming features include the integration of service dependency information, additional operational metadata, and conversational interfaces to improve user experience. Moreover, Expedia is exploring STAR’s potential within its chaos engineering practices, aiming to analyze the outcomes of controlled failure experiments using the platform. Currently, prompt management, tracing, and evaluation functionalities are supported through Langfuse, while system performance is continuously assessed through expert reviews and user feedback.
By democratizing access to observability tools and providing engineers with more effective means to conduct investigations, Expedia Group’s STAR could set new industry standards in incident management. The company’s commitment to iterative improvement showcases a strategic approach to not only resolving issues as they arise but also bolstering resilience in their operational frameworks.
Inspired by: Source
