In a recent examination of its operational efficiencies, Cloudflare revealed challenges within its billing pipeline stemming from ClickHouse’s query planning stage. This bottleneck, characterized by contention, significantly impacted multiple downstream services essential for billing and fraud detection. The analysis led to an upgrade of ClickHouse, substituting an exclusive lock with a shared lock and optimizing processes such as part filtering to enhance performance.
Cloudflare’s billing pipeline plays a crucial role in its fraud systems. After a major migration that drastically increased the volume of data parts, daily aggregation jobs on ClickHouse faced severe slowdowns. Interestingly, traditional performance metrics—like I/O, memory consumption, and the number of rows scanned—remained within expected ranges, masking the underlying issue.
Average SELECT query durations. Source: Cloudflare blog
Utilizing ClickHouse since its early days—before it featured built-in data expiration—Cloudflare handles a staggering amount of data, exceeding hundreds of petabytes. To manage this effectively, the company implemented its own retention system, partitioning data in the Ready-Analytics table daily. This approach allowed the deletion of data older than 31 days, preventing bloat. According to James Morrison and Christian Endres, senior distributed systems engineers at Cloudflare:
This system is popular, with hundreds of applications using it. It had already grown to more than 2 PiB of data by December 2024, and an ingestion rate of millions of rows per second. But it had one critical flaw: its retention policy.
As Cloudflare migrated to a new partitioning structure within ClickHouse, they included customer namespaces in the process. This enhancement allowed for more granular data retention management on a per-tenant basis. ClickHouse itself is an open-source analytical platform widely adopted for scenarios involving logs, metrics, analytics, and real-time reporting. However, the team discovered that while the number of parts accessed per query should stay constant, the actual performance declined significantly due to unexpected hurdles. Summarizing the plight, Morrison and Endres noted:
A huge amount of time was being spent in query planning. This is the phase before execution when ClickHouse decides which parts to read (…) 45% of the sampled CPU time was being spent in a single function called filterPartsByPartition (…) The problem wasn’t CPU-bound work; it was massive lock contention. More than half of our query duration was spent waiting to acquire a single mutex (MergeTreeData) that protects the table’s list of parts.
To alleviate the significant slowdown, the Cloudflare engineering team implemented three key modifications: they patched ClickHouse to replace the exclusive lock with a shared lock, eliminated the per-query replication of the complete parts list, and improved part filtering to ensure not every query had to scan the full list. Collectively, these enhancements led to a sharp reduction in query latency, successfully stabilizing performance even as the part count continued to rise. Morrison and Endres remarked:
After deploying this patch in March 2026, query durations dropped by 50%. More importantly, this finally breaks correlation of query durations with the number of parts.
Source: Cloudflare blog
While the Cloudflare team celebrated their success in stabilizing query performance and resolving immediate billing issues, they remain cautious about the potential long-term viability of their partitioning strategy. As data parts continue to grow, operational challenges may persist. They observed that the increasing metadata load also posed dilemmas for ZooKeeper, which coordinates ClickHouse clusters. Reflecting on their experience, they discussed an “uneasy truce” regarding the sustainability of their current architecture.
In a broader context, industry insights shared by Edydh Marquez Avila, a field engineer at Park Place Technologies, shed light on the systemic nature of modern infrastructure failures. On platforms like Reddit and LinkedIn, practitioners have acknowledged that such challenges often arise not from resource constraints but from coordination layers within databases. Avila stated:
Cloudflare’s ClickHouse investigation is a good reminder that modern infrastructure failures increasingly happen in coordination layers, not obvious resource limits (…) The interesting signal is broader than ClickHouse itself (…) High-level telemetry alone is no longer enough for diagnosing large-scale systems under concurrency. Low-level execution visibility still matters.
Following their successful enhancements, Cloudflare contributed the changes back to the ClickHouse project, resulting in their integration into future releases starting from version 25.11, showcasing their commitment to advancing community-driven solutions in data management.
Inspired by: Source
