DuckDB Introduces Quack: A Revolutionary Remote Protocol Over HTTP

DuckDB, the lightweight but powerful analytical database, has recently unveiled Quack, an innovative remote protocol designed to enable multiple instances of DuckDB to connect and collaborate on the same database over a network. This groundbreaking protocol marks a significant shift for a database that has primarily focused on local and embedded functionality, transforming DuckDB into a more versatile and scalable solution.

What is Quack?

Quack is a remote protocol that allows applications to access DuckDB databases using standard HTTP connections. This seamless integration facilitates simultaneous access by multiple users while maintaining DuckDB’s lightweight architecture. By preserving the database’s SQL compatibility, Quack efficiently supports data sharing, concurrent users, and remote analytics—without necessitating switching to more cumbersome traditional database systems.

Speed and Simplicity

One of the standout features of Quack is its performance. DuckDB claims that the new protocol can transfer substantial datasets about 3.5 times faster than Arrow Flight, outperforming conventional options like PostgreSQL. The simplicity of Quack also sets it apart; it’s designed to handle small queries more effectively by enabling a query to be sent and results returned in a single network round trip.

Source: DuckDB blog

Why Not Arrow Flight SQL?

The DuckDB team decided against using Arrow Flight SQL—a protocol built around the Arrow in-memory format—because they wanted complete control over data transfer processes and the evolution of the protocol itself. This decision reflects their commitment to continuous innovation in data systems without being restricted by externally controlled formats.

As they put it, “In order to be able to keep innovating in data systems, we cannot allow ourselves to be restricted by formats that are controlled externally.” This commitment ensures that DuckDB remains adaptable and relevant in the fast-evolving landscape of data analytics.

Increased Use Cases with Quack

With the introduction of Quack, DuckDB is set to expand into a plethora of new applications. The team observes that as the concept of data lakes becomes more prevalent, centralizing data states across multiple users takes precedence over localized querying. Future integrations, such as DuckLake, will also enable DuckDB to act as a remotely accessible catalog server, further enhancing its capabilities.

Optimizing for Multi-User Environments

Quack has garnered a positive response from the developer community. Many view it as a step toward more robust, multi-user analytics workflows while keeping the lightweight nature of DuckDB intact. For instance, Ryan Glover from Lattice Engineering expressed enthusiasm about using DuckDB in his firm’s internal app framework, noting that Quack resolves scalability challenges.

Enhancing Performance and Features

DuckDB has plans to further refine Quack, aiming for improved performance and feature sets in the next major release. DuckDB v2.0 is expected in 2026, with enhancements such as:

• Better support for remote databases
• Increased transaction throughput
• Customizable protocol extensions
• Replication features

Revolutionary Architectural Paradigms

The potential for DuckDB instances to communicate with one another creates exciting prospects for modern AI and data engineering. As Amir Sefati notes, combining DuckDB with object storage, DuckLake, Parquet, and vector databases lays the foundation for a pragmatic architecture that simplifies complexities in data management.

Furthermore, the application scenarios are vast, from a browser tab communicating directly with a DuckDB server to facilitating query forwarding between notebooks. Developers like Mehdi Ouazza emphasize that the Quack protocol finally enables multi-writer support, dispelling previous misconceptions.

Getting Started with Quack

For those eager to leverage the capabilities of Quack, support for the new protocol is accessible through the Quack extension in both DuckDB instances. The latest DuckDB v1.5.3 supports Quack as an autoloadable core extension along with features for DuckLake Catalog integration.

As DuckDB continues to grow and evolve, Quack stands as a pivotal development, pushing the boundaries of what can be achieved with remote database interactions while retaining the user-friendly simplicity that DuckDB is known for. Whether you are a data engineer, analyst, or application developer, staying informed about these advancements can open doors to innovative solutions in your data workflows.

Inspired by: Source