Dolt 2.0: Revolutionizing Version-Controlled SQL Databases

DoltHub has recently launched Dolt 2.0, a significant update to its open-source, version-controlled SQL database. This latest iteration brings a host of enhancements designed to optimize storage and improve performance, particularly for large datasets and advanced data types.

What Is Dolt?

Dolt is a MySQL-compatible SQL database that includes built-in Git-style version control. This innovative tool offers capabilities like branching, merging, cloning, and diffing, allowing for enhanced data handling and management. By storing data using content-addressed Prolly Trees, Dolt excels in row-level versioning and efficient structural sharing across multiple versions, resulting in quick diffs and merges.

Key Features of Dolt 2.0

After three years since its initial major release, Dolt 2.0 introduces several groundbreaking features:

Automated Storage Optimization

One of the standout enhancements is the introduction of automated garbage collection and archive compression, now enabled by default. These features help reduce unnecessary disk usage, a critical aspect as explained by Tim Sehn, founder and CEO of DoltHub:

Dolt makes a lot of disk garbage, especially during import. Dolt is copy-on-write so all intermediate committed transaction state is preserved to disk. Any intermediate state that is not in a Dolt commit is garbage and can be collected (…) Dolt must preserve all history in the commit graph on disk. Adding extra garbage can eat through your disk very quickly.

Improved On-Disk Format

Dolt 2.0 incorporates a new on-disk format known as “archives”, which aims to cut storage footprints by 30-50%. This is achieved through dictionary compression, significantly enhancing the database’s efficiency.

Performance Boosts

Previously, Dolt faced performance challenges compared to MySQL, being approximately 10X slower on reads and 20X slower on writes. However, after extensive optimization efforts, Dolt now claims to be 13% faster on writes and 5% faster on reads than MySQL, based on sysbench benchmarks.

Vector Data Support

Dolt 2.0 also introduces beta support for version-controlled vector indexes, utilizing MariaDB’s Vector type. This feature is particularly noteworthy, as Dolt is the only database to provide version control for vector data. The team is actively working on addressing remaining gaps in the read-path to fully roll out this functionality.

Comparing Dolt with Other Data Versioning Tools

While Dolt is making strides in the version-controlled database landscape, it’s important to recognize alternatives like LakeFS and Nessie. LakeFS offers data versioning solutions tailored for data lakes, whereas Nessie provides a transactional catalog with Git-like semantics. Both have their unique approaches but lack some of the extensive SQL capabilities that Dolt offers.

Community and Ecosystem

Dolt is openly available on GitHub under the Apache 2.0 license, promoting collaboration and user engagement within the developer community. A companion tool, DoltgreSQL, aims to bring PostgreSQL compatibility to Dolt, aligning with the same storage engine and version control interfaces, though it is still in beta.

Future of Data Management

The adoption of Git-like workflows in database management is on the rise. As Simon Späti aptly notes in his articles:

Git-like workflows are becoming table stakes. Maybe not today or tomorrow, but with the right tools and changes in workflow we can achieve significantly better change management, testing on production data, fast rollbacks, isolated experiments, and most importantly, peace of mind when deploying changes.

As Dolt continues to evolve, its potential to reshape how we view and manage data in relational databases becomes increasingly evident, making this tool one to watch for tech enthusiasts and data professionals alike.