Understanding the Future of Interoperable Applications: Insights from Seph Gentle’s Closing Talk
At a recent local-first software conference, Seph Gentle delivered a thought-provoking closing talk that highlighted the challenges and opportunities in the realm of interoperable applications. He argued that the future hinges on adopting a more robust and sustainable data format, navigating between the limitations of traditional local software and the issues associated with centralised cloud-based services.
- Understanding the Future of Interoperable Applications: Insights from Seph Gentle’s Closing Talk
- The Local-First Software Paradigm
- The Need for a Durable Schema Format
- Innovative Approaches: Embedding Schemas
- Experimental Implementation: Schemaboi
- Compatibility Beyond the Basics
- Schema Evolution Safety Guidelines
- Practical Considerations for Implementation
The Local-First Software Paradigm
Gentle framed the current software landscape as a dichotomy—on one side, we have local software, which is generally bound to a single device. On the other side is cloud-based software, which often restricts user control over data, effectively holding it hostage. He suggested that to bridge this gap, we need to rethink how applications handle data by conceptualising a local-first operating system. This innovative approach entails creating programs that share a common data layer, allowing them to adapt independently while maintaining compatibility.
The Need for a Durable Schema Format
A major theme of Gentle’s presentation was the critical need for a durable schema format. He advocated for a data format that is:
- Self-describing: This means that the format can convey its structure and requirements without needing additional documentation.
- Statically typed: This ensures that data types are defined at compile time, leading to fewer runtime errors and improved reliability.
- Long-term compatible: The format should maintain compatibility across versions and be capable of preserving data in the face of software evolution.
Moreover, Gentle emphasized that the format should efficiently handle foreign data—the pieces of information that the current system does not recognize—while being optimized for network synchronization and data streaming.
Innovative Approaches: Embedding Schemas
To achieve these ambitious goals, Gentle introduced the idea of embedding the schema directly into the file header. This significant shift means that any tool can inspect and interpret data without needing a separately distributed schema file, making it more accessible and efficient. He contrasted this approach with existing methods like Protocol Buffers and Cap’n Proto, which require pre-shared schemas, and criticized JSON for being overly verbose and weakly typed.
More Read
Experimental Implementation: Schemaboi
To showcase these principles, Gentle discussed an experimental Rust implementation named schemaboi. In this system, the schema is integrated within the first bytes of every file, merging seamlessly with the application’s internal schema when the file is opened. This results in a type system that supports basic types—like booleans, integers, and floats—as well as more complex structures including lists, maps, and a coherent struct-and-enum model. Notably, this design ensures that missing fields are populated with default values and unknown fields are retained, safeguarding data integrity.
Compatibility Beyond the Basics
Gentle elaborated on various forms of compatibility essential for fostering a flourishing ecosystem of applications. He discussed:
- Backward Compatibility: The ability to open old files using new applications.
- Forward Compatibility: The capacity to open new files using older applications.
- Sideways Compatibility: This innovative concept allows different developers to add fields independently, eliminating the need for central coordination.
To facilitate this, Gentle strongly advocated for the use of globally unique field identifiers instead of simple incrementing tags, which can lead to collisions in parallel development.
Schema Evolution Safety Guidelines
In addressing the complexities of schema evolution, he laid out safety guidelines to follow. Adding optional fields or new enum variants is generally safe and should be encouraged. However, high-risk actions such as making fields mandatory, renaming them, or altering a field’s type could compromise data integrity and should be avoided. Gentle emphasized that incremental enhancements often yield far more enduring solutions, pointing to the long-term success of established systems like email and HTTP.
Practical Considerations for Implementation
Towards the end of his talk, Gentle raised practical considerations like implementing canonical encoding for content-addressable systems and leveraging ATProto-style must_use fields. Preliminary benchmarks revealed that his proposed format is significantly smaller and faster than traditional formats like JSON, although he acknowledged that the project remains experimental. His vision ultimately points toward a future where users possess true ownership of their data, independent of the applications that created it.
Inspired by: Source
