4.1 KiB
Hypercore
Hypercore, formerly known as Dat, is a distributed append-only log.
Dat was created in 2013 as a protocol designed to help data analysts collaborate on changes to data sets, but has since expanded to other use cases. Dat was renamed Hypercore in 2020 to reflect the extent of the changes. The older Dat protocol was "a p2p hypermedia protocol that provides public-key-addressed file archives which can be synced securely and browsed on-demand."
Hypercore has moved away from a distributed file system architecture. It focuses on the p2p distribution of an append-only log, and its main purpose is to be a building block for other applications. The switch to hypercore was motivated by performance and scaling improvements, and the addition of mounts for nested hyperdrives. In addition, a DHT was added for peer lookup.
Identity
Networking
Hypercore uses the Hyperswarm networking module. Hyperswarm combines a Kademlia-based DHT for global discovery with MDNS to discover peers on local networks. Users join the swarm for a “topic” and query periodically for other peers who are in the topic. When ready to connect, Hyperswarm helps create a socket between them using either UTP or TCP. A set of bootstrap servers is used to help connect a new node to the network.
Data
Hypercore's primary data structure is a signed append-only log. Hypercore focuses on mutable data by organizing content under keys. A cryptographic keypair is used to sign the root of the Merkle tree (tree log?) as data is appended to it. Files stored under a key can easily change at any time. To track changes, Hypercore keeps a version log of the dataset over time. Only the original creator of a Hypercore can append new data, making it a single-writer data structure. Multi-writer data structures can be built by using multiple hypercores that reference each other's elements.
Hyperdrive is the filesystem implemented on hypercore. It is an append-only hash trie, which functions like a folder of files. An improvement added to Hypercore that was not present in Dat is the ability to mount hypercores like a folder of files, giving others access to subsets of data.
A node must be actively sharing a topic for it to be available. To ensure persistence, nodes can pin their own data on a server that stays online, or can pay a pinning service to host it.
Moderation & Reputation
Social & Discovery
Nodes create keys for topics. Data they want to associate with that topic can be added to the signed log.
Hypercore optionally uses DNS to give short-names to keys, so that users can have a "foo.com" address.
Privacy & Access Control
By default, Hyperswarm connections are unencrypted. Hyperswarm does not hide user IPs or mask metadata. However, once Hyperswarm has established a connection, Hypercore wraps it in a Noise protocol stream to add end-to-end encryption.
Interoperability
Scalability
Study on scalability of Dat vs HTTP for video streaming: http://www.diva-portal.org/smash/get/diva2:1318896/FULLTEXT01.pdf
Metrics
Governance & Business Models
The Hypercore protocol's development is coordinated through the Dat Foundation nonprofit, and led by two companies, Blue Link Labs (which develops Beaker browser), and Hyperdivision consulting.
Implementations & Applications
The Beaker browser is a p2p web browser for Hypercore sites.
Applications using Hypercore include:
- Mapeo, an offline-friendly mapping software to support indigenous land rights
- Cabal, a p2p chat platform
- Peer-to, an online art exhibition only available on the through the Beaker browser