Dat was created in 2013 as a protocol designed to help data analysts collaborate on changes to data sets, but has since expanded to other use cases. Dat was renamed Hypercore in 2020 to reflect the extent of the changes. The older Dat protocol was "a p2p hypermedia protocol that provides public-key-addressed file archives which can be synced securely and browsed on-demand."
Hypercore has moved away from a distributed file system architecture. It focuses on the p2p distribution of an append-only log, and its main purpose is to be a building block for other applications. The switch to hypercore was motivated by performance and scaling improvements, and the addition of [mounts](https://blog.hypercore-protocol.org/posts/announcing-hyperdrive-10/) for nested hyperdrives. In addition, a DHT was added for peer lookup.
Hypercore uses the [Hyperswarm](https://hypercore-protocol.org/#hyperswarm) networking module. Hyperswarm combines a Kademlia-based DHT for global discovery with MDNS to discover peers on local networks. Users [join the swarm](https://pfrazee.hashbase.io/blog/hyperswarm) for a “topic” and query periodically for other peers who are in the topic. When ready to connect, Hyperswarm helps create a socket between them using either UTP or TCP. A set of bootstrap servers is used to help connect a new node to the network.
Hypercore's primary data structure is a signed append-only log. Hypercore focuses on mutable data by organizing content under keys. A cryptographic keypair is used to sign the root of the Merkle tree (tree log?) as data is appended to it. Files stored under a key can easily change at any time. To track changes, Hypercore keeps a version log of the dataset over time. Only the original creator of a Hypercore can append new data, making it a [single-writer data structure](https://mafinto.sh/blog/introducing-hypercore-8.html). Multi-writer data structures can be built by using multiple hypercores that reference each other's elements.
Hyperdrive is the filesystem implemented on hypercore. It is an append-only hash trie, which functions like a folder of files. An improvement added to Hypercore that was not present in Dat is the ability to mount hypercores like a folder of files, giving others access to subsets of data.
A node must be actively sharing a topic for it to be available. To ensure persistence, nodes can [pin their own data](https://docs.dat.foundation/docs/dat-server) on a server that stays online, or can pay a pinning service to host it.
By default, Hyperswarm connections are unencrypted. Hyperswarm does not hide user IPs or mask metadata. However, once Hyperswarm has established a connection, Hypercore wraps it in a [Noise protocol](https://noiseprotocol.org/) stream to add end-to-end encryption.
The Hypercore protocol's development is coordinated through the Dat Foundation nonprofit, and led by two companies, Blue Link Labs (which develops Beaker browser), and Hyperdivision consulting.
