From aae86651b3047952cc7581011c3860e791193bd7 Mon Sep 17 00:00:00 2001 From: Golda Velez Date: Sun, 24 May 2020 23:19:23 +0000 Subject: [PATCH 01/33] Solid section of ecosystem review --- protocols/solid.md | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/protocols/solid.md b/protocols/solid.md index d309077..2ca6098 100644 --- a/protocols/solid.md +++ b/protocols/solid.md @@ -2,22 +2,64 @@ ### Identity +Solid uses a [Internationalised Resource Identifier (IRI)](https://tools.ietf.org/html/rfc3987) as a WebID, requires an IdentityProvider; can be multiple Identity Providers. + +The rights on a Pod are attached to WebIDs. WebIDs may refer to a person, a family or an organization. Instead of creating an account on each app, the app finds your WebID. + +### Networking/Message passing + ### Data Storage/Message Persistance +Pod is a data storage space. All a user's data is stored here. Applications read and write data into the Pod depending on the authorisations granted by the user or users associated to that Pod. A person may have multiple pods, for example for work and for home. Data may be replicated across pods. + +Users may self-host their pods (on their own server) or use a "Pod Provider". + +Users may also switch pods easily without losing their data (such as their contacts and chat history. ### Moderation/Reputation ### Social/Discovery +WebID allows user addressing across all Solid-conforming apps + +There does not seem to be a Solid-based social network as of this writing, some discussion about creating one. Instead, Solid aims to create tools that enable building of decentralized social applications. + ### Privacy and Access Control +The owner of the pod controls access to who is authorized to read or write data. However if the user revokes access of an app to their data, the app can't further read or update the user's data, but they would still have the data that the user sent out prior to the revocation. + +Data is always encrypted when in transition between an app and your pod. Note that pod providers don't necessarily have to encrypt the data on your pod. ### Monetization +Pod providers may or may not charge for hosting a pod. This will be determined by the market. They are hoping several business models emerge, some where the user pays for storage, and some funded by advertising for instance, like on the current Web platforms. Not clear which models are currently in use. + +### User experience (if applies) +The Solid ecosystem includes apps built to use solid data pods. +Example apps: + [dokieli](https://dokie.li/) for publishing articles/posts +[exhaustive list of apps built on Solid](https://solidproject.org/use-solid/apps) + ### Interop with other systems +Solid doesn't have its own sense of messages per se, but a Solid pod could be used for example in the implementation of an ActivityPub server. + +The minimum requirement to be 'doing Solid' appears to be the use of JSON-LD (linked data) - discussion of [a subset of JSON-LD](https://stephank.nl/p/2018-10-20-a-proposal-for-standardising-a-subset-of-json-ld.html) + +Some work on ActivityPub that is +[discussion of links between ActivityPub and Solid](https://socialhub.activitypub.rocks/t/which-links-between-activitypub-and-solid-project/529/8) ### Scalability +Published paper [Solid: A Platform for Decentralized Social +Applications Based on Linked Data](https://pdfs.semanticscholar.org/5ac9/3548fd0628f7ff8ff65b5878d04c79c513c4.pdf) indicated sub-linear scaling with number of users/pods. ### Metrics ### In the wild ### Links + +[A Demonstration of the Solid Platform for Social Web +Applications](http://crosscloud.org/2016/www-mansour-pdf.pdf) + +[Solid: A Platform for Decentralized Social +Applications Based on Linked Data](https://pdfs.semanticscholar.org/5ac9/3548fd0628f7ff8ff65b5878d04c79c513c4.pdf) + +(contributions also from Rahul Kothari) From e8b83812ca8cd5540a18df75a7e6d2f52cfd18b8 Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Tue, 26 May 2020 19:41:22 -0700 Subject: [PATCH 02/33] Add applications folder, add files --- applications/aether.md | 0 applications/diaspora.md | 0 applications/iris.md | 0 applications/mastodon.md | 0 {protocols => applications}/peergos.md | 0 applications/zeronet.md | 0 6 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 applications/aether.md create mode 100644 applications/diaspora.md create mode 100644 applications/iris.md create mode 100644 applications/mastodon.md rename {protocols => applications}/peergos.md (100%) create mode 100644 applications/zeronet.md diff --git a/applications/aether.md b/applications/aether.md new file mode 100644 index 0000000..e69de29 diff --git a/applications/diaspora.md b/applications/diaspora.md new file mode 100644 index 0000000..e69de29 diff --git a/applications/iris.md b/applications/iris.md new file mode 100644 index 0000000..e69de29 diff --git a/applications/mastodon.md b/applications/mastodon.md new file mode 100644 index 0000000..e69de29 diff --git a/protocols/peergos.md b/applications/peergos.md similarity index 100% rename from protocols/peergos.md rename to applications/peergos.md diff --git a/applications/zeronet.md b/applications/zeronet.md new file mode 100644 index 0000000..e69de29 From 871956686e7d54c8d94db00eb7e22aa89fcd78f4 Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Tue, 26 May 2020 22:17:13 -0700 Subject: [PATCH 03/33] Mv libp2p to topics --- protocols/libp2p.md | 4 ---- topics/network-structure.md | 4 ++++ 2 files changed, 4 insertions(+), 4 deletions(-) delete mode 100644 protocols/libp2p.md diff --git a/protocols/libp2p.md b/protocols/libp2p.md deleted file mode 100644 index 978f30f..0000000 --- a/protocols/libp2p.md +++ /dev/null @@ -1,4 +0,0 @@ - -[pubsub in the libp2p project](https://docs.libp2p.io/concepts/publish-subscribe/) - -[GossipSub](https://research.protocol.ai/posts/201912-resnetlab-launch/PL-TechRep-gossipsub-v0.1-Dec30.pdf) discussion of gossipsub as a decentralized scalable routing protocol \ No newline at end of file diff --git a/topics/network-structure.md b/topics/network-structure.md index 2d814ce..7f4b297 100644 --- a/topics/network-structure.md +++ b/topics/network-structure.md @@ -10,3 +10,7 @@ Hybrid federated/p2p approaches should be considered - i.e. a client-server arch - i.e. Matrix's new P2P work - categorization of blockchain light clients? + +[pubsub in the libp2p project](https://docs.libp2p.io/concepts/publish-subscribe/) + +[GossipSub](https://research.protocol.ai/posts/201912-resnetlab-launch/PL-TechRep-gossipsub-v0.1-Dec30.pdf) discussion of gossipsub as a decentralized scalable routing protocol From b759fa13ae24d6c01cf80f0949d32024f31447cd Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Tue, 26 May 2020 22:18:46 -0700 Subject: [PATCH 04/33] Add mls to security in topics --- protocols/mls.md => topics/security.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename protocols/mls.md => topics/security.md (100%) diff --git a/protocols/mls.md b/topics/security.md similarity index 100% rename from protocols/mls.md rename to topics/security.md From bc861e20d1a0c3253d1a6049aa101159e408c171 Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Tue, 26 May 2020 22:26:15 -0700 Subject: [PATCH 05/33] Add topics, reorg files --- applications/planetary.md | 0 applications/riot.md | 0 applications/steemit.md | 0 overview.md | 2 +- topics/data.md | 0 topics/filtering.md | 0 topics/governance.md | 0 topics/identity.md | 0 topics/moderation.md | 0 topics/monetization.md | 0 topics/zk-SNARK-and-decentralized-identity.md | 15 --------------- 11 files changed, 1 insertion(+), 16 deletions(-) create mode 100644 applications/planetary.md create mode 100644 applications/riot.md create mode 100644 applications/steemit.md create mode 100644 topics/data.md create mode 100644 topics/filtering.md create mode 100644 topics/governance.md create mode 100644 topics/identity.md create mode 100644 topics/moderation.md create mode 100644 topics/monetization.md delete mode 100644 topics/zk-SNARK-and-decentralized-identity.md diff --git a/applications/planetary.md b/applications/planetary.md new file mode 100644 index 0000000..e69de29 diff --git a/applications/riot.md b/applications/riot.md new file mode 100644 index 0000000..e69de29 diff --git a/applications/steemit.md b/applications/steemit.md new file mode 100644 index 0000000..e69de29 diff --git a/overview.md b/overview.md index 67987c1..a5a8e77 100644 --- a/overview.md +++ b/overview.md @@ -46,7 +46,7 @@ This section will be structured as comparisons between how protocols handle thes - Data models - Data availability -#### Money +#### Monetization - Business models - Payment diff --git a/topics/data.md b/topics/data.md new file mode 100644 index 0000000..e69de29 diff --git a/topics/filtering.md b/topics/filtering.md new file mode 100644 index 0000000..e69de29 diff --git a/topics/governance.md b/topics/governance.md new file mode 100644 index 0000000..e69de29 diff --git a/topics/identity.md b/topics/identity.md new file mode 100644 index 0000000..e69de29 diff --git a/topics/moderation.md b/topics/moderation.md new file mode 100644 index 0000000..e69de29 diff --git a/topics/monetization.md b/topics/monetization.md new file mode 100644 index 0000000..e69de29 diff --git a/topics/zk-SNARK-and-decentralized-identity.md b/topics/zk-SNARK-and-decentralized-identity.md deleted file mode 100644 index d5252ec..0000000 --- a/topics/zk-SNARK-and-decentralized-identity.md +++ /dev/null @@ -1,15 +0,0 @@ -[Zero-knowledge Proof of Identity paper](https://eprint.iacr.org/2019/546.pdf) - "turning X.509 certificates into anonymous credentials." - - -[business level description of zkSNARK](https://medium.com/@SamRadOfficial/why-zk-snarks-are-crucial-for-blockchain-data-privacy-e02f4bc1d352) -"zk-SNARK is a form of cryptography that proves possession of certain information without revealing that information, and without any interaction between the prover and verifier." - - -[Introduction to zk-SNARKs with Examples](https://media.consensys.net/introduction-to-zksnarks-with-examples-3283b554fc3b) - -It looks like the libraries available are still fairly early stage, but the excitement around it should make them move fairly fast - -Open question - how can this be integrated into a universal-use protocol that may include participants that do not use zkSNARK - -can we use it with different levels of uncertainty from different external providers who might identify people in looser ways? - From c292a0d0dab1d812b638a8cf6efd02c4acd6351a Mon Sep 17 00:00:00 2001 From: Golda Velez Date: Wed, 27 May 2020 23:40:58 +0000 Subject: [PATCH 06/33] Update solid.md --- protocols/solid.md | 82 +++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 73 insertions(+), 9 deletions(-) diff --git a/protocols/solid.md b/protocols/solid.md index 2ca6098..bb0e963 100644 --- a/protocols/solid.md +++ b/protocols/solid.md @@ -1,3 +1,4 @@ + # Solid ### Identity @@ -8,6 +9,14 @@ The rights on a Pod are attached to WebIDs. WebIDs may refer to a person, a fami ### Networking/Message passing +[Linked Data Notifications](https://www.w3.org/TR/ldn/) + +Solid allows inboxes for any resource, ie actors or articles +Example: [annotations related to a particular article](https://linkedresearch.org/annotation/csarven.ca/%23i/87bc9a28-9f94-4b1b-a4b9-503899795f6e) + +Currently smart clients retrieve messages with pull/get, rather than server push. + + ### Data Storage/Message Persistance Pod is a data storage space. All a user's data is stored here. Applications read and write data into the Pod depending on the authorisations granted by the user or users associated to that Pod. A person may have multiple pods, for example for work and for home. Data may be replicated across pods. @@ -17,12 +26,20 @@ Users may also switch pods easily without losing their data (such as their conta ### Moderation/Reputation +access control is currently thru ACLs, can plug in different methods; could possibly be swapped out if desired + +an agent with a webid can have control or other levels of access to a resource + ### Social/Discovery WebID allows user addressing across all Solid-conforming apps +Resources (including WebID's -they have uris eg. https://csarven.ca/#i ) are self-describing. They are hosted on pods and have HTTP endpoints. With permissions, normal spidering is possible (from article, you will find author, etc) + There does not seem to be a Solid-based social network as of this writing, some discussion about creating one. Instead, Solid aims to create tools that enable building of decentralized social applications. +Any app that can parse the format can read the inbox, and could send responses. + ### Privacy and Access Control The owner of the pod controls access to who is authorized to read or write data. However if the user revokes access of an app to their data, the app can't further read or update the user's data, but they would still have the data that the user sent out prior to the revocation. @@ -37,29 +54,76 @@ Example apps: [dokieli](https://dokie.li/) for publishing articles/posts [exhaustive list of apps built on Solid](https://solidproject.org/use-solid/apps) - ### Interop with other systems Solid doesn't have its own sense of messages per se, but a Solid pod could be used for example in the implementation of an ActivityPub server. The minimum requirement to be 'doing Solid' appears to be the use of JSON-LD (linked data) - discussion of [a subset of JSON-LD](https://stephank.nl/p/2018-10-20-a-proposal-for-standardising-a-subset-of-json-ld.html) +For example, dokieli (a solid app) is able to send notifications to ActivityPub + Some work on ActivityPub that is -[discussion of links between ActivityPub and Solid](https://socialhub.activitypub.rocks/t/which-links-between-activitypub-and-solid-project/529/8) +[discussion of links between ActivityPub and Solid](https://socialhub.activitypub.rocks/t/which-links-between-activitypub-and-solid-project/529/8) - there is work to close the gap between the specs. + +Compatible with existing web; WebAccessControl/ACLs etc are described in RDF + +Some efforts are being made to make WebIDs compatible with DIDs + +[Linked Data Notifications](https://www.w3.org/TR/ldn/) + +Web Access Control ### Scalability -Published paper [Solid: A Platform for Decentralized Social -Applications Based on Linked Data](https://pdfs.semanticscholar.org/5ac9/3548fd0628f7ff8ff65b5878d04c79c513c4.pdf) indicated sub-linear scaling with number of users/pods. + ### Metrics ### In the wild +[Solid Apps](https://solidproject.org/use-solid/apps) + +### Compare/contrast +Generally: +ActivityPub - smart server +Solid - smart client ### Links -[A Demonstration of the Solid Platform for Social Web -Applications](http://crosscloud.org/2016/www-mansour-pdf.pdf) +* [Solid specification development](https://github.com/solid/specification) +* [Solid Ecosystem Editors Draft] https://solid.github.io/specification/) +* [Solid Chat] https://gitter.im/solid/specification) +* [Solid Community Group Meetings](https://www.w3.org/community/solid/wiki/Meetings) -[Solid: A Platform for Decentralized Social -Applications Based on Linked Data](https://pdfs.semanticscholar.org/5ac9/3548fd0628f7ff8ff65b5878d04c79c513c4.pdf) +[LDN Test Reports and Summary](https://linkedresearch.org/ldn/tests/summary) -(contributions also from Rahul Kothari) +[Linked Research on the Decentralised Web](https://csarven.ca/linked-research-decentralised-web#decentralised-linked-research-application) + + +[A Demonstration of the Solid Platform for Social Web Applications](http://crosscloud.org/2016/www-mansour-pdf.pdf) + + +(contributions also from Rahul Kothari and Sarven Capadisli) + +### Payload example +``` +{ + "@context": { + "@language": "en", + "sioc": "http://rdfs.org/sioc/ns#", + "foaf": "http://xmlns.com/foaf/0.1/" + }, + "@id": "", + "@type": "sioc:Comment", + "sioc:reply_of": { "@id": "http://example.org/article" }, + "sioc:created_at": { + "@type": "http://www.w3.org/2001/XMLSchema#dateTime", + "@value": "2015-12-23T16:44:21Z" + }, + "sioc:content": "This is a great article!", + "sioc:has_creator": { + "@id": "http://example.org/profile", + "@type": "sioc:UserAccount", + "sioc:account_of": { "@id": "http://example.org/profile#alice" }, + "sioc:avatar": { "@id": "http://example.org/profile/avatar.png" }, + "foaf:name": "Alice" + } +} +``` \ No newline at end of file From 3e1bf3b950399859e8ca9c66b3ee0cf2c9947dae Mon Sep 17 00:00:00 2001 From: Golda Velez Date: Sun, 31 May 2020 04:56:40 +0000 Subject: [PATCH 07/33] Update iris.md --- applications/iris.md | 51 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/applications/iris.md b/applications/iris.md index e69de29..c1197ee 100644 --- a/applications/iris.md +++ b/applications/iris.md @@ -0,0 +1,51 @@ +# IRIS + +## Overview +IRIS is a WoT (Web of Trust) based social networking app. It includes libraries for reading messages and indexes, and several demo applications for chat and microblogging. + +### Identity +Uses WoT attestations to link human readable names to key-pair and other identity attributes. + +### Networking/Message passing +Uses GUN for networking and data storage. (stores locally on users' machines without a server) + +### Data Storage/Message Persistance +Also uses IPFS for attachments and message backups. + +### Moderation/Reputation +Users only see messages in their WoT, that is, from users who have been upvoted by someone in a chain from someone they upvoted. Downvotes are also possible. + +### Social/Discovery +Real name search. + +### Privacy and Access Control +A keypair is created for each account. + +### Monetization +Donation supported. + +### User experience (if applies) +Similar to other microblogging platforms, with mobile & desktop clients. Relies on finding/knowing your contacts in order to see messages. It is necessary to reach out to contacts to ask them to add you to see your messages. + +### Interop with other systems +Provides a library for the message layer that could be reused. + +Also allows importing content from other sources, based on attestations of the author's identity. +"For example, a crawler can import and sign other people's messages from Twitter. Only the users who trust the crawler will see the messages." (not sure how to find the crawler in order to trust it?) + +### Scalability + +### Metrics +?? + +### In the wild +Demos: + [https://iris.to/](https://iris.to/) decentralized chat + [https://social.iris.to/](https://social.iris.to/) Angular version + +### Links + +Code: [github.com/irislib/iris](https://github.com/irislib/iris) + +[Learning to Trust Strangers](https://medium.com/@mmalmi/learning-to-trust-strangers-167b652a654f) + From b710f5a2acabc1ba527c9436e6029d8901f5997f Mon Sep 17 00:00:00 2001 From: Mark Nadal Date: Sun, 31 May 2020 22:30:15 +0000 Subject: [PATCH 08/33] Update gun.md --- protocols/gun.md | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/protocols/gun.md b/protocols/gun.md index 31064d1..d2cfaa3 100644 --- a/protocols/gun.md +++ b/protocols/gun.md @@ -3,7 +3,7 @@ ## Stats - 11K+ [github](https://github.com/amark/gun) stars - - 3M ~ 30M [downloads](https://www.jsdelivr.com/package/npm/gun) monthly + - 10M ~ 30M [downloads](https://www.jsdelivr.com/package/npm/gun) monthly ## Whitepaper @@ -82,12 +82,33 @@ Iris and NAB (see above link). ### Social/Discovery +See "identity" section. + ### Privacy and Access Control +1. Cryptographic keys are assigned to roles, groups, or data points. +2. This information is either used to derive a shared ECDH secret to decrypt (read) +3. Or used to load collaborative multi-writer edits (signed). + +For more information, please watch our [Cartoon Cryptography](https://gun.eco/docs/Cartoon-Cryptography) explainer. + ### Interop with other systems +Via plugins, such as backup storage on centralized databases or file systems, or transport layers to other protocols. + ### Scalability + - We've seen a [70% cost savings](https://twitter.com/cagataycali/status/1245442681251205120) on projects with 1M+ page loads a month. + - We've seen in-production deployments with 5M+ monthly active users without increasing infrastructure costs. + ### Metrics +See "stats" section at top. + ### In the wild + + - [Internet Archive](https://news.ycombinator.com/item?id=17685682) (library meta-data) + - [HackerNoon](https://hackernoon.com/state-of-hacker-noon-2019-2020-8w1ls3axx) (annotations) + - [********]() (popular niche alternative to YouTube, cannot disclose) (live comments) + - [Meething](https://meething.space/) (Mozilla backed secure & decentralized video conferencing powered by GUN) + - Iris, Notabug, DTube, others... From a3c556fb756c6542698d5074c857b9f99ef2be7d Mon Sep 17 00:00:00 2001 From: Golda Velez Date: Mon, 1 Jun 2020 01:42:43 +0000 Subject: [PATCH 09/33] Add new file --- protocols/i2p.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 protocols/i2p.md diff --git a/protocols/i2p.md b/protocols/i2p.md new file mode 100644 index 0000000..976626d --- /dev/null +++ b/protocols/i2p.md @@ -0,0 +1,30 @@ +# I2P + +### Identity + +### Networking/Message passing + +### Data Storage/Message Persistance + +### Moderation/Reputation + +### Social/Discovery + +### Privacy and Access Control + +### Monetization + +### User experience (if applies) + +### Interop with other systems +in particular - what is the minimum requirement for a message to enter or leave the system? + +### Scalability + +### Metrics + +### In the wild + +### Links + +https://geti2p.net/en/ \ No newline at end of file From f43f4e0963019a3a0783a3bc088bf5b8ac3984e5 Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Tue, 2 Jun 2020 12:11:03 -0700 Subject: [PATCH 10/33] Add network structure --- protocols/template.md | 3 ++- protocols/xmpp.md | 0 topics/network-structure.md | 37 +++++++++++++++++++++++++++++-------- 3 files changed, 31 insertions(+), 9 deletions(-) create mode 100644 protocols/xmpp.md diff --git a/protocols/template.md b/protocols/template.md index 0d3b19b..17c4e44 100644 --- a/protocols/template.md +++ b/protocols/template.md @@ -4,7 +4,7 @@ ### Networking/Message passing -### Data Storage/Message Persistance +### Data Storage/Message Persistence ### Moderation/Reputation @@ -17,6 +17,7 @@ ### User experience (if applies) ### Interop with other systems + in particular - what is the minimum requirement for a message to enter or leave the system? ### Scalability diff --git a/protocols/xmpp.md b/protocols/xmpp.md new file mode 100644 index 0000000..e69de29 diff --git a/topics/network-structure.md b/topics/network-structure.md index 7f4b297..101c92f 100644 --- a/topics/network-structure.md +++ b/topics/network-structure.md @@ -1,16 +1,37 @@ # Network Structure -Federated approach to decentralized network architectures can be categorized as federation where you pass messages between systems (SMTP, XMPP, ActivityPub), and federation where you replicate data between systems (Git, Matrix, NNTP). +### Federated networks -P2p networks can be analyzed on a spectrum of fully distributed and p2p, to partially distributed. Supernode architectures, and points of logical centralization in p2p systems, should be addressed. +Federated approaches to decentralized network architectures can be categorized as federation where you pass messages between systems (SMTP, XMPP, ActivityPub), and federation where you replicate data between systems (Git, Matrix, NNTP). -- i.e. Textile cafes peers as p2p network infrastructure (https://docs.textile.io/concepts/cafes/) +#### Passing messages between systems -Hybrid federated/p2p approaches should be considered - i.e. a client-server architecture, where you can optionally run the server client-side. +Federated networks that pass messages between systems include SMTP (email), XMPP (chat), and ActivityPub (Mastodon). This is a straightforward approach for systems mainly concerned with one-to-one communications, such as email and chat, because messages do not get replicated across servers that do not concern them. -- i.e. Matrix's new P2P work -- categorization of blockchain light clients? +ActivityPub, a federated social protocol that passes messages between participating servers, is used in one-to-many social applications like Mastodon, a decentralized Twitter alternative. Unlike Twitter, there is [no global shared state across all servers](https://docs.joinmastodon.org/user/network/), so there is no way to browse all public posts. The federated timeline shows public posts that the user's server knows about. The majority of posts surfaced are from people other users of the server follow. -[pubsub in the libp2p project](https://docs.libp2p.io/concepts/publish-subscribe/) +#### Replicating data between systems -[GossipSub](https://research.protocol.ai/posts/201912-resnetlab-launch/PL-TechRep-gossipsub-v0.1-Dec30.pdf) discussion of gossipsub as a decentralized scalable routing protocol +Matrix is a federated protocol that replicates data between systems. Messages in Matrix are replicated over all the servers whose users are participating in a given conversation - similarly to how commits are replicated between Git repositories. The conversation history of the room being replicated across servers allows nodes to resync to the conversation if they go down. To ensure privacy of conversations in rooms, since those conversations are stored across many servers, Matrix has prioritized e2e encryption. + +If bluesky adopts a federated architecture, further work should be done to assess how message passing and message replicating systems affect scalability. Certain types of messages, such as popular topics, could be widely replicated to ensure global availability, and others, such as messages between two users, could be shared only with participating servers. + +### P2p networks + +P2p networks can be analyzed on a spectrum of fully distributed and p2p, to partially distributed. Many p2p systems have supernode architectures, and points of logical centralization. + +Ssb is a fully distributed p2p social protocol that avoids [logical centralization and singletons](https://handbook.scuttlebutt.nz/stories/design-challenge-avoid-centralization-and-singletons), as a design experiment and matter of principle. Ssb was designed for social applications, where data can be passed along follow relationships among users. Data is propagated through a gossip protocol among people who follow each other. As part of its commitment to full decentralization, ssb has no DHT, no universal human-readable namespace, and comes with no bootstrap nodes. New users seeking to connect to the network must often use a [pub server](https://github.com/ssbc/ssb-server/wiki/pub-servers), which is a peer that is always online, has a public IP address, and offers invite codes to new users. + +In IPFS and Hypercore, the focus is on content and not social relationships, so there are central directories to aid in discoverability of content, and nodes that guarantee persistence of content if the original host is offline. 'Gateways' are essentially supernodes, and DHTs used for discovery of content are points of logical centralization. + +Content in IPFS and Hypercore is accessible over HTTP through gateways. These are nodes that run the p2p protocol and allow HTTP requests to access content in the p2p network through it. Some of these gateways, like the one provided by the IPFS developers themselves, are quite large and become points of centralization. However, anyone can [configure a gateway](https://medium.com/@rossbulat/introduction-to-ipfs-set-up-nodes-on-your-network-with-http-gateways-10e21ea689a4) and run one themselves, as the system is permissionless. + +DHTs, distributed hash tables, keep track of who has what data in the network. The data itself may be spread out across many different nodes, but the [DHT](https://docs.ipfs.io/concepts/dht/) serves as the central index of where it is. + +### Hybrid networks + +Hybrid federated/p2p approaches also exist, such as a client-server architecture where you can optionally run the server client-side. + +Matrix's new p2p implementation takes a hybrid approach, moving away from a strictly federated model towards p2p functionality. The new p2p matrix client essentially runs the server client-side, allowing it to participate in the existing federated Matrix ecosystem as a p2p node. + +Light clients in blockchain architectures could also be considered an example of a hybrid approach to p2p network participation. Full nodes participate fully in the blockchain network, storing the entire history and validating all blocks and transactions as they come in. Light clients do not store the full history of the chain, which can be many GBs, but request the relevant history from a full node. This allows light clients to transact without having to store the entire history themselves. From 41dcdca1e2aebbbf3100d52eb4da5c66b4101570 Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Tue, 2 Jun 2020 13:25:46 -0700 Subject: [PATCH 11/33] Finish ssb stub article --- protocols/ssb.md | 93 +++++++++++++++++++++++++++++++++++++------ protocols/template.md | 5 ++- 2 files changed, 84 insertions(+), 14 deletions(-) diff --git a/protocols/ssb.md b/protocols/ssb.md index e1bb134..8b36efa 100644 --- a/protocols/ssb.md +++ b/protocols/ssb.md @@ -2,28 +2,97 @@ [ssb repo](https://github.com/dominictarr/scalable-secure-scuttlebutt) -Ssb, or secure-scuttlebutt, is a distributed gossip protocol designed for social sharing. Every node has a partial view of the network, which makes it hard to get a count of how many total users there are, but according to a network crawl run by a developer in Nov 2019, there are around 16,000 nodes on ssb. Users are distributed across a few different client apps that work on desktop (Patchwork) and mobile (Manyverse, Planetary). +Ssb, or secure-scuttlebutt, is a distributed gossip protocol designed for social sharing. Identities are cryptographic key pairs, feeds are a signed append-only sequence of messages, and nodes use a gossip protocol to disseminate content. Feeds can be thought of as essentially personal blockchains. -### Data Model - -Each post is appended to the last, in an append-only log establishing chronological ordering from the very first. Because every post is chained to the last, there’s currently no way to delete or edit posts. When you follow a user, you will begin to store and sync their posts. An ssb application is constantly sharing data with other nodes in the background while you use it. +Users are distributed across a few different client apps that work on desktop (Patchwork) and mobile (Manyverse, Planetary). ### Identity -Every user has a public/private keypair which is used to sign posts, verifying their authenticity. +A user's identity is their ed25519 key pair which is used to sign posts, verifying their authenticity. -In ssb, users are identified by public keys. My public key is: +The public key is used as the identifier. For example: `@3QHXrXl762sf7P/Q1RMtscA7IRipfUFnE5tpie5McvE=.ed25519` -Users can pick a human-readable nickname that is associated with their key, but nicknames are not unique since there is no global registry. Others can mention me by my nickname @arcalinea, but someone else could also use the same name. +Users can pick a human-readable nickname that is associated with their key, but nicknames are not unique because there is no global registry. Identity is verified based on a web-of-trust - if others that a user knows attest to the authenticity of an account, that account is assumed to be authentic. -Key management is one of the biggest challenges, as users inevitably lose and forget their passwords. Users are in complete control of their identity. That means if they lose their cryptographic key, they can permanently lose access to their account. Keys are also currently stored on devices, which makes it impossible to sign in to one account across multiple devices — a basic feature of social networks users have come to expect. -To attempt to address the problem of key management, a project in the ssb ecosystem, Dark Crystal, has implemented a social key recovery system. It splits keys into shards to store with family and friends who can be trusted to help reconstruct a lost key. +### Data Storage/Message Persistence -### Moderation +Each post is appended to the last, in an append-only log establishing chronological ordering from the very first. Because every post is chained to the last, there’s currently no way to delete or edit posts. When a follow relationship is initiated, the posts of the user being followed begins to be synced to the follower's node. Those messages and files are stored locally on the user's computer, indefinitely, for applications running ssb to read. An ssb application is constantly sharing data with other nodes in the background. -At the ssb protocol level, there is a “flag” feature to send a strong negative signal about bad actors. There is no global moderation, and no specialized moderators. Applications built on top of ssb allow users to “block” and “ignore”. A block in ssb functions more strongly than a block in centralized networks because it means that blocked users no longer have their data passed through those nodes. If enough people block a user or group of users, their part of the network will become cut off from the rest. +Each message contains: + +A signature +The signing public key +A content-hash of the previous message +A sequence number +A timestamp +An identifier of the hashing algorithm in use (currently only "sha256" is supported) +A content object + +### Networking/Message Passing + +Nodes request all messages in the feed that are newer than the latest message they know about. Scuttlebot, the networking component of ssb, maintains a table of known peers which it cycles through asking for updates for all followed feeds. + +Messages are passed through the ssb network via a gossip protocol. Messages may be passed through third parties (unless the user is blocked by that party), which improves availability. Pubs, which are always online, are a necessary component to improve data availability in the ssb ecosystem. Otherwise, user's nodes which are hosting content would go offline too often, preventing access to the latest state. + +### Moderation/Reputation + +There is no global moderation, and no specialized moderators in ssb. At the protocol level, there is a “flag” feature to send a strong negative signal about bad actors. Applications built on top of ssb allow users to “block” and “ignore”. An ignore will simply not show that data to the user's node, although their node will continue to pass their data through the network. A block will cause the user's node to refuse to replicate data from that feed, segmenting it off from their portion of the network. If enough people block a user or group of users, their part of the network will become partitioned from the rest. + +### Social/Discovery + +There is no global feed of content in ssb. All content is surfaced through social discovery. Out-of-band sharing, sending an ssb link through another channel, can also surface new content. + +Ssb clients decide how many hops away from primary follow relationships should be shown. The default is 2 hops, data from friends and FoaFs (friend of a friend's). Messages from 3 hops out are replicated to keep them available for others, but not shown in the user interface. + +"Pubs" are essentially the bootstrap nodes and mail-bots of ssb. They are bot-user nodes with public IPs which stay online and offer invite codes to new users. They also follow users and rebroadcast the messages to other peers. Ssb has no DHT or NAT-traversal utilities, so users must connect to a Pub to distribute their messages. Pubs ensure uptime and availability. Identity is not tied to pubs, unlike homeservers in Matrix or ActivityPub, and a user can join one or multiple pubs. + +### Privacy/Access Control + +Ssb applications can easily support encrypted DMs, as identities are cryptographic keypairs. Whoever controls the private key of an identity can publish to that feed. Messages canot be faked, omitted, or re-ordered, due to the signed append-only log nature of the feed. ### Monetization -Maintainers of p2p networks do not have to pay for hosting costs, since there are no servers and the network naturally grows in capacity as new users join. Developers who want to work on more than a volunteer basis need to find their own funding. The ssb ecosystem is supported through a variety of grants, donations, income from side projects and consulting, and a few companies that have raised money to build applications on ssb. +The ssb ecosystem is supported through a variety of grants, donations, income from side projects and consulting, and a few companies that have raised money to build applications on ssb, including Planetary. + +Pubs, the most resource-intensive nodes, are currently volunteer supported, but at scale could encounter incentive problems. + +### User Experience + +Key management is one of the biggest challenges of ssb, as users inevitably lose and forget their passwords. Users are in complete control of their identity. That means if they lose their cryptographic key, they can permanently lose access to their account. Keys are also currently stored on devices, which makes it impossible to sign in to one account across multiple devices — a basic feature of social networks users have come to expect. + +To attempt to address the problem of key management, a project in the ssb ecosystem, [Dark Crystal](https://darkcrystal.pw), has implemented a social key recovery system. It splits keys into shards to store with family and friends who can be trusted to help reconstruct a lost key. + +Getting connected to ssb is also much more cumbersome than the on-boarding experience for social applications. The p2p bootstrapping process introduces many frictions. First, new users much join a pub to connect to the network after they download an ssb application. Then, there is a long period of waiting time during the initial sync when logs are being downloaded, like the syncing time of a blockchain. A user that has not opened an ssb application in awhile will encounter this initial sync again while their node catches up to the state of the network. + +Another design choice that defies user expectations is the inability to edit or delete content. Because of the append-only nature of ssb feeds, there is no ability to permanently delete a piece of content. Applications can work around this by honoring edit or delete messages appended to the feed, but the original content stays in the append-only log that is shared among all nodes, and other applications could choose not to honor such messages. + +### Interoperability + +Ssb applications generally do not bridge to other applications, although a proof-of-concept experiment involving cross-posting to Twitter, and importing tweets into ssb, has been tried. + +[Discussion of tradeoffs of using IPFS for blob data storage in ssb](https://github.com/ssbc/ssb-server/issues/454) + +### Scalability + +Because ssb is entirely p2p, new users add capacity to the network as they will also be hosting and sharing content. The one point of possible centralization, pub servers, would need to be expanded to keep up with a sudden influx of new users. + +Another potential scalability issue is the size of the append-only log feeds stored on a user's device growing over time. + +### Metrics + +Every node has a partial view of the network, which makes it hard to get a count of how many total users there are, but [according to a network crawl](https://twitter.com/andrestaltz/status/1191821144040574984) run by a developer in Nov 2019, there are around 16,000 nodes on ssb. + +### Tradeoffs + +Feeds being append-only logs means content cannot be easily edited or deleted, but ensures the network converges towards the same state. It is a design choice that maximizes partition tolerance at the expense of mutability. + +Ssb focuses more on moving lightweight social data rather than large data, unlike protocols like bittorrent, Hypercore/Dat, and IPFS. + +### Links + +[Overview](https://scuttlebot.io/more/protocols/secure-scuttlebutt.html) +[Ssb concepts](https://handbook.scuttlebutt.nz/concepts/) +[Article on Manyverse and ssb](https://www.zdnet.com/article/manyverse-and-scuttlebutt-a-human-centric-technology-stack-for-social-applications/) +[Dark Crystal ssb protocol docs](https://darkcrystal.pw/ssb-protocol-docs.html) +[3box comparison of p2p dbs](https://medium.com/3box/3box-research-comparing-distributed-databases-gun-orbitdb-and-scuttlebutt-2e3b5da34ef3) diff --git a/protocols/template.md b/protocols/template.md index 0d3b19b..863017e 100644 --- a/protocols/template.md +++ b/protocols/template.md @@ -10,13 +10,14 @@ ### Social/Discovery -### Privacy and Access Control +### Privacy/Access Control ### Monetization ### User experience (if applies) -### Interop with other systems +### Interoperability + in particular - what is the minimum requirement for a message to enter or leave the system? ### Scalability From 6fab013322ac1c04b32d866b969947cbf678ccca Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Tue, 2 Jun 2020 13:40:23 -0700 Subject: [PATCH 12/33] Add drafts --- applications/aether.md | 43 +++++++++++++++++++++++++++++++++++ applications/diaspora.md | 35 ++++++++++++++++++++++++++++ applications/riot.md | 0 applications/steemit.md | 49 ++++++++++++++++++++++++++++++++++++++++ applications/zeronet.md | 25 ++++++++++++++++++++ protocols/template.md | 11 ++++----- 6 files changed, 156 insertions(+), 7 deletions(-) delete mode 100644 applications/riot.md diff --git a/applications/aether.md b/applications/aether.md index e69de29..01b2018 100644 --- a/applications/aether.md +++ b/applications/aether.md @@ -0,0 +1,43 @@ +# Aether + +Aether is a Reddit-like p2p social network, designed with ephemerality and mutability in mind. + +### Identity + +Identities in Aether are based on keypairs, like in ssb. + +Aether has a strategy for multiple device logins under development — you can store and sync encrypted keys from a remote backend. + +### Networking/Message passing + +### Data Storage/Message Persistence + +Aether's data structure is a DAG (directed acyclic graph). It allows insertion and deletion of content. + +Posts are ephemeral. After a certain period of inactivity on a post, it automatically gets dropped by nodes on the network. + +### Moderation/Reputation + +Each sub-community has its own moderators, which communities elect or impeach themselves. Like sub-reddits, communities largely govern themselves. + +Whitelists and blacklists are used to moderate the collection of communities. When users join, the default is to only display communities on a vetted whitelist, to ensure the safety and comfort of a user's initial experience. + +### Spam + +Aether combats spam by requiring users to complete a small "proof-of-work", a brief hashing function, before they post. This rate-limits the amount of posts that a user can send out to the network, making it more expensive to flood it with spam. + +### Social/Discovery + +### Privacy/Access Control + +### Monetization + +The p2p version is supported through funding from the self-hosted Aether Pro version. + +### User experience + +### Interoperability + +### Scalability + +### Metrics diff --git a/applications/diaspora.md b/applications/diaspora.md index e69de29..261312e 100644 --- a/applications/diaspora.md +++ b/applications/diaspora.md @@ -0,0 +1,35 @@ +# Diaspora + +Diaspora was a federated social network released in 2010. It is still in existence, although mostly unused. + +The federated nodes, called "pods", are hosted by different individuals and institutions. + +### Identity + +### Networking/Message passing + +### Data Storage/Message Persistence + +### Moderation/Reputation + +Like Mastodon, much of the moderation is done at the server level. When Isis joined the Diaspora network in 2014 after they were censored by Twitter, all the larger pods moved to block their server. + +### Social/Discovery + +Like Twitter, Diaspora included hashtags and mentions. + +### Privacy and Access Control + +### Monetization + +### User experience (if applies) + +### Interoperability + +Friendica instances are a part of the diaspora network, and natively supports the Diaspora protocol. + +### Scalability + +### Metrics + +### Links diff --git a/applications/riot.md b/applications/riot.md deleted file mode 100644 index e69de29..0000000 diff --git a/applications/steemit.md b/applications/steemit.md index e69de29..997dcfd 100644 --- a/applications/steemit.md +++ b/applications/steemit.md @@ -0,0 +1,49 @@ +# Steemit + +The Steem cryptocurrency was created for content monetization in social sites. Steemit, a Reddit/Medium-style social network, was the first site built to use Steem. Later, several other platforms including Dtube, a decentralized Youtube, also used Steem for monetization. + +User identities and post data are stored on the Steem blockchain. + +### Identity + +User identities are stored on the Steem blockchain. New account creation requires an email and phone number, and must go through a review process + +A Steemit account functions as a cryptocurrency wallet, and users are responsible for their own key management. There is no account recovery available, and funds can be lost or stolen if the key is compromised. Accounts cannot be deactivated or deleted, since they are permanently stored on the Steem blockchain. + +### Networking/Message passing + +### Data Storage/Message Persistence + +Post data is stored on the Steem blockchain. + +### Moderation/Reputation + +Reputation determines the weight of votes in the network. Older accounts have more reputation. + +### Spam + +Since there is a monetary incentive to create fake accounts to upvote posts, a centralized spam prevention method was necessary to combat abuse. The new account creation process which requires review of an email and phone number is the method used by Steemit. + +### Social/Discovery + +### Privacy and Access Control + +### Monetization + +Steemit mined 80% of Steem in the first week. Steemit benefited from the appreciation of Steem during the 2017 cryptocurrency run-up, but did not manage the bear market well and had to lay off much of its staff. When a company’s monetization strategy depends on a volatile asset, its leaders have to be prudent portfolio managers as well as good operators. + +Other than depending on Steem price appreciation, Steemit monetizes through users promoting their posts. When users perform certain actions on Steemit, they earn Steem. Creating posts that get upvoted qualifies users to earn from a rewards pool. Upvoting posts that later become popular can earn voters a curation reward. Votes are weighted by reputation, which accumulates with age, so older accounts of early adopters have more power in the network. This, as well as the fact that Steem tokens could be mined easily early on, means that Steemit’s incentives are geared towards early adopters. + +### User experience (if applies) + +### Interoperability + +what is the minimum requirement for a message to enter or leave the system? + +### Scalability + +### Metrics + +There are over a million users on Steemit. + +### Links diff --git a/applications/zeronet.md b/applications/zeronet.md index e69de29..88c3edc 100644 --- a/applications/zeronet.md +++ b/applications/zeronet.md @@ -0,0 +1,25 @@ +# Zeronet + +Zeronet is a browser for a decentralized network built on BitTorrent and Bitcoin, and run over Tor for privacy. Every Zeronet site address is a Bitcoin public key, so visitors can send bitcoin directly to the owner. It is essentially a darknet, or another version of Tor hidden services. + +### Identity + +Accounts are essentially Bitcoin wallets. The owner of a site signs the files with the private key, and the public key is the site address. + +### Networking/Message passing + +Zeronet uses the BitTorrent network to find peers that are seeding the site to download the site content from. + +### Data Storage/Message Persistence + +### Moderation/Reputation + +### Social/Discovery + +### Privacy and Access Control + +### Monetization + +### Metrics + +### Links diff --git a/protocols/template.md b/protocols/template.md index 0d3b19b..5a4462b 100644 --- a/protocols/template.md +++ b/protocols/template.md @@ -1,10 +1,8 @@ -# Name - ### Identity ### Networking/Message passing -### Data Storage/Message Persistance +### Data Storage/Message Persistence ### Moderation/Reputation @@ -16,13 +14,12 @@ ### User experience (if applies) -### Interop with other systems -in particular - what is the minimum requirement for a message to enter or leave the system? +### Interoperability + +what is the minimum requirement for a message to enter or leave the system? ### Scalability ### Metrics -### In the wild - ### Links From ebaf36213fc5dba607cca0b6c72f9af4b888bcce Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Wed, 3 Jun 2020 18:22:09 -0700 Subject: [PATCH 13/33] Finish Aether --- applications/aether.md | 52 +++++++++++++++++++++++++++++++----------- 1 file changed, 39 insertions(+), 13 deletions(-) diff --git a/applications/aether.md b/applications/aether.md index 01b2018..d24d851 100644 --- a/applications/aether.md +++ b/applications/aether.md @@ -1,26 +1,38 @@ # Aether -Aether is a Reddit-like p2p social network, designed with ephemerality and mutability in mind. +Aether is a Reddit-like p2p social network - a public discussion forum designed with ephemerality and mutability in mind. It is available as a desktop application. Aether's p2p protocol is called Mim. In practice, Aether is the only application using Mim, but it could support other applications. ### Identity -Identities in Aether are based on keypairs, like in ssb. - -Aether has a strategy for multiple device logins under development — you can store and sync encrypted keys from a remote backend. +Identities in Aether are based on keypairs, like in ssb. As with ssb, the human-readable usernames are not unique, although users can choose a custom nickname. Multi-device usage is currently possible, but difficult (requires manually porting a user config file). There is a strategy for multiple device logins under development that would allow users to store and sync encrypted keys from a remote backend. ### Networking/Message passing +Aether is a flood network where every node stores everything. All nodes download content as well as provide it to other nodes. Communities with objectionable content can optionally be blocked by the user, so that content is not stored on their machine. + +Aether comes with a list of bootstrap nodes. Nodes find each other through a probabilistic search over a network table. This is a random process in which no patterns of data distribution can be discerned, so only the last hop of where data came from is visible, not the source. Once nodes are connected, they sync their data. + +Syncing nodes request caches of data from others. Every node stores pre-packaged answers to common queries. Data from each endpoint (boards, threads, posts), will be cached every day, and stored for 7 days, after which it is no longer cached. Bootstrapping nodes do not download the entire history of the network - only the last 7 days by default. + +New posts have a 0 to 10 minute delay, as it takes time to traverse the p2p network. + ### Data Storage/Message Persistence -Aether's data structure is a DAG (directed acyclic graph). It allows insertion and deletion of content. +Aether's data structure is a DAG (directed acyclic graph). It allows insertion and updates of content. Deletion of content is achieved by inserting a blanck update. The backend keeps a local SQLite database up-to-date with the state of the network. The frontend compiles and presents the graph data. -Posts are ephemeral. After a certain period of inactivity on a post, it automatically gets dropped by nodes on the network. +Posts are ephemeral. After a certain period of inactivity on a post (the default is 6 months from the last reference), it automatically gets deleted by nodes on the network. If a post continues to referenced, it stays up, so actively discussed topics do not disappear. ### Moderation/Reputation -Each sub-community has its own moderators, which communities elect or impeach themselves. Like sub-reddits, communities largely govern themselves. +Each sub-community has its own moderators, which communities elect or impeach themselves. Like sub-reddits, communities largely govern themselves. Mods (moderators) can approve or delete posts. -Whitelists and blacklists are used to moderate the collection of communities. When users join, the default is to only display communities on a vetted whitelist, to ensure the safety and comfort of a user's initial experience. +Users can report posts. Reports go to mods of the community. Users elect moderators in communities, and can impeach them. If a user votes to impeach a mod, the mod's actions will be ineffective for that user, even if the vote wasn't won. All mod decisions are public history. + +Moderator elections begin once a community has more than 100 active users within the last 2 weeks. Voting thresholds for moderators is a 51% positive vote of at least 5% of the community's members. To prevent hostile takeovers, the population of communities is determined on a 2-week trailing basis, and mods can temporarily lock a community to new entrants. + +Users who wish to be mods can enable 'mod mode', which allows them to proactively attempt to approve or delete posts. Their moderation history can be observed by others considering electing them as an official mod. + +Whitelists and blacklists are used to moderate the collection of communities. When users join, the default is to only display SFW communities on a vetted whitelist, to ensure the safety and comfort of a user's initial experience. ### Spam @@ -28,16 +40,30 @@ Aether combats spam by requiring users to complete a small "proof-of-work", a br ### Social/Discovery +Users can search communities, content, and people. The user interface has tabs to discover newest content and popular content. + +There is no follow functionality for other users at present. + ### Privacy/Access Control +All posts are public. + ### Monetization -The p2p version is supported through funding from the self-hosted Aether Pro version. - -### User experience +The Aether p2p version is supported through Aether Pro, a SaaS version of Aether designed for private work deployments. ### Interoperability -### Scalability +Aether p2p is working on implementing a chat that bridges to Matrix p2p. -### Metrics +### User Experience & Scalability + +Aether is currently only available on desktop. There is no mobile app, as mobile devices have limited disk space and energy stores to run the p2p network. + +However, a mobile app is under development that uses the following method: Running an Aether host in a docker image on a home server, and allowing mobile devices to connect to it. + +Creating a mobile app in a way that requires users to run home servers, rather than simply providing full nodes that would allow a mobile app to download content, is a design decision that maximizes decentralization at the expense of user convenience. Every mobile user running a home server will also still be providing capacity to the network to help it scale in a purely p2p manner. + +### Links + +[Mims protocol](https://getaether.net/mim-docs/) From 6d6d130ae5d02ab95680ea5e9c5672f588d91bb2 Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Wed, 3 Jun 2020 18:34:20 -0700 Subject: [PATCH 14/33] Finish zeronet --- applications/zeronet.md | 38 ++++++++++++++++++++++++++++++++++---- 1 file changed, 34 insertions(+), 4 deletions(-) diff --git a/applications/zeronet.md b/applications/zeronet.md index 88c3edc..3090cc8 100644 --- a/applications/zeronet.md +++ b/applications/zeronet.md @@ -1,25 +1,55 @@ # Zeronet -Zeronet is a browser for a decentralized network built on BitTorrent and Bitcoin, and run over Tor for privacy. Every Zeronet site address is a Bitcoin public key, so visitors can send bitcoin directly to the owner. It is essentially a darknet, or another version of Tor hidden services. +Zeronet is a browser for a decentralized network built on BitTorrent and Bitcoin. Instead of having IP addresses, Zeronet site addresses are Bitcoin public keys. + +Example sites created on Zeronet include ZeroTalk (like Reddit), ZeroBlog (microblogging), ZeroMail (encrypted mail), and ZeroMe (p2p social network). + +ZeroMe is a proof-of-concept that demonstrates how to build a Twitter-like site in a decentralized browser. It has not received wide usage. ### Identity -Accounts are essentially Bitcoin wallets. The owner of a site signs the files with the private key, and the public key is the site address. +The creator of a site signs the files with the private key, and the public key is the site address. This is a Bitcoin key which can be exported to a wallet, allowing funds to be sent to the address and collected by the site creator. Zeronet domains end in `.bit`. + +ZeroId is an authorization provider that lets you interact with sites without contacting the owner, and is used for sites with user interaction like ZeroMe. ### Networking/Message passing -Zeronet uses the BitTorrent network to find peers that are seeding the site to download the site content from. +Zeronet uses the BitTorrent network to find peers that are seeding the site to download the site content from. When a user visits a site, they download the site files. Once they've visited, they start serving that site as well, seeding it to others. ### Data Storage/Message Persistence +Zeronet uses BitTorrent trackers and its own variety of trackers (zero://). + ### Moderation/Reputation +Blacklists are opt-in. + +ZeroId provides some control over user accounts to fight spam, by limiting the number of registrations from an IP address. + ### Social/Discovery +There are a few search engines for Zeronet, which have scraped and index the network. Zeronet addresses are commonly shared out-of-band. + ### Privacy and Access Control +Sites that take user input, like ZeroMe, ask the user for permission to grant read/write access. + +### User Experience + +Zeronet's decentralized hosting design allows for one-click site cloning. Popular sites scale with demand, as visitors become seeders of site content as well. + +### Interoperability + +Despite being compared to Beaker Browser and IPFS, Zeronet does not interoperate with other sites, or even with BitTorrent. + +Zeronet can be run over Tor for privacy. but it does not support .onion sites. I2P is not supported. + ### Monetization -### Metrics +The Zeronet project relies on tips. + +An interesting element of monetization in Zeronet is that the fact that site addresses are bitcoin addresses, which means that site owners can be tipped directly to the address. The owner can retrieve the funds by importing the private key into a bitcoin wallet. ### Links + +[Zeronet](zeronet.io) From 19d3f55caa05c38c26db480f530e6fde4d67079a Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Wed, 3 Jun 2020 18:42:40 -0700 Subject: [PATCH 15/33] Add diaspora --- applications/diaspora.md | 45 +++++++++++++++++++++++++++++++++------- 1 file changed, 38 insertions(+), 7 deletions(-) diff --git a/applications/diaspora.md b/applications/diaspora.md index 261312e..11c1a21 100644 --- a/applications/diaspora.md +++ b/applications/diaspora.md @@ -1,35 +1,66 @@ # Diaspora -Diaspora was a federated social network released in 2010. It is still in existence, although mostly unused. +Diaspora was a federated social network released in 2010. It is still in existence, although not widely used. It does not use the ActivityPub federation protocol. In the diaspora protocol, messages are passed between servers. -The federated nodes, called "pods", are hosted by different individuals and institutions. +The federated nodes, called "pods", are hosted by different individuals and institutions. User accounts, which are tied to pods, are called "seeds". ### Identity +Users joining Diaspora pick a pod to register their identity with. User identities contain the username, the hostname, and the port if their server does not listen on the default ports. + +`alice:example.org` + +It is not possible to move a user account to another pod once created. + ### Networking/Message passing +Diaspora uses the Webfinger protocol to discover users from other pods. User information is returned via hCard, an open microformat standard for identity. + +Messages sent between servers are serialized to XML, then signed using the Salmon Magic Signature protocol. + ### Data Storage/Message Persistence +User data in Diaspora is tied to their pod server. A user can export all of their data at any time, to reduce the risk of their pod going down. + ### Moderation/Reputation -Like Mastodon, much of the moderation is done at the server level. When Isis joined the Diaspora network in 2014 after they were censored by Twitter, all the larger pods moved to block their server. +Like Mastodon, moderation is done at the server level. When ISIS joined the Diaspora network in 2014 after they were censored by Twitter, all the larger pods moved to block their server. + +In this [Github issue about admin reports](https://github.com/diaspora/diaspora/issues/7316), it is possible to see how communities struggle when tools for moderation are not built into federated networks from the start. This user requested a way to forward spam reports to the source pod, as that was the only way to remove content from the entire network. Without this feature, pod administrators resorted to manually searching for and contacting other administrators. ### Social/Discovery -Like Twitter, Diaspora included hashtags and mentions. +Like Twitter, Diaspora includes hashtags and mentions. ### Privacy and Access Control +Profiles in Diaspora can be public or private. + +Private messages in Diaspora are encrypted. + +Diaspora allows users to group their contacts into "aspects". Limited posts will only be shown to contacts in the selected aspect. + +Communication between pods is encrypted, but data stored on pods is not. Administrators could access all profile and post data. + ### Monetization -### User experience (if applies) +Diaspora was initially funded through a kickstarter that raised $200,000. ### Interoperability -Friendica instances are a part of the diaspora network, and natively supports the Diaspora protocol. +Friendica instances are a part of the diaspora network, and natively support the Diaspora protocol. -### Scalability +Diaspora posts can be propagated to accounts on [WordPress, Twitter, and Tumblr](https://wiki.diasporafoundation.org/Integrating_other_social_networks). + +Diaspora has not integrated with ActivityPub, despite the similarities of approaches to federated social. Discussion of this topic can be found on the [Discourse forum](https://discourse.diasporafoundation.org/t/lets-talk-about-activitypub/741). ### Metrics +[Number of Diaspora users](https://fediverse.party/en/diaspora) + ### Links + +[Diaspora Federation Protocol](https://diaspora.github.io/diaspora_federation/) +[Webfinger](https://tools.ietf.org/html/rfc7033) +[Salmon Magic Signatures](https://cdn.rawgit.com/salmon-protocol/salmon-protocol/master/draft-panzer-magicsig-01.html) +[Discourse Forum](https://discourse.diasporafoundation.org/) From 5eafe7036a0f83b19b141b402a4609a1b43b3acf Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Fri, 5 Jun 2020 16:44:46 -0700 Subject: [PATCH 16/33] s/in the wild/implementations --- protocols/gun.md | 46 +++++++++++++++++++++---------------------- protocols/matrix.md | 7 +++---- protocols/solid.md | 41 ++++++++++++++++++++------------------ protocols/template.md | 2 ++ 4 files changed, 49 insertions(+), 47 deletions(-) diff --git a/protocols/gun.md b/protocols/gun.md index d2cfaa3..736cc98 100644 --- a/protocols/gun.md +++ b/protocols/gun.md @@ -2,8 +2,8 @@ ## Stats - - 11K+ [github](https://github.com/amark/gun) stars - - 10M ~ 30M [downloads](https://www.jsdelivr.com/package/npm/gun) monthly +- 11K+ [github](https://github.com/amark/gun) stars +- 10M ~ 30M [downloads](https://www.jsdelivr.com/package/npm/gun) monthly ## Whitepaper @@ -11,19 +11,19 @@ https://stanford.edu/~nadal/A-Decentralized-Data-Synchronization-Protocol.pdf ## Stack - - [ERA](https://www.youtube.com/watch?v=1HJdrBk3BlE) (economic system) - - [AXE](http://axe.eco) (routing) - - [DAM](https://gun.eco/docs/DAM) (networking) - - [RAD](https://gun.eco/docs/RAD) (storage) - - [SEA](https://gun.eco/docs/SEA) (security) - - [GUN](http://gun.eco) (data) - - [HAM](https://gun.eco/distributed/matters.html) (conflict resolution) +- [ERA](https://www.youtube.com/watch?v=1HJdrBk3BlE) (economic system) +- [AXE](http://axe.eco) (routing) +- [DAM](https://gun.eco/docs/DAM) (networking) +- [RAD](https://gun.eco/docs/RAD) (storage) +- [SEA](https://gun.eco/docs/SEA) (security) +- [GUN](http://gun.eco) (data) +- [HAM](https://gun.eco/distributed/matters.html) (conflict resolution) ## Bluesky - - Creating customizable interfaces: See [Party](https://party.lol/) which reuses any existing website (gmail, twitter, Facebook, reddit, etc.) for E2EE messages. - - Communities setting norms: See my article on [emotionally intelligent social networks](https://hackernoon.com/a-new-kind-of-social-network-emotional-intelligence-e45dcddb1bdb), also check out the wild [Notabug](https://notabug.io/t/notabug/comments/59382d2a08b7d7073415b5b6ae29dfe617690d74/welcome-to-notabug) experiment. - - Keeping public conversation from fragmenting: Check out ["How to fix the internet"](https://hackernoon.com/what-is-wrong-with-the-internet-and-how-to-fix-it-c67w32no) by [Martti Malmi](https://twitter.com/marttimalmi) (Satoshi's 1st Bitcoin contributor) which is exactly this. +- Creating customizable interfaces: See [Party](https://party.lol/) which reuses any existing website (gmail, twitter, Facebook, reddit, etc.) for E2EE messages. +- Communities setting norms: See my article on [emotionally intelligent social networks](https://hackernoon.com/a-new-kind-of-social-network-emotional-intelligence-e45dcddb1bdb), also check out the wild [Notabug](https://notabug.io/t/notabug/comments/59382d2a08b7d7073415b5b6ae29dfe617690d74/welcome-to-notabug) experiment. +- Keeping public conversation from fragmenting: Check out ["How to fix the internet"](https://hackernoon.com/what-is-wrong-with-the-internet-and-how-to-fix-it-c67w32no) by [Martti Malmi](https://twitter.com/marttimalmi) (Satoshi's 1st Bitcoin contributor) which is exactly this. ## Network structure @@ -79,16 +79,15 @@ Iris and NAB (see above link). - User level (yes) - Content level (yes) - ### Social/Discovery See "identity" section. ### Privacy and Access Control -1. Cryptographic keys are assigned to roles, groups, or data points. -2. This information is either used to derive a shared ECDH secret to decrypt (read) -3. Or used to load collaborative multi-writer edits (signed). +1. Cryptographic keys are assigned to roles, groups, or data points. +2. This information is either used to derive a shared ECDH secret to decrypt (read) +3. Or used to load collaborative multi-writer edits (signed). For more information, please watch our [Cartoon Cryptography](https://gun.eco/docs/Cartoon-Cryptography) explainer. @@ -98,17 +97,16 @@ Via plugins, such as backup storage on centralized databases or file systems, or ### Scalability - - We've seen a [70% cost savings](https://twitter.com/cagataycali/status/1245442681251205120) on projects with 1M+ page loads a month. - - We've seen in-production deployments with 5M+ monthly active users without increasing infrastructure costs. +- We've seen a [70% cost savings](https://twitter.com/cagataycali/status/1245442681251205120) on projects with 1M+ page loads a month. +- We've seen in-production deployments with 5M+ monthly active users without increasing infrastructure costs. ### Metrics See "stats" section at top. -### In the wild +### Implementations - - [Internet Archive](https://news.ycombinator.com/item?id=17685682) (library meta-data) - - [HackerNoon](https://hackernoon.com/state-of-hacker-noon-2019-2020-8w1ls3axx) (annotations) - - [********]() (popular niche alternative to YouTube, cannot disclose) (live comments) - - [Meething](https://meething.space/) (Mozilla backed secure & decentralized video conferencing powered by GUN) - - Iris, Notabug, DTube, others... +- [Internet Archive](https://news.ycombinator.com/item?id=17685682) (library meta-data) +- [HackerNoon](https://hackernoon.com/state-of-hacker-noon-2019-2020-8w1ls3axx) (annotations) +- [Meething](https://meething.space/) (Mozilla backed secure & decentralized video conferencing powered by GUN) +- Iris, Notabug, DTube, others... diff --git a/protocols/matrix.md b/protocols/matrix.md index be2d6fd..a83e31f 100644 --- a/protocols/matrix.md +++ b/protocols/matrix.md @@ -2,13 +2,13 @@ Matrix is a protocol for replicating a signed history of JSON objects in realtime across a set of nodes with (optional) end-to-end encryption. -It's designed to support multiple communication cases - Chat, VoIP, IoT, VR/AR, Social etc, but so far effort has been concentrated on providing a federated chat experience with good UX at scale. The public network currently (Feb 2020) has 14.7M known addressable users, with many others in private federations or on servers which don't report stats. Matrix supports multiple clients (most notably [Riot](https://riot.im), the flagship app from the core team), and has bridges to many other chat systems (IRC, Slack, Discord, Telegram etc). +It's designed to support multiple communication cases - Chat, VoIP, IoT, VR/AR, Social etc, but so far effort has been concentrated on providing a federated chat experience with good UX at scale. The public network currently (Feb 2020) has 14.7M known addressable users, with many others in private federations or on servers which don't report stats. Matrix supports multiple clients (most notably [Riot](https://riot.im), the flagship app from the core team), and has bridges to many other chat systems (IRC, Slack, Discord, Telegram etc). Note: In particular, we may be interested in the [Proposal for Aggregation via Relations](https://github.com/matrix-org/matrix-doc/blob/matthew/msc1849/proposals/1849-aggregations.md) ### Identity -Matrix has a more flexible identity solution than most decentralised protocols - users have a Matrix user ID, but can also use 3rd party IDs. A Matrix account can link to ids such as email addresses, social accounts, and phone numbers. A globally federated cluster of trusted identity servers verify and replicate the mappings, although this is a stopgap solution until a fully decentralised identity solution is adopted. +Matrix has a more flexible identity solution than most decentralised protocols - users have a Matrix user ID, but can also use 3rd party IDs. A Matrix account can link to ids such as email addresses, social accounts, and phone numbers. A globally federated cluster of trusted identity servers verify and replicate the mappings, although this is a stopgap solution until a fully decentralised identity solution is adopted. ### Data Storage/Message Persistance @@ -40,7 +40,6 @@ Bridges that allow communication with IRC, Slack, Discord, Telegram and others ### Metrics -### In the wild +### Implementations ### Links - diff --git a/protocols/solid.md b/protocols/solid.md index bb0e963..c304804 100644 --- a/protocols/solid.md +++ b/protocols/solid.md @@ -1,4 +1,3 @@ - # Solid ### Identity @@ -16,11 +15,11 @@ Example: [annotations related to a particular article](https://linkedresearch.or Currently smart clients retrieve messages with pull/get, rather than server push. - ### Data Storage/Message Persistance + Pod is a data storage space. All a user's data is stored here. Applications read and write data into the Pod depending on the authorisations granted by the user or users associated to that Pod. A person may have multiple pods, for example for work and for home. Data may be replicated across pods. -Users may self-host their pods (on their own server) or use a "Pod Provider". +Users may self-host their pods (on their own server) or use a "Pod Provider". Users may also switch pods easily without losing their data (such as their contacts and chat history. @@ -32,36 +31,40 @@ an agent with a webid can have control or other levels of access to a resource ### Social/Discovery -WebID allows user addressing across all Solid-conforming apps +WebID allows user addressing across all Solid-conforming apps -Resources (including WebID's -they have uris eg. https://csarven.ca/#i ) are self-describing. They are hosted on pods and have HTTP endpoints. With permissions, normal spidering is possible (from article, you will find author, etc) +Resources (including WebID's -they have uris eg. https://csarven.ca/#i ) are self-describing. They are hosted on pods and have HTTP endpoints. With permissions, normal spidering is possible (from article, you will find author, etc) There does not seem to be a Solid-based social network as of this writing, some discussion about creating one. Instead, Solid aims to create tools that enable building of decentralized social applications. Any app that can parse the format can read the inbox, and could send responses. ### Privacy and Access Control + The owner of the pod controls access to who is authorized to read or write data. However if the user revokes access of an app to their data, the app can't further read or update the user's data, but they would still have the data that the user sent out prior to the revocation. -Data is always encrypted when in transition between an app and your pod. Note that pod providers don't necessarily have to encrypt the data on your pod. +Data is always encrypted when in transition between an app and your pod. Note that pod providers don't necessarily have to encrypt the data on your pod. ### Monetization -Pod providers may or may not charge for hosting a pod. This will be determined by the market. They are hoping several business models emerge, some where the user pays for storage, and some funded by advertising for instance, like on the current Web platforms. Not clear which models are currently in use. + +Pod providers may or may not charge for hosting a pod. This will be determined by the market. They are hoping several business models emerge, some where the user pays for storage, and some funded by advertising for instance, like on the current Web platforms. Not clear which models are currently in use. ### User experience (if applies) + The Solid ecosystem includes apps built to use solid data pods. Example apps: - [dokieli](https://dokie.li/) for publishing articles/posts +[dokieli](https://dokie.li/) for publishing articles/posts [exhaustive list of apps built on Solid](https://solidproject.org/use-solid/apps) - + ### Interop with other systems + Solid doesn't have its own sense of messages per se, but a Solid pod could be used for example in the implementation of an ActivityPub server. The minimum requirement to be 'doing Solid' appears to be the use of JSON-LD (linked data) - discussion of [a subset of JSON-LD](https://stephank.nl/p/2018-10-20-a-proposal-for-standardising-a-subset-of-json-ld.html) For example, dokieli (a solid app) is able to send notifications to ActivityPub -Some work on ActivityPub that is +Some work on ActivityPub that is [discussion of links between ActivityPub and Solid](https://socialhub.activitypub.rocks/t/which-links-between-activitypub-and-solid-project/529/8) - there is work to close the gap between the specs. Compatible with existing web; WebAccessControl/ACLs etc are described in RDF @@ -74,35 +77,35 @@ Web Access Control ### Scalability - ### Metrics -### In the wild +### Implementations + [Solid Apps](https://solidproject.org/use-solid/apps) ### Compare/contrast + Generally: ActivityPub - smart server Solid - smart client ### Links -* [Solid specification development](https://github.com/solid/specification) -* [Solid Ecosystem Editors Draft] https://solid.github.io/specification/) -* [Solid Chat] https://gitter.im/solid/specification) -* [Solid Community Group Meetings](https://www.w3.org/community/solid/wiki/Meetings) +- [Solid specification development](https://github.com/solid/specification) +- [Solid Ecosystem Editors Draft] https://solid.github.io/specification/) +- [Solid Chat] https://gitter.im/solid/specification) +- [Solid Community Group Meetings](https://www.w3.org/community/solid/wiki/Meetings) [LDN Test Reports and Summary](https://linkedresearch.org/ldn/tests/summary) [Linked Research on the Decentralised Web](https://csarven.ca/linked-research-decentralised-web#decentralised-linked-research-application) - [A Demonstration of the Solid Platform for Social Web Applications](http://crosscloud.org/2016/www-mansour-pdf.pdf) - (contributions also from Rahul Kothari and Sarven Capadisli) ### Payload example + ``` { "@context": { @@ -126,4 +129,4 @@ Solid - smart client "foaf:name": "Alice" } } -``` \ No newline at end of file +``` diff --git a/protocols/template.md b/protocols/template.md index 5d4a26e..1f57110 100644 --- a/protocols/template.md +++ b/protocols/template.md @@ -22,4 +22,6 @@ what is the minimum requirement for a message to enter or leave the system? ### Metrics +### Implementations + ### Links From 39997a25f5cdc91bd8aad063fb4d593e2c91d54f Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Mon, 8 Jun 2020 11:40:05 -0700 Subject: [PATCH 17/33] Edit gun --- protocols/gun.md | 70 +++++++++++++++--------------------------------- 1 file changed, 21 insertions(+), 49 deletions(-) diff --git a/protocols/gun.md b/protocols/gun.md index 736cc98..13ace05 100644 --- a/protocols/gun.md +++ b/protocols/gun.md @@ -1,15 +1,8 @@ # GUN -## Stats +GUN is a decentralized graph database, and an ecosystem of tools. -- 11K+ [github](https://github.com/amark/gun) stars -- 10M ~ 30M [downloads](https://www.jsdelivr.com/package/npm/gun) monthly - -## Whitepaper - -https://stanford.edu/~nadal/A-Decentralized-Data-Synchronization-Protocol.pdf - -## Stack +The stack includes: - [ERA](https://www.youtube.com/watch?v=1HJdrBk3BlE) (economic system) - [AXE](http://axe.eco) (routing) @@ -19,48 +12,28 @@ https://stanford.edu/~nadal/A-Decentralized-Data-Synchronization-Protocol.pdf - [GUN](http://gun.eco) (data) - [HAM](https://gun.eco/distributed/matters.html) (conflict resolution) -## Bluesky +### Network structure -- Creating customizable interfaces: See [Party](https://party.lol/) which reuses any existing website (gmail, twitter, Facebook, reddit, etc.) for E2EE messages. -- Communities setting norms: See my article on [emotionally intelligent social networks](https://hackernoon.com/a-new-kind-of-social-network-emotional-intelligence-e45dcddb1bdb), also check out the wild [Notabug](https://notabug.io/t/notabug/comments/59382d2a08b7d7073415b5b6ae29dfe617690d74/welcome-to-notabug) experiment. -- Keeping public conversation from fragmenting: Check out ["How to fix the internet"](https://hackernoon.com/what-is-wrong-with-the-internet-and-how-to-fix-it-c67w32no) by [Martti Malmi](https://twitter.com/marttimalmi) (Satoshi's 1st Bitcoin contributor) which is exactly this. - -## Network structure - -Decentralized/p2p. - -Uses data replication via message passing. +Decentralized/p2p. Uses data replication via message passing. It daisy-chains, [routes](https://gun.eco/docs/Routing), & relays messages. -Think of it as doing also what the internet, routers, switches, etc. already does. - -But also works over most existing transports: WebRTC, UDP multicast, TCP/HTTP -> Websocket, etc. +It also works over most existing transports: WebRTC, UDP multicast, TCP/HTTP -> Websocket, etc. ## Identity +GUN supports multi-device login that is cryptographically secure yet uses "normal" UX. (?) This technique can be extended to secure password resets (no servers). + +There is a password recovery mechanism using a 3-Friend-Authorization approach. + Iris adds a [reputation system](https://medium.com/@mmalmi/learning-to-trust-strangers-167b652a654f) to SEA which provides the underlying cryptographic Security, Encryption, & Authorization. -The most important differentiating factor is that we support multi-device login that is cryptographically secure yet uses "normal" UX. - -This technique can be extended to secure password resets (no servers) - -And, password recovery using a 3-Friend-Authorization approach that we cutely call 3FA (please don't use this with your Bitcoins). - -- Decentralized identity (ECDSA) -- Key management (Iris) - - Key verification (Iris) - - Key backup (GUN+SEA) -- Reputation, Trust (Iris) -- Account recovery (3FA) -- Privacy (ECDH, AES) - #### Data - Data models (graphs, key/value, document, table, relational, hyper-graphs) - Data availability ([AP](https://gun.eco/docs/CAP-Theorem), strongly eventually consistent) -#### Money +#### Monetization - Business models (Open Source, donation, enterprise, + decentralized PaaS, dBaaS, SaaS, Marketplace) - Payment (AXE bandwidth credits) @@ -68,7 +41,6 @@ And, password recovery using a 3-Friend-Authorization approach that we cutely ca #### Filtering - Queries (GUN, [GraphQL](https://github.com/brysgo/graphql-gun), historical prototypes of SQL, Mango [Mongo]) -- Curation (Iris, also see [NAB](https://notabug.io/t/notabug/comments/59382d2a08b7d7073415b5b6ae29dfe617690d74/welcome-to-notabug)) #### Moderation @@ -79,30 +51,24 @@ Iris and NAB (see above link). - User level (yes) - Content level (yes) -### Social/Discovery - -See "identity" section. - ### Privacy and Access Control 1. Cryptographic keys are assigned to roles, groups, or data points. 2. This information is either used to derive a shared ECDH secret to decrypt (read) 3. Or used to load collaborative multi-writer edits (signed). -For more information, please watch our [Cartoon Cryptography](https://gun.eco/docs/Cartoon-Cryptography) explainer. +### Interoperablity -### Interop with other systems - -Via plugins, such as backup storage on centralized databases or file systems, or transport layers to other protocols. +Plugins, such as backup storage on centralized databases or file systems, can be used to extend GUN. ### Scalability -- We've seen a [70% cost savings](https://twitter.com/cagataycali/status/1245442681251205120) on projects with 1M+ page loads a month. -- We've seen in-production deployments with 5M+ monthly active users without increasing infrastructure costs. +A user reported a [70% cost savings](https://twitter.com/cagataycali/status/1245442681251205120) on cloud bills on a project with 1M+ page loads a month. ### Metrics -See "stats" section at top. +- 11K+ [github](https://github.com/amark/gun) stars +- 10M ~ 30M monthly [downloads](https://www.jsdelivr.com/package/npm/gun) ### Implementations @@ -110,3 +76,9 @@ See "stats" section at top. - [HackerNoon](https://hackernoon.com/state-of-hacker-noon-2019-2020-8w1ls3axx) (annotations) - [Meething](https://meething.space/) (Mozilla backed secure & decentralized video conferencing powered by GUN) - Iris, Notabug, DTube, others... + [Party](https://party.lol/) + [Notabug](https://notabug.io/t/notabug/comments/59382d2a08b7d7073415b5b6ae29dfe617690d74/welcome-to-notabug) + +### Links + +[Whitepaper](https://stanford.edu/~nadal/A-Decentralized-Data-Synchronization-Protocol.pdf) From 513e07bc26207956e935086128900b9d1db603b3 Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Tue, 9 Jun 2020 14:30:21 -0700 Subject: [PATCH 18/33] Edit gun --- protocols/gun.md | 90 ++++++++++++++++++++++-------------------------- 1 file changed, 41 insertions(+), 49 deletions(-) diff --git a/protocols/gun.md b/protocols/gun.md index 13ace05..a34be92 100644 --- a/protocols/gun.md +++ b/protocols/gun.md @@ -1,61 +1,40 @@ -# GUN +# Gun -GUN is a decentralized graph database, and an ecosystem of tools. +Gun is a distributed database with a graph structure. Related libraries include a constellation of tools for merging conflicting data and handling routing and security. -The stack includes: +In GunDB's graph store, entries are [JSON objects under UUID keys](https://gun.eco/docs/Porting-GUN). Data is stored in browser localStorage by default, with backup "supernode" servers to ensure persistence. Peers connect over websockets to other peers, and choose what data to synchronize and persist. -- [ERA](https://www.youtube.com/watch?v=1HJdrBk3BlE) (economic system) -- [AXE](http://axe.eco) (routing) -- [DAM](https://gun.eco/docs/DAM) (networking) -- [RAD](https://gun.eco/docs/RAD) (storage) -- [SEA](https://gun.eco/docs/SEA) (security) -- [GUN](http://gun.eco) (data) -- [HAM](https://gun.eco/distributed/matters.html) (conflict resolution) +### Identity + +Gun's SEA module provides the capability to create a [public/private keypair](https://github.com/amark/gun/wiki/Auth) for a user. ### Network structure -Decentralized/p2p. Uses data replication via message passing. +Gun uses websockets to transfer data between peers. When a client puts something in a graph, a message is sent to the network and other clients that have subscribed to that data pick it up and update their state. Those who are offline will eventually receive those updates. -It daisy-chains, [routes](https://gun.eco/docs/Routing), & relays messages. +Gun uses a flooding algorithm for message propagation, called [DAM](https://gun.eco/docs/DAM) (Daisy-chain ad-hoc mesh-network). DAM's API allows for the swapping out of transport layers (websockets, WebRTC, etc). A proposed future [routing](https://gun.eco/docs/Routing) protocol, the [AXE protocol](https://web.stanford.edu/~nadal/A-Decentralized-Data-Synchronization-Protocol.pdf), includes a mechanism for monetization through bandwidth-based payments. -It also works over most existing transports: WebRTC, UDP multicast, TCP/HTTP -> Websocket, etc. +### Data -## Identity +Gun saves data two ways, in a client browser's localStorage, and through [RAD](https://gun.eco/docs/RAD), a storage adaptor that stores data on disk using a radix tree. Clients subscribe to the data they need from the "superpeer", a persistent node storing data on disk that should be run with most applications to ensure availability of data when other browser-based peers are offline. Superpeers are servers running node.js and persisting data on disk. Clients may serve the data they have to other clients as well. -GUN supports multi-device login that is cryptographically secure yet uses "normal" UX. (?) This technique can be extended to secure password resets (no servers). +Peers connect to graphs relevant to their application's logic. There is no universally shared graph across the network. -There is a password recovery mechanism using a 3-Friend-Authorization approach. +GunDB uses CRDTs (Conflict-free Replicated Data Types). Conflicts are handled by a [conflict resolution algorithm](https://gun.eco/docs/Conflict-Resolution-with-Guns) that uses lexical sort. Gun is eventually consistent, not strongly consistent, meaning that peers will eventually converge upon the last updated value. -Iris adds a [reputation system](https://medium.com/@mmalmi/learning-to-trust-strangers-167b652a654f) to SEA which provides the underlying cryptographic Security, Encryption, & Authorization. - -#### Data - -- Data models (graphs, key/value, document, table, relational, hyper-graphs) -- Data availability ([AP](https://gun.eco/docs/CAP-Theorem), strongly eventually consistent) - -#### Monetization - -- Business models (Open Source, donation, enterprise, + decentralized PaaS, dBaaS, SaaS, Marketplace) -- Payment (AXE bandwidth credits) +Gun focuses on mutability by not using an append-only CRDT, instead allowing insertion and deletion into the graph. [Deletion](https://stackoverflow.com/questions/37758618/how-to-delete-data-in-gun-db) works by inserting `null` at the deleted node, de-referencing it. #### Filtering -- Queries (GUN, [GraphQL](https://github.com/brysgo/graphql-gun), historical prototypes of SQL, Mango [Mongo]) - -#### Moderation - -Iris and NAB (see above link). - -- Protocol level (no) -- Server level (no) -- User level (yes) -- Content level (yes) +There is a [GraphQL](https://github.com/brysgo/graphql-gun) API for the gun p2p graph database. ### Privacy and Access Control -1. Cryptographic keys are assigned to roles, groups, or data points. -2. This information is either used to derive a shared ECDH secret to decrypt (read) -3. Or used to load collaborative multi-writer edits (signed). +Privacy and access control can be built on top of GunDB's graph by encrypting data for the intended user. Gun's encryption utilities, [SEA](https://gun.eco/docs/SEA) (Security, Encryption, and Authorization), intend to make it easier to add private space to applications. + +For access control, cryptographic keys are assigned to roles, groups, or data points. This information is either used to derive a shared ECDH secret to decrypt (read), or to load collaborative multi-writer edits (signed). + +Iris, a decentralized social network building on Gun, has a [library](https://github.com/irislib/iris-lib) for encrypted GUN channels and contact list management. ### Interoperablity @@ -63,22 +42,35 @@ Plugins, such as backup storage on centralized databases or file systems, can be ### Scalability -A user reported a [70% cost savings](https://twitter.com/cagataycali/status/1245442681251205120) on cloud bills on a project with 1M+ page loads a month. - ### Metrics - 11K+ [github](https://github.com/amark/gun) stars - 10M ~ 30M monthly [downloads](https://www.jsdelivr.com/package/npm/gun) +### Monetization + +The GUN protocol is developed by a VC-funded company. + ### Implementations -- [Internet Archive](https://news.ycombinator.com/item?id=17685682) (library meta-data) -- [HackerNoon](https://hackernoon.com/state-of-hacker-noon-2019-2020-8w1ls3axx) (annotations) -- [Meething](https://meething.space/) (Mozilla backed secure & decentralized video conferencing powered by GUN) -- Iris, Notabug, DTube, others... - [Party](https://party.lol/) - [Notabug](https://notabug.io/t/notabug/comments/59382d2a08b7d7073415b5b6ae29dfe617690d74/welcome-to-notabug) +- [HackerNoon](https://hackernoon.com/state-of-hacker-noon-2019-2020-8w1ls3axx) integrated Gun for annotations +- [Meething](https://meething.space/) is a Mozilla backed secure & decentralized video conferencing that uses GUN +- [Party](https://party.lol/), an encrypted messaging browser extension, uses Gun +- [Notabug](https://notabug.io/t/notabug/comments/59382d2a08b7d7073415b5b6ae29dfe617690d74/welcome-to-notabug), a decentralized Reddit clone, uses Gun +- [DTube](https://d.tube/) uses GunDB for messaging +- [Iris](https://irislib.github.io/), a web-of-trust based social network, is built on Gun + +### Related + +Comparisons can be made to other databases designed for p2p use cases: + +[OrbitDB](https://github.com/orbitdb/orbit-db) - see under [IPFS.md](IPFS.md) + +[HyperDB](https://github.com/mafintosh/hyperdb) - see under [hypercore.md](hypercore.md) + +[KappaDB](https://github.com/kappa-db) ### Links -[Whitepaper](https://stanford.edu/~nadal/A-Decentralized-Data-Synchronization-Protocol.pdf) +[Site](gun.eco) +[Documentation Clarification](https://github.com/amark/gun/issues/597) From c6b4053efea37980ec86b1b188147186ad81aee2 Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Tue, 9 Jun 2020 15:00:42 -0700 Subject: [PATCH 19/33] Add line about ssb --- protocols/ssb.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/protocols/ssb.md b/protocols/ssb.md index 8b36efa..fb03c7a 100644 --- a/protocols/ssb.md +++ b/protocols/ssb.md @@ -4,6 +4,8 @@ Ssb, or secure-scuttlebutt, is a distributed gossip protocol designed for social sharing. Identities are cryptographic key pairs, feeds are a signed append-only sequence of messages, and nodes use a gossip protocol to disseminate content. Feeds can be thought of as essentially personal blockchains. +Ssb is based on the idea that your social network mirrors your actual communication network, and your network peers mirror your actual peers. + Users are distributed across a few different client apps that work on desktop (Patchwork) and mobile (Manyverse, Planetary). ### Identity From 20b24f026b80f11ebfcc8b5a79941e7d9bc4f9d2 Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Tue, 9 Jun 2020 20:39:29 -0700 Subject: [PATCH 20/33] Add orbitdb --- protocols/IPFS.md | 24 ++++++++++-------------- 1 file changed, 10 insertions(+), 14 deletions(-) diff --git a/protocols/IPFS.md b/protocols/IPFS.md index afd99c2..2f22027 100644 --- a/protocols/IPFS.md +++ b/protocols/IPFS.md @@ -1,6 +1,7 @@ # IPFS ## Overview + IPFS is a content-addressed protocol for peer-to-peer hypermedia storage and distribution. It builds on top of libp2p (as the peer-to-peer transport layer) and IPLD (as the data model for content-addressed merkle dags). It’s main purpose is to add, share, find, and transfer files in a globally distributed file system without any central controller. IPFS is entirely open source, as are libp2p and IPLD - supported by an OSS community of over 4 thousand contributors. The IPFS protocol continues to evolve through the main reference implementations in go and JS, stewarded by the core implementations working group. The IPFS Alpha launched in February 2015, and the ecosystem has since grown to serve millions of monthly users through hundreds of applications including social networking platforms (Peepeth, Akasha, Peergos), content distribution networks (Dtube, Everipedia, Audius), decentralized identity solutions (Microsoft ION, 3box, ENS) and a host of other projects (Textile, Terminal, Infura, AnyType, etc). The majority of users today use IPFS as a content addressable storage layer to back their decentralized applications, using libp2p for networking and IPLD as a low-level data model. Many groups, like Textile, orbitdb, and 3box, build additional layers of tooling on top of IPFS to support a wider range of developers. @@ -17,7 +18,6 @@ Ability to adapt and evolve to adjust to new needs (future-proofing) These challenges need to be solved at the network fabric level in order to preserve a baseline of what the values of the network are. - ### Network architecture & Connectivity IPFS is fully peer-to-peer. When joining the network, nodes bootstrap off of long-lived peers or those in their local area network. A node running the IPFS protocol can opt-in to be part of the main public network and/or some other alternative network, either independently or simultaneously. Nodes joining the main public network will join the DHT as either clients (consumers) or servers (participants in providing the content routing service), or by directly connecting to nodes they’re interested in either by peer id or through subscribing to relevant pubsub channels. All nodes in the IPFS network use libp2p to arrange peer-to-peer connections, a project that is now used by other decentralized networks such as Polkadot, ETH2 and more (including Matrix, which is experimenting with it to become full p2p). @@ -26,13 +26,11 @@ IPFS is a distributed protocol, what this means is that every single node gets t Peers dial to each other using a multiaddr, a self-describing address that lets peers know a node’s prefered way to be dialed. All connections in IPFS are end-to-end encrypted and authenticated using modern cryptographic primitives (Public/Private Key Crypto). - ### Identity -IPFS is frequently used as a content addressed storage system for decentralized identity solutions (like Microsoft ION for example). This allows IPFS to flexibly serve a variety of opinionated privacy constraints and application-layer preferences around reputation, trust, and anonymity. - -At the protocol layer, IPFS peer ids are pseudonymous - and can be reset as needed for increased privacy. IPNS and provider records both rely on time-based revocation after which records expire. +IPFS is frequently used as a content addressed storage system for decentralized identity solutions (like Microsoft ION for example). This allows IPFS to flexibly serve a variety of opinionated privacy constraints and application-layer preferences around reputation, trust, and anonymity. +At the protocol layer, IPFS peer ids are pseudonymous - and can be reset as needed for increased privacy. IPNS and provider records both rely on time-based revocation after which records expire. ### Data layer @@ -42,20 +40,17 @@ IPFS doesn’t have any built-in persistence incentives, however it is compatibl Since each user is responsible for pinning their own data, deleting data just requires all the users with the data to stop hosting it. This is good for censorship-resistance (individual nodes with valuable data being blacklisted by an authoritarian regime won’t delete all copies), and also good for agreed content deletion (where all data hosts can unpin or avoid resolving content deemed bad). - ### Monetization & Business models IPFS is fully open source and free to use. Each individual in the network is responsible for persisting the data they care about by either adding their own resources (run a node) or incentivizing another group to persist their data (pay a pinning service). Therefore, the network grows in capacity as new users join. Services that rely on IPFS are incentivized to participate in the public DHT as servers to improve performance and availability of their data - and all participating nodes help with peer-to-peer data transfer and routing. A number of developers in the IPFS ecosystem are supported by companies who have raised money to build projects on top of IPFS - including Protocol Labs, Textile, Anytype, Infura, 3box, Audius, and many others. While many open source developers contribute pro bono part time, other groups are funded through grants or bounties by one or many of these organizations. - ### Curation/Discovery -IPFS generally uses a DHT for finding content in the network. Each data host advertises the data they’re storing once per day, which can be looked up by consumers through the DHT. Nodes also discover peers through their local area network, and by bootstrapping with other nodes in the network. The DHT is also used for bootstrapping pubsub channels which groups can subscribe to for topic-based updates to content they care about. - -These tools can be mixed and matched by applications developing on IPFS to create very flexible structures for curating and sharing data. A great example of this is Textile [threads](https://docs.textile.io/concepts/threads/) and [buckets](https://docs.textile.io/concepts/buckets/) - which are both higher-level structures built on IPFS. +IPFS generally uses a DHT for finding content in the network. Each data host advertises the data they’re storing once per day, which can be looked up by consumers through the DHT. Nodes also discover peers through their local area network, and by bootstrapping with other nodes in the network. The DHT is also used for bootstrapping pubsub channels which groups can subscribe to for topic-based updates to content they care about. +These tools can be mixed and matched by applications developing on IPFS to create very flexible structures for curating and sharing data. A great example of this is Textile [threads](https://docs.textile.io/concepts/threads/) and [buckets](https://docs.textile.io/concepts/buckets/) - which are both higher-level structures built on IPFS. ### Moderation & Reputation @@ -63,17 +58,18 @@ Libp2p has some low-level primitives around connection management which can be u In terms of content moderation - each IPFS node is in full control of the data it pins, and we have early designs for how to implement an autonomy-preserving content moderation system by which nodes can subscribe to denylists from entities they trust to help avoid or filter unwanted content - ### Scalability -The IPFS public network currently has hundreds of thousands of nodes, but there are also many private networks running IPFS without connecting to the main DHT. Most nodes participate as DHT clients, using the network to find desired content or propagate messages or data to other peers. +The IPFS public network currently has hundreds of thousands of nodes, but there are also many private networks running IPFS without connecting to the main DHT. Most nodes participate as DHT clients, using the network to find desired content or propagate messages or data to other peers. Other groups have built distributed search indexes over the public DHT either through incentivized curation or by introspecting public data announced to the wider network. - ### Governance The core implementations working group is responsible for reviewing and merging/rejecting internal and external contributions to the IPFS protocol, rather than through broader consensus. There is also a wider community of 4000+ OSS contributors helping improve and test IPFS. -IPFS contributors interface closely with contributors to both libp2p and IPLD, since many features or improvements require cross-cutting collaboration, however all 3 protocols are independently stewarded and have their own unique end users they optimize for. +IPFS contributors interface closely with contributors to both libp2p and IPLD, since many features or improvements require cross-cutting collaboration, however all 3 protocols are independently stewarded and have their own unique end users they optimize for. +### Related + +[OrbitDB](https://github.com/orbitdb/orbit-db) is a serverless, distributed, p2p database. It uses IPFS as its data storage, and IPFS PubSub to sync databases with peers. It uses CRDTs to resolve conflicts. From c13d3fbba13de162af6b02fb0aea03e8bd414288 Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Tue, 9 Jun 2020 20:39:45 -0700 Subject: [PATCH 21/33] Mv peergos --- {applications => protocols}/peergos.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename {applications => protocols}/peergos.md (100%) diff --git a/applications/peergos.md b/protocols/peergos.md similarity index 100% rename from applications/peergos.md rename to protocols/peergos.md From 0b22175ad45648b681e84ceb3b6ec99cdddc7dfb Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Tue, 9 Jun 2020 20:40:20 -0700 Subject: [PATCH 22/33] Add stubs --- topics/filtering.md | 5 +++++ topics/security.md | 2 ++ 2 files changed, 7 insertions(+) diff --git a/topics/filtering.md b/topics/filtering.md index e69de29..65169d7 100644 --- a/topics/filtering.md +++ b/topics/filtering.md @@ -0,0 +1,5 @@ +# Filtering + +In decentralized networks, whether federated or p2p, there is often no global search functionality. + +This issue being discussed in Mastodon: https://github.com/tootsuite/mastodon/issues/9529 diff --git a/topics/security.md b/topics/security.md index dfbd95e..614168f 100644 --- a/topics/security.md +++ b/topics/security.md @@ -7,3 +7,5 @@ Addresses key establishment, authentication, and confidentiality service [Inside MLS, the New Protocol for Secure Enterprise Messaging](https://www.darkreading.com/perimeter/inside-mls-the-new-protocol-for-secure-enterprise-messaging/d/d-id/1335075) [messaginglayersecurity.rocks](https://messaginglayersecurity.rocks/) + +Object capability security authorization system From f9545eab3ec3c47599399b35ff2c8d872d343e9e Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Tue, 2 Jun 2020 11:17:38 -0700 Subject: [PATCH 23/33] Finish Matrix overview --- protocols/matrix.md | 58 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 45 insertions(+), 13 deletions(-) diff --git a/protocols/matrix.md b/protocols/matrix.md index a83e31f..e2cdebe 100644 --- a/protocols/matrix.md +++ b/protocols/matrix.md @@ -4,42 +4,74 @@ Matrix is a protocol for replicating a signed history of JSON objects in realtim It's designed to support multiple communication cases - Chat, VoIP, IoT, VR/AR, Social etc, but so far effort has been concentrated on providing a federated chat experience with good UX at scale. The public network currently (Feb 2020) has 14.7M known addressable users, with many others in private federations or on servers which don't report stats. Matrix supports multiple clients (most notably [Riot](https://riot.im), the flagship app from the core team), and has bridges to many other chat systems (IRC, Slack, Discord, Telegram etc). -Note: In particular, we may be interested in the [Proposal for Aggregation via Relations](https://github.com/matrix-org/matrix-doc/blob/matthew/msc1849/proposals/1849-aggregations.md) +### Data Storage/Message Persistance + +Matrix messages are stored in per-conversation Merkle DAG data structures, and conversations are replicated across all participating servers. Matrix is architecturally most similar to Git. + +### Network Structure + +Matrix has a federated and a p2p version. + +Federated + +Currently, all messages are sent out full-mesh within a conversation. A node broadcasts in parallel to every other node present in the room. Experimental work on stochastic spanning tree "fan-out" approaches are being researched. + +Most users in the public federation are concentrated in a single server, although there are many smaller servers as well. + +P2p + +Matrix has released a p2p version that runs client-side over libp2p. + +https://fosdem.org/2020/schedule/event/dip_p2p_matrix/ ### Identity -Matrix has a more flexible identity solution than most decentralised protocols - users have a Matrix user ID, but can also use 3rd party IDs. A Matrix account can link to ids such as email addresses, social accounts, and phone numbers. A globally federated cluster of trusted identity servers verify and replicate the mappings, although this is a stopgap solution until a fully decentralised identity solution is adopted. +Matrix has a more flexible identity solution than most decentralised protocols - users have a Matrix user ID, but can also use 3rd party IDs. -### Data Storage/Message Persistance +A matrix user id includes the username along with the user's homeserver. For example: -JSON objects representing pieces of the conversation are replicated across nodes. +`@bob:matrix.org` + +As for 3rd party ids, a Matrix account can link to ids such as email addresses, social accounts, and phone numbers. A globally federated cluster of trusted identity servers verify and replicate the mappings, although this is considered a stopgap solution until a fully decentralised identity solution is adopted. ### Moderation/Reputation -The Matrix team has also been working intensively on tools for moderation, detailed [here](https://matrix.org/docs/guides/moderation/), and is in the final stages of releasing a [P2P variant of the protocol](https://fosdem.org/2020/schedule/event/dip_p2p_matrix/). +The Matrix team has been working intensively on tools for moderation, detailed [here](https://matrix.org/docs/guides/moderation/). + +Most moderation takes place at the room level. Rooms have moderators who can remove undesirable content. A redaction, or 'remove message', can request all participating servers to remove a message. Users can also remove their own messages. However, due to the decentralized nature of Matrix, if the client does not remove the redaction, the message will not be removed. Moderators can kick or ban users from rooms. They can also ban servers from rooms, in cases where malicious traffic must be avoided. + +Users in a room have 'power levels', a number between 0 and 100 that indicates how much power the user has in the room. A higher level has more permissions. Moderators have ability to change power level permissions. Users can report abusive content to the server admin, which has the ability to remove it. Users can also block other users to prevent them from contacting them. ### Social/Discovery +Most conversation on Matrix takes place through rooms, which people must be invited to. + +Features supporting more advanced social functionality are still under research and development, such as this proposal for tracking events related to existing events: +[Proposal for Aggregation via Relations](https://github.com/matrix-org/matrix-doc/blob/matthew/msc1849/proposals/1849-aggregations.md) + ### User experience -Currently the primary user experience is in Riot, a slack-like chat app +Currently the primary client is Riot, a Slack-like chat app. ### Privacy and Access Control +Matrix recently introduced end-to-end encryption by default for private messages. This was on the roadmap since the beginning, because conversations are replicated over every server participating in a room, and there is no guarantee against servers looking into conversations. + +[olm e2e encryption](https://matrix.org/blog/2016/11/21/matrixs-olm-end-to-end-encryption-security-assessment-released-and-implemented-cross-platform-on-riot-at-last) +[encryption for private messages](https://matrix.org/blog/2020/05/06/cross-signing-and-end-to-end-encryption-by-default-is-here) + ### Monetization and Governance -Matrix is governed by [The Matrix.org Foundation CIC](https://matrix.org/foundation) (a UK non-profit foundation), with the largest contributor being [New Vector](https://vector.im), a startup formed by the original Matrix dev team, which raised $8.5M series A in October 2019. - -? Income is primarily from contracts with governments and large clients ? +Matrix is governed by [The Matrix.org Foundation CIC](https://matrix.org/foundation) (a UK non-profit foundation), with the largest contributor being [New Vector](https://vector.im), a startup formed by the original Matrix dev team, which raised $8.5M series A in October 2019. Income primarily comes from contracts with governments and large clients. ### Interop with other systems -Bridges that allow communication with IRC, Slack, Discord, Telegram and others +Matrix can be bridged with IRC, Slack, Discord, Telegram and others. ### Scalability -### Metrics - -### Implementations +Scalability study: https://arxiv.org/pdf/1910.06295.pdf ### Links + +[matrix.org](https://matrix.org/) From 50363dad0ac9e9c9daca632ff640b9c15a4275d9 Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Tue, 9 Jun 2020 15:07:14 -0700 Subject: [PATCH 24/33] Edit --- protocols/matrix.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/protocols/matrix.md b/protocols/matrix.md index e2cdebe..eddf72f 100644 --- a/protocols/matrix.md +++ b/protocols/matrix.md @@ -2,7 +2,7 @@ Matrix is a protocol for replicating a signed history of JSON objects in realtime across a set of nodes with (optional) end-to-end encryption. -It's designed to support multiple communication cases - Chat, VoIP, IoT, VR/AR, Social etc, but so far effort has been concentrated on providing a federated chat experience with good UX at scale. The public network currently (Feb 2020) has 14.7M known addressable users, with many others in private federations or on servers which don't report stats. Matrix supports multiple clients (most notably [Riot](https://riot.im), the flagship app from the core team), and has bridges to many other chat systems (IRC, Slack, Discord, Telegram etc). +Matrix is designed to support multiple communication cases - Chat, VoIP, IoT, VR/AR, Social etc, but so far effort has been concentrated on providing a federated chat experience with good UX at scale. ### Data Storage/Message Persistance @@ -12,13 +12,13 @@ Matrix messages are stored in per-conversation Merkle DAG data structures, and c Matrix has a federated and a p2p version. -Federated +#### Federated Currently, all messages are sent out full-mesh within a conversation. A node broadcasts in parallel to every other node present in the room. Experimental work on stochastic spanning tree "fan-out" approaches are being researched. Most users in the public federation are concentrated in a single server, although there are many smaller servers as well. -P2p +#### P2p Matrix has released a p2p version that runs client-side over libp2p. @@ -26,13 +26,13 @@ https://fosdem.org/2020/schedule/event/dip_p2p_matrix/ ### Identity -Matrix has a more flexible identity solution than most decentralised protocols - users have a Matrix user ID, but can also use 3rd party IDs. +Matrix has a more flexible identity solution than most decentralized protocols - users have a Matrix user ID, but can also use 3rd party IDs. A matrix user id includes the username along with the user's homeserver. For example: `@bob:matrix.org` -As for 3rd party ids, a Matrix account can link to ids such as email addresses, social accounts, and phone numbers. A globally federated cluster of trusted identity servers verify and replicate the mappings, although this is considered a stopgap solution until a fully decentralised identity solution is adopted. +As for 3rd party ids, a Matrix account can link to ids such as email addresses, social accounts, and phone numbers. A globally federated cluster of trusted identity servers verify and replicate the mappings, although this is considered a stopgap solution until a fully decentralized identity solution is adopted. ### Moderation/Reputation @@ -49,10 +49,6 @@ Most conversation on Matrix takes place through rooms, which people must be invi Features supporting more advanced social functionality are still under research and development, such as this proposal for tracking events related to existing events: [Proposal for Aggregation via Relations](https://github.com/matrix-org/matrix-doc/blob/matthew/msc1849/proposals/1849-aggregations.md) -### User experience - -Currently the primary client is Riot, a Slack-like chat app. - ### Privacy and Access Control Matrix recently introduced end-to-end encryption by default for private messages. This was on the roadmap since the beginning, because conversations are replicated over every server participating in a room, and there is no guarantee against servers looking into conversations. @@ -68,6 +64,12 @@ Matrix is governed by [The Matrix.org Foundation CIC](https://matrix.org/foundat Matrix can be bridged with IRC, Slack, Discord, Telegram and others. +### Applications + +The public network currently (Feb 2020) has 14.7M known addressable users, with many others in private federations or on servers which don't report stats. + +Matrix supports multiple clients (most notably [Riot](https://riot.im), the flagship app from the core team), and has bridges to many other chat systems (IRC, Slack, Discord, Telegram etc). + ### Scalability Scalability study: https://arxiv.org/pdf/1910.06295.pdf From 3774ba3a52db6867fb34b8520bda0dc3ae863d0b Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Tue, 9 Jun 2020 17:01:39 -0700 Subject: [PATCH 25/33] Edit --- protocols/matrix.md | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/protocols/matrix.md b/protocols/matrix.md index eddf72f..823c8c8 100644 --- a/protocols/matrix.md +++ b/protocols/matrix.md @@ -14,25 +14,27 @@ Matrix has a federated and a p2p version. #### Federated -Currently, all messages are sent out full-mesh within a conversation. A node broadcasts in parallel to every other node present in the room. Experimental work on stochastic spanning tree "fan-out" approaches are being researched. +In the federated version of Matrix, all messages are currently sent out full-mesh within a conversation. A node broadcasts in parallel to every other node present in the room. Experimental work on stochastic spanning tree "fan-out" approaches to improve efficiency are being researched. -Most users in the public federation are concentrated in a single server, although there are many smaller servers as well. +30% of publicly visible users are on the matrix.org home server. There are many smaller servers as well. #### P2p -Matrix has released a p2p version that runs client-side over libp2p. +Matrix has released a p2p version that runs client-side. P2p Matrix avoids the problem of homeservers accumulating metadata, and simplifies signup by not requiring new users to pick a homeserver. The new p2p implementation runs the homeserver on the client. The p2p network is currently separate from the federated network, but the end goal is to connect the two in a hybrid federated/p2p model. Network transports being considered for p2p Matrix include libp2p, Yggdrasil, or hyperswarm. -https://fosdem.org/2020/schedule/event/dip_p2p_matrix/ +https://matrix.org/blog/2020/06/02/introducing-p-2-p-matrix ### Identity -Matrix has a more flexible identity solution than most decentralized protocols - users have a Matrix user ID, but can also use 3rd party IDs. +Matrix has a more flexible identity solution than many decentralized protocols - users have a Matrix user ID, but can also use 3rd party IDs. -A matrix user id includes the username along with the user's homeserver. For example: +A matrix user ID includes the username along with the user's homeserver. For example: `@bob:matrix.org` -As for 3rd party ids, a Matrix account can link to ids such as email addresses, social accounts, and phone numbers. A globally federated cluster of trusted identity servers verify and replicate the mappings, although this is considered a stopgap solution until a fully decentralized identity solution is adopted. +As for 3rd party IDs, a Matrix account can link to IDs such as email addresses, social accounts, and phone numbers. A globally federated cluster of trusted identity servers verify and replicate the mappings, although this is considered a stopgap solution until a fully decentralized identity solution is adopted. + +User IDs used in conversations will soon be decoupled from permanent IDs, allowing one to decorrelate users from their messages. ### Moderation/Reputation @@ -44,13 +46,15 @@ Users in a room have 'power levels', a number between 0 and 100 that indicates h ### Social/Discovery -Most conversation on Matrix takes place through rooms, which people must be invited to. +All conversations on Matrix take place through rooms, which people either join (if public), peek into (if viewable), or are invited to. -Features supporting more advanced social functionality are still under research and development, such as this proposal for tracking events related to existing events: +Features supporting more advanced social functionality are being developed, such as this proposal for tracking events related to existing events: [Proposal for Aggregation via Relations](https://github.com/matrix-org/matrix-doc/blob/matthew/msc1849/proposals/1849-aggregations.md) ### Privacy and Access Control +Matrix homeservers have access to metadata about conversations, because the homeservers of all users in a given conversation have to store that conversation's metadata. P2p Matrix mitigates this privacy issue. + Matrix recently introduced end-to-end encryption by default for private messages. This was on the roadmap since the beginning, because conversations are replicated over every server participating in a room, and there is no guarantee against servers looking into conversations. [olm e2e encryption](https://matrix.org/blog/2016/11/21/matrixs-olm-end-to-end-encryption-security-assessment-released-and-implemented-cross-platform-on-riot-at-last) @@ -66,7 +70,7 @@ Matrix can be bridged with IRC, Slack, Discord, Telegram and others. ### Applications -The public network currently (Feb 2020) has 14.7M known addressable users, with many others in private federations or on servers which don't report stats. +The public network currently (Feb 2020) has 17.9M known addressable users (as of June 2020), with more in private federations or on servers which don't report stats. Matrix supports multiple clients (most notably [Riot](https://riot.im), the flagship app from the core team), and has bridges to many other chat systems (IRC, Slack, Discord, Telegram etc). From 4efda5aaca14bc7caead79d2c89af86860d655dc Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Tue, 9 Jun 2020 20:43:45 -0700 Subject: [PATCH 26/33] s/home server/homeserver --- protocols/matrix.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/protocols/matrix.md b/protocols/matrix.md index 823c8c8..1893f39 100644 --- a/protocols/matrix.md +++ b/protocols/matrix.md @@ -16,7 +16,7 @@ Matrix has a federated and a p2p version. In the federated version of Matrix, all messages are currently sent out full-mesh within a conversation. A node broadcasts in parallel to every other node present in the room. Experimental work on stochastic spanning tree "fan-out" approaches to improve efficiency are being researched. -30% of publicly visible users are on the matrix.org home server. There are many smaller servers as well. +30% of publicly visible users are on the matrix.org homeserver. There are many smaller servers as well. #### P2p From a54d68d71253041bf38fbf678473f913618609e2 Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Tue, 9 Jun 2020 20:55:59 -0700 Subject: [PATCH 27/33] Rm i2p --- protocols/i2p.md | 30 ------------------------------ 1 file changed, 30 deletions(-) delete mode 100644 protocols/i2p.md diff --git a/protocols/i2p.md b/protocols/i2p.md deleted file mode 100644 index 976626d..0000000 --- a/protocols/i2p.md +++ /dev/null @@ -1,30 +0,0 @@ -# I2P - -### Identity - -### Networking/Message passing - -### Data Storage/Message Persistance - -### Moderation/Reputation - -### Social/Discovery - -### Privacy and Access Control - -### Monetization - -### User experience (if applies) - -### Interop with other systems -in particular - what is the minimum requirement for a message to enter or leave the system? - -### Scalability - -### Metrics - -### In the wild - -### Links - -https://geti2p.net/en/ \ No newline at end of file From 534c1faf31fa58a438775825b3954c3c038008fd Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Tue, 9 Jun 2020 21:58:27 -0700 Subject: [PATCH 28/33] edit --- protocols/gun.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/protocols/gun.md b/protocols/gun.md index a34be92..94d0a02 100644 --- a/protocols/gun.md +++ b/protocols/gun.md @@ -2,7 +2,7 @@ Gun is a distributed database with a graph structure. Related libraries include a constellation of tools for merging conflicting data and handling routing and security. -In GunDB's graph store, entries are [JSON objects under UUID keys](https://gun.eco/docs/Porting-GUN). Data is stored in browser localStorage by default, with backup "supernode" servers to ensure persistence. Peers connect over websockets to other peers, and choose what data to synchronize and persist. +In GunDB's graph store, entries are [JSON objects under UUID keys](https://gun.eco/docs/Porting-GUN). Data is stored in browser localStorage by default, with backup "supernode" servers to ensure persistence. Peers connect to other peers, and choose what data to synchronize and persist. ### Identity @@ -10,7 +10,7 @@ Gun's SEA module provides the capability to create a [public/private keypair](ht ### Network structure -Gun uses websockets to transfer data between peers. When a client puts something in a graph, a message is sent to the network and other clients that have subscribed to that data pick it up and update their state. Those who are offline will eventually receive those updates. +Gun transfers data between peers. When a client puts something in a graph, a message is sent to the network and other clients that have subscribed to that data pick it up and update their state. Those who are offline will eventually receive those updates. Gun uses a flooding algorithm for message propagation, called [DAM](https://gun.eco/docs/DAM) (Daisy-chain ad-hoc mesh-network). DAM's API allows for the swapping out of transport layers (websockets, WebRTC, etc). A proposed future [routing](https://gun.eco/docs/Routing) protocol, the [AXE protocol](https://web.stanford.edu/~nadal/A-Decentralized-Data-Synchronization-Protocol.pdf), includes a mechanism for monetization through bandwidth-based payments. From 27467776f6a9b39278d74f89cc7d6a4771940bec Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Wed, 10 Jun 2020 15:43:58 -0700 Subject: [PATCH 29/33] Edit --- protocols/gun.md | 47 +++++++++++++++++++++++++++-------------------- 1 file changed, 27 insertions(+), 20 deletions(-) diff --git a/protocols/gun.md b/protocols/gun.md index 94d0a02..64f5841 100644 --- a/protocols/gun.md +++ b/protocols/gun.md @@ -1,40 +1,44 @@ -# Gun +# GUN -Gun is a distributed database with a graph structure. Related libraries include a constellation of tools for merging conflicting data and handling routing and security. +GUN is a distributed database with a graph structure. It includes a library of tools for merging conflicting data and handling routing, security, and storage. -In GunDB's graph store, entries are [JSON objects under UUID keys](https://gun.eco/docs/Porting-GUN). Data is stored in browser localStorage by default, with backup "supernode" servers to ensure persistence. Peers connect to other peers, and choose what data to synchronize and persist. +In GUN's graph store, entries are [JSON objects under UUID keys](https://gun.eco/docs/Porting-GUN). + +Data is stored in the browser by default, with backup "superpeers" to ensure persistence. Peers connect to other peers, and choose what data to synchronize and persist. ### Identity -Gun's SEA module provides the capability to create a [public/private keypair](https://github.com/amark/gun/wiki/Auth) for a user. +GUN's SEA (Security, Encryption, Authorization) module provides the capability to create a [public/private keypair](https://github.com/amark/gun/wiki/Auth) for a user. ### Network structure -Gun transfers data between peers. When a client puts something in a graph, a message is sent to the network and other clients that have subscribed to that data pick it up and update their state. Those who are offline will eventually receive those updates. +[DAM](https://gun.eco/docs/DAM) (Daisy-chain ad-hoc mesh-network) is the message [routing](https://gun.eco/docs/Routing) algorithm used by GUN to sync data between peers. DAM's API allows for the swapping out of transport layers (websockets, WebRTC, multicast UDP etc). -Gun uses a flooding algorithm for message propagation, called [DAM](https://gun.eco/docs/DAM) (Daisy-chain ad-hoc mesh-network). DAM's API allows for the swapping out of transport layers (websockets, WebRTC, etc). A proposed future [routing](https://gun.eco/docs/Routing) protocol, the [AXE protocol](https://web.stanford.edu/~nadal/A-Decentralized-Data-Synchronization-Protocol.pdf), includes a mechanism for monetization through bandwidth-based payments. +When a peer puts something in a graph, a message is sent to the network and other peers that have subscribed to that data pick it up and update their state. Those who are offline will eventually receive those updates. -### Data +A proposed future evolution of the routing protocol is called [AXE](https://web.stanford.edu/~nadal/A-Decentralized-Data-Synchronization-Protocol.pdf). It cuts off unnecessary connections, includes a DHT, and has a token for monetization through bandwidth-based payments. -Gun saves data two ways, in a client browser's localStorage, and through [RAD](https://gun.eco/docs/RAD), a storage adaptor that stores data on disk using a radix tree. Clients subscribe to the data they need from the "superpeer", a persistent node storing data on disk that should be run with most applications to ensure availability of data when other browser-based peers are offline. Superpeers are servers running node.js and persisting data on disk. Clients may serve the data they have to other clients as well. +### Data Storage + +Peers subscribe to the data they need and the network retrieves it from any peer (including browsers, where GUN stores data in localStorage). Running always-online peers, such as a "superpeer", is recommended for most applications to ensure availability of data when most browser-based peers may be offline. A superpeer is an IP addressable machine running node.js that persists data to disk. [RAD](https://gun.eco/docs/RAD), GUN's storage adaptor, saves data to disk using a radix tree. Peers connect to graphs relevant to their application's logic. There is no universally shared graph across the network. -GunDB uses CRDTs (Conflict-free Replicated Data Types). Conflicts are handled by a [conflict resolution algorithm](https://gun.eco/docs/Conflict-Resolution-with-Guns) that uses lexical sort. Gun is eventually consistent, not strongly consistent, meaning that peers will eventually converge upon the last updated value. +GUN uses a CRDT (Conflict-free Replicated Data Type) to merge data. Conflicts are handled by a [conflict resolution algorithm](https://gun.eco/docs/Conflict-Resolution-with-Guns) that uses lexical sort. GUN is eventually consistent, meaning that peers will eventually converge upon the last updated value. -Gun focuses on mutability by not using an append-only CRDT, instead allowing insertion and deletion into the graph. [Deletion](https://stackoverflow.com/questions/37758618/how-to-delete-data-in-gun-db) works by inserting `null` at the deleted node, de-referencing it. +GUN focuses on mutability by not using an append-only log, which implements updates, insertions, and deletion as a layer on top. [Deletion](https://stackoverflow.com/questions/37758618/how-to-delete-data-in-gun-db) in GUN works by overwriting bytes with `null`, or by de-referencing portions of a graph. #### Filtering -There is a [GraphQL](https://github.com/brysgo/graphql-gun) API for the gun p2p graph database. +There is a [GraphQL](https://github.com/brysgo/graphql-gun) API for the gun p2p graph database. SQL and Mango (MongoDB) queries were available in the past, but deprecated. ### Privacy and Access Control -Privacy and access control can be built on top of GunDB's graph by encrypting data for the intended user. Gun's encryption utilities, [SEA](https://gun.eco/docs/SEA) (Security, Encryption, and Authorization), intend to make it easier to add private space to applications. +Privacy and access control can be built on top of the GUN database graph by encrypting data for the intended user. GUN's encryption utilities, [SEA](https://gun.eco/docs/SEA) (Security, Encryption, and Authorization), intend to make it easier to add private space to applications. -For access control, cryptographic keys are assigned to roles, groups, or data points. This information is either used to derive a shared ECDH secret to decrypt (read), or to load collaborative multi-writer edits (signed). +Cryptographic keypairs are assigned to roles, groups, or data points. This information is either used to derive a shared ECDH secret to decrypt (read), or to load collaborative multi-writer edits (signed). -Iris, a decentralized social network building on Gun, has a [library](https://github.com/irislib/iris-lib) for encrypted GUN channels and contact list management. +Iris, a decentralized social network building on GUN, has a [library](https://github.com/irislib/iris-lib) for end-to-end encrypted chat channels and private contact list management. ### Interoperablity @@ -49,16 +53,19 @@ Plugins, such as backup storage on centralized databases or file systems, can be ### Monetization -The GUN protocol is developed by a VC-funded company. +The GUN protocol is developed by a [VC-funded company](https://era.eco/#step1), which funds the development of Iris as well. Business models include consulting and integrations. Future business models include a proposed paid service on AXE, a blockchain-based tokenized bandwidth incentive network. ### Implementations -- [HackerNoon](https://hackernoon.com/state-of-hacker-noon-2019-2020-8w1ls3axx) integrated Gun for annotations +GUN is used for p2p chat/social apps, encrypted video conferencing, realtime GPS tracking, and AR/VR multiplayer games, among other applications. + +- [Internet Archive](https://news.ycombinator.com/item?id=17685682) uses GUN for their [dWeb library](https://github.com/internetarchive/dweb-transports) metadata +- [HackerNoon](https://hackernoon.com/state-of-hacker-noon-2019-2020-8w1ls3axx) integrated GUN for annotations - [Meething](https://meething.space/) is a Mozilla backed secure & decentralized video conferencing that uses GUN -- [Party](https://party.lol/), an encrypted messaging browser extension, uses Gun -- [Notabug](https://notabug.io/t/notabug/comments/59382d2a08b7d7073415b5b6ae29dfe617690d74/welcome-to-notabug), a decentralized Reddit clone, uses Gun -- [DTube](https://d.tube/) uses GunDB for messaging -- [Iris](https://irislib.github.io/), a web-of-trust based social network, is built on Gun +- [Party](https://party.lol/), an encrypted messaging browser extension, uses GUN +- [Notabug](https://notabug.io/t/notabug/comments/59382d2a08b7d7073415b5b6ae29dfe617690d74/welcome-to-notabug), a decentralized Reddit clone, uses GUN +- [DTube](https://d.tube/) uses GUN for messaging +- [Iris](https://irislib.github.io/), is a web-of-trust based social network built on GUN ### Related From 4b9a5a35659dd6b7859bf832f8c117de4a38acf0 Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Wed, 10 Jun 2020 17:35:45 -0700 Subject: [PATCH 30/33] Edits --- protocols/gun.md | 46 +++++++++++++++++++++------------------------- 1 file changed, 21 insertions(+), 25 deletions(-) diff --git a/protocols/gun.md b/protocols/gun.md index 64f5841..14cd57e 100644 --- a/protocols/gun.md +++ b/protocols/gun.md @@ -1,32 +1,36 @@ # GUN -GUN is a distributed database with a graph structure. It includes a library of tools for merging conflicting data and handling routing, security, and storage. +GUN is a decentralized graph database with a conflict resolution algorithm (CRDT) and synchronization protocol. It includes a library of tools for merging conflicting data and handling routing, security, and storage. -In GUN's graph store, entries are [JSON objects under UUID keys](https://gun.eco/docs/Porting-GUN). +In GUN's graph store, entries are [javascript objects under UUID keys](https://gun.eco/docs/Porting-GUN). Objects can be data of any type, including key-value, files, JSON, documents, tables, relational, and graph or hyper-graph data. + +There is a public space and a user space. In the public space are all graphs without a public key as their ID. In the user space, graphs are signed with the user's keys, and their IDs must include the user's public key. Data is stored in the browser by default, with backup "superpeers" to ensure persistence. Peers connect to other peers, and choose what data to synchronize and persist. ### Identity -GUN's SEA (Security, Encryption, Authorization) module provides the capability to create a [public/private keypair](https://github.com/amark/gun/wiki/Auth) for a user. +Gun's [User System](https://gun.eco/docs/Auth) creates a username and password. Usernames are global but not unique. + +[Multi-device login](https://gun.eco/docs/Auth) is handled by encrypting a user's crytographic keypair, which is stored in the GUN graph. Keypairs are not derived from the password. PBDKF proof is derived from the password, and AES is derived from that to encrypt the keypair. GUN treats this method as "secure enough" for applications in which private keys do not control financial information. "Auth" is doing a GUN query for that account, subscribing to it, and then attempts to brute force decrypt the keys of all accounts that match that username. Once loaded once, it's cached on that device, loading from localstorage or local harddrive. + +GUN's SEA (Security, Encryption, Authorization) module provides the capability to directly create a [public/private keypair](https://gun.eco/docs/SEA) for a user, without a username and account. ### Network structure -[DAM](https://gun.eco/docs/DAM) (Daisy-chain ad-hoc mesh-network) is the message [routing](https://gun.eco/docs/Routing) algorithm used by GUN to sync data between peers. DAM's API allows for the swapping out of transport layers (websockets, WebRTC, multicast UDP etc). +GUN uses a gossip protocol along with a topic-based PubSub protocol to sync data between peers. GUN peers fall back to the [gossip-based protocol](https://gun.eco/docs/DAM) when the more optimized PubSub [routing](https://gun.eco/docs/Routing) protocol fails. Messages are routed across different transport layers (websockets, WebRTC, multicast UDP etc). -When a peer puts something in a graph, a message is sent to the network and other peers that have subscribed to that data pick it up and update their state. Those who are offline will eventually receive those updates. +Peers subscribe to graphs relevant to their application's logic, although the global GUN graph is accessible to all peers. -A proposed future evolution of the routing protocol is called [AXE](https://web.stanford.edu/~nadal/A-Decentralized-Data-Synchronization-Protocol.pdf). It cuts off unnecessary connections, includes a DHT, and has a token for monetization through bandwidth-based payments. +Planned future network upgrades include the addition of a DHT. A [tokenized incentivized mesh proposal](https://web.stanford.edu/~nadal/A-Decentralized-Data-Synchronization-Protocol.pdf) is also on the roadmap. ### Data Storage Peers subscribe to the data they need and the network retrieves it from any peer (including browsers, where GUN stores data in localStorage). Running always-online peers, such as a "superpeer", is recommended for most applications to ensure availability of data when most browser-based peers may be offline. A superpeer is an IP addressable machine running node.js that persists data to disk. [RAD](https://gun.eco/docs/RAD), GUN's storage adaptor, saves data to disk using a radix tree. -Peers connect to graphs relevant to their application's logic. There is no universally shared graph across the network. +GUN uses a CRDT (Conflict-free Replicated Data Type) to merge data. Conflicts are handled by a [conflict resolution algorithm](https://gun.eco/docs/Conflict-Resolution-with-Guns) that uses lexical sort. GUN is [strongly eventually consistent](https://pages.lip6.fr/Marc.Shapiro/slides/CRDTs%20Google%20Zurich-2011-09.pdf), meaning that peers will eventually converge upon the last updated value when nodes that are offline eventually receive updates. -GUN uses a CRDT (Conflict-free Replicated Data Type) to merge data. Conflicts are handled by a [conflict resolution algorithm](https://gun.eco/docs/Conflict-Resolution-with-Guns) that uses lexical sort. GUN is eventually consistent, meaning that peers will eventually converge upon the last updated value. - -GUN focuses on mutability by not using an append-only log, which implements updates, insertions, and deletion as a layer on top. [Deletion](https://stackoverflow.com/questions/37758618/how-to-delete-data-in-gun-db) in GUN works by overwriting bytes with `null`, or by de-referencing portions of a graph. +GUN focuses on mutability by not using an append-only log, which implements updates, insertions, and deletion as a layer on top. [Deletion](https://stackoverflow.com/questions/37758618/how-to-delete-data-in-gun-db) in GUN works by overwriting bytes with `null`, or by de-referencing portions of a graph. A content-addressed graph space is used to implement immutable, append-only data. #### Filtering @@ -34,11 +38,11 @@ There is a [GraphQL](https://github.com/brysgo/graphql-gun) API for the gun p2p ### Privacy and Access Control -Privacy and access control can be built on top of the GUN database graph by encrypting data for the intended user. GUN's encryption utilities, [SEA](https://gun.eco/docs/SEA) (Security, Encryption, and Authorization), intend to make it easier to add private space to applications. +Access control is built into the [User system](https://gun.eco/docs/Auth) and can be combined with [SEA](https://gun.eco/docs/SEA), GUN's encryption utilities, for more advanced use cases. Cryptographic keypairs are assigned to roles, groups, or data points. This information is either used to derive a shared ECDH secret to decrypt (read), or to load collaborative multi-writer edits (signed). -Iris, a decentralized social network building on GUN, has a [library](https://github.com/irislib/iris-lib) for end-to-end encrypted chat channels and private contact list management. +[Iris-lib](https://github.com/irislib/iris-lib) provides an API for end-to-end encrypted chat channels and private contact list management. ### Interoperablity @@ -46,6 +50,8 @@ Plugins, such as backup storage on centralized databases or file systems, can be ### Scalability +Test relays (superpeers) on GUN can handle about 10k simultaneous connections: http://guntest.herokuapp.com/stats.html + ### Metrics - 11K+ [github](https://github.com/amark/gun) stars @@ -53,7 +59,7 @@ Plugins, such as backup storage on centralized databases or file systems, can be ### Monetization -The GUN protocol is developed by a [VC-funded company](https://era.eco/#step1), which funds the development of Iris as well. Business models include consulting and integrations. Future business models include a proposed paid service on AXE, a blockchain-based tokenized bandwidth incentive network. +The GUN protocol is developed by a [VC-funded company](https://era.eco/#step1), which funds the development of Iris as well. The business model is based on consulting and integrations. Future business models include a proposed paid service through a blockchain-based [tokenized bandwidth incentive network](https://web.stanford.edu/~nadal/A-Decentralized-Data-Synchronization-Protocol.pdf). ### Implementations @@ -62,21 +68,11 @@ GUN is used for p2p chat/social apps, encrypted video conferencing, realtime GPS - [Internet Archive](https://news.ycombinator.com/item?id=17685682) uses GUN for their [dWeb library](https://github.com/internetarchive/dweb-transports) metadata - [HackerNoon](https://hackernoon.com/state-of-hacker-noon-2019-2020-8w1ls3axx) integrated GUN for annotations - [Meething](https://meething.space/) is a Mozilla backed secure & decentralized video conferencing that uses GUN -- [Party](https://party.lol/), an encrypted messaging browser extension, uses GUN +- [Party](https://party.lol/) and [Maskbook](https://maskbook.com/), encrypted browser extensions, use GUN - [Notabug](https://notabug.io/t/notabug/comments/59382d2a08b7d7073415b5b6ae29dfe617690d74/welcome-to-notabug), a decentralized Reddit clone, uses GUN -- [DTube](https://d.tube/) uses GUN for messaging +- [Unstoppable Domains](https://unstoppabledomains.com/chat) and [DTube](https://d.tube/) use GUN for messaging - [Iris](https://irislib.github.io/), is a web-of-trust based social network built on GUN -### Related - -Comparisons can be made to other databases designed for p2p use cases: - -[OrbitDB](https://github.com/orbitdb/orbit-db) - see under [IPFS.md](IPFS.md) - -[HyperDB](https://github.com/mafintosh/hyperdb) - see under [hypercore.md](hypercore.md) - -[KappaDB](https://github.com/kappa-db) - ### Links [Site](gun.eco) From 52eba2a8b0f548c0d57df13cfafaabc6eb31ddc5 Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Wed, 10 Jun 2020 17:38:00 -0700 Subject: [PATCH 31/33] Edit --- protocols/gun.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/protocols/gun.md b/protocols/gun.md index 14cd57e..f087f9b 100644 --- a/protocols/gun.md +++ b/protocols/gun.md @@ -2,12 +2,10 @@ GUN is a decentralized graph database with a conflict resolution algorithm (CRDT) and synchronization protocol. It includes a library of tools for merging conflicting data and handling routing, security, and storage. -In GUN's graph store, entries are [javascript objects under UUID keys](https://gun.eco/docs/Porting-GUN). Objects can be data of any type, including key-value, files, JSON, documents, tables, relational, and graph or hyper-graph data. +In GUN's graph store, entries are [javascript objects under UUID keys](https://gun.eco/docs/Porting-GUN). Objects can be data of any type, including key-value, files, JSON, documents, tables, relational, and graph or hyper-graph data. Data is stored in the browser by default, with backup "superpeers" to ensure persistence. Peers connect to other peers, and choose what data to synchronize and persist. There is a public space and a user space. In the public space are all graphs without a public key as their ID. In the user space, graphs are signed with the user's keys, and their IDs must include the user's public key. -Data is stored in the browser by default, with backup "superpeers" to ensure persistence. Peers connect to other peers, and choose what data to synchronize and persist. - ### Identity Gun's [User System](https://gun.eco/docs/Auth) creates a username and password. Usernames are global but not unique. From 2aa205850b4b8ebc04315383c0fcc8e4c91cb377 Mon Sep 17 00:00:00 2001 From: Jay Graber Date: Wed, 10 Jun 2020 17:40:56 -0700 Subject: [PATCH 32/33] Rm link --- protocols/gun.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/protocols/gun.md b/protocols/gun.md index f087f9b..f2ffe7a 100644 --- a/protocols/gun.md +++ b/protocols/gun.md @@ -73,5 +73,4 @@ GUN is used for p2p chat/social apps, encrypted video conferencing, realtime GPS ### Links -[Site](gun.eco) -[Documentation Clarification](https://github.com/amark/gun/issues/597) +- [Site](gun.eco) From eb350b6d8d531fd849aa1c32244c12575a0aed05 Mon Sep 17 00:00:00 2001 From: Golda Velez Date: Thu, 11 Jun 2020 04:38:33 +0000 Subject: [PATCH 33/33] Update governance.md (WIP) --- topics/governance.md | 55 ++++++++++++++++++++++++++++++++++++++++++++ topics/identity.md | 44 +++++++++++++++++++++++++++++++++++ 2 files changed, 99 insertions(+) diff --git a/topics/governance.md b/topics/governance.md index e69de29..e0dbf6d 100644 --- a/topics/governance.md +++ b/topics/governance.md @@ -0,0 +1,55 @@ +# Decentralized Governance + +May sound like an oxymoron - but in order for the decentralized web to work, decisions have to be made about protocols, moderation, design and many other topics. + +The purpose of this doc is to overview how existing projects govern themselves, and how decisions and compromises have been made in the past. + +### SocialWeb +Negotiations between numerous parties ending with the design of ActivityPub. Consensus based? + +### Mastodon +Federated system. Each server admin can create their own moderation rules as well as a theme for their server, publish their TOS which may include rules about whether data can leave the server, etc. Users choose which server to join. + +No formal project-wide code of conduct/bylaws, more work talks + +### Matrix +Uses a vote of confidence in a nonprofit foundation to periodically review the leaders. In the interim functions similar to a standard nonprofit with leadership making decisions? + +Servers in Matrix also have terms of use that users agree to when they join. Rooms may have moderators who can remove people from individual rooms. + +### Links and more examples +(some are from the discussions at Mastodon on whether to have a formal structure) + +[How the Node.js Foundation Utilizes Participatory Governance to Build Its Community] (https://thenewstack.io/node-js-foundation-utilizing-participatory-governance-models/) - more about opening commit rights to the repo earlier and more quickly? + +[Debian Constitution](https://www.debian.org/devel/constitution) + +[Roads and Bridges](https://www.fordfoundation.org/work/learning/research-reports/roads-and-bridges-the-unseen-labor-behind-our-digital-infrastructure/) <- that link is the summary, it is a full eBook + +[Apache Foundation Governance](https://www.apache.org/foundation/governance/) +"The Apache Way - merit, consensus, community, charity" +Similar to a standard corporation. + +[Wikipedia original statement of principles](https://en.wikipedia.org/w/index.php?oldid=409315229) +"Wikipedia's success to date is 100% a function of our open community...Newcomers are always to be welcomed." + +### Blockchain governance +(blockchain, because of the inherent ability to count and vote, has a lot of possible governance solutions; most blockchain projects do not actually use blockchain for governance however) + +Dash and Decred are two that may be of interest + +[Decentralized Network Governance: Blockchain Technology and the Future of Regulation](https://www.frontiersin.org/articles/10.3389/fbloc.2020.00012/full) + +concepts: DAOs (decentralized autonomous organizations) + +[Bitcoin and the Rise of Decentralized Autonomous Organizations](https://www.researchgate.net/publication/326816667_Bitcoin_and_the_Rise_of_Decentralized_Autonomous_Organizations) + + +### Of possible interest + +[SlicingPie - a 'perfectly fair' equity split](https://slicingpie.com/) + +[Gitcoin](https://gitcoin.co/) - bounties for git issues - could include decision making equity + +[Code is Law](https://www.harvardmagazine.com/2000/01/code-is-law-html) - 2000 essay by Lawrence Lessig, not directly applicable but has some important concepts +"Information from Bosnia or East Timor can flow freely to the world because the Net makes it hard for governments in those countries to control how information flows....We should interrogate the architecture of cyberspace as we interrogate the code of Congress....Unless we do, or unless we learn how, the relevance of our constitutional tradition will fade. The importance of our commitment to fundamental values, through a self-consciously enacted constitution, will fade." \ No newline at end of file diff --git a/topics/identity.md b/topics/identity.md index e69de29..24dcee9 100644 --- a/topics/identity.md +++ b/topics/identity.md @@ -0,0 +1,44 @@ +# Identity + +(trace online accounts across various kinds of contiunity) + +## Use Cases/Motivations + +Why do we care about identity anyway? + + - ability to control access to private data and over an account (ie need to identify myself, this is me, this is my account) + - ability to communicate with a person across platforms, ie 'send this message to X on whatever platform he is on' (ie need to find another person by their identity as a person) + - ability to track reputation and credibility of a user across platforms + - ability to identify things other than humans, ie accounts, corporations, organizations, documents + +## Decentralized identity + + - [DIDs or Decentralized Identifiers are a W3C standard](https://www.w3.org/TR/did-core/) + Simple example: `did:example:123456789abcdefghi` + ^^ scheme identifier + ^^ did method + ^^ method-specific identifier + + +## Pseudonymity + + - reputation requires a long term handle of some sort + +## Key management + +## Key verification + +## Key backup + +## Reputation, Trust + +## Social graph proof + - Handshake problem and collusion protection (see IRIS) + +## Account recovery + +## Privacy + +## Links + +[What are Decentralized Identifiers](https://www.evernym.com/blog/what-are-decentralized-identifiers-dids/)