I’m always so fascinated by the peer-to-peer (or p2p) networks that power the various blockchain ecosystems. Unknown nodes come together, communicate, and push enough data to decide on a global state.
This was part of the reason that I started diving into the world of consensus for my phd/research. In future, I’m keen to dive into the worlds of consensus! So keep your eyes posted.
But for now, let’s start with the networking - so that we can send messages to each other.
Peer-to-Peer (P2P) Networks
Peer-to-peer networks are formed when nodes communicate directly with each other forming a mesh-like structure, rather than communicating, or relaying, through a central or designated server. Usually this means that messages are disseminated in a gossip-style pattern, where nodes send information to their directly connected peers, which is then relayed to that peer’s connected peers (it can be thought of spreading in the same way an epidemic spreads). There are different strategies and patterns that can be used to make the network as optimal as possible, which we will cover below.
When we think of the scale of blockchains, the p2p nature of the network makes sense. If we look at the larger blockchain networks, such as Bitcoin and Ethereum, they have an estimated 5k-15k nodes active in the network. If a dedicated central server was responsible for the message and data dissemination, it would (1) be under high load and have high bandwidth + processing requirements, or (2) could be the possible target for a denial of service attack if it were to go down.
P2P networks have been used in a number of different scenarios, from BitTorrent to games to chat and video applications. The benefit in the blockchain, is that it provides coverage that has the ability to handle the churn and bandwidth that is observed in blockchain networks.
Joining the network
When it comes to joining the P2P network, it gets interesting, and there are a number of challenges that need to be considered. But first off, how do we join?
Most of the blockchain networks use dedicated sets of seed/boot nodes whose entire role is to provide connections into the network. When a node first starts up, it contacts the boot node, which then responds with a set of addresses for the original node to try connecting to. Each time a node connects to another node, they can exchange messages to swap the sets of peers they have seen, so that they can each build up more of a network of nodes to connect to. Eventually, they will have enough connections to get information - and will actively maintain connections (by disconnecting, reconnecting, and connecting to new nodes) by keeping a list and ensuring a healthy connection.
Discovery
The process of finding other peers and learning more about the network is usually called discovery, and it has subtle differences for each implementation. The general idea is that you go through three steps, repeating step 2 and 3:
- Reach out to an initial node/set of nodes that are known for the network. These nodes are the ‘boot/seed’ nodes that have knowledge of other nodes on the network and point you to a list of peers to connect to.
- From that list of peers, try to establish a connection with each node (as required), and perform the protocol. Part of that protocol, if in the discovery phase, will be to exchange a list of peers to connect to, this will provide more nodes to connect to in order for you to form your own network of peers.
- Continue going through the list of peers until you have connected to sufficiently enough peers. A list of peers should be maintained in the case that one drops out or is providing poor data – then the cycle will continue back to step 2.
There are a few techniques used; but let’s go through the main ones:
- Kademlia (paper): a distributed hash table that buckets peers based on their distance. In Ethereum, and others, kademlia is used for the bucketing of peers and the discovery, not the hash table for lookups. DHTs are very common in bittorent-style protocols.
- PEER messages: connetcting with each peer, asking for a set of peers that are known, and continuing down to finding potential connections.
- Advertised Peers: a list of peers is well-known or advertised so that newcomers can join the network directly connecting with the known peers (and trying a randomised peer selection, etc.).
(There are many more, but this just scratches the surface).
Types of networks
You’ve probably heard about different blockchain network types: ‘permissionless’, ‘permissioned’, ‘consortium’, ‘private’, etc. These dictate how (and who) nodes are able to join the network, and can be referred to as the openness of the network.
Permissionless: aligning with the original, unstructured Bitcoin model – any node can join the network by querying the bootnode and participating in the protocol. They can mine, they can relay messages, they can send transactions. Essentially an open network that allows anyone to join and run the blockchain.
Permissioned or Private: a private network that requires registration and has strict protocols in place to joining the network. These are often used in closed, business-focused, or banking-focused blockchain networks where they operate a small set of private nodes. They often have a fixed set of nodes that perform the consensus operations.
Consortium: A (mainly private) network for a geo-distributed blockchain amongst multiple parties that do not want interact on the public blockchain. In the past, this has been the go-to for banks testing settlement between each other using blockchain infrastructure, as it offers a blockchain for all participants in a closed environment. It also requires registration and acceptance into the network.
Hybrid-style: There are other types, hybrids of permissioned/permissionless where nodes can join the network and not participate in the block building, or are only relaying transactions as RPC endpoints, or, have rotating sets of block buidlers but require a form of kyc/registration to do so.
Why is P2P a good reason for blockchains?
Blockchain, in its nature, is a massively distributed network of nodes that are working together. For decentralization, this means that anyone, at any time, is able to join in the network and participate. It also means that nodes will get a copy of the messages that are being passed around, which (depending on how they are operating) allows them to verify and process as they wish.
Imagine if blockchains weren’t as decentralized. If a single machine, or small subset of machines, were the critical points of keeping the network alive - how would that work? If all messages are relayed to others through these central points? That would be (1) a big burden on those machines, and would need hubs like the large data-centres to facilitate the communication, and (2) a potential point of failure for the
It’s also important to think about the messaging patterns used within the p2p network. For all-to-all communication, it could become overly bandwidth-hungry as everyone is receiving and relaying messages to everyone. With a gossip-style messaging system, everyone only needs to connect to a few others, and eventually the message will bubble through.
Types of Nodes
Mentioned before, it really depends on what type of node you’re running to how much data you need to process.
- Full Nodes: The full nodes operate the main clients, participate in the gossiping of blocks, transactions, (etc.) and hold copies of the data. These nodes usually prune their state to keep recency, and will be able to retrieve any block data from the underlying database.
- Full nodes can either be relaying (see relay nodes), or, consensus participating nodes - such as miners/validators/etc.
- Archive Nodes: Full nodes that store all the information without pruning unused, or older, data. These archive nodes usually also keep all data (from tracing, to full block data) so that they can be collected by anyone.
- Relay Nodes: Nodes in the network that act as relayers - they might not be participating in the consensus, but they are there to send messages across the network.
- RPC Nodes: Becoming much more common lately, these RPC nodes provide RPC endpoints for other (client/wallet-like) applications to connect and send data through - acting as a hub to send data. These can either be self-hosted, or, provided as part of a service.
- Light Clients: A light node that communicates on the gossip layer, but only storing the highest abstracted data - being able to verify and process the chain, but requesting for full block information when necessary. Often they work with block headers to form their chain, as they can validate the necessary information to know that a chain is valid.
- Note: these are sometimes called Light nodes or (in some networks) SPV nodes.
The open problems
The underlying peer-to-peer network that forms the backbone of the blockchain. That does, however, mean that there’s a lot of data going through a lot of nodes. The nodes on the network are heterogeneous - meaning they’re going to have vastly different capabilities of bandwidth and processing capacity.
- How can we better optimise the data that’s being spread through the network?
- Aggregation?
- Push-based (sending messages) vs Pull-based (requesting messages)
- Sending proofs vs entire datasets.
- Fighting duplicates?
I’ve been very interested in the data produced by ProbeLab, showing the health of various aspects in the p2p networks. It also gives us some good ideas in to what we need to look for.
The network problems
There are a few problems that protocols should understand, especially around peer management, but that’s quite a big deep dive too. ;)
- Eclipsing: When a node connects to the system, it is given new nodes to connect to. It will also have a numer of nodes that connect to it. If a node goes offline or is unable to connect to peers, it will build a new set of directly connected peers. If those peers are “sybils” (one actor imitating multiple personas), then the node will believe that it is receiving information from the whole network, but in reality it’s only getting information from one person, and can be censored (or controlled).
- Peer Reputation: Peers come from diverse areas, they could be extremely geographically close, or they could be remote nodes in a rural area. As well as this, there may be slight delays, or peers that are acting nefariously. There must be a way to have reptuation between peers, to let others know “hey this peer isn’t giving me the correct information”.
- Delays: many of the blockchain consensus expect messages to come through within some delta (either known or unknown) - as they’re synchronous/partially-synchronous. When long delays are experienced, then multiple blocks could be proposed at the same height, or we could have a split horizon, or make way for double spends.
The future of P2P networks?
I am extremely excited for what the future holds, and I can see a lot of my time moving towards how we can optimise for low-bandwidth and various strategies we can find to improve.