本文说明 CometBFT Peers 如何被标识,以及它们如何彼此建立连接。

节点身份

CometBFT 对等节点应以公钥的形式维护长期持久的身份。 每个节点都有一个 ID,定义为 peer.ID == peer.PubKey.Address(),其中 Address 使用 crypto 包中定义的方案。 单个节点 ID 可以关联多个 IP 地址,但一个节点在任意时刻只会连接其中一个。 尝试连接某个节点时,我们使用 PeerURL:<ID>@<IP>:<PORT>。 我们会尝试连接该节点的 IP:PORT,并通过认证加密验证其持有与 <ID> 对应的私钥。 这可以防止对等层上的中间人攻击。

连接

所有 p2p 连接都使用 TCP。 与节点成功建立 TCP 连接后,会执行两次握手:一次用于认证加密,另一次用于 CometBFT 版本协商。 这两个握手都有可配置的超时时间(通常应快速完成)。

认证加密握手

CometBFT 使用 X25519 密钥进行 Diffie-Hellman 密钥交换,并使用 chacha20poly1305 进行加密,从而实现 Station-to-Station 协议。 该协议的早期版本(0.32 及以下)存在可塑性攻击问题,活跃的中间人攻击者可能破坏机密性,具体见 Prime, Order Please! Revisiting Small Subgroup and Invalid Curve Attacks on Protocols using Diffie-Hellman。 我们引入了 Merlin 这一基于 keccak 的 transcript 哈希协议依赖,以确保不可塑性。 流程如下:
  • 生成一组临时 X25519 密钥对
  • 将临时公钥发送给对端
  • 等待接收对端的临时公钥
  • 使用字符串 TENDERMINT_SECRET_CONNECTION_TRANSCRIPT_HASH 创建一个新的 Merlin Transcript
  • 对临时密钥进行排序,并将较大的那个以标签 EPHEMERAL_UPPER_PUBLIC_KEY 加入 Merlin transcript,将较小的那个以标签 EPHEMERAL_LOWER_PUBLIC_KEY 加入 Merlin transcript。
  • 使用对端的临时公钥和本地的临时私钥计算 Diffie-Hellman 共享密钥
  • 将带有标签 DH_SECRET 的 DH 密钥加入 transcript。
  • 按如下方式生成两个用于加密的密钥(发送和接收)以及一个用于认证的 challenge:
    • 创建一个 hkdf-sha256 实例,其中 key 为 Diffie-Hellman 共享密钥,info 参数为 TENDERMINT_SECRET_CONNECTION_KEY_AND_CHALLENGE_GEN
    • 从 hkdf-sha256 获取 64 字节输出
    • 如果我们持有较小的临时公钥,则前 32 字节用作接收密钥,后 32 字节用作发送密钥;否则相反。
  • 接收和发送分别使用独立的 nonce。两个 nonce 都从 0 开始,并应支持完整的 96 位 nonce 范围
  • 从现在开始,所有通信都按 1400 字节帧(外加编码开销)进行加密,使用各自对应的密钥和 nonce。每次使用后,nonce 都递增 1。
  • 至此我们已有加密通道,但仍需完成身份认证
  • 从 merlin transcript 中提取一个带标签 SECRET_CONNECTION_MAC 的 32 字节 challenge
  • 使用持久私钥对从 hkdf 获取的公共 challenge 进行签名
  • 将 amino 编码后的持久公钥和签名发送给对端
  • 等待接收对端的持久公钥和签名
  • 使用对端的持久公钥验证其对 challenge 的签名
如果这是一个出站连接(即我们主动拨号对端),并且我们使用了某个节点 ID, 那么最后还要验证对端的持久公钥是否与我们拨号使用的节点 ID 对应, 即 peer.PubKey.Address() == <ID>。 此时连接已完成认证,所有流量都已加密。 注意:只有拨号方能够认证对端的身份, 但这正是我们关心的点,因为加入网络时我们希望确保自己连接到的是目标节点(而不是遭遇 MITM)。

Peer 过滤

继续之前,我们会检查新节点的 ID 是否与我们自己或某个已存在节点相同。 如果相同,就会断开连接。 我们还会根据一个可选白名单检查节点的地址和公钥,该白名单可通过 ABCI 应用管理。 如果启用了白名单而该节点不符合条件,连接将被终止。

CometBFT 版本握手

CometBFT 版本握手允许节点之间交换各自的 NodeInfo:
type NodeInfo struct {
  Version    p2p.Version
  ID         p2p.ID
  ListenAddr string

  Network    string
  SoftwareVersion    string
  Channels   []int8

  Moniker    string
  Other      NodeInfoOther
}

type Version struct {
 P2P uint64
 Block uint64
 App uint64
}

type NodeInfoOther struct {
 TxIndex          string
 RPCAddress       string
}
在以下情况下会断开连接:
  • peer.NodeInfo.ID 不等于 peerConn.ID
  • peer.NodeInfo.Version.Block 与我们的不匹配
  • peer.NodeInfo.Network 与我们的不相同
  • peer.Channels 与我们已知的 Channels 没有交集
  • peer.NodeInfo.ListenAddr 格式错误,或者其 DNS 主机名无法解析
到这一步,如果我们尚未断开连接,则该节点有效。 它会通过 AddPeer 方法加入 switch,并因此加入所有 reactors。 注意,每个 reactor 都可能处理多个通道。

连接活动

节点被加入后,某个 reactor 的入站消息会通过该 reactor 的 Receive 方法处理,而出站消息则由各个 peer 上的 Reactors 直接发送。 典型的 reactor 会为每个 peer 维护一个或多个处理此事的 go-routine。
This document explains how CometBFT Peers are identified and how they connect to one another.

Peer Identity

CometBFT peers are expected to maintain long-term persistent identities in the form of a public key. Each peer has an ID defined as peer.ID == peer.PubKey.Address(), where Address uses the scheme defined in crypto package. A single peer ID can have multiple IP addresses associated with it, but a node will only ever connect to one at a time. When attempting to connect to a peer, we use the PeerURL: <ID>@<IP>:<PORT>. We will attempt to connect to the peer at IP:PORT, and verify, via authenticated encryption, that it is in possession of the private key corresponding to <ID>. This prevents man-in-the-middle attacks on the peer layer.

Connections

All p2p connections use TCP. Upon establishing a successful TCP connection with a peer, two handshakes are performed: one for authenticated encryption, and one for CometBFT versioning. Both handshakes have configurable timeouts (they should complete quickly).

Authenticated Encryption Handshake

CometBFT implements the Station-to-Station protocol using X25519 keys for Diffie-Helman key-exchange and chacha20poly1305 for encryption. Previous versions of this protocol (0.32 and below) suffered from malleability attacks whereas an active man in the middle attacker could compromise confidentiality as described in Prime, Order Please! Revisiting Small Subgroup and Invalid Curve Attacks on Protocols using Diffie-Hellman. We have added dependency on the Merlin a keccak based transcript hashing protocol to ensure non-malleability. It goes as follows:
  • generate an ephemeral X25519 keypair
  • send the ephemeral public key to the peer
  • wait to receive the peer’s ephemeral public key
  • create a new Merlin Transcript with the string “TENDERMINT_SECRET_CONNECTION_TRANSCRIPT_HASH”
  • Sort the ephemeral keys and add the high labeled “EPHEMERAL_UPPER_PUBLIC_KEY” and the low keys labeled “EPHEMERAL_LOWER_PUBLIC_KEY” to the Merlin transcript.
  • compute the Diffie-Hellman shared secret using the peers ephemeral public key and our ephemeral private key
  • add the DH secret to the transcript labeled DH_SECRET.
  • generate two keys to use for encryption (sending and receiving) and a challenge for authentication as follows:
    • create a hkdf-sha256 instance with the key being the diffie hellman shared secret, and info parameter as TENDERMINT_SECRET_CONNECTION_KEY_AND_CHALLENGE_GEN
    • get 64 bytes of output from hkdf-sha256
    • if we had the smaller ephemeral pubkey, use the first 32 bytes for the key for receiving, the second 32 bytes for sending; else the opposite.
  • use a separate nonce for receiving and sending. Both nonces start at 0, and should support the full 96 bit nonce range
  • all communications from now on are encrypted in 1400 byte frames (plus encoding overhead), using the respective secret and nonce. Each nonce is incremented by one after each use.
  • we now have an encrypted channel, but still need to authenticate
  • extract a 32 bytes challenge from merlin transcript with the label “SECRET_CONNECTION_MAC”
  • sign the common challenge obtained from the hkdf with our persistent private key
  • send the amino encoded persistent pubkey and signature to the peer
  • wait to receive the persistent public key and signature from the peer
  • verify the signature on the challenge using the peer’s persistent public key
If this is an outgoing connection (we dialed the peer) and we used a peer ID, then finally verify that the peer’s persistent public key corresponds to the peer ID we dialed, ie. peer.PubKey.Address() == <ID>. The connection has now been authenticated. All traffic is encrypted. Note: only the dialer can authenticate the identity of the peer, but this is what we care about since when we join the network we wish to ensure we have reached the intended peer (and are not being MITMd).

Peer Filter

Before continuing, we check if the new peer has the same ID as ourselves or an existing peer. If so, we disconnect. We also check the peer’s address and public key against an optional whitelist which can be managed through the ABCI app - if the whitelist is enabled and the peer does not qualify, the connection is terminated.

CometBFT Version Handshake

The CometBFT Version Handshake allows the peers to exchange their NodeInfo:
type NodeInfo struct {
  Version    p2p.Version
  ID         p2p.ID
  ListenAddr string

  Network    string
  SoftwareVersion    string
  Channels   []int8

  Moniker    string
  Other      NodeInfoOther
}

type Version struct {
 P2P uint64
 Block uint64
 App uint64
}

type NodeInfoOther struct {
 TxIndex          string
 RPCAddress       string
}
The connection is disconnected if:
  • peer.NodeInfo.ID is not equal peerConn.ID
  • peer.NodeInfo.Version.Block does not match ours
  • peer.NodeInfo.Network is not the same as ours
  • peer.Channels does not intersect with our known Channels.
  • peer.NodeInfo.ListenAddr is malformed or is a DNS host that cannot be resolved
At this point, if we have not disconnected, the peer is valid. It is added to the switch and hence all reactors via the AddPeer method. Note that each reactor may handle multiple channels.

Connection Activity

Once a peer is added, incoming messages for a given reactor are handled through that reactor’s Receive method, and output messages are sent directly by the Reactors on each peer. A typical reactor maintains per-peer go-routine(s) that handle this.