大纲

概览与基本概念

ABCI 2.0 与 ABCI

↑ 返回大纲 应用程序的主要职责是执行由共识决定(也称为最终确定)的区块。这些已决定的区块是共识向(复制的)应用程序输出的主要结果。在 ABCI 中,应用程序只会在决定时刻与共识交互。这种受限的交互模式使应用程序无法实现许多特性,包括许多如今比 ABCI 初次编写时理解得更清楚的可扩展性改进。比如,许多为提升可扩展性而提出的思路都可以归结为“让区块提议者承担工作,这样网络就不必承担”。这包括交易级签名聚合、状态转换证明等优化。此外,在当前范式下还无法实现许多新的安全属性,因为应用程序无法要求验证者做的不仅仅是执行最终确定区块中包含的交易。这包括门限密码学、对 IBC 连接尝试的保证等特性。 ABCI 2.0 通过允许应用程序在共识执行的三个关键位置介入来解决这些限制:(a)即将创建新提案时,(b)即将验证提案时,以及(c)即将发送/接收一个(precommit)投票时。新接口允许区块提议者通过 PrepareProposal 方法在区块中执行依赖应用程序的工作(a);允许验证者通过 ProcessProposal 方法在提议区块中执行依赖应用程序的工作和检查(b);也允许应用程序通过 ExtendVote 和 VerifyVoteExtensions 方法要求其验证者做的不仅仅是验证区块(c)。 此外,ABCI 2.0 将 合并为 FinalizeBlock,作为一种更简化、更高效的方式,将已决定区块交付给应用程序。

方法概览

↑ 返回大纲 方法可以分为四类:共识、内存池、信息 和 状态同步。

共识/区块执行方法

新区块链第一次启动时,CometBFT 会调用 InitChain。从那以后,每个区块一旦被决定,就会执行 FinalizeBlock 方法,从而更新应用程序状态。在一次用于决定给定高度区块的共识实例执行期间,以及在调用 FinalizeBlock 方法之前,PrepareProposal、ProcessProposal、ExtendVote 和 VerifyVoteExtension 这些方法可能会被调用多次。有关这些方法可能的调用序列,请参见CometBFT 的预期行为。
  • InitChain: 此方法初始化区块链。 CometBFT 会在创世时调用它一次。
  • PrepareProposal: 它允许区块提议者在提出区块之前,在区块中执行依赖应用程序的工作。 例如,这使得可以对区块进行批处理优化,而实证已表明这是提升性能的关键组成部分。每当 CometBFT 即将广播 Proposal 消息且 validValue 为 nil 时,都会调用 PrepareProposal 方法。 CometBFT 会从内存池收集待处理交易,生成区块头,并据此创建一个待提议的区块。随后,它会使用这个新创建的提案(称为原始提案)调用 RequestPrepareProposal。应用程序可以在 ResponsePrepareProposal 中返回(可能)被修改过的提案(称为已准备提案)之前,对原始提案进行修改,例如重排、添加或移除交易。 修改原始提案的逻辑可以是非确定性的。
  • ProcessProposal: 它允许验证者在提议区块上执行依赖应用程序的工作。这使得即时执行等特性成为可能,并允许应用程序拒绝无效区块。 当 CometBFT 收到提案且 validValue 为 nil 时会调用它。此时应用程序不能修改提案,但如果提案无效,可以拒绝它。如果发生这种情况,共识算法将对该提案 prevote nil,这会对 CometBFT 的活性产生重要影响。作为一般规则,即使通过 ProcessProposal 传入的已准备提案有一部分无效(例如某笔无效交易),应用程序也应当接受它;应用程序可以在区块执行时忽略已准备提案中的无效部分。 ProcessProposal 中的逻辑必须是确定性的。
  • ExtendVote: 它允许应用程序让其验证者在共识中承担的不仅仅是验证工作。ExtendVote 允许应用程序将对共识算法而言不透明的非确定性数据附加到 precommit 消息(投票的最后一轮)中。这些数据称为投票扩展,会与其所扩展的投票一起被广播和接收,并会在下一个高度、本地进程作为提议者的轮次中提供给应用程序。 当共识算法即将发送一个非 nil 的 precommit 消息时,CometBFT 会调用 ExtendVote。如果此时应用程序没有投票扩展信息可提供,它会返回一个长度为 0 的字节数组作为投票扩展。 ExtendVote 中的逻辑可以是非确定性的。
  • VerifyVoteExtension: 它允许验证者验证附加到 precommit 消息上的投票扩展数据。如果验证失败,整个 precommit 消息都会被视为无效,并被共识算法忽略。这会对活性造成负面影响;也就是说,如果正确的验证者反复无法验证投票扩展,即使有足够多(+2/3)的验证者为该区块发送了 precommit 投票,共识算法也可能无法最终确定区块。因此,VerifyVoteExtension 应特别谨慎地实现。 作为一般规则,检测到无效投票扩展的应用程序应在 ResponseVerifyVoteExtension 中接受它,并在自身逻辑中忽略它。当前高度下,当某个进程接收到带有投票扩展(可能为空)的 precommit 消息时,CometBFT 会调用它。对于在该高度已经结束后、但仍在等待累积更多 precommit 投票时接收到的 precommit 投票,不会调用它。 VerifyVoteExtension 中的逻辑必须是确定性的。
  • FinalizeBlock: 它将一个已决定区块交付给应用程序。应用程序必须以确定性的方式执行区块中的交易,并据此更新其状态。通过 ResponseFinalizeBlock 中相应参数返回的、对区块和交易结果的密码学承诺,会被包含在下一个区块的区块头中。当新块被决定时,CometBFT 会调用它。 当使用某个区块调用 FinalizeBlock 时,由 CometBFT 运行的共识算法保证至少有一个非拜占庭验证者已经对该区块运行过 ProcessProposal。
  • Commit: 指示应用程序持久化其状态。它是 CometBFT 崩溃恢复机制的基础组成部分,可确保恢复后 CometBFT 与应用程序之间的同步。CometBFT 会在持久化对 ResponseFinalizeBlock 调用返回的数据之后立即调用它。此时,应用程序可以丢弃任何状态或数据,只保留执行已决定区块中交易后得到的结果。

内存池方法

  • CheckTx: 此方法允许应用程序验证交易。验证既可以是无状态的(例如检查签名),也可以是有状态的(例如检查账户余额)。具体执行哪种验证由应用程序决定。如果交易通过验证,CometBFT 会将其加入内存池;否则该交易会被丢弃。 当 CometBFT 收到一笔新交易时,无论它来自外部用户(例如客户端)还是另一节点,都会调用它。此外,CometBFT 还可以配置为:在对某个区块调用 Commit 之后,对内存池中所有待处理交易重新执行 CheckTx。

信息方法

  • Info: 用于在恢复时进行握手,或在使用状态同步启动时,让 CometBFT 与应用程序保持同步。
  • Query: 此方法可用于向应用程序查询应用程序状态相关信息。

状态同步方法

状态同步允许新节点通过发现、获取并应用状态机(应用程序)快照,而不是重放历史区块,来快速完成引导。更多细节请参见状态同步文档。 新节点会在 P2P 网络中发现并向其他节点请求快照。接收到来自对等节点的快照请求的 CometBFT 节点,会在其应用程序上调用 ListSnapshots。应用程序会返回本地可用快照的列表。 请注意,该列表不包含实际快照,而是关于快照的元数据:拍摄快照时的高度、应用程序特定的验证数据等(更多细节见快照数据类型)。在从某个对等节点接收到可用快照列表后,新节点可以通过 OfferSnapshot 方法,将列表中的任意快照提供给其本地应用程序。此时应用程序可以检查快照元数据的有效性。 快照可能非常大,因此会被拆分成更小的“块”,这些块可以重新组装成完整快照。一旦应用程序接受某个快照并开始恢复它,CometBFT 就会从现有节点获取快照“块”。提供这些“块”的节点会使用 LoadSnapshotChunk 方法从其本地应用程序中取出这些块。 随着新节点接收到这些“块”,它会使用 ApplySnapshotChunk 按顺序将它们应用到本地应用程序。当所有块都应用完成后,会通过一次 Info 查询获取应用程序的 AppHash。为了确保同步正确进行,CometBFT 会将本地应用程序的 AppHash 与区块链上存储的 AppHash 进行比较(后者通过轻客户端验证完成验证)。 总结如下:

其他方法

此外,还有一个在每条连接上都会调用的 Flush 方法,以及一个用于调试的 Echo 方法。 有关如何跨连接管理状态的更多细节,请参见管理应用程序状态一节。

提案超时

PrepareProposal 位于共识算法的关键路径上,也就是说,在执行此方法期间,CometBFT 无法推进处理。因此,如果应用程序准备提案耗时较长,TimeoutPropose 的默认值可能不足以容纳该方法的执行,验证者节点可能会超时并 prevote nil。在这种情况下,该提案很可能会被拒绝,并且需要开启一个新轮次。 每个高度进入新轮次时,超时时间都会自动增加;如果 PrepareProposal 的执行时间是有界的,那么最终 TimeoutPropose 会足够长,以容纳 PrepareProposal 的执行。 然而,依赖这种自适应机制可能导致性能下降,因此建议运维人员调整 CometBFT 配置文件中 TimeoutPropose 的初始值,以适配所部署具体应用程序的需求。 如果应用程序实现了即时执行,这一点尤其重要。为了实现这项技术,提议者需要在 PrepareProposal 中执行即将被提议的区块,这可能会比 TimeoutPropose 更耗时。

确定性状态机复制

↑ 返回大纲 ABCI 应用程序必须实现确定性的有限状态机,才能被 CometBFT 共识引擎安全地复制。这意味着区块执行必须严格确定:给定相同的有序交易集合,所有节点在所有连续的 FinalizeBlock 调用中都将计算出完全相同的响应。这一点至关重要,因为这些响应会通过 Merkle 根或直接方式被包含在下一个区块的区块头中,因此所有节点必须对其具体内容达成完全一致。 因此,建议不要将应用程序状态暴露给任何外部用户或进程,除非是通过与 CometBFT 这类共识引擎建立的 ABCI 连接。应用程序只能依据区块执行(FinalizeBlock 调用)的输入改变自身状态,而不能通过任何其他类型的请求修改状态。只有这样才能确保所有节点看到相同的交易并计算出相同的结果。 实现即时执行的应用程序(即在 PrepareProposal 中执行即将被提议的区块,或在 ProcessProposal 中执行需要验证的区块)会在区块被决定之前产生一个新的候选状态。处理这些提议区块导致的状态变更,在 FinalizeBlock 确认该提议区块已被决定并且对其调用 Commit 之前,绝不能替换先前状态。 对于那些为了节省 FinalizeBlock 阶段时间而快速接受区块,并在剩余共识步骤并行期间乐观执行区块的应用程序,同样如此;它们只能在 Commit 中应用状态变更。 另外,投票扩展或对其的验证(通过 ExtendVote 或 VerifyVoteExtension)绝不能对当前状态产生副作用。只有当其数据在一次 RequestPrepareProposal 调用中被提供时,才能使用这些数据,但同样不能对应用程序状态产生副作用。 如果状态机中存在某种非确定性,随着节点对区块头正确值产生分歧,共识最终会失败。必须修复这种非确定性并重启节点。 应用程序中的非确定性来源可能包括:
  • 硬件故障
    • 宇宙射线、过热等
  • 节点依赖状态
    • 随机数
    • 时间
  • 规范不足
    • 库版本变化
    • 竞争条件
    • 浮点数
    • JSON 或 protobuf 序列化
    • 遍历哈希表/map/dictionary
  • 外部来源
    • 文件系统
    • 网络调用(例如某个外部 REST API 服务)
原始讨论见 #56。 请注意,某些方法(例如 Query 和 FinalizeBlock)可能会以 Info、Log 和/或 Events 字段的形式返回非确定性数据。Log 旨在承载应用程序记录器的原始输出,而 Info 则用于返回任何附加信息。这些字段不会被纳入区块头计算,因此不需要节点就这些内容达成一致。关于某个字段是否必须确定,请参见各字段的说明。

事件

↑ 返回大纲 FinalizeBlock 方法在其 Response* 顶层包含一个 events 字段,并为区块中包含的每笔交易各自包含一个 events 字段。应用程序可以对这个 ABCI 2.0 方法作出响应:为每笔已执行交易返回一个事件列表,并为区块本身返回一个通用事件列表。事件允许应用程序将元数据与交易和区块关联起来。通过 FinalizeBlock 返回的事件不会以任何方式影响共识算法,而是用于支撑 CometBFT 状态的订阅和查询。 一个 Event 包含一个 type 和一个 EventAttributes 列表,后者是键值字符串对,用于表示该方法(或交易)执行期间发生事情的元数据。Event 的值可用于根据执行期间发生的情况,对交易和区块建立索引。 每个事件都有一个 type,用于对特定 Response* 或 Tx 的事件进行分类。一个 Response* 或 Tx 可以包含多个 type 值重复的事件,其中每个独立条目都用于对特定事件的属性进行分类。事件属性中的每个键和值,以及事件类型本身,都必须是 UTF-8 编码字符串。
message Event {
  string                  type       = 1;
  repeated EventAttribute attributes = 2;
}
Event 的属性由 key、value 和 index 标记组成。index 标记用于通知 CometBFT 索引器对该属性建立索引。 type 和 attributes 字段是非确定性的,可能在网络中的不同节点之间有所不同。
message EventAttribute {
  string key   = 1;
  string value = 2;
  bool   index = 3;  // nondeterministic
}
示例:
 abci.ResponseFinalizeBlock{
  // ...
 Events: []abci.Event{
  {
   Type: "validator.provisions",
   Attributes: []abci.EventAttribute{
    abci.EventAttribute{Key: "address", Value: "...", Index: true},
    abci.EventAttribute{Key: "amount", Value: "...", Index: true},
    abci.EventAttribute{Key: "balance", Value: "...", Index: true},
   },
  },
  {
   Type: "validator.provisions",
   Attributes: []abci.EventAttribute{
    abci.EventAttribute{Key: "address", Value: "...", Index: true},
    abci.EventAttribute{Key: "amount", Value: "...", Index: false},
    abci.EventAttribute{Key: "balance", Value: "...", Index: false},
   },
  },
  {
   Type: "validator.slashed",
   Attributes: []abci.EventAttribute{
    abci.EventAttribute{Key: "address", Value: "...", Index: false},
    abci.EventAttribute{Key: "amount", Value: "...", Index: true},
    abci.EventAttribute{Key: "reason", Value: "...", Index: true},
   },
  },
  // ...
 },
}

证据

↑ 返回大纲 CometBFT 的安全模型依赖于使用不当行为证据。证据是对网络参与者恶意行为的不可辩驳证明。检测这类恶意行为是 CometBFT 的职责。当检测到恶意行为时,CometBFT 会将不当行为证据通过 gossip 传播给其他节点,并在一部分验证者完成验证后,将这些证据提交到链上。随后,这些证据会通过 ABCI++ 传递给应用程序。处理不当行为证据并实施惩罚则是应用程序的职责。 证据有两种形式:重复投票和轻客户端攻击。更多信息可参见数据结构或问责。 EvidenceType 的 protobuf 格式如下:
enum EvidenceType {
  UNKNOWN               = 0;
  DUPLICATE_VOTE        = 1;
  LIGHT_CLIENT_ATTACK   = 2;
}

错误

↑ 返回大纲 Query 和 CheckTx 方法在其 Response* 中包含一个 Code 字段。Code 字段用于承载应用程序特定的响应码。响应码为 0 表示没有错误,任何其他响应码都表示 CometBFT 认为发生了错误。 这些方法还会向 CometBFT 返回一个 Codespace 字符串。该字段用于区分应用程序不同域返回的 Code 值。Codespace 是 Code 的命名空间。 Echo、Info、Commit 和 InitChain 方法不会返回错误。这些方法中的错误代表严重问题,而 CometBFT 没有合理的处理方式。如果其中任一方法出现错误,应用程序必须崩溃,以确保该错误能被运维人员安全处理。 FinalizeBlock 是一个特殊情况。它包含若干属于 ExecTxResult 类型的 Code 和 Codespace 字段。每个代码都用于报告其所附带交易相关的错误。然而,FinalizeBlock 本身不会在顶层返回错误,因此,对 Echo、Info 和 InitChain 所作的关于严重问题的同样考虑也适用于这里。 CometBFT 对非零响应码的处理方式如下所述。

CheckTx

当 CometBFT 收到一个 Code 非零的 ResponseCheckTx 时,相关交易不会被加入 CometBFT 的内存池;如果它已经在内存池中,也会被移除。

ExecTxResult(作为 FinalizeBlock 的一部分)

ExecTxResult 类型将交易结果从应用程序传递给 CometBFT。当 CometBFT 收到一个包含 Code 非零的 ExecTxResult 的 ResponseFinalizeBlock 时,会记录该响应码。客户端可以查询历史 Code 值。由于该交易已经属于一个已决定区块,Code 不会影响共识。

Query

当 CometBFT 收到一个 Code 非零的 ResponseQuery 时,该代码会被直接返回给发起查询的客户端。

Outline

Overview and basic concepts

ABCI 2.0 vs. ABCI

↑ Back to Outline The Application’s main role is to execute blocks decided (a.k.a. finalized) by consensus. The decided blocks are the consensus’s main output to the (replicated) Application. With ABCI, the application only interacts with consensus at decision time. This restricted mode of interaction prevents numerous features for the Application, including many scalability improvements that are now better understood than when ABCI was first written. For example, many ideas proposed to improve scalability can be boiled down to “make the block proposers do work, so the network does not have to”. This includes optimizations such as transaction level signature aggregation, state transition proofs, etc. Furthermore, many new security properties cannot be achieved in the current paradigm, as the Application cannot require validators to do more than executing the transactions contained in finalized blocks. This includes features such as threshold cryptography, and guaranteed IBC connection attempts. ABCI 2.0 addresses these limitations by allowing the application to intervene at three key places of consensus execution: (a) at the moment a new proposal is to be created, (b) at the moment a proposal is to be validated, and (c) at the moment a (precommit) vote is sent/received. The new interface allows block proposers to perform application-dependent work in a block through the PrepareProposal method (a); and validators to perform application-dependent work and checks in a proposed block through the ProcessProposal method (b); and applications to require their validators to do more than just validate blocks through the ExtendVote and VerifyVoteExtensions methods (c). Furthermore, ABCI 2.0 coalesces into FinalizeBlock, as a simplified, efficient way to deliver a decided block to the Application.

Methods overview

↑ Back to Outline Methods can be classified into four categories: consensus, mempool, info, and state-sync.

Consensus/block execution methods

The first time a new blockchain is started, CometBFT calls InitChain. From then on, method FinalizeBlock is executed upon the decision of each block, resulting in an updated Application state. During the execution of an instance of consensus, which decides the block for a given height, and before method FinalizeBlock is called, methods PrepareProposal, ProcessProposal, ExtendVote, and VerifyVoteExtension may be called several times. See CometBFT’s expected behavior for details on the possible call sequences of these methods.
  • InitChain: This method initializes the blockchain. CometBFT calls it once upon genesis.
  • PrepareProposal: It allows the block proposer to perform application-dependent work in a block before proposing it. This enables, for instance, batch optimizations to a block, which has been empirically demonstrated to be a key component for improved performance. Method PrepareProposal is called every time CometBFT is about to broadcast a Proposal message and validValue is nil. CometBFT gathers outstanding transactions from the mempool, generates a block header, and uses them to create a block to propose. Then, it calls RequestPrepareProposal with the newly created proposal, called raw proposal. The Application can make changes to the raw proposal, such as reordering, adding and removing transactions, before returning the (potentially) modified proposal, called prepared proposal in the ResponsePrepareProposal. The logic modifying the raw proposal MAY be non-deterministic.
  • ProcessProposal: It allows a validator to perform application-dependent work in a proposed block. This enables features such as immediate block execution, and allows the Application to reject invalid blocks. CometBFT calls it when it receives a proposal and validValue is nil. The Application cannot modify the proposal at this point but can reject it if invalid. If that is the case, the consensus algorithm will prevote nil on the proposal, which has strong liveness implications for CometBFT. As a general rule, the Application SHOULD accept a prepared proposal passed via ProcessProposal, even if a part of the proposal is invalid (e.g., an invalid transaction); the Application can ignore the invalid part of the prepared proposal at block execution time. The logic in ProcessProposal MUST be deterministic.
  • ExtendVote: It allows applications to let their validators do more than just validate within consensus. ExtendVote allows applications to include non-deterministic data, opaque to the consensus algorithm, to precommit messages (the final round of voting). The data, called vote extension, will be broadcast and received together with the vote it is extending, and will be made available to the Application in the next height, in the rounds where the local process is the proposer. CometBFT calls ExtendVote when the consensus algorithm is about to send a non-nil precommit message. If the Application does not have vote extension information to provide at that time, it returns a 0-length byte array as its vote extension. The logic in ExtendVote MAY be non-deterministic.
  • VerifyVoteExtension: It allows validators to validate the vote extension data attached to a precommit message. If the validation fails, the whole precommit message will be deemed invalid and ignored by consensus algorithm. This has a negative impact on liveness, i.e., if vote extensions repeatedly cannot be verified by correct validators, the consensus algorithm may not be able to finalize a block even if sufficiently many (+2/3) validators send precommit votes for that block. Thus, VerifyVoteExtension should be implemented with special care. As a general rule, an Application that detects an invalid vote extension SHOULD accept it in ResponseVerifyVoteExtension and ignore it in its own logic. CometBFT calls it when a process receives a precommit message with a (possibly empty) vote extension, for the current height. It is not called for precommit votes received after the height is concluded but while waiting to accumulate more precommit votes. The logic in VerifyVoteExtension MUST be deterministic.
  • FinalizeBlock: It delivers a decided block to the Application. The Application must execute the transactions in the block deterministically and update its state accordingly. Cryptographic commitments to the block and transaction results, returned via the corresponding parameters in ResponseFinalizeBlock, are included in the header of the next block. CometBFT calls it when a new block is decided. When calling FinalizeBlock with a block, the consensus algorithm run by CometBFT guarantees that at least one non-byzantine validator has run ProcessProposal on that block.
  • Commit: Instructs the Application to persist its state. It is a fundamental part of CometBFT’s crash-recovery mechanism that ensures the synchronization between CometBFT and the Application upon recovery. CometBFT calls it just after having persisted the data returned by calls to ResponseFinalizeBlock. The Application can now discard any state or data except the one resulting from executing the transactions in the decided block.

Mempool methods

  • CheckTx: This method allows the Application to validate transactions. Validation can be stateless (e.g., checking signatures ) or stateful (e.g., account balances). The type of validation performed is up to the application. If a transaction passes the validation, then CometBFT adds it to the mempool; otherwise the transaction is discarded. CometBFT calls it when it receives a new transaction either coming from an external user (e.g., a client) or another node. Furthermore, CometBFT can be configured to call re-CheckTx on all outstanding transactions in the mempool after calling Commit for a block.

Info methods

  • Info: Used to sync CometBFT with the Application during a handshake that happens upon recovery, or on startup when state-sync is used.
  • Query: This method can be used to query the Application for information about the application state.

State-sync methods

State sync allows new nodes to rapidly bootstrap by discovering, fetching, and applying state machine (application) snapshots instead of replaying historical blocks. For more details, see the state sync documentation. New nodes discover and request snapshots from other nodes in the P2P network. A CometBFT node that receives a request for snapshots from a peer will call ListSnapshots on its Application. The Application returns the list of locally available snapshots. Note that the list does not contain the actual snapshots but metadata about them: height at which the snapshot was taken, application-specific verification data and more (see snapshot data type for more details). After receiving a list of available snapshots from a peer, the new node can offer any of the snapshots in the list to its local Application via the OfferSnapshot method. The Application can check at this point the validity of the snapshot metadata. Snapshots may be quite large and are thus broken into smaller “chunks” that can be assembled into the whole snapshot. Once the Application accepts a snapshot and begins restoring it, CometBFT will fetch snapshot “chunks” from existing nodes. The node providing “chunks” will fetch them from its local Application using the LoadSnapshotChunk method. As the new node receives “chunks” it will apply them sequentially to the local application with ApplySnapshotChunk. When all chunks have been applied, the Application’s AppHash is retrieved via an Info query. To ensure that the sync proceeded correctly, CometBFT compares the local Application’s AppHash to the AppHash stored on the blockchain (verified via light client verification). In summary:
  • ListSnapshots: Used by nodes to discover available snapshots on peers.
  • OfferSnapshot: When a node receives a snapshot from a peer, CometBFT uses this method to offer the snapshot to the Application.
  • LoadSnapshotChunk: Used by CometBFT to retrieve snapshot chunks from the Application to send to peers.
  • ApplySnapshotChunk: Used by CometBFT to hand snapshot chunks to the Application.

Other methods

Additionally, there is a Flush method that is called on every connection, and an Echo method that is used for debugging. More details on managing state across connections can be found in the section on Managing Application State.

Proposal timeout

PrepareProposal stands on the consensus algorithm critical path, i.e., CometBFT cannot make progress while this method is being executed. Hence, if the Application takes a long time preparing a proposal, the default value of TimeoutPropose might not be sufficient to accommodate the method’s execution and validator nodes might time out and prevote nil. The proposal, in this case, will probably be rejected and a new round will be necessary. Timeouts are automatically increased for each new round of a height and, if the execution of PrepareProposal is bound, eventually TimeoutPropose will be long enough to accommodate the execution of PrepareProposal. However, relying on this self adaptation could lead to performance degradation and, therefore, operators are suggested to adjust the initial value of TimeoutPropose in CometBFT’s configuration file, in order to suit the needs of the particular application being deployed. This is particularly important if applications implement immediate execution. To implement this technique, proposers need to execute the block being proposed within PrepareProposal, which could take longer than TimeoutPropose.

Deterministic State-Machine Replication

↑ Back to Outline ABCI applications must implement deterministic finite-state machines to be securely replicated by the CometBFT consensus engine. This means block execution must be strictly deterministic: given the same ordered set of transactions, all nodes will compute identical responses, for all successive FinalizeBlock calls. This is critical because the responses are included in the header of the next block, either via a Merkle root or directly, so all nodes must agree on exactly what they are. For this reason, it is recommended that application state is not exposed to any external user or process except via the ABCI connections to a consensus engine like CometBFT. The Application must only change its state based on input from block execution (FinalizeBlock calls), and not through any other kind of request. This is the only way to ensure all nodes see the same transactions and compute the same results. Applications that implement immediate execution (execute the blocks that are about to be proposed, in PrepareProposal, or that require validation, in ProcessProposal) produce a new candidate state before a block is decided. The state changes caused by processing those proposed blocks must never replace the previous state until FinalizeBlock confirms that the proposed block was decided and Commit is invoked for it. The same is true to Applications that quickly accept blocks and execute the blocks optimistically in parallel with the remaining consensus steps to save time during FinalizeBlock; they must only apply state changes in Commit. Additionally, vote extensions or the validation thereof (via ExtendVote or VerifyVoteExtension) must never have side effects on the current state. They can only be used when their data is provided in a RequestPrepareProposal call but, again, without side effects to the app state. If there is some non-determinism in the state machine, consensus will eventually fail as nodes disagree over the correct values for the block header. The non-determinism must be fixed and the nodes restarted. Sources of non-determinism in applications may include:
  • Hardware failures
    • Cosmic rays, overheating, etc.
  • Node-dependent state
    • Random numbers
    • Time
  • Underspecification
    • Library version changes
    • Race conditions
    • Floating point numbers
    • JSON or protobuf serialization
    • Iterating through hash-tables/maps/dictionaries
  • External Sources
    • Filesystem
    • Network calls (eg. some external REST API service)
See #56 for the original discussion. Note that some methods (e.g., Query and FinalizeBlock) may return non-deterministic data in the form of Info, Log and/or Events fields. The Log is intended for the literal output from the Application’s logger, while the Info is any additional info that should be returned. These fields are not included in block header computations, so we don’t need agreement on them. See each field’s description on whether it must be deterministic or not.

Events

↑ Back to Outline Method FinalizeBlock includes an events field at the top level in its Response*, and one events field per transaction included in the block. Applications may respond to this ABCI 2.0 method with an event list for each executed transaction, and a general event list for the block itself. Events allow applications to associate metadata with transactions and blocks. Events returned via FinalizeBlock do not impact the consensus algorithm in any way and instead exist to power subscriptions and queries of CometBFT state. An Event contains a type and a list of EventAttributes, which are key-value string pairs denoting metadata about what happened during the method’s (or transaction’s) execution. Event values can be used to index transactions and blocks according to what happened during their execution. Each event has a type which is meant to categorize the event for a particular Response* or Tx. A Response* or Tx may contain multiple events with duplicate type values, where each distinct entry is meant to categorize attributes for a particular event. Every key and value in an event’s attributes must be UTF-8 encoded strings along with the event type itself.
message Event {
  string                  type       = 1;
  repeated EventAttribute attributes = 2;
}
The attributes of an Event consist of a key, a value, and an index flag. The index flag notifies the CometBFT indexer to index the attribute. The type and attributes fields are non-deterministic and may vary across different nodes in the network.
message EventAttribute {
  string key   = 1;
  string value = 2;
  bool   index = 3;  // nondeterministic
}
Example:
 abci.ResponseFinalizeBlock{
  // ...
 Events: []abci.Event{
  {
   Type: "validator.provisions",
   Attributes: []abci.EventAttribute{
    abci.EventAttribute{Key: "address", Value: "...", Index: true},
    abci.EventAttribute{Key: "amount", Value: "...", Index: true},
    abci.EventAttribute{Key: "balance", Value: "...", Index: true},
   },
  },
  {
   Type: "validator.provisions",
   Attributes: []abci.EventAttribute{
    abci.EventAttribute{Key: "address", Value: "...", Index: true},
    abci.EventAttribute{Key: "amount", Value: "...", Index: false},
    abci.EventAttribute{Key: "balance", Value: "...", Index: false},
   },
  },
  {
   Type: "validator.slashed",
   Attributes: []abci.EventAttribute{
    abci.EventAttribute{Key: "address", Value: "...", Index: false},
    abci.EventAttribute{Key: "amount", Value: "...", Index: true},
    abci.EventAttribute{Key: "reason", Value: "...", Index: true},
   },
  },
  // ...
 },
}

Evidence

↑ Back to Outline CometBFT’s security model relies on the use of evidences of misbehavior. An evidence is an irrefutable proof of malicious behavior by a network participant. It is the responsibility of CometBFT to detect such malicious behavior. When malicious behavior is detected, CometBFT will gossip evidences of misbehavior to other nodes and commit the evidences to the chain once they are verified by a subset of validators. These evidences will then be passed on to the Application through ABCI++. It is the responsibility of the Application to handle evidence of misbehavior and exercise punishment. There are two forms of evidence: Duplicate Vote and Light Client Attack. More information can be found in either data structures or accountability. EvidenceType has the following protobuf format:
enum EvidenceType {
  UNKNOWN               = 0;
  DUPLICATE_VOTE        = 1;
  LIGHT_CLIENT_ATTACK   = 2;
}

Errors

↑ Back to Outline The Query and CheckTx methods include a Code field in their Response*. Field Code is meant to contain an application-specific response code. A response code of 0 indicates no error. Any other response code indicates to CometBFT that an error occurred. These methods also return a Codespace string to CometBFT. This field is used to disambiguate Code values returned by different domains of the Application. The Codespace is a namespace for the Code. Methods Echo, Info, Commit and InitChain do not return errors. An error in any of these methods represents a critical issue that CometBFT has no reasonable way to handle. If there is an error in one of these methods, the Application must crash to ensure that the error is safely handled by an operator. Method FinalizeBlock is a special case. It contains a number of Code and Codespace fields as part of type ExecTxResult. Each of these codes reports errors related to the transaction it is attached to. However, FinalizeBlock does not return errors at the top level, so the same considerations on critical issues made for Echo, Info, and InitChain also apply here. The handling of non-zero response codes by CometBFT is described below.

CheckTx

When CometBFT receives a ResponseCheckTx with a non-zero Code, the associated transaction will not be added to CometBFT’s mempool or it will be removed if it is already included.

ExecTxResult (as part of FinalizeBlock)

The ExecTxResult type delivers transaction results from the Application to CometBFT. When CometBFT receives a ResponseFinalizeBlock containing an ExecTxResult with a non-zero Code, the response code is logged. Past Code values can be queried by clients. As the transaction was part of a decided block, the Code does not influence consensus.

Query

When CometBFT receives a ResponseQuery with a non-zero Code, this code is returned directly to the client that initiated the query.