CometBFT debug kill

CometBFT 提供了一个 debug 子命令,允许你在收集有用信息并打包为压缩归档的同时,终止一个正在运行的 CometBFT 进程。 这些信息包括所使用的配置、共识状态、网络状态、节点状态、WAL,甚至还有进程退出前的栈跟踪。 在调试异常的 CometBFT 进程时,这些文件非常适合用于排查问题。
cometbft debug kill <pid> </path/to/out.zip> --home=</path/to/app.d>
该命令会将调试信息写入一个压缩归档中。归档将包含以下内容:
├── config.toml
├── consensus_state.json
├── net_info.json
├── stacktrace.out
├── status.json
└── wal
在底层实现上,debug kill 会从 /status、/net_info 和 /dump_consensus_state 这几个 HTTP 端点获取信息,并使用 -6 终止进程,以捕获 goroutine 转储。

CometBFT debug dump

debug dump 子命令允许你按固定时间间隔将调试数据导出为压缩归档。 这些归档除了包含共识状态、网络信息、节点状态和 WAL 之外,还会包含 goroutine 和堆内存的 profile。
cometbft debug dump </path/to/out> --home=</path/to/app.d>
它的行为与 kill 类似,但只会轮询节点,并按设定频率每隔若干秒将调试数据写入指定目标目录下的压缩归档。 每个归档将包含:
├── consensus_state.json
├── goroutine.out
├── heap.out
├── net_info.json
├── status.json
└── wal
注意:只有在提供了 profile 地址且该地址可正常使用时,才会写出 goroutine.out 和 heap.out。 该命令是阻塞式的,并会记录所有错误。

CometBFT Inspect

CometBFT 包含一个 inspect 命令,用于通过 CometBFT RPC 查询 CometBFT 的状态存储和区块存储。 当 CometBFT 共识引擎检测到状态不一致时,它会导致整个 CometBFT 进程崩溃。 在这种状态不一致的情况下,运行 CometBFT 的节点将无法启动。 inspect 命令只会运行 CometBFT RPC 端点中的一个子集,用于查询区块存储和状态存储。 inspect 允许运维人员查询状态的只读视图。 inspect 完全不会运行共识引擎,因此可以用于调试那些因状态不一致而崩溃的进程。

运行 inspect

在 CometBFT 崩溃所在的机器上,使用以下命令启动 inspect 工具:
cometbft inspect --home=</path/to/app.d>
inspect 会使用你在 CometBFT 配置文件中指定的数据目录。 inspect 还会在你在 CometBFT 配置文件中指定的地址上运行 RPC 服务器。

使用 inspect

当 inspect 服务器运行后,你可以访问那些对调试至关重要的 RPC 端点。 调用 /status、/consensus_state 和 /dump_consensus_state 这几个 RPC 端点,将返回有关 CometBFT 共识状态的有用信息。 要启动 inspect 进程,请运行:
cometbft inspect

RPC 端点

可以通过向 RPC 端口发起请求来获取可用 RPC 端点列表。 对于运行在 127.0.0.1:26657 上的 inspect 进程,可在浏览器中访问 http://127.0.0.1:26657/ 以获取已启用的 RPC 端点列表。 有关 CometBFT RPC 端点的更多信息,请参阅 RPC 文档。

CometBFT debug kill

CometBFT comes with a debug sub-command that allows you to kill a live CometBFT process while collecting useful information in a compressed archive. The information includes the configuration used, consensus state, network state, the node’s status, the WAL, and even the stack trace of the process before exit. These files can be useful to examine when debugging a faulty CometBFT process.
cometbft debug kill <pid> </path/to/out.zip> --home=</path/to/app.d>
will write debug info into a compressed archive. The archive will contain the following:
├── config.toml
├── consensus_state.json
├── net_info.json
├── stacktrace.out
├── status.json
└── wal
Under the hood, debug kill fetches info from /status, /net_info, and /dump_consensus_state HTTP endpoints, and kills the process with -6, which captures the goroutine dump.

CometBFT debug dump

The debug dump sub-command allows you to dump debugging data into compressed archives at a regular interval. These archives contain the goroutine and heap profiles in addition to the consensus state, network info, node status, and the WAL.
cometbft debug dump </path/to/out> --home=</path/to/app.d>
will perform similarly to kill except it only polls the node and dumps debugging data every frequency seconds to a compressed archive under a given destination directory. Each archive will contain:
├── consensus_state.json
├── goroutine.out
├── heap.out
├── net_info.json
├── status.json
└── wal
Note: goroutine.out and heap.out will only be written if a profile address is provided and is operational. This command is blocking and will log any error.

CometBFT Inspect

CometBFT includes an inspect command for querying CometBFT’s state store and block store over CometBFT RPC. When the CometBFT consensus engine detects inconsistent state, it will crash the entire CometBFT process. While in this inconsistent state, a node running CometBFT will not start up. The inspect command runs only a subset of CometBFT’s RPC endpoints for querying the block store and state store. inspect allows operators to query a read-only view of the state. inspect does not run the consensus engine at all and can therefore be used to debug processes that have crashed due to inconsistent state.

Running inspect

Start up the inspect tool on the machine where CometBFT crashed using:
cometbft inspect --home=</path/to/app.d>
inspect will use the data directory specified in your CometBFT configuration file. inspect will also run the RPC server at the address specified in your CometBFT configuration file.

Using inspect

With the inspect server running, you can access RPC endpoints that are critically important for debugging. Calling the /status, /consensus_state, and /dump_consensus_state RPC endpoints will return useful information about the CometBFT consensus state. To start the inspect process, run
cometbft inspect

RPC endpoints

The list of available RPC endpoints can be found by making a request to the RPC port. For an inspect process running on 127.0.0.1:26657, navigate your browser to http://127.0.0.1:26657/ to retrieve the list of enabled RPC endpoints. Additional information on the CometBFT RPC endpoints can be found in the RPC documentation.