数据库

默认情况下,CometBFT 使用 syndtr/goleveldb 包作为其进程内键值数据库。如果你希望获得最高性能,最佳做法可能是安装 LevelDB 的真实 C 实现,并通过 make build COMETBFT_BUILD_OPTIONS=cleveldb 编译 CometBFT 以使用它。详情请参阅安装说明。 CometBFT 会在 $CMTHOME/data 中维护多个彼此独立的数据库:
  • blockstore.db:保存完整区块链,存储区块、区块提交以及区块元数据,并按区块高度建立索引。用于同步新节点。
  • evidence.db:存储所有已验证的不当行为证据。
  • state.db:存储当前区块链状态(即高度、验证者、共识参数)。仅当共识参数或验证者发生变化时才会增长。同时也用于在区块处理期间临时存储中间结果。
  • tx_index.db:按交易哈希和区块高度为交易建立索引。如果应用在 FinalizeBlock 响应中加入了交易结果,这些结果也会被建立索引。
默认情况下,CometBFT 只会按交易哈希和高度建立交易索引。如果你希望对结果事件建立索引,详情请参阅交易索引。 应用可以向节点运营者暴露区块裁剪策略。请阅读你的应用文档以了解更多细节。 应用可以使用状态同步 来帮助节点快速完成引导。

日志

默认日志级别(log_level = "main:info,state:info,statesync:info,*:error")足以满足正常运行模式。关于如何配置 log_level 配置变量的详细说明,请阅读这篇文章。部分模块列表可见这里。如果你正在调试 CometBFT,或者被要求提供 debug 日志级别的日志,可以使用 --log_level="*:debug" 启动 CometBFT。

预写日志(WAL)

CometBFT 为共识(cs.wal)和 mempool(mempool.wal)使用预写日志。两个 WAL 的最大大小都是 1GB,并会自动轮转。

共识 WAL

consensus.wal 用于确保我们能够在共识状态机的任意时刻从崩溃中恢复。 它会将所有共识消息(超时、提案、区块分片或投票)写入单个文件,并在处理来自其自身验证者的消息之前将内容刷新到磁盘。由于 CometBFT 验证者预期永远不会签署相互冲突的投票,因此 WAL 能确保我们始终可以在不依赖网络、也无需重新签署任何共识消息的前提下,以确定性方式恢复到最新的共识状态。 如果你的 consensus.wal 已损坏,请参阅下文。

Mempool WAL

mempool.wal 会在执行 CheckTx 之前记录所有传入交易,但除此之外不会被程序以任何方式使用。它更像是一种手动保障机制。请注意,mempool 不提供持久性保证:如果交易被发送到一个或多个节点,而这些节点在能够提议该交易之前崩溃,那么该交易可能永远不会进入区块链。客户端必须通过 websocket 订阅、轮询查询,或使用 /broadcast_tx_commit 来监控自己的交易。在最坏情况下,可以手动从 mempool WAL 重新发送交易。 基于上述原因,mempool.wal 默认是禁用的。要启用它,请将 mempool.wal_dir 设置为你希望存放 WAL 的位置(例如 data/mempool.wal)。

DoS 暴露面与缓解

验证者应当部署哨兵节点架构 来防止拒绝服务攻击。

P2P

CometBFT 点对点系统的核心是 MConnection。每个连接都有 MaxPacketMsgPayloadSize,即最大数据包大小,并带有有界的发送与接收队列。还可以对每个连接的发送与接收速率施加限制(SendRate、RecvRate)。 打开的 P2P 连接数量可能会变得相当大,并触及操作系统的打开文件数限制(因为在基于 UNIX 的系统中,TCP 连接被视为文件)。应当为节点分配足够大的打开文件数限制,例如通过 ulimit -n 8192 或其他部署相关机制设置为 8192。

RPC

攻击暴露面与缓解

通常不建议将 RPC 端点公开暴露,尤其当相关节点是验证者节点时更是如此,因为 CometBFT RPC 当前并未提供高级安全特性。在缺乏适当保护的情况下公开暴露 RPC 端点,会使相关节点容易受到多种攻击。 如果节点的 RPC 端点必须公开暴露,运营者需要自行确保已经采取合适措施来缓解攻击。缓解措施示例包括但不限于:
  • 永远不要公开暴露验证者的 RPC 端点(也就是说,如果确实必须暴露 RPC 端点,请确保只在全节点上这样做,并配备适当保护)
  • 正确使用限流、身份认证和缓存(例如通过 nginx 这类反向代理和/或 Cloudflare 这类 DDoS 防护服务提供)
  • 仅暴露相关使用场景绝对必要的特定端点(可通过 nginx/Cloudflare 等进行配置)
如果运营者无法获得保护节点 RPC 端点所需的专业支持,强烈建议永远不要公开暴露这些端点。 在任何情况下,都不应将任何不安全 RPC 端点公开暴露。

返回多条记录的端点

默认情况下,返回多条记录的端点会限制为返回 30 个元素(最多 100 个)。更多信息请参阅RPC 文档。

调试 CometBFT

如果你需要调试 CometBFT,首先应该做的很可能是查看日志。请参阅如何阅读日志,其中解释了某些日志语句的含义。 如果粗略浏览日志之后,情况仍然不清楚,下一步可以尝试查询 /status RPC 端点。它会提供必要信息:节点是否正在同步、当前所处高度等。
curl http(s)://{ip}:{rpcPort}/status
/dump_consensus_state 会为你提供共识状态的详细概览(提议者、最新验证者、对等节点状态)。借助它,你应该能够判断例如网络为何会停止。
curl http(s)://{ip}:{rpcPort}/dump_consensus_state
该端点还有一个简化版本 /consensus_state,它只返回当前高度下观察到的投票。 如果在查看日志以及上述端点之后,你仍然不知道发生了什么,可以考虑使用 cometbft debug kill 子命令。该命令会抓取所有可用信息并终止进程。具体格式请参阅调试。 你可以自己检查生成的归档文件,或者在 Github 上创建 issue。不过,在创建 issue 之前,请务必先确认是否不存在已有 issue。

监控 CometBFT

每个 CometBFT 实例都有一个标准的 /health RPC 端点:如果一切正常,它会返回 200(OK);如果出现问题,则返回 500(或无响应)。 其他有用的端点包括前面提到的 /status、/net_info 和 /validators。 CometBFT 还可以上报并提供 Prometheus 指标。请参阅指标。 cometbft debug dump 子命令可用于定期将有用信息导出到归档文件中。更多信息请参阅调试。

当我的应用挂掉时会发生什么

你应当在进程监督器(例如 systemd 或 runit)下运行 CometBFT。这样可以确保 CometBFT 始终处于运行状态(即使发生错误)。 回到最初的问题,如果你的应用挂掉了,CometBFT 会 panic。在进程监督器重启你的应用之后,CometBFT 应该能够成功重新连接。对它来说,重启顺序无关紧要。

信号处理

我们会捕获 SIGINT 和 SIGTERM,并尽量进行妥善清理。对于其他信号,我们使用 Go 的默认行为: Go 程序中的信号默认行为。

数据损坏

注意: 请确保你已经备份了 CometBFT 数据目录。

可能原因

请记住,大多数损坏都是由硬件问题引起的:
  • 带有故障或老化电池备份的 RAID 控制器,再加上意外断电
  • 启用了写回缓存的机械硬盘,再加上意外断电
  • 断电保护能力不足的廉价 SSD,再加上意外断电
  • 有缺陷的内存
  • 有缺陷或过热的 CPU
其他原因还可能包括:
  • 配置了 fsync=off 的数据库系统,再加上操作系统崩溃或断电
  • 配置为使用写屏障的文件系统,再叠加忽略写屏障的存储层。LVM 是一个典型问题来源。
  • CometBFT bug
  • 操作系统 bug
  • 管理员操作失误(例如直接修改 CometBFT 数据目录内容)
(来源:https://wiki.postgresql.org/wiki/Corruption)

WAL 损坏

如果共识 WAL 在最新高度处损坏,而你正在尝试启动 CometBFT,重放过程会因 panic 而失败。 从数据损坏中恢复可能既困难又耗时。这里有两种你可以采取的方法:
  1. 删除 WAL 文件并重启 CometBFT。它会尝试与其他对等节点同步。
  2. 尝试手动修复 WAL 文件:
  1. 为损坏的 WAL 文件创建备份:
    cp "$CMTHOME/data/cs.wal/wal" > /tmp/corrupted_wal_backup
    
  2. 使用 ./scripts/wal2json 创建可读版本:
    ./scripts/wal2json/wal2json "$CMTHOME/data/cs.wal/wal" > /tmp/corrupted_wal
    
  3. 查找包含 “CORRUPTED MESSAGE” 的那一行。
  4. 结合损坏消息前一条消息、后一条消息以及日志内容,尝试重建该消息。如果后续消息也被标记为损坏(如果长度头损坏,或者某些写入未成功落盘到 WAL 中,即发生截断,就可能出现这种情况),那么请删除从损坏消息开始的所有行,然后重启 CometBFT。
    $EDITOR /tmp/corrupted_wal
    
  5. 编辑完成后,执行以下命令将该文件重新转换为二进制格式:
    ./scripts/json2wal/json2wal /tmp/corrupted_wal  $CMTHOME/data/cs.wal/wal
    

硬件

处理器与内存

实际规格会因负载和验证者数量而有所不同,但最低要求为:
  • 1GB RAM
  • 25GB 磁盘空间
  • 1.4 GHz CPU
对于高交易吞吐量的应用,建议优先使用 SSD 磁盘。 推荐配置:
  • 2GB RAM
  • 100GB SSD
  • x64 2.0 GHz 2v CPU
目前,CometBFT 会存储全部历史数据,因此随着时间推移可能需要大量磁盘空间。但我们计划实现状态同步(参见这个 issue)。因此,保存所有历史区块将不再是必要的。

在 32 位架构(或 ARM)上进行验证者签名

我们的 ed25519 和 secp256k1 实现都要求常量时间的 uint64 乘法运算。非固定时间的密码学实现可能会(并且已经)在 ed25519 和 secp256k1 中泄露私钥。32 位 x86 平台的硬件上不存在这种能力(来源),并且它还依赖编译器来保证其为常量时间。目前尚不清楚 Golang 编译器是否能对所有实现都正确做到这一点。 我们不支持也不建议在 32 位架构、“VIA Nano 2000 Series”,以及 ARM 章节中被评为 “S-” 的架构上运行验证者。

操作系统

得益于 Go 语言,CometBFT 可以编译到广泛的操作系统上( $OS/$ARCH 组合列表可在这里找到)。 虽然我们不偏向任何特定操作系统,但相比桌面操作系统(如 Mac OS),更安全、更稳定的 Linux 服务器发行版(如 CentOS)应当是更优选择。

杂项

注意:如果你打算在公共网络环境中使用 CometBFT,请务必阅读 Cosmos 网络中验证者的硬件建议。

配置参数

  • p2p.flush_throttle_timeout
  • p2p.max_packet_msg_payload_size
  • p2p.send_rate
  • p2p.recv_rate
如果你要在私有环境中使用 CometBFT,并且对等节点之间拥有私有高速网络,那么降低 flush throttle timeout 并提高其他参数是合理的。
[p2p]

send_rate=20000000 # 2MB/s
recv_rate=20000000 # 2MB/s
flush_throttle_timeout=10
max_packet_msg_payload_size=10240 # 10KB
  • mempool.recheck
每个区块之后,CometBFT 都会重新检查 mempool 中剩余的每一笔交易,以判断该区块中已提交的交易是否影响了应用状态,因此剩余的一些交易可能会变为无效。如果这不适用于你的应用,可以通过设置 mempool.recheck=false 来禁用它。
  • mempool.broadcast
将其设置为 false 会阻止 mempool 在交易被打包进区块之前向其他对等节点转发这些交易。这意味着在交易进入区块之前,只有你发送该交易的那个对等节点能看到它。
  • consensus.skip_timeout_commit
当涉及经济激励时,我们希望 skip_timeout_commit=false,因为提议者应该等待收到更多投票。但如果你不在意这一点,并希望获得最快的共识速度,就可以跳过它。在公共部署中(例如 Cosmos Hub),默认会保持为 false;而在企业应用中,将其设置为 true 并没有问题。
  • consensus.peer_gossip_sleep_duration
你可以尝试减少节点在检查是否有内容要发送给其对等节点之前的休眠时间。
  • consensus.timeout_commit
你也可以尝试降低 timeout_commit(即在提议下一个区块之前的休眠时间)。
  • p2p.addr_book_strict
默认情况下,CometBFT 会在将对等节点地址保存到地址簿之前检查该地址是否可路由。若 IP 有效且位于允许范围内,则该地址会被视为可路由。 而在私有网络或本地网络中,通常并不满足这一条件,因为你的 IP 范围通常是受限且私有的。在这种情况下,你需要将 addr_book_strict 设置为 false(关闭它)。
  • rpc.max_open_connections
默认情况下,并发连接数会受到限制,因为大多数操作系统只提供有限数量的文件描述符。 如果你希望接受更多连接,就需要提高这些限制。 用于调优系统以支持打开更多连接的 sysctl 参数 进程的文件数限制也必须提高,例如通过 ulimit -n 8192。 ……对于 N 个连接,例如 50k:
kern.maxfiles=10000+2*N         # BSD
kern.maxfilesperproc=100+2*N    # BSD
kern.ipc.maxsockets=10000+2*N   # BSD
fs.file-max=10000+2*N           # Linux
net.ipv4.tcp_max_orphans=N      # Linux

# For load-generating clients.
net.ipv4.ip_local_port_range="10000  65535"  # Linux.
net.inet.ip.portrange.first=10000  # BSD/Mac.
net.inet.ip.portrange.last=65535   # (Enough for N < 55535)
net.ipv4.tcp_tw_reuse=1         # Linux
net.inet.tcp.maxtcptw=2*N       # BSD

# If using netfilter on Linux:
net.netfilter.nf_conntrack_max=N
echo $((N/8)) > /sys/module/nf_conntrack/parameters/hashsize
还有一个用于限制 gRPC 连接数的类似选项:rpc.grpc_max_open_connections。

Database

By default, CometBFT uses the syndtr/goleveldb package for its in-process key-value database. If you want maximal performance, it may be best to install the real C implementation of LevelDB and compile CometBFT to use that using make build COMETBFT_BUILD_OPTIONS=cleveldb. See the install instructions for details. CometBFT keeps multiple distinct databases in the $CMTHOME/data:
  • blockstore.db: Keeps the entire blockchain - stores blocks, block commits, and block metadata, each indexed by height. Used to sync new peers.
  • evidence.db: Stores all verified evidence of misbehavior.
  • state.db: Stores the current blockchain state (i.e. height, validators, consensus params). Only grows if consensus params or validators change. Also used to temporarily store intermediate results during block processing.
  • tx_index.db: Indexes transactions by tx hash and height. The tx results are indexed if they are added to the FinalizeBlock response in the application.
By default, CometBFT will only index transactions by their hash and height. If you want the result events to be indexed, see indexing transactions for details. Applications can expose block pruning strategies to the node operator. Please read the documentation of your application to find out more details. Applications can use state sync to help nodes bootstrap quickly.

Logging

Default logging level (log_level = "main:info,state:info,statesync:info,*:error") should suffice for normal operation mode. Read this post for details on how to configure the log_level config variable. Some of the modules can be found here. If you’re trying to debug CometBFT or asked to provide logs with debug logging level, you can do so by running CometBFT with --log_level="*:debug".

Write Ahead Logs (WAL)

CometBFT uses write ahead logs for the consensus (cs.wal) and the mempool (mempool.wal). Both WALs have a max size of 1GB and are automatically rotated.

Consensus WAL

The consensus.wal is used to ensure we can recover from a crash at any point in the consensus state machine. It writes all consensus messages (timeouts, proposals, block parts, or votes) to a single file, flushing to disk before processing messages from its own validator. Since CometBFT validators are expected to never sign a conflicting vote, the WAL ensures we can always recover deterministically to the latest state of the consensus without using the network or re-signing any consensus messages. If your consensus.wal is corrupted, see below.

Mempool WAL

The mempool.wal logs all incoming transactions before running CheckTx, but is otherwise not used in any programmatic way. It’s just a kind of manual safeguard. Note the mempool provides no durability guarantees - a tx sent to one or many nodes may never make it into the blockchain if those nodes crash before being able to propose it. Clients must monitor their transactions by subscribing over websockets, polling for them, or using /broadcast_tx_commit. In the worst case, transactions can be resent from the mempool WAL manually. For the above reasons, the mempool.wal is disabled by default. To enable, set mempool.wal_dir to where you want the WAL to be located (e.g. data/mempool.wal).

DoS Exposure and Mitigation

Validators are supposed to set up Sentry Node Architecture to prevent Denial-of-Service attacks.

P2P

The core of the CometBFT peer-to-peer system is MConnection. Each connection has MaxPacketMsgPayloadSize, which is the maximum packet size and bounded send & receive queues. One can impose restrictions on send & receive rate per connection (SendRate, RecvRate). The number of open P2P connections can become quite large and hit the operating system’s open file limit (since TCP connections are considered files on UNIX-based systems). Nodes should be given a sizable open file limit, e.g. 8192, via ulimit -n 8192 or other deployment-specific mechanisms.

RPC

Attack Exposure and Mitigation

It is generally not recommended for RPC endpoints to be exposed publicly, and especially so if the node in question is a validator, as the CometBFT RPC does not currently provide advanced security features. Public exposure of RPC endpoints without appropriate protection can make the associated node vulnerable to a variety of attacks. It is entirely up to operators to ensure, if nodes’ RPC endpoints have to be exposed publicly, that appropriate measures have been taken to mitigate against attacks. Some examples of mitigation measures include, but are not limited to:
  • Never publicly exposing the RPC endpoints of validators (i.e. if the RPC endpoints absolutely have to be exposed, ensure you do so only on full nodes and with appropriate protection)
  • Correct usage of rate-limiting, authentication, and caching (e.g. as provided by reverse proxies like nginx and/or DDoS protection services like Cloudflare)
  • Only exposing the specific endpoints absolutely necessary for the relevant use cases (configurable via nginx/Cloudflare/etc.)
If no expertise is available to the operator to assist with securing nodes’ RPC endpoints, it is strongly recommended to never expose those endpoints publicly. Under no condition should any of the unsafe RPC endpoints ever be exposed publicly.

Endpoints Returning Multiple Entries

Endpoints returning multiple entries are limited by default to return 30 elements (100 max). See the RPC Documentation for more information.

Debugging CometBFT

If you ever have to debug CometBFT, the first thing you should probably do is check out the logs. See How to read logs, where we explain what certain log statements mean. If, after skimming through the logs, things are still not clear, the next thing to try is querying the /status RPC endpoint. It provides the necessary info: whether the node is syncing or not, what height it is on, etc.
curl http(s)://{ip}:{rpcPort}/status
/dump_consensus_state will give you a detailed overview of the consensus state (proposer, latest validators, peer states). From it, you should be able to figure out why, for example, the network had halted.
curl http(s)://{ip}:{rpcPort}/dump_consensus_state
There is a reduced version of this endpoint - /consensus_state, which returns just the votes seen at the current height. If, after consulting with the logs and the above endpoints, you still have no idea what’s happening, consider using the cometbft debug kill subcommand. This command will scrape all the available info and kill the process. See Debugging for the exact format. You can inspect the resulting archive yourself or create an issue on Github. Before opening an issue, however, be sure to check if there’s no existing issue already.

Monitoring CometBFT

Each CometBFT instance has a standard /health RPC endpoint, which responds with 200 (OK) if everything is fine and 500 (or no response) if something is wrong. Other useful endpoints include the previously mentioned /status, /net_info, and /validators. CometBFT can also report and serve Prometheus metrics. See Metrics. The cometbft debug dump subcommand can be used to periodically dump useful information into an archive. See Debugging for more information.

What happens when my app dies

You are supposed to run CometBFT under a process supervisor (like systemd or runit). It will ensure CometBFT is always running (despite possible errors). Getting back to the original question, if your application dies, CometBFT will panic. After a process supervisor restarts your application, CometBFT should be able to reconnect successfully. The order of restart does not matter for it.

Signal handling

We catch SIGINT and SIGTERM and try to clean up nicely. For other signals we use the default behavior in Go: Default behavior of signals in Go programs.

Corruption

NOTE: Make sure you have a backup of the CometBFT data directory.

Possible causes

Remember that most corruption is caused by hardware issues:
  • RAID controllers with faulty/worn out battery backup, and an unexpected power loss
  • Hard disk drives with write-back cache enabled, and an unexpected power loss
  • Cheap SSDs with insufficient power-loss protection, and an unexpected power loss
  • Defective RAM
  • Defective or overheating CPU(s)
Other causes can be:
  • Database systems configured with fsync=off and an OS crash or power loss
  • Filesystems configured to use write barriers plus a storage layer that ignores write barriers. LVM is a particular culprit.
  • CometBFT bugs
  • Operating system bugs
  • Admin error (e.g., directly modifying CometBFT data-directory contents)
(Source: https://wiki.postgresql.org/wiki/Corruption)

WAL Corruption

If consensus WAL is corrupted at the latest height and you are trying to start CometBFT, replay will fail with panic. Recovering from data corruption can be hard and time-consuming. Here are two approaches you can take:
  1. Delete the WAL file and restart CometBFT. It will attempt to sync with other peers.
  2. Try to repair the WAL file manually:
  1. Create a backup of the corrupted WAL file:
    cp "$CMTHOME/data/cs.wal/wal" > /tmp/corrupted_wal_backup
    
  2. Use ./scripts/wal2json to create a human-readable version:
    ./scripts/wal2json/wal2json "$CMTHOME/data/cs.wal/wal" > /tmp/corrupted_wal
    
  3. Search for a “CORRUPTED MESSAGE” line.
  4. By looking at the previous message and the message after the corrupted one and looking at the logs, try to rebuild the message. If the subsequent messages are marked as corrupted too (this may happen if the length header got corrupted or some writes did not make it to the WAL ~ truncation), then remove all the lines starting from the corrupted one and restart CometBFT.
    $EDITOR /tmp/corrupted_wal
    
  5. After editing, convert this file back into binary form by running:
    ./scripts/json2wal/json2wal /tmp/corrupted_wal  $CMTHOME/data/cs.wal/wal
    

Hardware

Processor and Memory

While actual specs vary depending on the load and validator count, minimal requirements are:
  • 1GB RAM
  • 25GB of disk space
  • 1.4 GHz CPU
SSD disks are preferable for applications with high transaction throughput. Recommended:
  • 2GB RAM
  • 100GB SSD
  • x64 2.0 GHz 2v CPU
While for now, CometBFT stores all the history and it may require significant disk space over time, we are planning to implement state syncing (See this issue). So, storing all the past blocks will not be necessary.

Validator signing on 32 bit architectures (or ARM)

Both our ed25519 and secp256k1 implementations require constant time uint64 multiplication. Non-constant time crypto can (and has) leaked private keys on both ed25519 and secp256k1. This doesn’t exist in hardware on 32 bit x86 platforms (source), and it depends on the compiler to enforce that it is constant time. It’s unclear at this point whether the Golang compiler does this correctly for all implementations. We do not support nor recommend running a validator on 32 bit architectures OR the “VIA Nano 2000 Series”, and the architectures in the ARM section rated “S-”.

Operating Systems

CometBFT can be compiled for a wide range of operating systems thanks to the Go language (the list of $OS/$ARCH pairs can be found here). While we do not favor any operating system, more secure and stable Linux server distributions (like CentOS) should be preferred over desktop operating systems (like Mac OS).

Miscellaneous

NOTE: If you are going to use CometBFT in a public domain, make sure you read hardware recommendations for a validator in the Cosmos network.

Configuration parameters

  • p2p.flush_throttle_timeout
  • p2p.max_packet_msg_payload_size
  • p2p.send_rate
  • p2p.recv_rate
If you are going to use CometBFT in a private domain and you have a private high-speed network among your peers, it makes sense to lower flush throttle timeout and increase other params.
[p2p]

send_rate=20000000 # 2MB/s
recv_rate=20000000 # 2MB/s
flush_throttle_timeout=10
max_packet_msg_payload_size=10240 # 10KB
  • mempool.recheck
After every block, CometBFT rechecks every transaction left in the mempool to see if transactions committed in that block affected the application state, so some of the transactions left may become invalid. If that does not apply to your application, you can disable it by setting mempool.recheck=false.
  • mempool.broadcast
Setting this to false will stop the mempool from relaying transactions to other peers until they are included in a block. It means only the peer you send the tx to will see it until it is included in a block.
  • consensus.skip_timeout_commit
We want skip_timeout_commit=false when there is economics on the line because proposers should wait to hear for more votes. But if you don’t care about that and want the fastest consensus, you can skip it. It will be kept false by default for public deployments (e.g. Cosmos Hub) while for enterprise applications, setting it to true is not a problem.
  • consensus.peer_gossip_sleep_duration
You can try to reduce the time your node sleeps before checking if there’s something to send its peers.
  • consensus.timeout_commit
You can also try lowering timeout_commit (time we sleep before proposing the next block).
  • p2p.addr_book_strict
By default, CometBFT checks whether a peer’s address is routable before saving it to the address book. The address is considered as routable if the IP is valid and within allowed ranges. This may not be the case for private or local networks, where your IP range is usually strictly limited and private. In that case, you need to set addr_book_strict to false (turn it off).
  • rpc.max_open_connections
By default, the number of simultaneous connections is limited because most OSes give you a limited number of file descriptors. If you want to accept a greater number of connections, you will need to increase these limits. Sysctls to tune the system to be able to open more connections The process file limits must also be increased, e.g. via ulimit -n 8192. …for N connections, such as 50k:
kern.maxfiles=10000+2*N         # BSD
kern.maxfilesperproc=100+2*N    # BSD
kern.ipc.maxsockets=10000+2*N   # BSD
fs.file-max=10000+2*N           # Linux
net.ipv4.tcp_max_orphans=N      # Linux

# For load-generating clients.
net.ipv4.ip_local_port_range="10000  65535"  # Linux.
net.inet.ip.portrange.first=10000  # BSD/Mac.
net.inet.ip.portrange.last=65535   # (Enough for N < 55535)
net.ipv4.tcp_tw_reuse=1         # Linux
net.inet.tcp.maxtcptw=2*N       # BSD

# If using netfilter on Linux:
net.netfilter.nf_conntrack_max=N
echo $((N/8)) > /sys/module/nf_conntrack/parameters/hashsize
A similar option exists for limiting the number of gRPC connections - rpc.grpc_max_open_connections.