cosmovisor 是一个用于 Cosmos SDK 应用二进制文件的进程管理器,可在链升级时自动切换应用二进制文件。 它会轮询由 x/upgrade 模块在升级高度创建的 upgrade-info.json 文件,然后可以自动下载新的二进制文件、停止当前二进制文件、从旧二进制文件切换到新二进制文件,并最终使用新二进制文件重启节点。

设计

Cosmovisor 被设计为 Cosmos SDK 应用的包装器:
  • 它会将参数传递给关联的应用(由 DAEMON_NAME 环境变量配置)。 运行 cosmovisor run arg1 arg2 .... 将会执行 app arg1 arg2 ...;
  • 它会在需要时通过重启和升级来管理应用;
  • 它使用环境变量进行配置,而不是位置参数。
注意:如果应用的新版本没有设置为执行原地存储迁移,则必须先手动运行迁移,然后才能使用新二进制文件重启 cosmovisor。因此,我们建议应用采用原地存储迁移。
只有最新版本的 cosmovisor 会被积极开发和维护。
v1.0.0 之前的版本存在一个可能导致 DoS 的漏洞。请升级到最新版本。

贡献

Cosmovisor 是 Cosmos SDK 单仓库的一部分,但它是一个独立模块,拥有自己的发布节奏。 发布分支采用 release/cosmovisor/vA.B.x 格式,其中 A 和 B 是数字(例如 release/cosmovisor/v1.3.x)。发布标签采用以下格式:cosmovisor/vA.B.C。

设置

安装

你可以从 GitHub releases 下载 Cosmovisor。 要安装最新版 cosmovisor,运行以下命令:
go install cosmossdk.io/tools/cosmovisor/cmd/cosmovisor@latest
要安装特定版本,可以指定版本号:
go install cosmossdk.io/tools/cosmovisor/cmd/[email protected]
运行 cosmovisor version 以检查 cosmovisor 版本。 或者,如果要从源码构建,只需运行 make cosmovisor。生成的二进制文件将位于 tools/cosmovisor。
使用 go install 安装 cosmovisor 时,会显示正确的 cosmovisor 版本。 从源码构建(make cosmovisor)或通过其他方式安装 cosmovisor 时,则不会显示正确的版本。

命令行参数和环境变量

传递给 cosmovisor 的第一个参数是要让 cosmovisor 执行的操作。可选项包括:
  • help、--help 或 -h - 输出 cosmovisor 帮助信息,并检查你的 cosmovisor 配置。
  • run - 使用其余提供的参数运行已配置的二进制文件。
  • version - 输出 cosmovisor 版本,同时使用 version 参数运行该二进制文件。
  • config - 显示当前 cosmovisor 配置,即显示 cosmovisor 正在使用的环境变量值。
  • add-upgrade - 手动向 cosmovisor 添加一个升级。此命令允许你轻松将某次升级对应的二进制文件添加到 cosmovisor 中。
  • add-batch-upgrade - 一次性添加多个升级。
  • show-upgrade-info - 显示来自 upgrade-info.json 文件的当前升级信息。
所有传递给 cosmovisor run 的参数都会被传递给应用二进制文件(作为子进程)。cosmovisor 会将子进程的 /dev/stdout 和 /dev/stderr 作为自己的输出返回。因此,cosmovisor run 不能接受除应用二进制文件本身支持的参数之外的其他命令行参数。 cosmovisor 从环境变量或其配置文件中读取配置(使用 --cosmovisor-config <path>):
  • DAEMON_HOME 是保存 cosmovisor/ 目录的位置,该目录包含创世二进制文件、升级二进制文件,以及与每个二进制文件关联的其他辅助文件(例如 $HOME/.gaiad、$HOME/.regend、$HOME/.simd 等)。
  • DAEMON_NAME 是二进制文件本身的名称(例如 gaiad、regend、simd 等)。
  • DAEMON_ALLOW_DOWNLOAD_BINARIES(可选),如果设置为 true,将启用新二进制文件的自动下载(出于安全原因,这主要面向全节点而非验证者)。默认情况下,cosmovisor 不会自动下载新二进制文件。
  • DAEMON_DOWNLOAD_MUST_HAVE_CHECKSUM(可选,默认 = false),如果为 true,cosmovisor 将要求升级计划中为待下载的二进制文件提供校验和。如果为 false,cosmovisor 不会强制要求提供校验和,但如果提供了仍会进行校验。
  • DAEMON_RESTART_AFTER_UPGRADE(可选,默认 = true),如果为 true,升级成功后会使用相同的命令行参数和标志重新启动子进程(但使用新的二进制文件)。否则(false),cosmovisor 会在升级后停止运行,并要求系统管理员手动重启。注意,这里的重启只发生在升级之后,不会在子进程出错后自动重启。
  • DAEMON_RESTART_DELAY(可选,默认无),允许节点运营者定义节点停机(用于升级)与在指定时间后重新启动之间的延迟。该值必须是一个时长(例如 1s)。
  • DAEMON_SHUTDOWN_GRACE(可选,默认无),如果设置了该值,会先向二进制文件发送中断信号,并等待指定时间,以便在发送 kill 信号前完成清理或将缓存刷新到磁盘。该值必须是一个时长(例如 1s)。
  • DAEMON_POLL_INTERVAL(可选,默认 300 毫秒),是轮询升级计划文件的时间间隔。该值必须是一个时长(例如 1s)。
  • DAEMON_DATA_BACKUP_DIR 用于设置自定义备份目录。如果未设置,则使用 DAEMON_HOME。
  • UNSAFE_SKIP_BACKUP(默认 false),如果设置为 true,则直接升级而不执行备份。否则(false,默认),会在尝试升级前备份数据。默认值 false 在发生故障以及需要通过备份回滚时非常有用,也更推荐使用。我们建议使用默认备份选项 UNSAFE_SKIP_BACKUP=false。
  • DAEMON_PREUPGRADE_MAX_RETRIES(默认 0)。在收到退出状态 31 后,对 pre-upgrade 进行重试的最大次数。默认值 0 表示单次返回退出状态 31 就会立即导致升级失败。在重试次数耗尽后,Cosmovisor 会将该次升级标记为失败。
  • DAEMON_GRPC_ADDRESS(可选,默认 localhost:9090)。节点的 gRPC 地址,由 prepare-upgrade 命令和批量升级监视器使用。
  • COSMOVISOR_DISABLE_LOGS(默认 false)。如果设置为 true,将完全禁用 Cosmovisor 的日志(但不会禁用底层进程的日志)。例如,当你执行的某个 Cosmovisor 子命令返回有效 JSON,而你随后需要解析它时,这会很有用,因为 Cosmovisor 追加的日志会使输出不再是有效 JSON。
  • COSMOVISOR_COLOR_LOGS(默认 true)。如果设置为 true,将为 Cosmovisor 日志添加颜色(但不会影响底层进程)。
  • COSMOVISOR_TIMEFORMAT_LOGS(默认 kitchen)。如果设置为某个值(layout|ansic|unixdate|rubydate|rfc822|rfc822z|rfc850|rfc1123|rfc1123z|rfc3339|rfc3339nano|kitchen),将为 Cosmovisor 日志添加时间戳前缀(但不会影响底层进程)。
  • COSMOVISOR_CUSTOM_PREUPGRADE(默认 \`)。如果设置了该值,则会在升级之前,使用参数 [ upgrade.Name, upgrade.Height ]运行DAEMONHOME/cosmovisor/DAEMON_HOME/cosmovisor/COSMOVISOR_CUSTOM_PREUPGRADE`。这会执行一个自定义脚本(独立于链守护进程的 pre-upgrade 命令,且在其之前执行)。
  • COSMOVISOR_DISABLE_RECASE(默认 false)。如果设置为 true,则升级目录必须与升级计划名称完全匹配,不做任何大小写转换。

目录布局

$DAEMON_HOME/cosmovisor 应完全归 cosmovisor 及其所控制的子进程所有。目录内容组织如下:
.
├── current -> genesis or upgrades/<name>
├── genesis
│   └── bin
│       └── $DAEMON_NAME
└── upgrades
│   └── <name>
│       ├── bin
│       │   └── $DAEMON_NAME
│       └── upgrade-info.json
└── preupgrade.sh (optional)
cosmovisor/ 目录为应用的每个版本都包含一个子目录(即 genesis 或 upgrades/<name>)。每个子目录中都包含应用二进制文件(即 bin/$DAEMON_NAME)以及与该二进制文件关联的其他辅助文件。current 是一个指向当前活动目录(即 genesis 或 upgrades/<name>)的符号链接。upgrades/<name> 中的 name 变量是升级模块计划中指定的升级名称经过 URI 编码并转换为小写后的结果。请注意,升级名称路径会被规范化为小写:例如,MyUpgrade 会被规范化为 myupgrade,其路径为 upgrades/myupgrade。 请注意,$DAEMON_HOME/cosmovisor 只存储应用二进制文件。cosmovisor 二进制文件本身可以存放在任何常见位置(例如 /usr/local/bin)。应用仍会将其数据存储在默认数据目录(例如 $HOME/.simapp)或通过 --home 标志指定的数据目录中。$DAEMON_HOME 依赖于数据目录,必须设置为与数据目录相同的目录,因此你的配置最终会类似如下:
.simapp
├── config
├── data
└── cosmovisor

使用

系统管理员需要负责:
  • 安装 cosmovisor 二进制文件
  • 配置主机的 init 系统(例如 systemd、launchd 等)
  • 正确设置环境变量
  • 创建 <DAEMON_HOME>/cosmovisor 目录
  • 创建 <DAEMON_HOME>/cosmovisor/genesis/bin 目录
  • 创建 <DAEMON_HOME>/cosmovisor/upgrades/<name>/bin 目录
  • 将不同版本的 <DAEMON_NAME> 可执行文件放入对应的 bin 目录中。
cosmovisor 会在首次启动时(即不存在 current 链接时)将 current 链接指向 genesis,然后在正确的时间点处理二进制文件切换,这样系统管理员就可以提前数天完成准备,并在升级时从容应对。 为了支持可下载的二进制文件,需要将每个升级二进制文件打包成 tarball,并通过规范 URL 提供下载。此外,也可以将包含创世二进制文件和所有可用升级二进制文件的 tarball 一并打包并提供,这样从头开始同步全节点所需的所有二进制文件都可以方便地下载。 DAEMON 专属的代码和操作(例如 CometBFT 配置、应用数据库、区块同步等)都会按预期工作。应用二进制文件的指令,例如命令行标志和环境变量,也都会按预期工作。

初始化

cosmovisor init <path to executable> 命令会创建使用 cosmovisor 所需的目录结构。 它会执行以下操作:
  • 如果 <DAEMON_HOME>/cosmovisor 文件夹尚不存在,则创建它
  • 如果 <DAEMON_HOME>/cosmovisor/genesis/bin 文件夹尚不存在,则创建它
  • 将提供的可执行文件复制到 <DAEMON_HOME>/cosmovisor/genesis/bin/<DAEMON_NAME>
  • 创建指向 genesis 文件夹的 current 链接
它使用 DAEMON_HOME 和 DAEMON_NAME 环境变量来确定目录位置和可执行文件名。 cosmovisor init 命令专门用于初始化 cosmovisor,不应与链自身的 init 命令混淆(例如 cosmovisor run init)。

检测升级

cosmovisor 会轮询 $DAEMON_HOME/data/upgrade-info.json 文件,以检查新的升级指令。当检测到升级且区块链到达升级高度时,x/upgrade 模块会在 BeginBlocker 中创建该文件。 检测升级时会应用以下启发式规则:
  • 启动时,cosmovisor 对当前正在运行的升级了解不多,除了当前二进制文件位于 current/bin/。它会尝试读取 current/upgrade-info.json 文件,以获取当前升级名称的信息。
  • 如果 cosmovisor/current/upgrade-info.json 和 data/upgrade-info.json 都不存在,那么 cosmovisor 会等待 data/upgrade-info.json 文件出现来触发升级。
  • 如果 cosmovisor/current/upgrade-info.json 不存在,但 data/upgrade-info.json 存在,那么 cosmovisor 会假定 data/upgrade-info.json 中的内容是一个有效的升级请求。在这种情况下,cosmovisor 会立即尝试根据 data/upgrade-info.json 中的 name 属性执行升级。
  • 否则,cosmovisor 会等待 upgrade-info.json 发生变化。一旦文件中记录了新的升级名称,cosmovisor 就会触发升级机制。
当升级机制被触发时,cosmovisor 将会:
  1. 如果启用了 DAEMON_ALLOW_DOWNLOAD_BINARIES,首先自动将新的二进制文件下载到 cosmovisor/<name>/bin(其中 <name> 是 upgrade-info.json:name 属性);
  2. 更新 current 符号链接,使其指向新目录,并将 data/upgrade-info.json 保存到 cosmovisor/current/upgrade-info.json。

添加升级二进制文件

cosmovisor 提供了一个 add-upgrade 命令,用于方便地将某个二进制文件关联到升级。它会在 cosmovisor/upgrades/<name> 中创建一个新目录,并将提供的可执行文件复制到 cosmovisor/upgrades/<name>/bin/<DAEMON_NAME>。 使用 --upgrade-height 标志可以指定在什么高度切换二进制文件,而无需通过治理提案。 这使得紧急协同升级成为可能,即必须在特定高度切换二进制文件,但没有时间走治理提案流程。
--upgrade-height 会创建一个 upgrade-info.json 文件。这意味着,如果在通过 --upgrade-height 指定的高度之前,链通过治理提案执行了升级,那么治理提案会覆盖由 add-upgrade --upgrade-height <height> 创建的 upgrade-info.json 计划。 使用 --upgrade-height 时请考虑这一点。

自动下载

通常,cosmovisor 要求系统管理员在升级发生前将所有相关二进制文件放到磁盘上。不过,对于不需要这种控制、希望实现自动化部署的用户(例如正在同步非验证者全节点并希望尽量减少维护工作),还有另一种选择。 注意:我们不建议使用自动下载,因为它不会提前验证二进制文件是否可用。如果下载二进制文件时出现任何问题,cosmovisor 会停止,并且不会重新启动 App(这可能导致链停摆)。 如果 DAEMON_ALLOW_DOWNLOAD_BINARIES 设置为 true,并且在触发升级时找不到本地二进制文件,cosmovisor 将尝试根据 data/upgrade-info.json 文件中 info 属性里的指令,自行下载并安装该二进制文件。该文件由 x/upgrade 模块构造,包含来自升级 Plan 对象的数据。Plan 有一个 info 字段,预期使用以下两种有效格式之一来指定下载信息:
  1. 在升级计划的 info 字段中,以 JSON 格式在 `

更新应用

将应用更新到最新版本(例如 v0.50.0)。
迁移计划使用 x/upgrade 模块定义,并在升级模块中说明。迁移可以执行任何确定性的状态变更。用于将 simapp 从 v0.47 升级到 v0.50 的迁移计划定义在 simapp/upgrade.go 中。
构建新的 simd 二进制:
make build
添加新的 simd 二进制和升级名称:
迁移名称必须与迁移计划中定义的名称一致。
cosmovisor add-upgrade v047-to-v050 ./build/simd
打开一个新的终端窗口,并提交升级提案,同时完成保证金和投票操作(这些命令必须在 20 秒内相继执行):
./build/simd tx upgrade software-upgrade v047-to-v050 --title upgrade --summary upgrade --upgrade-height 200 --upgrade-info "{}" --no-validate --from validator --yes
./build/simd tx gov deposit 1 10000000stake --from validator --yes
./build/simd tx gov vote 1 yes --from validator --yes
升级会在高度 200 时自动发生。注意:如果你的测试过程耗时更长,可能需要修改上面示例中的升级高度。

升级前处理

Cosmovisor 支持自定义升级前处理。当你需要在执行升级之前,为较新版本实现必需的应用配置变更时,应使用升级前处理。如果未实现升级前处理,升级会正常继续。 在应用二进制完成升级之前,Cosmovisor 会调用一个可由应用实现的 pre-upgrade 命令。pre-upgrade 命令不接收任何命令行参数,并且应以下列退出码结束:
退出状态码在 Cosmovisor 中的处理方式
0pre-upgrade 命令执行成功。Cosmovisor 继续升级。
1pre-upgrade 命令未实现。Cosmovisor 正常继续升级。
30pre-upgrade 命令失败。Cosmovisor 使整个升级失败。
31pre-upgrade 命令失败。Cosmovisor 会持续重试,直到返回退出码 1 或 30,或者 DAEMON_PREUPGRADE_MAX_RETRIES 规定的重试次数耗尽(此时升级失败)。
退出码 31 允许的重试次数通过 DAEMON_PREUPGRADE_MAX_RETRIES 配置(默认值为 0,表示不重试,即单次返回 exit-31 就会立即导致升级失败)。 pre-upgrade 命令实现示例:
func preUpgradeCommand() *cobra.Command {
    return &cobra.Command{
        Use:   "pre-upgrade",
        Short: "Pre-upgrade command",
        Run: func(cmd *cobra.Command, args []string) {
            if err := HandlePreUpgrade(); err != nil {
                os.Exit(30)
            }
            os.Exit(0)
        },
    }
}
在根命令中注册它:
rootCmd.AddCommand(
    // ..
    preUpgradeCommand(),
)
如果不使用 Cosmovisor,请先安装新的二进制,然后在启动它之前运行 <new-appd> pre-upgrade。pre-upgrade 命令属于新的二进制,而不是旧的。
cosmovisor is a process manager for Cosmos SDK application binaries that automates application binary switch at chain upgrades. It polls the upgrade-info.json file that is created by the x/upgrade module at upgrade height, and then can automatically download the new binary, stop the current binary, switch from the old binary to the new one, and finally restart the node with the new binary.

Design

Cosmovisor is designed to be used as a wrapper for a Cosmos SDK app:
  • it will pass arguments to the associated app (configured by DAEMON_NAME env variable). Running cosmovisor run arg1 arg2 .... will run app arg1 arg2 ...;
  • it will manage an app by restarting and upgrading if needed;
  • it is configured using environment variables, not positional arguments.
Note: If new versions of the application are not set up to run in-place store migrations, migrations will need to be run manually before restarting cosmovisor with the new binary. For this reason, we recommend applications adopt in-place store migrations.
Only the latest version of cosmovisor is actively developed/maintained.
Versions prior to v1.0.0 have a vulnerability that could lead to a DOS. Please upgrade to the latest version.

Contributing

Cosmovisor is part of the Cosmos SDK monorepo, but it’s a separate module with its own release schedule. Release branches have the following format release/cosmovisor/vA.B.x, where A and B are a number (e.g. release/cosmovisor/v1.3.x). Releases are tagged using the following format: cosmovisor/vA.B.C.

Setup

Installation

You can download Cosmovisor from the GitHub releases. To install the latest version of cosmovisor, run the following command:
go install cosmossdk.io/tools/cosmovisor/cmd/cosmovisor@latest
To install a specific version, you can specify the version:
go install cosmossdk.io/tools/cosmovisor/cmd/[email protected]
Run cosmovisor version to check the cosmovisor version. Alternatively, for building from source, simply run make cosmovisor. The binary will be located in tools/cosmovisor.
Installing cosmovisor using go install will display the correct cosmovisor version. Building from source (make cosmovisor) or installing cosmovisor by other means won’t display the correct version.

Command Line Arguments And Environment Variables

The first argument passed to cosmovisor is the action for cosmovisor to take. Options are:
  • help, --help, or -h - Output cosmovisor help information and check your cosmovisor configuration.
  • run - Run the configured binary using the rest of the provided arguments.
  • version - Output the cosmovisor version and also run the binary with the version argument.
  • config - Display the current cosmovisor configuration, that means displaying the environment variables value that cosmovisor is using.
  • add-upgrade - Add an upgrade manually to cosmovisor. This command allow you to easily add the binary corresponding to an upgrade in cosmovisor.
  • add-batch-upgrade - Add multiple upgrades at once.
  • show-upgrade-info - Show the current upgrade info from the upgrade-info.json file.
All arguments passed to cosmovisor run will be passed to the application binary (as a subprocess). cosmovisor will return /dev/stdout and /dev/stderr of the subprocess as its own. For this reason, cosmovisor run cannot accept any command-line arguments other than those available to the application binary. cosmovisor reads its configuration from environment variables, or its configuration file (use --cosmovisor-config <path>):
  • DAEMON_HOME is the location where the cosmovisor/ directory is kept that contains the genesis binary, the upgrade binaries, and any additional auxiliary files associated with each binary (e.g. $HOME/.gaiad, $HOME/.regend, $HOME/.simd, etc.).
  • DAEMON_NAME is the name of the binary itself (e.g. gaiad, regend, simd, etc.).
  • DAEMON_ALLOW_DOWNLOAD_BINARIES (optional), if set to true, will enable auto-downloading of new binaries (for security reasons, this is intended for full nodes rather than validators). By default, cosmovisor will not auto-download new binaries.
  • DAEMON_DOWNLOAD_MUST_HAVE_CHECKSUM (optional, default = false), if true cosmovisor will require that a checksum is provided in the upgrade plan for the binary to be downloaded. If false, cosmovisor will not require a checksum to be provided, but still check the checksum if one is provided.
  • DAEMON_RESTART_AFTER_UPGRADE (optional, default = true), if true, restarts the subprocess with the same command-line arguments and flags (but with the new binary) after a successful upgrade. Otherwise (false), cosmovisor stops running after an upgrade and requires the system administrator to manually restart it. Note restart is only after the upgrade and does not auto-restart the subprocess after an error occurs.
  • DAEMON_RESTART_DELAY (optional, default none), allow a node operator to define a delay between the node halt (for upgrade) and backup by the specified time. The value must be a duration (e.g. 1s).
  • DAEMON_SHUTDOWN_GRACE (optional, default none), if set, send interrupt to binary and wait the specified time to allow for cleanup/cache flush to disk before sending the kill signal. The value must be a duration (e.g. 1s).
  • DAEMON_POLL_INTERVAL (optional, default 300 milliseconds), is the interval length for polling the upgrade plan file. The value must be a duration (e.g. 1s).
  • DAEMON_DATA_BACKUP_DIR option to set a custom backup directory. If not set, DAEMON_HOME is used.
  • UNSAFE_SKIP_BACKUP (defaults to false), if set to true, upgrades directly without performing a backup. Otherwise (false, default) backs up the data before trying the upgrade. The default value of false is useful and recommended in case of failures and when a backup needed to rollback. We recommend using the default backup option UNSAFE_SKIP_BACKUP=false.
  • DAEMON_PREUPGRADE_MAX_RETRIES (defaults to 0). The maximum number of times to retry pre-upgrade after exit status of 31. With the default of 0, a single exit-31 result immediately fails the upgrade. After retries are exhausted, Cosmovisor fails the upgrade.
  • DAEMON_GRPC_ADDRESS (optional, default localhost:9090). The gRPC address of the node, used by the prepare-upgrade command and the batch upgrade watcher.
  • COSMOVISOR_DISABLE_LOGS (defaults to false). If set to true, this will disable Cosmovisor logs (but not the underlying process) completely. This may be useful, for example, when a Cosmovisor subcommand you are executing returns a valid JSON you are then parsing, as logs added by Cosmovisor make this output not a valid JSON.
  • COSMOVISOR_COLOR_LOGS (defaults to true). If set to true, this will colorize Cosmovisor logs (but not the underlying process).
  • COSMOVISOR_TIMEFORMAT_LOGS (defaults to kitchen). If set to a value (layout|ansic|unixdate|rubydate|rfc822|rfc822z|rfc850|rfc1123|rfc1123z|rfc3339|rfc3339nano|kitchen), this will add timestamp prefix to Cosmovisor logs (but not the underlying process).
  • COSMOVISOR_CUSTOM_PREUPGRADE (defaults to “). If set, this will run DAEMON_HOME/cosmovisor/DAEMON\_HOME/cosmovisor/COSMOVISOR_CUSTOM_PREUPGRADE prior to upgrade with the arguments [ upgrade.Name, upgrade.Height ]. Executes a custom script (separate and prior to the chain daemon pre-upgrade command)
  • COSMOVISOR_DISABLE_RECASE (defaults to false). If set to true, the upgrade directory will expected to match the upgrade plan name without any case changes

Folder Layout

$DAEMON_HOME/cosmovisor is expected to belong completely to cosmovisor and the subprocesses that are controlled by it. The folder content is organized as follows:
.
├── current -> genesis or upgrades/<name>
├── genesis
│   └── bin
│       └── $DAEMON_NAME
└── upgrades
│   └── <name>
│       ├── bin
│       │   └── $DAEMON_NAME
│       └── upgrade-info.json
└── preupgrade.sh (optional)
The cosmovisor/ directory includes a subdirectory for each version of the application (i.e. genesis or upgrades/<name>). Within each subdirectory is the application binary (i.e. bin/$DAEMON_NAME) and any additional auxiliary files associated with each binary. current is a symbolic link to the currently active directory (i.e. genesis or upgrades/<name>). The name variable in upgrades/<name> is the lowercased URI-encoded name of the upgrade as specified in the upgrade module plan. Note that the upgrade name path are normalized to be lowercased: for instance, MyUpgrade is normalized to myupgrade, and its path is upgrades/myupgrade. Please note that $DAEMON_HOME/cosmovisor only stores the application binaries. The cosmovisor binary itself can be stored in any typical location (e.g. /usr/local/bin). The application will continue to store its data in the default data directory (e.g. $HOME/.simapp) or the data directory specified with the --home flag. $DAEMON_HOME is dependent of the data directory and must be set to the same directory as the data directory, you will end up with a configuration like the following:
.simapp
├── config
├── data
└── cosmovisor

Usage

The system administrator is responsible for:
  • installing the cosmovisor binary
  • configuring the host’s init system (e.g. systemd, launchd, etc.)
  • appropriately setting the environmental variables
  • creating the <DAEMON_HOME>/cosmovisor directory
  • creating the <DAEMON_HOME>/cosmovisor/genesis/bin folder
  • creating the <DAEMON_HOME>/cosmovisor/upgrades/<name>/bin folders
  • placing the different versions of the <DAEMON_NAME> executable in the appropriate bin folders.
cosmovisor will set the current link to point to genesis at first start (i.e. when no current link exists) and then handle switching binaries at the correct points in time so that the system administrator can prepare days in advance and relax at upgrade time. In order to support downloadable binaries, a tarball for each upgrade binary will need to be packaged up and made available through a canonical URL. Additionally, a tarball that includes the genesis binary and all available upgrade binaries can be packaged up and made available so that all the necessary binaries required to sync a fullnode from start can be easily downloaded. The DAEMON specific code and operations (e.g. CometBFT config, the application db, syncing blocks, etc.) all work as expected. The application binaries’ directives such as command-line flags and environment variables also work as expected.

Initialization

The cosmovisor init <path to executable> command creates the folder structure required for using cosmovisor. It does the following:
  • creates the <DAEMON_HOME>/cosmovisor folder if it doesn’t yet exist
  • creates the <DAEMON_HOME>/cosmovisor/genesis/bin folder if it doesn’t yet exist
  • copies the provided executable file to <DAEMON_HOME>/cosmovisor/genesis/bin/<DAEMON_NAME>
  • creates the current link, pointing to the genesis folder
It uses the DAEMON_HOME and DAEMON_NAME environment variables for folder location and executable name. The cosmovisor init command is specifically for initializing cosmovisor, and should not be confused with a chain’s init command (e.g. cosmovisor run init).

Detecting Upgrades

cosmovisor is polling the $DAEMON_HOME/data/upgrade-info.json file for new upgrade instructions. The file is created by the x/upgrade module in BeginBlocker when an upgrade is detected and the blockchain reaches the upgrade height. The following heuristic is applied to detect the upgrade:
  • When starting, cosmovisor doesn’t know much about currently running upgrade, except the binary which is current/bin/. It tries to read the current/upgrade-info.json file to get information about the current upgrade name.
  • If neither cosmovisor/current/upgrade-info.json nor data/upgrade-info.json exist, then cosmovisor will wait for data/upgrade-info.json file to trigger an upgrade.
  • If cosmovisor/current/upgrade-info.json doesn’t exist but data/upgrade-info.json exists, then cosmovisor assumes that whatever is in data/upgrade-info.json is a valid upgrade request. In this case cosmovisor tries immediately to make an upgrade according to the name attribute in data/upgrade-info.json.
  • Otherwise, cosmovisor waits for changes in upgrade-info.json. As soon as a new upgrade name is recorded in the file, cosmovisor will trigger an upgrade mechanism.
When the upgrade mechanism is triggered, cosmovisor will:
  1. if DAEMON_ALLOW_DOWNLOAD_BINARIES is enabled, start by auto-downloading a new binary into cosmovisor/<name>/bin (where <name> is the upgrade-info.json:name attribute);
  2. update the current symbolic link to point to the new directory and save data/upgrade-info.json to cosmovisor/current/upgrade-info.json.

Adding Upgrade Binary

cosmovisor has an add-upgrade command that allows to easily link a binary to an upgrade. It creates a new folder in cosmovisor/upgrades/<name> and copies the provided executable file to cosmovisor/upgrades/<name>/bin/<DAEMON_NAME>. Using the --upgrade-height flag allows you to specify at which height the binary should be switched, without going via a governance proposal. This enables support for an emergency coordinated upgrades where the binary must be switched at a specific height, but there is no time to go through a governance proposal.
--upgrade-height creates an upgrade-info.json file. This means if a chain upgrade via governance proposal is executed before the specified height with --upgrade-height, the governance proposal will overwrite the upgrade-info.json plan created by add-upgrade --upgrade-height <height>. Take this into consideration when using --upgrade-height.

Auto-Download

Generally, cosmovisor requires that the system administrator place all relevant binaries on disk before the upgrade happens. However, for people who don’t need such control and want an automated setup (maybe they are syncing a non-validating fullnode and want to do little maintenance), there is another option. NOTE: we don’t recommend using auto-download because it doesn’t verify in advance if a binary is available. If there will be any issue with downloading a binary, the cosmovisor will stop and won’t restart an App (which could lead to a chain halt). If DAEMON_ALLOW_DOWNLOAD_BINARIES is set to true, and no local binary can be found when an upgrade is triggered, cosmovisor will attempt to download and install the binary itself based on the instructions in the info attribute in the data/upgrade-info.json file. The files is constructed by the x/upgrade module and contains data from the upgrade Plan object. The Plan has an info field that is expected to have one of the following two valid formats to specify a download:
  1. Store an os/architecture -> binary URI map in the upgrade plan info field as JSON under the "binaries" key. For example:
    {
      "binaries": {
        "linux/amd64": "https://example.com/gaia.zip?checksum=sha256:aec070645fe53ee3b3763059376134f058cc337247c978add178b6ccdfb0019f"
      }
    }
    
    You can include multiple binaries at once to ensure more than one environment will receive the correct binaries:
    {
      "binaries": {
        "linux/amd64": "https://example.com/gaia.zip?checksum=sha256:aec070645fe53ee3b3763059376134f058cc337247c978add178b6ccdfb0019f",
        "linux/arm64": "https://example.com/gaia.zip?checksum=sha256:aec070645fe53ee3b3763059376134f058cc337247c978add178b6ccdfb0019f",
        "darwin/amd64": "https://example.com/gaia.zip?checksum=sha256:aec070645fe53ee3b3763059376134f058cc337247c978add178b6ccdfb0019f"
      }
    }
    
    When submitting this as a proposal ensure there are no spaces. An example command using gaiad could look like:
    > gaiad tx upgrade software-upgrade Vega \
    --title Vega \
    --deposit 100uatom \
    --upgrade-height 7368420 \
    --upgrade-info '{"binaries":{"linux/amd64":"https://github.com/cosmos/gaia/releases/download/v6.0.0-rc1/gaiad-v6.0.0-rc1-linux-amd64","linux/arm64":"https://github.com/cosmos/gaia/releases/download/v6.0.0-rc1/gaiad-v6.0.0-rc1-linux-arm64","darwin/amd64":"https://github.com/cosmos/gaia/releases/download/v6.0.0-rc1/gaiad-v6.0.0-rc1-darwin-amd64"}}' \
    --summary "upgrade to Vega" \
    --gas 400000 \
    --from user \
    --chain-id test \
    --home test/val2 \
    --node tcp://localhost:36657 \
    --yes
    
  2. Store a link to a file that contains all information in the above format (e.g. if you want to specify lots of binaries, changelog info, etc. without filling up the blockchain). For example:
    https://example.com/testnet-1001-info.json?checksum=sha256:deaaa99fda9407c4dbe1d04bd49bab0cc3c1dd76fa392cd55a9425be074af01e
    
When cosmovisor is triggered to download the new binary, cosmovisor will parse the "binaries" field, download the new binary with go-getter, and unpack the new binary in the upgrades/<name> folder so that it can be run as if it was installed manually. Note that for this mechanism to provide strong security guarantees, all URLs should include a SHA 256/512 checksum. This ensures that no false binary is run, even if someone hacks the server or hijacks the DNS. go-getter will always ensure the downloaded file matches the checksum if it is provided. go-getter will also handle unpacking archives into directories (in this case the download link should point to a zip file of all data in the bin directory). To properly create a sha256 checksum on linux, you can use the sha256sum utility. For example:
sha256sum ./testdata/repo/zip_directory/autod.zip
The result will look something like the following: 29139e1381b8177aec909fab9a75d11381cab5adf7d3af0c05ff1c9c117743a7. You can also use sha512sum if you would prefer to use longer hashes, or md5sum if you would prefer to use broken hashes. Whichever you choose, make sure to set the hash algorithm properly in the checksum argument to the URL.

Preparing for an Upgrade

To prepare for an upgrade, use the prepare-upgrade command:
cosmovisor prepare-upgrade
This command performs the following actions:
  1. Retrieves upgrade information directly from the blockchain about the next scheduled upgrade.
  2. Downloads the new binary specified in the upgrade plan.
  3. Verifies the binary’s checksum (if required by configuration).
  4. Places the new binary in the appropriate directory for Cosmovisor to use during the upgrade.
This command requires gRPC to be enabled on the node (configured via DAEMON_GRPC_ADDRESS, default localhost:9090). The prepare-upgrade command logs the following:
  • The name and height of the upcoming upgrade
  • The URL from which the new binary is being downloaded
  • Confirmation of successful completion
Example output:
INFO Preparing for upgrade name=v1.0.0 height=1000000
INFO Downloading upgrade binary url=https://example.com/binary/v1.0.0?checksum=sha256:339911508de5e20b573ce902c500ee670589073485216bee8b045e853f24bce8
INFO Upgrade preparation complete name=v1.0.0 height=1000000
Note: The current way of downloading manually and placing the binary at the right place would still work.

Example: SimApp Upgrade

The following instructions provide a demonstration of cosmovisor using the simulation application (simapp) shipped with the Cosmos SDK’s source code. The following commands are to be run from within the cosmos-sdk repository.

Chain Setup

Let’s create a new chain using the v0.47.4 version of simapp (the Cosmos SDK demo app):
git checkout v0.47.4
make build
Clean ~/.simapp (never do this in a production environment):
./build/simd tendermint unsafe-reset-all
Set up app config:
./build/simd config chain-id test
./build/simd config keyring-backend test
./build/simd config broadcast-mode sync
Initialize the node and overwrite any previous genesis file (never do this in a production environment):
./build/simd init test --chain-id test --overwrite
For the sake of this demonstration, amend voting_period in genesis.json to a reduced time of 20 seconds (20s):
cat <<< $(jq '.app_state.gov.params.voting_period = "20s"' $HOME/.simapp/config/genesis.json) > $HOME/.simapp/config/genesis.json
Create a validator, and setup genesis transaction:
./build/simd keys add validator
./build/simd genesis add-genesis-account validator 1000000000stake --keyring-backend test
./build/simd genesis gentx validator 1000000stake --chain-id test
./build/simd genesis collect-gentxs

Prepare Cosmovisor and Start the Chain

Set the required environment variables:
export DAEMON_NAME=simd
export DAEMON_HOME=$HOME/.simapp
Set the optional environment variable to trigger an automatic app restart:
export DAEMON_RESTART_AFTER_UPGRADE=true
Initialize cosmovisor with the current binary:
cosmovisor init ./build/simd
Now you can run cosmovisor with simapp v0.47.4:
cosmovisor run start

Update App

Update app to the latest version (e.g. v0.50.0).
Migration plans are defined using the x/upgrade module and described in Upgrading Modules. Migrations can perform any deterministic state change.The migration plan to upgrade the simapp from v0.47 to v0.50 is defined in simapp/upgrade.go.
Build the new version simd binary:
make build
Add the new simd binary and the upgrade name:
The migration name must match the one defined in the migration plan.
cosmovisor add-upgrade v047-to-v050 ./build/simd
Open a new terminal window and submit an upgrade proposal along with a deposit and a vote (these commands must be run within 20 seconds of each other):
./build/simd tx upgrade software-upgrade v047-to-v050 --title upgrade --summary upgrade --upgrade-height 200 --upgrade-info "{}" --no-validate --from validator --yes
./build/simd tx gov deposit 1 10000000stake --from validator --yes
./build/simd tx gov vote 1 yes --from validator --yes
The upgrade will occur automatically at height 200. Note: you may need to change the upgrade height in the snippet above if your test play takes more time.

Pre-Upgrade Handling

Cosmovisor supports custom pre-upgrade handling. Use pre-upgrade handling when you need to implement application config changes that are required in the newer version before you perform the upgrade. If pre-upgrade handling is not implemented, the upgrade continues normally. Before the application binary is upgraded, Cosmovisor calls a pre-upgrade command that can be implemented by the application. The pre-upgrade command does not take in any command-line arguments and is expected to terminate with the following exit codes:
Exit status codeHow it is handled in Cosmovisor
0pre-upgrade command executed successfully. Cosmovisor continues the upgrade.
1pre-upgrade command is not implemented. Cosmovisor continues the upgrade normally.
30pre-upgrade command failed. Cosmovisor fails the entire upgrade.
31pre-upgrade command failed. Cosmovisor retries until exit code 1 or 30 are returned, or until DAEMON_PREUPGRADE_MAX_RETRIES retries are exhausted (at which point the upgrade fails).
The number of allowed retries for exit code 31 is configured via DAEMON_PREUPGRADE_MAX_RETRIES (defaults to 0, meaning no retries — a single exit-31 result immediately fails the upgrade). Sample pre-upgrade command implementation:
func preUpgradeCommand() *cobra.Command {
    return &cobra.Command{
        Use:   "pre-upgrade",
        Short: "Pre-upgrade command",
        Run: func(cmd *cobra.Command, args []string) {
            if err := HandlePreUpgrade(); err != nil {
                os.Exit(30)
            }
            os.Exit(0)
        },
    }
}
Register it in the root command:
rootCmd.AddCommand(
    // ..
    preUpgradeCommand(),
)
When not using Cosmovisor, install the new binary first, then run <new-appd> pre-upgrade before starting it. The pre-upgrade command is part of the new binary, not the old one.