​
变更记录

  • 2022 年 1 月 19 日：初始草案
  • 2022 年 4 月 29 日：更安全的扩展快照器接口

​
状态

已实现

​
摘要

本 ADR 概述了一种基于 hooks 的机制，使应用模块能够提供额外状态（位于 IAVL 树之外），以便在状态同步过程中使用。

​
背景

新客户端使用状态同步从对等节点下载模块状态快照。目前，快照由 SnapshotStoreItem 和 SnapshotIAVLItem 流组成，这意味着将状态定义在 IAVL 树之外的应用模块，无法将其状态纳入状态同步流程。 需要注意的是，即使模块状态数据位于树之外，为了保证确定性，我们仍要求将外部数据的哈希写入 IAVL 树中。

​
决策

基于现有实现的一个简单方案是新增两种消息类型：SnapshotExtensionMeta 和 SnapshotExtensionPayload。它们会追加到现有的 multi-store 流中，其中 SnapshotExtensionMeta 用作不同扩展之间的分隔符。由于 chunk 哈希已经能够保证数据完整性，因此我们不需要额外的分隔符来标记快照流的结束。 此外，我们为模块提供 Snapshotter 和 ExtensionSnapshotter 接口来实现快照器，它们将同时负责创建快照和恢复。每个模块可以拥有多个快照器；对于带有额外状态的模块，应将其实现为 ExtensionSnapshotter 扩展快照器。在设置应用时，快照 Manager 应调用 RegisterExtensions([]ExtensionSnapshotter…) 来注册所有扩展快照器。 
// SnapshotItem is an item contained in a rootmulti.Store snapshot.
// On top of the exsiting SnapshotStoreItem and SnapshotIAVLItem, we add two new options for the item.
message SnapshotItem {
  // item is the specific type of snapshot item.
  oneof item {
    SnapshotStoreItem        store             = 1;
    SnapshotIAVLItem         iavl              = 2 [(gogoproto.customname) = "IAVL"];
    SnapshotExtensionMeta    extension         = 3;
    SnapshotExtensionPayload extension_payload = 4;
  }
}

// SnapshotExtensionMeta contains metadata about an external snapshotter.
// One module may need multiple snapshotters, so each module may have multiple SnapshotExtensionMeta.
message SnapshotExtensionMeta {
  // the name of the ExtensionSnapshotter, and it is registered to snapshotter manager when setting up the application
  // name should be unique for each ExtensionSnapshotter as we need to alphabetically order their snapshots to get
  // deterministic snapshot stream.
  string name   = 1;
  // this is used by each ExtensionSnapshotter to decide the format of payloads included in SnapshotExtensionPayload message
  // it is used within the snapshotter/namespace, not global one for all modules
  uint32 format = 2;
}

// SnapshotExtensionPayload contains payloads of an external snapshotter.
message SnapshotExtensionPayload {
  bytes payload = 1;
}
创建快照流时，multistore 快照始终位于二进制流的开头，其他扩展快照则按对应 ExtensionSnapshotter 的名称按字母顺序排列。 快照流大致如下所示： 
// multi-store snapshot
{
    SnapshotStoreItem | SnapshotIAVLItem, ...
}
// extension1 snapshot
SnapshotExtensionMeta
{
    SnapshotExtensionPayload, ...
}
// extension2 snapshot
SnapshotExtensionMeta
{
    SnapshotExtensionPayload, ...
}
我们在快照 Manager 中新增一个 extensions 字段，用于保存扩展快照器。multistore 快照器是一个特殊情况，它不需要名称，因为它始终位于二进制流的开头。 
type Manager struct {
    store      *Store
	multistore types.Snapshotter
	extensions map[string]types.ExtensionSnapshotter
	mtx                sync.Mutex
	operation          operation
	chRestore          chan<- io.ReadCloser
	chRestoreDone      <-chan restoreDone
	restoreChunkHashes [][]byte
	restoreChunkIndex  uint32
}
对于实现了 ExtensionSnapshotter 接口的扩展快照器，需要在设置应用时调用 RegisterExtensions，将其名称注册到快照 Manager 中。这些快照器将同时负责创建快照和恢复。 
// RegisterExtensions register extension snapshotters to manager
func (m *Manager)

RegisterExtensions(extensions ...types.ExtensionSnapshotter)

error
在现有用于 multistore 的 Snapshotter 接口之上，我们为扩展快照器新增了 ExtensionSnapshotter 接口。该接口新增了三个函数签名：SnapshotFormat()、SupportedFormats() 和 SnapshotName()。 
// ExtensionPayloadReader read extension payloads,
// it returns io.EOF when reached either end of stream or the extension boundaries.
type ExtensionPayloadReader = func() ([]byte, error)

// ExtensionPayloadWriter is a helper to write extension payloads to underlying stream.
type ExtensionPayloadWriter = func([]byte)

error

// ExtensionSnapshotter is an extension Snapshotter that is appended to the snapshot stream.
// ExtensionSnapshotter has a unique name and manages its own internal formats.
type ExtensionSnapshotter interface {
	// SnapshotName returns the name of snapshotter, it should be unique in the manager.
	SnapshotName()

string

	// SnapshotFormat returns the default format used to take a snapshot.
	SnapshotFormat()

uint32

	// SupportedFormats returns a list of formats it can restore from.
	SupportedFormats() []uint32

	// SnapshotExtension writes extension payloads into the underlying protobuf stream.
	SnapshotExtension(height uint64, payloadWriter ExtensionPayloadWriter)

error

	// RestoreExtension restores an extension state snapshot,
	// the payload reader returns `io.EOF` when reached the extension boundaries.
	RestoreExtension(height uint64, format uint32, payloadReader ExtensionPayloadReader)

error
}

​
影响

通过这一实现，我们能够为维护在 IAVL 树之外的状态创建二进制 chunk 流快照，例如 CosmWasm blobs。与此同时，新客户端也能够从对等节点获取所有已实现对应接口模块的状态快照。

​
向后兼容性

本 ADR 引入了新的 proto 消息类型，在快照 Manager 中新增了 extensions 字段，并增加了新的 ExtensionSnapshotter 接口，因此如果存在扩展，这一变更不具备向后兼容性。 但对于那些所有模块都没有将状态数据存储在 IAVL 树之外的应用，其快照流仍然是向后兼容的。

​
正面影响

  • 通过实现扩展快照器，维护在 IAVL 树之外的状态（例如 CosmWasm blobs）也可以创建快照，并通过状态同步被新客户端获取。

​
负面影响

​
中性影响

  • 所有在 IAVL 树之外维护状态的模块都需要实现 ExtensionSnapshotter，并且快照 Manager 在设置应用时需要调用 RegisterExtensions。

​
后续讨论

当 ADR 处于 DRAFT 或 PROPOSED 阶段时，本节应包含未来迭代中需要解决的问题摘要（通常会引用 pull request 讨论中的评论）。 之后，本节也可以选择性列出作者或评审者在分析该 ADR 过程中发现的想法或改进点。

​
测试用例 [可选]

对于影响共识变更的 ADR，其实现必须提供测试用例。其他 ADR 则可以在适用时选择附上测试用例链接。

​
参考资料

​
Changelog

  • Jan 19, 2022: Initial Draft
  • Apr 29, 2022: Safer extension snapshotter interface

​
Status

Implemented

​
Abstract

This ADR outlines a hooks-based mechanism for application modules to provide additional state (outside of the IAVL tree) to be used during state sync.

​
Context

New clients use state-sync to download snapshots of module state from peers. Currently, the snapshot consists of a stream of SnapshotStoreItem and SnapshotIAVLItem, which means that application modules that define their state outside of the IAVL tree cannot include their state as part of the state-sync process. Note, Even though the module state data is outside of the tree, for determinism we require that the hash of the external data should be posted in the IAVL tree.

​
Decision

A simple proposal based on our existing implementation is that, we can add two new message types: SnapshotExtensionMeta and SnapshotExtensionPayload, and they are appended to the existing multi-store stream with SnapshotExtensionMeta acting as a delimiter between extensions. As the chunk hashes should be able to ensure data integrity, we don’t need a delimiter to mark the end of the snapshot stream. Besides, we provide Snapshotter and ExtensionSnapshotter interface for modules to implement snapshotters, which will handle both taking snapshot and the restoration. Each module could have mutiple snapshotters, and for modules with additional state, they should implement ExtensionSnapshotter as extension snapshotters. When setting up the application, the snapshot Manager should call RegisterExtensions([]ExtensionSnapshotter…) to register all the extension snapshotters. 
// SnapshotItem is an item contained in a rootmulti.Store snapshot.
// On top of the exsiting SnapshotStoreItem and SnapshotIAVLItem, we add two new options for the item.
message SnapshotItem {
  // item is the specific type of snapshot item.
  oneof item {
    SnapshotStoreItem        store             = 1;
    SnapshotIAVLItem         iavl              = 2 [(gogoproto.customname) = "IAVL"];
    SnapshotExtensionMeta    extension         = 3;
    SnapshotExtensionPayload extension_payload = 4;
  }
}

// SnapshotExtensionMeta contains metadata about an external snapshotter.
// One module may need multiple snapshotters, so each module may have multiple SnapshotExtensionMeta.
message SnapshotExtensionMeta {
  // the name of the ExtensionSnapshotter, and it is registered to snapshotter manager when setting up the application
  // name should be unique for each ExtensionSnapshotter as we need to alphabetically order their snapshots to get
  // deterministic snapshot stream.
  string name   = 1;
  // this is used by each ExtensionSnapshotter to decide the format of payloads included in SnapshotExtensionPayload message
  // it is used within the snapshotter/namespace, not global one for all modules
  uint32 format = 2;
}

// SnapshotExtensionPayload contains payloads of an external snapshotter.
message SnapshotExtensionPayload {
  bytes payload = 1;
}
When we create a snapshot stream, the multistore snapshot is always placed at the beginning of the binary stream, and other extension snapshots are alphabetically ordered by the name of the corresponding ExtensionSnapshotter. The snapshot stream would look like as follows: 
// multi-store snapshot
{
    SnapshotStoreItem | SnapshotIAVLItem, ...
}
// extension1 snapshot
SnapshotExtensionMeta
{
    SnapshotExtensionPayload, ...
}
// extension2 snapshot
SnapshotExtensionMeta
{
    SnapshotExtensionPayload, ...
}
We add an extensions field to snapshot Manager for extension snapshotters. The multistore snapshotter is a special one and it doesn’t need a name because it is always placed at the beginning of the binary stream. 
type Manager struct {
    store      *Store
	multistore types.Snapshotter
	extensions map[string]types.ExtensionSnapshotter
	mtx                sync.Mutex
	operation          operation
	chRestore          chan<- io.ReadCloser
	chRestoreDone      <-chan restoreDone
	restoreChunkHashes [][]byte
	restoreChunkIndex  uint32
}
For extension snapshotters that implement the ExtensionSnapshotter interface, their names should be registered to the snapshot Manager by calling RegisterExtensions when setting up the application. The snapshotters will handle both taking snapshot and restoration. 
// RegisterExtensions register extension snapshotters to manager
func (m *Manager)

RegisterExtensions(extensions ...types.ExtensionSnapshotter)

error
On top of the existing Snapshotter interface for the multistore, we add ExtensionSnapshotter interface for the extension snapshotters. Three more function signatures: SnapshotFormat(), SupportedFormats() and SnapshotName() are added to ExtensionSnapshotter. 
// ExtensionPayloadReader read extension payloads,
// it returns io.EOF when reached either end of stream or the extension boundaries.
type ExtensionPayloadReader = func() ([]byte, error)

// ExtensionPayloadWriter is a helper to write extension payloads to underlying stream.
type ExtensionPayloadWriter = func([]byte)

error

// ExtensionSnapshotter is an extension Snapshotter that is appended to the snapshot stream.
// ExtensionSnapshotter has a unique name and manages its own internal formats.
type ExtensionSnapshotter interface {
	// SnapshotName returns the name of snapshotter, it should be unique in the manager.
	SnapshotName()

string

	// SnapshotFormat returns the default format used to take a snapshot.
	SnapshotFormat()

uint32

	// SupportedFormats returns a list of formats it can restore from.
	SupportedFormats() []uint32

	// SnapshotExtension writes extension payloads into the underlying protobuf stream.
	SnapshotExtension(height uint64, payloadWriter ExtensionPayloadWriter)

error

	// RestoreExtension restores an extension state snapshot,
	// the payload reader returns `io.EOF` when reached the extension boundaries.
	RestoreExtension(height uint64, format uint32, payloadReader ExtensionPayloadReader)

error
}

​
Consequences

As a result of this implementation, we are able to create snapshots of binary chunk stream for the state that we maintain outside of the IAVL Tree, CosmWasm blobs for example. And new clients are able to fetch sanpshots of state for all modules that have implemented the corresponding interface from peer nodes.

​
Backwards Compatibility

This ADR introduces new proto message types, add an extensions field in snapshot Manager, and add new ExtensionSnapshotter interface, so this is not backwards compatible if we have extensions. But for applications that does not have the state data outside of the IAVL tree for any module, the snapshot stream is backwards-compatible.

​
Positive

  • State maintained outside of IAVL tree like CosmWasm blobs can create snapshots by implementing extension snapshotters, and being fetched by new clients via state-sync.

​
Negative

​
Neutral

  • All modules that maintain state outside of IAVL tree need to implement ExtensionSnapshotter and the snapshot Manager need to call RegisterExtensions when setting up the application.

​
Further Discussions

While an ADR is in the DRAFT or PROPOSED stage, this section should contain a summary of issues to be solved in future iterations (usually referencing comments from a pull-request discussion). Later, this section can optionally list ideas or improvements the author or reviewers found during the analysis of this ADR.

​
Test Cases [optional]

Test cases for an implementation are mandatory for ADRs that are affecting consensus changes. Other ADRs can choose to include links to test cases if applicable.

​
References