​
变更记录

  • 30/11/2022：提议中

​
状态

提议中 - 已实现

​
摘要

我们提议引入一个简化的模块存储层，利用 golang 泛型，让模块开发者能够以简单直接的方式处理模块存储，同时提供安全性、可扩展性和标准化能力。

​
背景

模块开发者目前被迫在各自模块中手动实现存储相关功能，这些功能包括但不限于：
  • 定义键到字节的格式。
  • 定义值到字节的格式。
  • 定义二级索引。
  • 定义对外暴露、用于处理存储的查询方法。
  • 定义用于处理存储写入的本地方法。
  • 处理 genesis 的导入与导出。
  • 为上述所有内容编写测试。
这带来了很多问题：
  • 它会阻碍开发者专注于最重要的部分：编写业务逻辑。
  • 键到字节的格式非常复杂，其定义也容易出错，例如：
    • 我该如何把时间格式化为字节，并确保这些字节可排序？
    • 在处理二级索引时，我如何确保不会发生命名空间冲突？
  • 缺乏标准化会给客户端带来很大困难，而当需要为状态中的对象提供证明时，这个问题会更加严重。客户端被迫维护一份对象路径列表，以便收集证明。

​
当前方案：ORM

SDK 目前针对这个问题提出的方案是 ORM。 尽管 ORM 提供了大量旨在解决这些特定问题的良好功能，但它也有一些缺点：
  • 它需要迁移。
  • 它使用最新的 protobuf golang API，而 SDK 目前仍主要使用 gogoproto。
  • 将 ORM 集成到模块中，需要开发者同时处理两个表示同一 API 对象的不同 golang 框架（golang protobuf + gogoproto）。
  • 它的学习曲线很陡，即使是简单的存储层也不例外，因为它要求开发者了解 protobuf 选项、自定义 cosmos-sdk 存储扩展以及工具下载。之后，他们仍然需要继续学习代码生成的 API。

​
CosmWasm 方案：cw-storage-plus

collections API 借鉴了 cw-storage-plus， 后者已经证明是处理 CosmWasm 合约存储的强大工具。 它简单、不需要额外工具，并且能够轻松处理复杂的存储结构（索引、快照等）。 该 API 直观且显式。

​
决策

我们提议将 collections API 移植到 cosmos-sdk 中，其实现当前位于 NibiruChain/collections。 Collections 实现了五种不同类型的存储处理器：
  • Map：处理简单的 key=>object 映射。
  • KeySet：充当一个 Set，只保留键而不保留对象（用例：允许列表）。
  • Item：始终只包含一个对象（用例：Params）。
  • Sequence：实现一个简单且始终递增的数字（用例：Nonces）。
  • IndexedMap：构建于 Map 和 KeySet 之上，允许为 Objects 及其二级键创建关联关系。
所有 collection API 都建立在简单的 Map 类型之上。 Collections 是完全泛型的，这意味着任何类型都可以作为 Key 和 Value。它既可以是 protobuf 对象，也可以不是。 实际上，Collections 类型将键和值的序列化职责委托给 collections API 的另一个组件：ValueEncoders 和 KeyEncoders。 ValueEncoders 负责将值转换为字节（仅对 Map 相关）。它提供了一个即插即用层，使我们能够调整对象的编码方式， 这对于切换序列化框架和提升性能非常重要。 Collections 已经内置了默认的 ValueEncoders，特别针对：protobuf 对象、特殊 SDK 类型（sdk.Int、sdk.Dec）。 KeyEncoders 负责将键转换为字节，collections 已经为一些 golang 基本类型提供了默认的 KeyEncoders （uint64、string、time.Time 等），以及一些广泛使用的 sdk 类型（sdk.Acc/Val/ConsAddress、sdk.Int/Dec 等）。 这些默认实现还提供了对正确字典序排序和命名空间冲突的安全保障。 collections API 的示例可见于：

​
影响

​
向后兼容性

ValueEncoders 和 KeyEncoders 的设计允许模块保留相同的 byte(key)=>byte(value) 映射，因此升级到新的存储层不会破坏状态。

​
正面影响

  • 该 ADR 的目标是从 SDK 中移除代码，而不是增加代码。仅将 x/staking 迁移到 collections，就会带来代码行数的净减少（即使把 collections 自身新增的代码计算在内）。
  • 简化并标准化 SDK 各模块中的存储层。
  • 不需要处理 protobuf。
  • 它是纯 golang 代码。
  • 基于 KeyEncoders 和 ValueEncoders 的泛化能力，使我们不必绑定到某一种数据序列化框架。
  • KeyEncoders 和 ValueEncoders 可以扩展以提供模式反射。

​
负面影响

  • 尽管 golang 泛型已经在生产环境中使用，但与 golang 的其他特性相比，它还没有经过同等充分的验证。
  • Collection 类型的实例化仍需改进。

​
中性影响

{neutral consequences}

​
后续讨论

  • 自动化 genesis 导入/导出（由于会造成 API 破坏，暂未实现）
  • 模式反射

​
参考资料

​
Changelog

  • 30/11/2022: PROPOSED

​
Status

PROPOSED - Implemented

​
Abstract

We propose a simplified module storage layer which leverages golang generics to allow module developers to handle module storage in a simple and straightforward manner, whilst offering safety, extensibility and standardisation.

​
Context

Module developers are forced into manually implementing storage functionalities in their modules, those functionalities include but are not limited to:
  • Defining key to bytes formats.
  • Defining value to bytes formats.
  • Defining secondary indexes.
  • Defining query methods to expose outside to deal with storage.
  • Defining local methods to deal with storage writing.
  • Dealing with genesis imports and exports.
  • Writing tests for all the above.
This brings in a lot of problems:
  • It blocks developers from focusing on the most important part: writing business logic.
  • Key to bytes formats are complex and their definition is error-prone, for example:
    • how do I format time to bytes in such a way that bytes are sorted?
    • how do I ensure when I don’t have namespace collisions when dealing with secondary indexes?
  • The lack of standardisation makes life hard for clients, and the problem is exacerbated when it comes to providing proofs for objects present in state. Clients are forced to maintain a list of object paths to gather proofs.

​
Current Solution: ORM

The current SDK proposed solution to this problem is ORM. While ORM offers a lot of good functionality aimed at solving these specific problems, it has some downsides:
  • It requires migrations.
  • It uses the newest protobuf golang API, whilst the SDK still mainly uses gogoproto.
  • Integrating ORM into a module would require the developer to deal with two different golang frameworks (golang protobuf + gogoproto) representing the same API objects.
  • It has a high learning curve, even for simple storage layers as it requires developers to have knowledge around protobuf options, custom cosmos-sdk storage extensions, and tooling download. Then after this they still need to learn the code-generated API.

​
CosmWasm Solution: cw-storage-plus

The collections API takes inspiration from cw-storage-plus, which has demonstrated to be a powerful tool for dealing with storage in CosmWasm contracts. It’s simple, does not require extra tooling, it makes it easy to deal with complex storage structures (indexes, snapshot, etc). The API is straightforward and explicit.

​
Decision

We propose to port the collections API, whose implementation lives in NibiruChain/collections to cosmos-sdk. Collections implements four different storage handlers types:
  • Map: which deals with simple key=>object mappings.
  • KeySet: which acts as a Set and only retains keys and no object (usecase: allow-lists).
  • Item: which always contains only one object (usecase: Params)
  • Sequence: which implements a simple always increasing number (usecase: Nonces)
  • IndexedMap: builds on top of Map and KeySet and allows to create relationships with Objects and Objects secondary keys.
All the collection APIs build on top of the simple Map type. Collections is fully generic, meaning that anything can be used as Key and Value. It can be a protobuf object or not. Collections types, in fact, delegate the duty of serialisation of keys and values to a secondary collections API component called ValueEncoders and KeyEncoders. ValueEncoders take care of converting a value to bytes (relevant only for Map). And offers a plug and play layer which allows us to change how we encode objects, which is relevant for swapping serialisation frameworks and enhancing performance. Collections already comes in with default ValueEncoders, specifically for: protobuf objects, special SDK types (sdk.Int, sdk.Dec). KeyEncoders take care of converting keys to bytes, collections already comes in with some default KeyEncoders for some privimite golang types (uint64, string, time.Time, …) and some widely used sdk types (sdk.Acc/Val/ConsAddress, sdk.Int/Dec, …). These default implementations also offer safety around proper lexicographic ordering and namespace-collision. Examples of the collections API can be found here:

​
Consequences

​
Backwards Compatibility

The design of ValueEncoders and KeyEncoders allows modules to retain the same byte(key)=>byte(value) mappings, making the upgrade to the new storage layer non-state breaking.

​
Positive

  • ADR aimed at removing code from the SDK rather than adding it. Migrating just x/staking to collections would yield to a net decrease in LOC (even considering the addition of collections itself).
  • Simplifies and standardises storage layers across modules in the SDK.
  • Does not require to have to deal with protobuf.
  • It’s pure golang code.
  • Generalisation over KeyEncoders and ValueEncoders allows us to not tie ourself to the data serialisation framework.
  • KeyEncoders and ValueEncoders can be extended to provide schema reflection.

​
Negative

  • Golang generics are not as battle-tested as other Golang features, despite being used in production right now.
  • Collection types instantiation needs to be improved.

​
Neutral

{neutral consequences}

​
Further Discussions

  • Automatic genesis import/export (not implemented because of API breakage)
  • Schema reflection

​
References