ADR 022: Custom BaseApp panic handling

变更记录

2020 年 4 月 24 日：初始草案
2021 年 9 月 14 日：被 ADR-045 取代

状态

已被 ADR-045 取代

背景

当前 BaseApp 的 runTx() panic 恢复实现不允许开发者编写自定义错误处理器。我们认为，这一方法可以设计得更灵活，让 Cosmos SDK 用户在无需重写整个 BaseApp 的情况下拥有更多定制选项。此外，sdk.ErrorOutOfGas 的错误处理是一个特殊情况，它也许可以与其他情况一起通过“标准”方式（中间件）处理。我们提出一种中间件方案，可帮助开发者实现以下场景：

添加外部日志记录（例如将报告发送到 Sentry 之类的外部服务）；
针对特定错误场景调用 panic；

它还会将 OutOfGas 情况和 default 情况都变为某个中间件。Default 情况会将 recovery 对象包装为错误并记录日志（中间件实现示例）。我们的项目在区块链节点旁运行了一个 sidecar 服务（智能合约虚拟机）。对于交易处理来说，节点与 sidecar 之间的连接必须保持稳定。当通信中断时，我们需要让节点崩溃，并在问题解决后重启它。这种行为使节点状态机的执行保持确定性。由于所有 keeper 的 panic 都会被 runTx 的 defer() 处理器捕获，我们必须调整 BaseApp 代码才能对其进行定制。

决策

设计

概览

我们建议不要把自定义错误处理硬编码进 BaseApp，而是使用一组可在外部定制的中间件，使开发者能够按需使用任意数量的自定义错误处理器。包含测试的实现可见这里。

实现细节

恢复处理器

新增了 RecoveryHandler 类型。输入参数 recoveryObj 是标准 Go 函数 recover() 从 builtin 包返回的对象。

type RecoveryHandler func(recoveryObj interface{
})

error

处理器应通过类型断言（或其他方法）判断该对象是否应由自己处理。如果输入对象无法被该 RecoveryHandler 处理（不是处理器的目标类型），则应返回 nil。如果输入对象已被处理，并且中间件链执行应当停止，则应返回非 nil 的错误。示例：

func exampleErrHandler(recoveryObj interface{
})

error {
    err, ok := recoveryObj.(error)
    if !ok {
    return nil
}
    if someSpecificError.Is(err) {
    panic(customPanicMsg)
}

else {
    return nil
}
}

该示例会中断应用执行，但它也可以像 OutOfGas 处理器那样补充错误上下文。

恢复中间件

我们还新增了一个中间件类型（装饰器）。该函数类型包装 RecoveryHandler，并返回执行链中的下一个中间件以及处理器的 error。这个类型用于将实际的 recovery() 对象处理与中间件链处理分离开来。

type recoveryMiddleware func(recoveryObj interface{
}) (recoveryMiddleware, error)

func newRecoveryMiddleware(handler RecoveryHandler, next recoveryMiddleware)

recoveryMiddleware {
    return func(recoveryObj interface{
}) (recoveryMiddleware, error) {
    if err := handler(recoveryObj); err != nil {
    return nil, err
}

return next, nil
}
}

该函数接收一个 recoveryObj 对象，并返回：

如果对象未被 RecoveryHandler 处理（不是目标类型），则返回（下一个 recoveryMiddleware，nil）；
如果输入对象已被处理，且链中的其他中间件不应继续执行，则返回（nil，非 nil 的 error）；
在无效行为的情况下返回（nil，nil）。这意味着 panic 恢复可能没有被正确处理；这可以通过始终在链最右侧使用一个 default 中间件来避免（它总是返回一个 error）；

OutOfGas 中间件示例：

func newOutOfGasRecoveryMiddleware(gasWanted uint64, ctx sdk.Context, next recoveryMiddleware)

recoveryMiddleware {
    handler := func(recoveryObj interface{
})

error {
    err, ok := recoveryObj.(sdk.ErrorOutOfGas)
    if !ok {
    return nil
}

return errorsmod.Wrap(
            sdkerrors.ErrOutOfGas, fmt.Sprintf(
                "out of gas in location: %v; gasWanted: %d, gasUsed: %d", err.Descriptor, gasWanted, ctx.GasMeter().GasConsumed(),
            ),
        )
}

return newRecoveryMiddleware(handler, next)
}

Default 中间件示例：

func newDefaultRecoveryMiddleware()

recoveryMiddleware {
    handler := func(recoveryObj interface{
})

error {
    return errorsmod.Wrap(
            sdkerrors.ErrPanic, fmt.Sprintf("recovered: %v\nstack:\n%v", recoveryObj, string(debug.Stack())),
        )
}

return newRecoveryMiddleware(handler, nil)
}

恢复处理流程

基础的中间件链处理流程如下：

func processRecovery(recoveryObj interface{
}, middleware recoveryMiddleware)

error {
    if middleware == nil {
    return nil
}

next, err := middleware(recoveryObj)
    if err != nil {
    return err
}
    if next == nil {
    return nil
}

return processRecovery(recoveryObj, next)
}

这样我们就可以创建一个从左到右执行的中间件链，最右侧的中间件是一个必须返回 error 的 default 处理器。

`BaseApp` 变更

default 中间件链必须存在于 BaseApp 对象中。BaseApp 修改如下：

type BaseApp struct {
    // ...
    runTxRecoveryMiddleware recoveryMiddleware
}

func NewBaseApp(...) {
    // ...
    app.runTxRecoveryMiddleware = newDefaultRecoveryMiddleware()
}

func (app *BaseApp)

runTx(...) {
    // ...
    defer func() {
    if r := recover(); r != nil {
    recoveryMW := newOutOfGasRecoveryMiddleware(gasWanted, ctx, app.runTxRecoveryMiddleware)

err, result = processRecovery(r, recoveryMW), nil
}

gInfo = sdk.GasInfo{
    GasWanted: gasWanted,
    GasUsed: ctx.GasMeter().GasConsumed()
}

}()
    // ...
}

开发者可以通过向 NewBaseapp 构造函数提供 AddRunTxRecoveryHandler 这一 BaseApp 选项参数来添加自定义 RecoveryHandler：

func (app *BaseApp)

AddRunTxRecoveryHandler(handlers ...RecoveryHandler) {
    for _, h := range handlers {
    app.runTxRecoveryMiddleware = newRecoveryMiddleware(h, app.runTxRecoveryMiddleware)
}
}

该方法会将处理器前置到现有链的最前面。

影响

正面

基于 Cosmos SDK 的项目开发者可以添加自定义 panic 处理器，以便：
- 为自定义 panic 来源（自定义 keeper 内部的 panic）补充错误上下文；
- 触发 panic()：将 recovery 对象透传给 Tendermint core；
- 执行其他必要处理；
开发者可以使用标准的 Cosmos SDK BaseApp 实现，而不必在自己的项目中重写它；
所提方案不会破坏当前“标准”的 runTx() 流程；

负面

对执行模型设计引入了变更。

中性

OutOfGas 错误处理器成为中间件之一；
默认 panic 处理器成为中间件之一；

参考资料

Changelog

2020 Apr 24: Initial Draft
2021 Sep 14: Superseded by ADR-045

Status

SUPERSEDED by ADR-045

Context

The current implementation of BaseApp does not allow developers to write custom error handlers during panic recovery runTx() method. We think that this method can be more flexible and can give Cosmos SDK users more options for customizations without the need to rewrite whole BaseApp. Also there’s one special case for sdk.ErrorOutOfGas error handling, that case might be handled in a “standard” way (middleware) alongside the others. We propose middleware-solution, which could help developers implement the following cases:

add external logging (let’s say sending reports to external services like Sentry);
call panic for specific error cases;

It will also make OutOfGas case and default case one of the middlewares. Default case wraps recovery object to an error and logs it (example middleware implementation). Our project has a sidecar service running alongside the blockchain node (smart contracts virtual machine). It is essential that node <-> sidecar connectivity stays stable for TXs processing. So when the communication breaks we need to crash the node and reboot it once the problem is solved. That behavior makes node’s state machine execution deterministic. As all keeper panics are caught by runTx’s defer() handler, we have to adjust the BaseApp code in order to customize it.

Decision

Design

Overview

Instead of hardcoding custom error handling into BaseApp we suggest using set of middlewares which can be customized externally and will allow developers use as many custom error handlers as they want. Implementation with tests can be found here.

Implementation details

Recovery handler

New RecoveryHandler type added. recoveryObj input argument is an object returned by the standard Go function recover() from the builtin package.

type RecoveryHandler func(recoveryObj interface{
})

error

Handler should type assert (or other methods) an object to define if object should be handled. nil should be returned if input object can’t be handled by that RecoveryHandler (not a handler’s target type). Not nil error should be returned if input object was handled and middleware chain execution should be stopped. An example:

func exampleErrHandler(recoveryObj interface{
})

error {
    err, ok := recoveryObj.(error)
    if !ok {
    return nil
}
    if someSpecificError.Is(err) {
    panic(customPanicMsg)
}

else {
    return nil
}
}

This example breaks the application execution, but it also might enrich the error’s context like the OutOfGas handler.

Recovery middleware

We also add a middleware type (decorator). That function type wraps RecoveryHandler and returns the next middleware in execution chain and handler’s error. Type is used to separate actual recovery() object handling from middleware chain processing.

type recoveryMiddleware func(recoveryObj interface{
}) (recoveryMiddleware, error)

func newRecoveryMiddleware(handler RecoveryHandler, next recoveryMiddleware)

recoveryMiddleware {
    return func(recoveryObj interface{
}) (recoveryMiddleware, error) {
    if err := handler(recoveryObj); err != nil {
    return nil, err
}

return next, nil
}
}

Function receives a recoveryObj object and returns:

(next recoveryMiddleware, nil) if object wasn’t handled (not a target type) by RecoveryHandler;
(nil, not nil error) if input object was handled and other middlewares in the chain should not be executed;
(nil, nil) in case of invalid behavior. Panic recovery might not have been properly handled; this can be avoided by always using a default as a rightmost middleware in the chain (always returns an error’);

OutOfGas middleware example:

func newOutOfGasRecoveryMiddleware(gasWanted uint64, ctx sdk.Context, next recoveryMiddleware)

recoveryMiddleware {
    handler := func(recoveryObj interface{
})

error {
    err, ok := recoveryObj.(sdk.ErrorOutOfGas)
    if !ok {
    return nil
}

return errorsmod.Wrap(
            sdkerrors.ErrOutOfGas, fmt.Sprintf(
                "out of gas in location: %v; gasWanted: %d, gasUsed: %d", err.Descriptor, gasWanted, ctx.GasMeter().GasConsumed(),
            ),
        )
}

return newRecoveryMiddleware(handler, next)
}

Default middleware example:

func newDefaultRecoveryMiddleware()

recoveryMiddleware {
    handler := func(recoveryObj interface{
})

error {
    return errorsmod.Wrap(
            sdkerrors.ErrPanic, fmt.Sprintf("recovered: %v\nstack:\n%v", recoveryObj, string(debug.Stack())),
        )
}

return newRecoveryMiddleware(handler, nil)
}

Recovery processing

Basic chain of middlewares processing would look like:

func processRecovery(recoveryObj interface{
}, middleware recoveryMiddleware)

error {
    if middleware == nil {
    return nil
}

next, err := middleware(recoveryObj)
    if err != nil {
    return err
}
    if next == nil {
    return nil
}

return processRecovery(recoveryObj, next)
}

That way we can create a middleware chain which is executed from left to right, the rightmost middleware is a default handler which must return an error.

BaseApp changes

The default middleware chain must exist in a BaseApp object. Baseapp modifications:

type BaseApp struct {
    // ...
    runTxRecoveryMiddleware recoveryMiddleware
}

func NewBaseApp(...) {
    // ...
    app.runTxRecoveryMiddleware = newDefaultRecoveryMiddleware()
}

func (app *BaseApp)

runTx(...) {
    // ...
    defer func() {
    if r := recover(); r != nil {
    recoveryMW := newOutOfGasRecoveryMiddleware(gasWanted, ctx, app.runTxRecoveryMiddleware)

err, result = processRecovery(r, recoveryMW), nil
}

gInfo = sdk.GasInfo{
    GasWanted: gasWanted,
    GasUsed: ctx.GasMeter().GasConsumed()
}

}()
    // ...
}

Developers can add their custom RecoveryHandlers by providing AddRunTxRecoveryHandler as a BaseApp option parameter to the NewBaseapp constructor:

func (app *BaseApp)

AddRunTxRecoveryHandler(handlers ...RecoveryHandler) {
    for _, h := range handlers {
    app.runTxRecoveryMiddleware = newRecoveryMiddleware(h, app.runTxRecoveryMiddleware)
}
}

This method would prepend handlers to an existing chain.

Consequences

Positive

Developers of Cosmos SDK based projects can add custom panic handlers to:
- add error context for custom panic sources (panic inside of custom keepers);
- emit panic(): passthrough recovery object to the Tendermint core;
- other necessary handling;
Developers can use standard Cosmos SDK BaseApp implementation, rather that rewriting it in their projects;
Proposed solution doesn’t break the current “standard” runTx() flow;

ADR 022: Custom BaseApp panic handling

变更记录

状态

背景

决策

设计

概览

实现细节

恢复处理器

恢复中间件

恢复处理流程

`BaseApp` 变更

影响

正面

负面

中性

参考资料

Changelog

Status

Context

Decision

Design

Overview

Implementation details

Recovery handler

Recovery middleware

Recovery processing

BaseApp changes

Consequences

Positive

Negative

Neutral

References

​变更记录

​状态

​背景

​决策

​设计

​概览

​实现细节

恢复处理器

恢复中间件

恢复处理流程

BaseApp 变更

​影响

​正面

​负面

​中性

​参考资料

​Changelog

​Status

​Context

​Decision

​Design

​Overview

​Implementation details

Recovery handler

Recovery middleware

Recovery processing

BaseApp changes

​Consequences

​Positive

​Negative

​Neutral

​References

变更记录

状态

背景

决策

设计

概览

实现细节

`BaseApp` 变更

影响

正面

负面

中性

参考资料

Changelog

Status

Context

Decision

Design

Overview

Implementation details

Consequences

Positive

Negative

Neutral

References