使用自定义指标和遥测收集有关您的应用程序和模块的关键信息。

概述

telemetry 包为 Cosmos SDK 应用程序提供基于 OpenTelemetry 的可观测性工具。它通过 OpenTelemetry 声明式配置 API,为追踪、指标和日志提供统一的初始化入口。 该包可实现:
  • 使用 YAML 配置文件初始化 OpenTelemetry SDK
  • 为 Cosmos SDK 旧版 go-metrics 包装器 API 提供向后兼容性
  • 内置主机、运行时和磁盘 I/O 指标的采集能力

快速开始

1. 启动本地遥测后端

docker run -p 3000:3000 -p 4317:4317 -p 4318:4318 --rm -ti grafana/otel-lgtm

2. 创建配置文件

创建一个 otel.yaml 文件:
file_format: "1.0-rc.3"
resource:
  attributes:
    - name: service.name
      value: my-cosmos-app

tracer_provider:
  processors:
    - batch:
        exporter:
          otlp_grpc:
            endpoint: http://localhost:4317

meter_provider:
  readers:
    - pull:
        exporter:
          prometheus/development:
            host: 0.0.0.0
            port: 9464

logger_provider:
  processors:
    - batch:
        exporter:
          otlp_grpc:
            endpoint: http://localhost:4317

extensions:
  instruments:
    host: {}
    runtime: {}
    diskio: {}
  propagators:
    - tracecontext

3. 初始化遥测

选项 A:环境变量(推荐) 将 OTEL_EXPERIMENTAL_CONFIG_FILE 设置为您的配置路径。这会在创建任何 meter/tracer 之前初始化 SDK,从而避免原子加载开销。
export OTEL_EXPERIMENTAL_CONFIG_FILE=/path/to/otel.yaml
选项 B:节点配置目录 现在会在 ~/.<node_home>/config/ 中生成一个空的 otel.yaml。将所需配置写入 otel.yaml。 选项 C:以编程方式初始化 SDK 会先尝试通过环境变量初始化,然后使用节点主目录中的配置进行初始化。 您也可以选择通过 telemetry.InitializeOpenTelemetry 函数自行初始化遥测:
err := telemetry.InitializeOpenTelemetry("/path/to/otel.yaml")
if err != nil {
    log.Fatal(err)
}
defer telemetry.Shutdown(context.Background())

配置

OpenTelemetry 配置

该包使用 OpenTelemetry 声明式配置规范。关键部分如下:
部分用途
resource服务标识和属性
tracer_provider追踪导出配置
meter_provider指标导出配置
logger_provider日志导出配置
有关包含可用选项的示例,请参见 OpenTelemetry 配置示例。

扩展

otel.yaml 配置文件中的 extensions 部分提供了标准 otelconf 尚未支持的附加功能:
extensions:
  # Optional file-based exporters
  trace_file: "/path/to/traces.json"
  metrics_file: "/path/to/metrics.json"
  metrics_file_interval: "10s"
  logs_file: "/path/to/logs.json"

  # Custom instrumentation additions
  instruments:
    host: {}
    runtime: {}
    diskio:
      disable_virtual_device_filter: true # removes the automatic filtering of virtual disks. Operating systems such as Linux typically add virtual disks, which can add duplication to disk io data. These disks usually take the form of loopback, RAID, partitions, etc.

  # Trace context propagation
  propagators:
    - tracecontext
    - baggage
    - b3
    - jaeger

自定义采集项

主机采集(host)

使用 go.opentelemetry.io/contrib/instrumentation/host 上报主机级指标:
  • CPU 使用率
  • 内存使用率
  • 网络 I/O
extensions:
  instruments:
    host: {}

运行时采集(runtime)

使用 go.opentelemetry.io/contrib/instrumentation/runtime 上报 Go 运行时指标:
  • Goroutine 数量
  • GC 统计信息
  • 内存分配
extensions:
  instruments:
    runtime: {}

磁盘 I/O 采集(diskio)

使用 gopsutil 上报磁盘 I/O 指标:
指标说明
system.disk.io读取/写入的字节数
system.disk.operations读/写操作次数
system.disk.io_timeI/O 操作耗时
system.disk.operation_time每次读/写操作耗时
system.disk.merged合并的读/写操作
extensions:
  instruments:
    diskio: {}
    # Or with options:
    diskio:
      disable_virtual_device_filter: true  # Include loopback, RAID, partitions on Linux
默认情况下,为避免 I/O 被重复计数,Linux 上的虚拟设备(loopback、RAID、分区)会被过滤掉。

传播器

为分布式追踪配置追踪上下文传播:
传播器说明
tracecontextW3C Trace Context(默认)
baggageW3C Baggage
b3Zipkin B3 单头部
b3multiZipkin B3 多头部
jaegerJaeger 传播

开发者用法

使用 Meters 和 Tracers

初始化完成后,使用标准 OpenTelemetry API:
import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/metric"
)

var (
    tracer = otel.Tracer("my-package")
    meter  = otel.Meter("my-package")

    myCounter metric.Int64Counter
)

func init() {
   var err error
   myCounter, err = meter.Int64Counter("my.counter")
   if err != nil {
	   panic(err)
   }
}

func MyFunction(ctx context.Context) error {
    ctx, span := tracer.Start(ctx, "MyFunction")
    defer span.End()

    myCounter.Add(ctx, 1)

    // ... your code
    return nil
}

关闭

应用退出时务必调用 Shutdown():
func (a *App) Close() {
    telemetry.Shutdown(ctx)
}

旧版 API(已弃用)

该包为 github.com/hashicorp/go-metrics 提供了向后兼容的包装器。这些包装器已弃用,用户应直接迁移到 OpenTelemetry API。

OpenTelemetry 桥接

Cosmos SDK v0.54.0+ 提供了一个桥接层,可将现有 go-metrics 发送到您在 OpenTelemetry 配置中定义的 meter provider。 要桥接您的指标,请将 app.toml 中的 metrics-sink 设置为 "otel"。

###############################################################################

###                         Telemetry Configuration                         ###
###############################################################################
[telemetry]


# other fields...

metrics-sink = "otel"

旧版配置

cfg := telemetry.Config{
    ServiceName:            "my-service",
    Enabled:                true,
    EnableHostname:         true,
    EnableHostnameLabel:    true,
    EnableServiceLabel:     true,
    PrometheusRetentionTime: 60,  // seconds
    GlobalLabels:           [][]string{{"chain_id", "cosmoshub-1"}},
    MetricsSink:            "otel",  // "mem", "statsd", "dogstatsd", "otel"
    StatsdAddr:             "localhost:8125",
}

m, err := telemetry.New(cfg)

旧版指标函数

以下函数均已弃用;请优先使用 OpenTelemetry:
// Counters
telemetry.IncrCounter(1.0, "tx", "count")
telemetry.IncrCounterWithLabels([]string{"tx", "count"}, 1.0, labels)

// Gauges
telemetry.SetGauge(42.0, "mempool", "size")
telemetry.SetGaugeWithLabels([]string{"mempool", "size"}, 42.0, labels)

// Timing
start := telemetry.Now()
// ... operation
telemetry.MeasureSince(start, "tx", "process_time")

// Module-specific helpers
telemetry.ModuleMeasureSince("bank", start, "send", "time")
telemetry.ModuleSetGauge("bank", 100.0, "balance", "total")

Metrics Sink 类型

Sink说明
mem带 SIGUSR1 转储支持的内存 sink(默认)
statsdStatsD 协议
dogstatsdDatadog DogStatsD
otelOpenTelemetry(桥接到已配置的 MeterProvider)

最佳实践

  1. 在生产环境使用环境变量初始化,以避免原子加载开销
  2. 始终调用 Shutdown(),确保指标/追踪被刷新
  3. 正确传递 context.Context,以确保 span 关联正确

查看遥测数据

在 Grafana LGTM 运行时:
  1. 打开 http://localhost:3000
  2. 使用 Drilldown 视图进行查看:
    • Traces:分布式追踪可视化
    • Metrics:指标查询与仪表板
    • Logs:结构化日志搜索

相关文档

Cosmos SDK 指标

以下指标由 Cosmos SDK 发出。
指标描述单位类型
tx_count通过 FinalizeBlock 处理的交易总数交易计数器
tx_successful通过 FinalizeBlock 处理的成功交易总数交易计数器
tx_failed通过 FinalizeBlock 处理的失败交易总数交易计数器
tx_gas_used单笔交易使用的 gas 总量gas仪表
tx_gas_wanted单笔交易请求的 gas 总量gas仪表
tx_msg_sendMsgSend 中发送的代币总量(按 denom 区分)代币仪表
tx_msg_withdraw_rewardMsgWithdrawDelegatorReward 中提取的代币总量(按 denom 区分)代币仪表
tx_msg_withdraw_commissionMsgWithdrawValidatorCommission 中提取的代币总量(按 denom 区分)代币仪表
tx_msg_delegateMsgDelegate 中委托的代币总量代币仪表
tx_msg_begin_unbondingMsgUndelegate 中取消委托的代币总量代币仪表
tx_msg_begin_begin_redelegateMsgBeginRedelegate 中重新委托的代币总量代币仪表
tx_msg_ibc_transferMsgTransfer 中通过 IBC 转移的代币总量(源链或汇链)代币仪表
ibc_transfer_packet_receiveFungibleTokenPacketData 中接收的代币总量(源链或汇链)代币仪表
new_account新创建账户总数账户计数器
gov_proposal治理提案总数提案计数器
gov_vote针对提案的治理投票总数投票计数器
gov_deposit针对提案的治理存入总数存入计数器
staking_delegate委托总数委托计数器
staking_undelegate取消委托总数取消委托计数器
staking_redelegate重新委托总数重新委托计数器
ibc_transfer_send从链上发送的 IBC 转账总数(源链或汇链)转账计数器
ibc_transfer_receive链上接收的 IBC 转账总数(源链或汇链)转账计数器
ibc_client_create创建的客户端总数创建计数器
ibc_client_update客户端更新总数更新计数器
ibc_client_upgrade客户端升级总数升级计数器
ibc_client_misbehaviour客户端误行为总数误行为计数器
ibc_connection_open-init连接 OpenInit 握手总数握手计数器
ibc_connection_open-try连接 OpenTry 握手总数握手计数器
ibc_connection_open-ack连接 OpenAck 握手总数握手计数器
ibc_connection_open-confirm连接 OpenConfirm 握手总数握手计数器
ibc_channel_open-init通道 OpenInit 握手总数握手计数器
ibc_channel_open-try通道 OpenTry 握手总数握手计数器
ibc_channel_open-ack通道 OpenAck 握手总数握手计数器
ibc_channel_open-confirm通道 OpenConfirm 握手总数握手计数器
ibc_channel_close-init通道 CloseInit 握手总数握手计数器
ibc_channel_close-confirm通道 CloseConfirm 握手总数握手计数器
tx_msg_ibc_recv_packet接收的 IBC 数据包总数数据包计数器
tx_msg_ibc_acknowledge_packet已确认的 IBC 数据包总数确认计数器
ibc_timeout_packetIBC 超时数据包总数超时计数器
store_iavl_getIAVL Store#Get 调用的耗时毫秒摘要
store_iavl_setIAVL Store#Set 调用的耗时毫秒摘要
store_iavl_hasIAVL Store#Has 调用的耗时毫秒摘要
store_iavl_deleteIAVL Store#Delete 调用的耗时毫秒摘要
store_iavl_commitIAVL Store#Commit 调用的耗时毫秒摘要
store_iavl_queryIAVL Store#Query 调用的耗时毫秒摘要

Gather relevant insights about your application and modules with custom metrics and telemetry.

Overview

The telemetry package provides observability tooling for Cosmos SDK applications using OpenTelemetry. It offers a unified initialization point for traces, metrics, and logs via the OpenTelemetry declarative configuration API. This package:
  • Initializes OpenTelemetry SDK using YAML configuration files
  • Provides backward compatibility with Cosmos SDK’s legacy go-metrics wrapper API
  • Includes built-in instrumentation for host, runtime, and disk I/O metrics

Quick Start

1. Start a Local Telemetry Backend

docker run -p 3000:3000 -p 4317:4317 -p 4318:4318 --rm -ti grafana/otel-lgtm

2. Create Configuration File

Create an otel.yaml file:
file_format: "1.0-rc.3"
resource:
  attributes:
    - name: service.name
      value: my-cosmos-app

tracer_provider:
  processors:
    - batch:
        exporter:
          otlp_grpc:
            endpoint: http://localhost:4317

meter_provider:
  readers:
    - pull:
        exporter:
          prometheus/development:
            host: 0.0.0.0
            port: 9464

logger_provider:
  processors:
    - batch:
        exporter:
          otlp_grpc:
            endpoint: http://localhost:4317

extensions:
  instruments:
    host: {}
    runtime: {}
    diskio: {}
  propagators:
    - tracecontext

3. Initialize Telemetry

Option A: Environment Variable (Recommended) Set OTEL_EXPERIMENTAL_CONFIG_FILE to your config path. This initializes the SDK before any meters/tracers are created, avoiding atomic load overhead.
export OTEL_EXPERIMENTAL_CONFIG_FILE=/path/to/otel.yaml
Option B: Node Config Directory An empty otel.yaml will now be generated in ~/.<node_home>/config/. Place the desired configuration in otel.yaml. Option C: Programmatic Initialization The SDK will first attempt to initialize via env var, then using the config in the node’s home directory. You may optionally initialize telemetry yourself using the telemetry.InitializeOpenTelemetry function:
err := telemetry.InitializeOpenTelemetry("/path/to/otel.yaml")
if err != nil {
    log.Fatal(err)
}
defer telemetry.Shutdown(context.Background())

Configuration

OpenTelemetry Configuration

The package uses the OpenTelemetry declarative configuration spec. Key sections:
SectionPurpose
resourceService identity and attributes
tracer_providerTrace export configuration
meter_providerMetrics export configuration
logger_providerLog export configuration
For examples containing available options, see the OpenTelemetry configuration examples.

Extensions

The extensions section of the otel.yaml configuration file provides additional features not yet supported by the standard otelconf:
extensions:
  # Optional file-based exporters
  trace_file: "/path/to/traces.json"
  metrics_file: "/path/to/metrics.json"
  metrics_file_interval: "10s"
  logs_file: "/path/to/logs.json"

  # Custom instrumentation additions
  instruments:
    host: {}
    runtime: {}
    diskio:
      disable_virtual_device_filter: true # removes the automatic filtering of virtual disks. Operating systems such as Linux typically add virtual disks, which can add duplication to disk io data. These disks usually take the form of loopback, RAID, partitions, etc.

  # Trace context propagation
  propagators:
    - tracecontext
    - baggage
    - b3
    - jaeger

Custom Instruments

Host Instrumentation (host)

Reports host-level metrics using go.opentelemetry.io/contrib/instrumentation/host:
  • CPU usage
  • Memory usage
  • Network I/O
extensions:
  instruments:
    host: {}

Runtime Instrumentation (runtime)

Reports Go runtime metrics using go.opentelemetry.io/contrib/instrumentation/runtime:
  • Goroutine count
  • GC statistics
  • Memory allocations
extensions:
  instruments:
    runtime: {}

Disk I/O Instrumentation (diskio)

Reports disk I/O metrics using gopsutil:
MetricDescription
system.disk.ioBytes read/written
system.disk.operationsRead/write operation counts
system.disk.io_timeTime spent on I/O operations
system.disk.operation_timeTime per read/write operation
system.disk.mergedMerged read/write operations
extensions:
  instruments:
    diskio: {}
    # Or with options:
    diskio:
      disable_virtual_device_filter: true  # Include loopback, RAID, partitions on Linux
By default, virtual devices (loopback, RAID, partitions) are filtered out on Linux to avoid double-counting I/O.

Propagators

Configure trace context propagation for distributed tracing:
PropagatorDescription
tracecontextW3C Trace Context (default)
baggageW3C Baggage
b3Zipkin B3 single header
b3multiZipkin B3 multi-header
jaegerJaeger propagation

Developer Usage

Using Meters and Tracers

After initialization, use standard OpenTelemetry APIs:
import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/metric"
)

var (
    tracer = otel.Tracer("my-package")
    meter  = otel.Meter("my-package")

    myCounter metric.Int64Counter
)

func init() {
   var err error
   myCounter, err = meter.Int64Counter("my.counter")
   if err != nil {
	   panic(err)
   }
}

func MyFunction(ctx context.Context) error {
    ctx, span := tracer.Start(ctx, "MyFunction")
    defer span.End()

    myCounter.Add(ctx, 1)

    // ... your code
    return nil
}

Shutdown

Always call Shutdown() when the application exits:
func (a *App) Close() {
    telemetry.Shutdown(ctx)
}

Legacy API (Deprecated)

The package provides backward-compatible wrappers for github.com/hashicorp/go-metrics. These are deprecated and users should migrate to OpenTelemetry APIs directly.

OpenTelemetry Bridge

Cosmos SDK v0.54.0+ provides a bridge to send existing go-metrics to the meter provider defined in your OpenTelemetry config. To bridge your metrics, set the metrics-sink in app.toml to “otel”.

###############################################################################
###                         Telemetry Configuration                         ###
###############################################################################
[telemetry]

# other fields...

metrics-sink = "otel"

Legacy Configuration

cfg := telemetry.Config{
    ServiceName:            "my-service",
    Enabled:                true,
    EnableHostname:         true,
    EnableHostnameLabel:    true,
    EnableServiceLabel:     true,
    PrometheusRetentionTime: 60,  // seconds
    GlobalLabels:           [][]string{{"chain_id", "cosmoshub-1"}},
    MetricsSink:            "otel",  // "mem", "statsd", "dogstatsd", "otel"
    StatsdAddr:             "localhost:8125",
}

m, err := telemetry.New(cfg)

Legacy Metrics Functions

All are deprecated; prefer OpenTelemetry:
// Counters
telemetry.IncrCounter(1.0, "tx", "count")
telemetry.IncrCounterWithLabels([]string{"tx", "count"}, 1.0, labels)

// Gauges
telemetry.SetGauge(42.0, "mempool", "size")
telemetry.SetGaugeWithLabels([]string{"mempool", "size"}, 42.0, labels)

// Timing
start := telemetry.Now()
// ... operation
telemetry.MeasureSince(start, "tx", "process_time")

// Module-specific helpers
telemetry.ModuleMeasureSince("bank", start, "send", "time")
telemetry.ModuleSetGauge("bank", 100.0, "balance", "total")

Metrics Sink Types

SinkDescription
memIn-memory sink with SIGUSR1 dump support (default)
statsdStatsD protocol
dogstatsdDatadog DogStatsD
otelOpenTelemetry (bridges to configured MeterProvider)

Best Practices

  1. Use environment variable initialization for production to avoid atomic load overhead
  2. Always call Shutdown() to ensure metrics/traces are flushed
  3. Thread context.Context properly for correct span correlation

Viewing Telemetry Data

With Grafana LGTM running:
  1. Open http://localhost:3000
  2. Use the Drilldown views to explore:
    • Traces: Distributed trace visualization
    • Metrics: Query and dashboard metrics
    • Logs: Structured log search

Cosmos SDK Metrics

The following metrics are emitted from the Cosmos SDK.
MetricDescriptionUnitType
tx_countTotal number of txs processed via FinalizeBlocktxcounter
tx_successfulTotal number of successful txs processed via FinalizeBlocktxcounter
tx_failedTotal number of failed txs processed via FinalizeBlocktxcounter
tx_gas_usedThe total amount of gas used by a txgasgauge
tx_gas_wantedThe total amount of gas requested by a txgasgauge
tx_msg_sendThe total amount of tokens sent in a MsgSend (per denom)tokengauge
tx_msg_withdraw_rewardThe total amount of tokens withdrawn in a MsgWithdrawDelegatorReward (per denom)tokengauge
tx_msg_withdraw_commissionThe total amount of tokens withdrawn in a MsgWithdrawValidatorCommission (per denom)tokengauge
tx_msg_delegateThe total amount of tokens delegated in a MsgDelegatetokengauge
tx_msg_begin_unbondingThe total amount of tokens undelegated in a MsgUndelegatetokengauge
tx_msg_begin_begin_redelegateThe total amount of tokens redelegated in a MsgBeginRedelegatetokengauge
tx_msg_ibc_transferThe total amount of tokens transferred via IBC in a MsgTransfer (source or sink chain)tokengauge
ibc_transfer_packet_receiveThe total amount of tokens received in a FungibleTokenPacketData (source or sink chain)tokengauge
new_accountTotal number of new accounts createdaccountcounter
gov_proposalTotal number of governance proposalsproposalcounter
gov_voteTotal number of governance votes for a proposalvotecounter
gov_depositTotal number of governance deposits for a proposaldepositcounter
staking_delegateTotal number of delegationsdelegationcounter
staking_undelegateTotal number of undelegationsundelegationcounter
staking_redelegateTotal number of redelegationsredelegationcounter
ibc_transfer_sendTotal number of IBC transfers sent from a chain (source or sink)transfercounter
ibc_transfer_receiveTotal number of IBC transfers received to a chain (source or sink)transfercounter
ibc_client_createTotal number of clients createdcreatecounter
ibc_client_updateTotal number of client updatesupdatecounter
ibc_client_upgradeTotal number of client upgradesupgradecounter
ibc_client_misbehaviourTotal number of client misbehaviorsmisbehaviourcounter
ibc_connection_open-initTotal number of connection OpenInit handshakeshandshakecounter
ibc_connection_open-tryTotal number of connection OpenTry handshakeshandshakecounter
ibc_connection_open-ackTotal number of connection OpenAck handshakeshandshakecounter
ibc_connection_open-confirmTotal number of connection OpenConfirm handshakeshandshakecounter
ibc_channel_open-initTotal number of channel OpenInit handshakeshandshakecounter
ibc_channel_open-tryTotal number of channel OpenTry handshakeshandshakecounter
ibc_channel_open-ackTotal number of channel OpenAck handshakeshandshakecounter
ibc_channel_open-confirmTotal number of channel OpenConfirm handshakeshandshakecounter
ibc_channel_close-initTotal number of channel CloseInit handshakeshandshakecounter
ibc_channel_close-confirmTotal number of channel CloseConfirm handshakeshandshakecounter
tx_msg_ibc_recv_packetTotal number of IBC packets receivedpacketcounter
tx_msg_ibc_acknowledge_packetTotal number of IBC packets acknowledgedacknowledgementcounter
ibc_timeout_packetTotal number of IBC timeout packetstimeoutcounter
store_iavl_getDuration of an IAVL Store#Get callmssummary
store_iavl_setDuration of an IAVL Store#Set callmssummary
store_iavl_hasDuration of an IAVL Store#Has callmssummary
store_iavl_deleteDuration of an IAVL Store#Delete callmssummary
store_iavl_commitDuration of an IAVL Store#Commit callmssummary
store_iavl_queryDuration of an IAVL Store#Query callmssummary