使用自定义指标和遥测收集有关您的应用程序和模块的关键信息。
概述
telemetry 包为 Cosmos SDK 应用程序提供基于 OpenTelemetry 的可观测性工具。它通过 OpenTelemetry 声明式配置 API,为追踪、指标和日志提供统一的初始化入口。
该包可实现:
- 使用 YAML 配置文件初始化 OpenTelemetry SDK
- 为 Cosmos SDK 旧版
go-metrics包装器 API 提供向后兼容性 - 内置主机、运行时和磁盘 I/O 指标的采集能力
快速开始
1. 启动本地遥测后端
2. 创建配置文件
创建一个otel.yaml 文件:
3. 初始化遥测
选项 A:环境变量(推荐) 将OTEL_EXPERIMENTAL_CONFIG_FILE 设置为您的配置路径。这会在创建任何 meter/tracer 之前初始化 SDK,从而避免原子加载开销。
~/.<node_home>/config/ 中生成一个空的 otel.yaml。将所需配置写入 otel.yaml。
选项 C:以编程方式初始化
SDK 会先尝试通过环境变量初始化,然后使用节点主目录中的配置进行初始化。
您也可以选择通过 telemetry.InitializeOpenTelemetry 函数自行初始化遥测:
配置
OpenTelemetry 配置
该包使用 OpenTelemetry 声明式配置规范。关键部分如下:| 部分 | 用途 |
|---|---|
resource | 服务标识和属性 |
tracer_provider | 追踪导出配置 |
meter_provider | 指标导出配置 |
logger_provider | 日志导出配置 |
扩展
otel.yaml 配置文件中的 extensions 部分提供了标准 otelconf 尚未支持的附加功能:
自定义采集项
主机采集(host)
使用 go.opentelemetry.io/contrib/instrumentation/host 上报主机级指标:
- CPU 使用率
- 内存使用率
- 网络 I/O
运行时采集(runtime)
使用 go.opentelemetry.io/contrib/instrumentation/runtime 上报 Go 运行时指标:
- Goroutine 数量
- GC 统计信息
- 内存分配
磁盘 I/O 采集(diskio)
使用 gopsutil 上报磁盘 I/O 指标:
| 指标 | 说明 |
|---|---|
system.disk.io | 读取/写入的字节数 |
system.disk.operations | 读/写操作次数 |
system.disk.io_time | I/O 操作耗时 |
system.disk.operation_time | 每次读/写操作耗时 |
system.disk.merged | 合并的读/写操作 |
传播器
为分布式追踪配置追踪上下文传播:| 传播器 | 说明 |
|---|---|
tracecontext | W3C Trace Context(默认) |
baggage | W3C Baggage |
b3 | Zipkin B3 单头部 |
b3multi | Zipkin B3 多头部 |
jaeger | Jaeger 传播 |
开发者用法
使用 Meters 和 Tracers
初始化完成后,使用标准 OpenTelemetry API:关闭
应用退出时务必调用Shutdown():
旧版 API(已弃用)
该包为github.com/hashicorp/go-metrics 提供了向后兼容的包装器。这些包装器已弃用,用户应直接迁移到 OpenTelemetry API。
OpenTelemetry 桥接
Cosmos SDK v0.54.0+ 提供了一个桥接层,可将现有 go-metrics 发送到您在 OpenTelemetry 配置中定义的 meter provider。 要桥接您的指标,请将app.toml 中的 metrics-sink 设置为 "otel"。
旧版配置
旧版指标函数
以下函数均已弃用;请优先使用 OpenTelemetry:Metrics Sink 类型
| Sink | 说明 |
|---|---|
mem | 带 SIGUSR1 转储支持的内存 sink(默认) |
statsd | StatsD 协议 |
dogstatsd | Datadog DogStatsD |
otel | OpenTelemetry(桥接到已配置的 MeterProvider) |
最佳实践
- 在生产环境使用环境变量初始化,以避免原子加载开销
- 始终调用
Shutdown(),确保指标/追踪被刷新 - 正确传递
context.Context,以确保 span 关联正确
查看遥测数据
在 Grafana LGTM 运行时:- 打开 http://localhost:3000
- 使用 Drilldown 视图进行查看:
- Traces:分布式追踪可视化
- Metrics:指标查询与仪表板
- Logs:结构化日志搜索
相关文档
Cosmos SDK 指标
以下指标由 Cosmos SDK 发出。| 指标 | 描述 | 单位 | 类型 |
|---|---|---|---|
tx_count | 通过 FinalizeBlock 处理的交易总数 | 交易 | 计数器 |
tx_successful | 通过 FinalizeBlock 处理的成功交易总数 | 交易 | 计数器 |
tx_failed | 通过 FinalizeBlock 处理的失败交易总数 | 交易 | 计数器 |
tx_gas_used | 单笔交易使用的 gas 总量 | gas | 仪表 |
tx_gas_wanted | 单笔交易请求的 gas 总量 | gas | 仪表 |
tx_msg_send | MsgSend 中发送的代币总量(按 denom 区分) | 代币 | 仪表 |
tx_msg_withdraw_reward | MsgWithdrawDelegatorReward 中提取的代币总量(按 denom 区分) | 代币 | 仪表 |
tx_msg_withdraw_commission | MsgWithdrawValidatorCommission 中提取的代币总量(按 denom 区分) | 代币 | 仪表 |
tx_msg_delegate | MsgDelegate 中委托的代币总量 | 代币 | 仪表 |
tx_msg_begin_unbonding | MsgUndelegate 中取消委托的代币总量 | 代币 | 仪表 |
tx_msg_begin_begin_redelegate | MsgBeginRedelegate 中重新委托的代币总量 | 代币 | 仪表 |
tx_msg_ibc_transfer | MsgTransfer 中通过 IBC 转移的代币总量(源链或汇链) | 代币 | 仪表 |
ibc_transfer_packet_receive | FungibleTokenPacketData 中接收的代币总量(源链或汇链) | 代币 | 仪表 |
new_account | 新创建账户总数 | 账户 | 计数器 |
gov_proposal | 治理提案总数 | 提案 | 计数器 |
gov_vote | 针对提案的治理投票总数 | 投票 | 计数器 |
gov_deposit | 针对提案的治理存入总数 | 存入 | 计数器 |
staking_delegate | 委托总数 | 委托 | 计数器 |
staking_undelegate | 取消委托总数 | 取消委托 | 计数器 |
staking_redelegate | 重新委托总数 | 重新委托 | 计数器 |
ibc_transfer_send | 从链上发送的 IBC 转账总数(源链或汇链) | 转账 | 计数器 |
ibc_transfer_receive | 链上接收的 IBC 转账总数(源链或汇链) | 转账 | 计数器 |
ibc_client_create | 创建的客户端总数 | 创建 | 计数器 |
ibc_client_update | 客户端更新总数 | 更新 | 计数器 |
ibc_client_upgrade | 客户端升级总数 | 升级 | 计数器 |
ibc_client_misbehaviour | 客户端误行为总数 | 误行为 | 计数器 |
ibc_connection_open-init | 连接 OpenInit 握手总数 | 握手 | 计数器 |
ibc_connection_open-try | 连接 OpenTry 握手总数 | 握手 | 计数器 |
ibc_connection_open-ack | 连接 OpenAck 握手总数 | 握手 | 计数器 |
ibc_connection_open-confirm | 连接 OpenConfirm 握手总数 | 握手 | 计数器 |
ibc_channel_open-init | 通道 OpenInit 握手总数 | 握手 | 计数器 |
ibc_channel_open-try | 通道 OpenTry 握手总数 | 握手 | 计数器 |
ibc_channel_open-ack | 通道 OpenAck 握手总数 | 握手 | 计数器 |
ibc_channel_open-confirm | 通道 OpenConfirm 握手总数 | 握手 | 计数器 |
ibc_channel_close-init | 通道 CloseInit 握手总数 | 握手 | 计数器 |
ibc_channel_close-confirm | 通道 CloseConfirm 握手总数 | 握手 | 计数器 |
tx_msg_ibc_recv_packet | 接收的 IBC 数据包总数 | 数据包 | 计数器 |
tx_msg_ibc_acknowledge_packet | 已确认的 IBC 数据包总数 | 确认 | 计数器 |
ibc_timeout_packet | IBC 超时数据包总数 | 超时 | 计数器 |
store_iavl_get | IAVL Store#Get 调用的耗时 | 毫秒 | 摘要 |
store_iavl_set | IAVL Store#Set 调用的耗时 | 毫秒 | 摘要 |
store_iavl_has | IAVL Store#Has 调用的耗时 | 毫秒 | 摘要 |
store_iavl_delete | IAVL Store#Delete 调用的耗时 | 毫秒 | 摘要 |
store_iavl_commit | IAVL Store#Commit 调用的耗时 | 毫秒 | 摘要 |
store_iavl_query | IAVL Store#Query 调用的耗时 | 毫秒 | 摘要 |
Gather relevant insights about your application and modules with custom metrics and telemetry.
Overview
Thetelemetry package provides observability tooling for Cosmos SDK applications using OpenTelemetry. It offers a unified initialization point for traces, metrics, and logs via the OpenTelemetry declarative configuration API.
This package:
- Initializes OpenTelemetry SDK using YAML configuration files
- Provides backward compatibility with Cosmos SDK’s legacy
go-metricswrapper API - Includes built-in instrumentation for host, runtime, and disk I/O metrics
Quick Start
1. Start a Local Telemetry Backend
2. Create Configuration File
Create anotel.yaml file:
3. Initialize Telemetry
Option A: Environment Variable (Recommended) SetOTEL_EXPERIMENTAL_CONFIG_FILE to your config path. This initializes the SDK before any meters/tracers are created, avoiding atomic load overhead.
otel.yaml will now be generated in ~/.<node_home>/config/. Place the desired configuration in otel.yaml.
Option C: Programmatic Initialization
The SDK will first attempt to initialize via env var, then using the config in the node’s home directory.
You may optionally initialize telemetry yourself using the telemetry.InitializeOpenTelemetry function:
Configuration
OpenTelemetry Configuration
The package uses the OpenTelemetry declarative configuration spec. Key sections:| Section | Purpose |
|---|---|
resource | Service identity and attributes |
tracer_provider | Trace export configuration |
meter_provider | Metrics export configuration |
logger_provider | Log export configuration |
Extensions
Theextensions section of the otel.yaml configuration file provides additional features not yet supported by the standard otelconf:
Custom Instruments
Host Instrumentation (host)
Reports host-level metrics using go.opentelemetry.io/contrib/instrumentation/host:
- CPU usage
- Memory usage
- Network I/O
Runtime Instrumentation (runtime)
Reports Go runtime metrics using go.opentelemetry.io/contrib/instrumentation/runtime:
- Goroutine count
- GC statistics
- Memory allocations
Disk I/O Instrumentation (diskio)
Reports disk I/O metrics using gopsutil:
| Metric | Description |
|---|---|
system.disk.io | Bytes read/written |
system.disk.operations | Read/write operation counts |
system.disk.io_time | Time spent on I/O operations |
system.disk.operation_time | Time per read/write operation |
system.disk.merged | Merged read/write operations |
Propagators
Configure trace context propagation for distributed tracing:| Propagator | Description |
|---|---|
tracecontext | W3C Trace Context (default) |
baggage | W3C Baggage |
b3 | Zipkin B3 single header |
b3multi | Zipkin B3 multi-header |
jaeger | Jaeger propagation |
Developer Usage
Using Meters and Tracers
After initialization, use standard OpenTelemetry APIs:Shutdown
Always callShutdown() when the application exits:
Legacy API (Deprecated)
The package provides backward-compatible wrappers forgithub.com/hashicorp/go-metrics. These are deprecated and users should migrate to OpenTelemetry APIs directly.
OpenTelemetry Bridge
Cosmos SDK v0.54.0+ provides a bridge to send existing go-metrics to the meter provider defined in your OpenTelemetry config. To bridge your metrics, set themetrics-sink in app.toml to “otel”.
Legacy Configuration
Legacy Metrics Functions
All are deprecated; prefer OpenTelemetry:Metrics Sink Types
| Sink | Description |
|---|---|
mem | In-memory sink with SIGUSR1 dump support (default) |
statsd | StatsD protocol |
dogstatsd | Datadog DogStatsD |
otel | OpenTelemetry (bridges to configured MeterProvider) |
Best Practices
- Use environment variable initialization for production to avoid atomic load overhead
- Always call
Shutdown()to ensure metrics/traces are flushed - Thread
context.Contextproperly for correct span correlation
Viewing Telemetry Data
With Grafana LGTM running:- Open http://localhost:3000
- Use the Drilldown views to explore:
- Traces: Distributed trace visualization
- Metrics: Query and dashboard metrics
- Logs: Structured log search
Related Documentation
Cosmos SDK Metrics
The following metrics are emitted from the Cosmos SDK.| Metric | Description | Unit | Type |
|---|---|---|---|
tx_count | Total number of txs processed via FinalizeBlock | tx | counter |
tx_successful | Total number of successful txs processed via FinalizeBlock | tx | counter |
tx_failed | Total number of failed txs processed via FinalizeBlock | tx | counter |
tx_gas_used | The total amount of gas used by a tx | gas | gauge |
tx_gas_wanted | The total amount of gas requested by a tx | gas | gauge |
tx_msg_send | The total amount of tokens sent in a MsgSend (per denom) | token | gauge |
tx_msg_withdraw_reward | The total amount of tokens withdrawn in a MsgWithdrawDelegatorReward (per denom) | token | gauge |
tx_msg_withdraw_commission | The total amount of tokens withdrawn in a MsgWithdrawValidatorCommission (per denom) | token | gauge |
tx_msg_delegate | The total amount of tokens delegated in a MsgDelegate | token | gauge |
tx_msg_begin_unbonding | The total amount of tokens undelegated in a MsgUndelegate | token | gauge |
tx_msg_begin_begin_redelegate | The total amount of tokens redelegated in a MsgBeginRedelegate | token | gauge |
tx_msg_ibc_transfer | The total amount of tokens transferred via IBC in a MsgTransfer (source or sink chain) | token | gauge |
ibc_transfer_packet_receive | The total amount of tokens received in a FungibleTokenPacketData (source or sink chain) | token | gauge |
new_account | Total number of new accounts created | account | counter |
gov_proposal | Total number of governance proposals | proposal | counter |
gov_vote | Total number of governance votes for a proposal | vote | counter |
gov_deposit | Total number of governance deposits for a proposal | deposit | counter |
staking_delegate | Total number of delegations | delegation | counter |
staking_undelegate | Total number of undelegations | undelegation | counter |
staking_redelegate | Total number of redelegations | redelegation | counter |
ibc_transfer_send | Total number of IBC transfers sent from a chain (source or sink) | transfer | counter |
ibc_transfer_receive | Total number of IBC transfers received to a chain (source or sink) | transfer | counter |
ibc_client_create | Total number of clients created | create | counter |
ibc_client_update | Total number of client updates | update | counter |
ibc_client_upgrade | Total number of client upgrades | upgrade | counter |
ibc_client_misbehaviour | Total number of client misbehaviors | misbehaviour | counter |
ibc_connection_open-init | Total number of connection OpenInit handshakes | handshake | counter |
ibc_connection_open-try | Total number of connection OpenTry handshakes | handshake | counter |
ibc_connection_open-ack | Total number of connection OpenAck handshakes | handshake | counter |
ibc_connection_open-confirm | Total number of connection OpenConfirm handshakes | handshake | counter |
ibc_channel_open-init | Total number of channel OpenInit handshakes | handshake | counter |
ibc_channel_open-try | Total number of channel OpenTry handshakes | handshake | counter |
ibc_channel_open-ack | Total number of channel OpenAck handshakes | handshake | counter |
ibc_channel_open-confirm | Total number of channel OpenConfirm handshakes | handshake | counter |
ibc_channel_close-init | Total number of channel CloseInit handshakes | handshake | counter |
ibc_channel_close-confirm | Total number of channel CloseConfirm handshakes | handshake | counter |
tx_msg_ibc_recv_packet | Total number of IBC packets received | packet | counter |
tx_msg_ibc_acknowledge_packet | Total number of IBC packets acknowledged | acknowledgement | counter |
ibc_timeout_packet | Total number of IBC timeout packets | timeout | counter |
store_iavl_get | Duration of an IAVL Store#Get call | ms | summary |
store_iavl_set | Duration of an IAVL Store#Set call | ms | summary |
store_iavl_has | Duration of an IAVL Store#Has call | ms | summary |
store_iavl_delete | Duration of an IAVL Store#Delete call | ms | summary |
store_iavl_commit | Duration of an IAVL Store#Commit call | ms | summary |
store_iavl_query | Duration of an IAVL Store#Query call | ms | summary |