How to Monitor Signer
We will use Grafana Cloud to observe and monitor both the Signer and its corresponding Stacks node.
Requirements
Grafana's application observability docs have a great quick-start. We will use:
Grafana Cloud to collect metrics and visualize them.
Grafana Alloy, on the Signer host, to push the metrics.
Creating a Grafana Cloud account
Before we begin, create a Grafana Cloud account (they offer a free tier that you can use).
Once done, access your dashboard and follow these steps:
Note the token GCLOUD_RW_API_KEY and the parameters GCLOUD_HOSTED_METRICS_URL and GCLOUD_HOSTED_METRICS_ID; we will use them later.
Configuring the Signer and the Stacks node
Ensure both your Signer configuration and your node configuration include the following lines:
# signer-config.toml
# ...
# Adjust to 0.0.0.0:30001 if running in Docker.
metrics_endpoint = "127.0.0.1:30001"# node-config.toml
[node]
# ...
# Adjust to 0.0.0.0:9153 if running in Docker.
prometheus_bind = "127.0.0.1:9153"The pre-compiled binaries already include the monitoring feature. However, if you are compiling the application binaries yourself, remember to enable the Cargo feature monitoring_prom while building them, for example:
cargo build --features monitoring_prom,slog_json --releaseOnce both binaries are running with the updated configuration, you can peek at the metrics being exposed:
curl 127.0.0.1:30001/metrics
# HELP stacks_signer_current_reward_cycle The current reward cycle
# TYPE stacks_signer_current_reward_cycle gauge
stacks_signer_current_reward_cycle 95
# HELP stacks_signer_node_rpc_call_latencies_histogram Time (seconds) measuring round-trip RPC call latency to the Stacks node
# TYPE stacks_signer_node_rpc_call_latencies_histogram histogram
...
stacks_signer_node_rpc_call_latencies_histogram_bucket{path="/v2/info",le="0.005"} 0
stacks_signer_node_rpc_call_latencies_histogram_bucket{path="/v2/info",le="0.01"} 0
stacks_signer_node_rpc_call_latencies_histogram_bucket{path="/v2/info",le="0.025"} 0
stacks_signer_node_rpc_call_latencies_histogram_bucket{path="/v2/info",le="0.05"} 985
stacks_signer_node_rpc_call_latencies_histogram_bucket{path="/v2/info",le="0.1"} 1194
...Also, you'll have a /info endpoint on the same port:
curl 127.0.0.1:30001/infoInstall Alloy
Follow these instructions to install Grafana Alloy.
On Debian-based distributions:
sudo apt install gpg
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install alloyConfigure Alloy
Edit the file /etc/alloy/config.alloy as follows, replacing the placeholders related to the prometheus endpoint with the parameters obtained when creating a Grafana Cloud account:
GCLOUD_HOSTED_METRICS_URLGCLOUD_HOSTED_METRICS_IDGCLOUD_RW_API_KEY
// For a full configuration reference, see https://grafana.com/docs/alloy
// For a default configuration, integrating all environmental variables from Grafana Cloud
// see https://storage.googleapis.com/cloud-onboarding/alloy/config/config.alloy
logging {
level = "warn"
}
prometheus.exporter.unix "default" {
include_exporter_metrics = true
disable_collectors = ["mdadm"]
}
prometheus.scrape "default" {
targets = array.concat(
prometheus.exporter.unix.default.targets,
[
{
// Self-collect metrics
job = "alloy",
__address__ = "127.0.0.1:12345",
},
{
// stacks-signer
job = "stacks-signer",
__address__ = "127.0.0.1:30001",
},
{
// stacks-node
job = "stacks-node",
__address__ = "127.0.0.1:9153",
},
],
)
forward_to = [prometheus.remote_write.metrics_service.receiver]
}
prometheus.remote_write "metrics_service" {
external_labels = {"instance" = constants.hostname}
endpoint {
# TODO: Edit the URL below with your Grafana production URL.
# should end with /api/prom/push
url = "<your GCLOUD_HOSTED_METRICS_URL>"
# TODO: Edit with your Grafana Cloud ID and Token
basic_auth {
username = "<your GCLOUD_HOSTED_METRICS_ID>"
password = "<your GCLOUD_RW_API_KEY>"
}
}
}Enable and start Alloy:
sudo systemctl daemon-reload
sudo systemctl enable alloy.service
sudo systemctl start alloy.serviceMetrics from your Signer and node will now start being pushed to Grafana Cloud.
Visualizing the metrics
You can now start building a dashboard to visualize the metrics.
Log in to Grafana Cloud and create a new Dashboard.
Pick the Prometheus instance you created before as the data source.
Create a new panel and pick
stacks_signer_current_reward_cyclefrom the metrics.
You should now be able to see Stacks' current reward cycle, as measured by the Signer, in the dashboard.
Grafana comes with powerful data visualization tools. You can read about how to query and transform data here, and find examples on how to build Prometheus queries.
This template will kick-start your dashboard.
Bonus: monitoring the host
Since we are here, we can also monitor the host itself. Debian-based distributions make it very easy for us by using node_exporter.
sudo apt install prometheus-node-exporter
sudo systemctl enable prometheus-node-exporter
sudo systemctl start prometheus-node-exporterThis will expose metrics on port 9100 of localhost.
We can now configure alloy to push them to Grafana. Edit your /etc/alloy/config.alloy file and add the following scrape target to the prometheus.scrape "default" targets list:
{
job = "node_exporter",
__address__ = "127.0.0.1:9100",
}Now reload alloy and check its status:
sudo systemctl reload alloy
sudo systemctl status alloynode_exporter provides a lot of metrics. Explore them through the Grafana Explorer or use one of the many prepared dashboards (e.g., this one) to see comprehensive information. Once you have a dashboard ready, you can also use it to configure alerts (e.g., on disk space, etc.).
Was this helpful?