How to Monitor a Signer

We will use to observe and monitor both the Signer and its corresponding Stacks node.

Requirements

Grafana's application observability docs have a . We will use:

Grafana Cloud to collect metrics and visualize them.
Grafana Alloy, on the Signer host, to push the metrics.

Creating a Grafana Cloud account

Before we begin, create a account (they offer a free tier that you can use).

Once done, access your dashboard and:

Click on "Connections", then
"Add new connection", and
select "Hosted Prometheus metrics".
Now select "Via Grafana Alloy", then
On step 2, select "Run Grafana Alloy" to generate an API token.

Note the token GCLOUD_RW_API_KEY and the parameters GCLOUD_HOSTED_METRICS_URL and GCLOUD_HOSTED_METRICS_ID, we will use them later.

Configuring the Signer and the Stacks node

Ensure both your Signer configuration and your node configuration include the following lines:

# signer-config.toml
# ...
# Adjust to 0.0.0.0:30001 if running in Docker.
metrics_endpoint = "127.0.0.1:30001"

# node-config.toml
[node]
# ...
# Adjust to 0.0.0.0:9153 if running in Docker.
prometheus_bind = "127.0.0.1:9153"

The pre-compiled binaries already include the monitoring feature. However, if you are compiling the application binaries yourself, remember to enable the Cargo feature monitoring_prom while building them, for example:

cargo build --features monitoring_prom,slog_json --release

Once both binaries are running with the updated configuration, you can peek at the metrics being exposed:

curl 127.0.0.1:30001/metrics

# HELP stacks_signer_current_reward_cycle The current reward cycle
# TYPE stacks_signer_current_reward_cycle gauge
stacks_signer_current_reward_cycle 95
# HELP stacks_signer_node_rpc_call_latencies_histogram Time (seconds) measuring round-trip RPC call latency to the Stacks node
# TYPE stacks_signer_node_rpc_call_latencies_histogram histogram
...
stacks_signer_node_rpc_call_latencies_histogram_bucket{path="/v2/info",le="0.005"} 0
stacks_signer_node_rpc_call_latencies_histogram_bucket{path="/v2/info",le="0.01"} 0
stacks_signer_node_rpc_call_latencies_histogram_bucket{path="/v2/info",le="0.025"} 0
stacks_signer_node_rpc_call_latencies_histogram_bucket{path="/v2/info",le="0.05"} 985
stacks_signer_node_rpc_call_latencies_histogram_bucket{path="/v2/info",le="0.1"} 1194
...

Also, you'll have a /info endpoint on the same port:

curl 127.0.0.1:30001/info

Install Alloy

On Debian-based distributions:

sudo apt install gpg
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install alloy

Configure Alloy

GCLOUD_HOSTED_METRICS_URL
GCLOUD_HOSTED_METRICS_ID
GCLOUD_RW_API_KEY

// For a full configuration reference, see https://grafana.com/docs/alloy
// For a default configuration, integrating all environmental variables from Grafana Cloud
// see https://storage.googleapis.com/cloud-onboarding/alloy/config/config.alloy

logging {
  level = "warn"
}

prometheus.exporter.unix "default" {
  include_exporter_metrics = true
  disable_collectors       = ["mdadm"]
}

prometheus.scrape "default" {
  targets = array.concat(
    prometheus.exporter.unix.default.targets,
    [
      {
        // Self-collect metrics
        job         = "alloy",
        __address__ = "127.0.0.1:12345",
      },
      {
        // stacks-signer
        job         = "stacks-signer",
        __address__ = "127.0.0.1:30001",
      },
      {
        // stacks-node
        job         = "stacks-node",
        __address__ = "127.0.0.1:9153",
      },
    ],
  )

  forward_to = [prometheus.remote_write.metrics_service.receiver]
}

prometheus.remote_write "metrics_service" {
  external_labels = {"instance" = constants.hostname}
  endpoint {
    # TODO: Edit the URL below with your Grafana production URL.
    # should end with /api/prom/push
    url = "<your GCLOUD_HOSTED_METRICS_URL>"

    # TODO: Edit with your Grafana Cloud ID and Token
    basic_auth {
      username = "<your GCLOUD_HOSTED_METRICS_ID>"
      password = "<your GCLOUD_RW_API_KEY>"
    }
  }
}

sudo systemctl daemon-reload
sudo systemctl enable alloy.service
sudo systemctl start alloy.service

Metrics from your Signer and node will now start being pushed to Grafana Cloud.

Visualizing the metrics

You can now start building a dashboard to visualize the metrics.

Log-in to Grafana Cloud and create a new Dashboard.
Pick the Prometheus instance you created before as the data source.
Create a new panel and pick stacks_signer_current_reward_cycle from the metrics.

You should now be able to see Stacks' current reward cycle, as measured by the Signer, into the dashboard.

Bonus: monitoring the host

sudo apt install prometheus-node-exporter
sudo systemctl enable prometheus-node-exporter
sudo systemctl start prometheus-node-exporter

This will expose metrics on port 9100 of localhost.

We can now configure alloy to push them to Grafana. Edit your /etc/alloy/config.alloy file and add the following:

prometheus.scrape "default" {
  targets = array.concat([

  ...

    {
        job         = "node_exporter",
        __address__ = "127.0.0.1:9100",
    }

  ...
])}

Now reload alloy and check its status:

sudo systemctl reload alloy
sudo systemctl status alloy

PreviousHow to Read Signer Logs NextBest practices for running a Signer

Last updated 2 months ago

Was this helpful?