Documentation Grafana Cloud Testing and synthetics Synthetic Monitoring Configure alerts Configure custom alerts

Configure custom alerts

You can use Grafana Alerting to view pre-configured Synthetic Monitoring alert rules, create custom alert rules, and set up alert notifications.

Create custom alert rules

You can create PromQL queries for check metrics to create custom Grafana alerts.

Example query: check if latency exceeds 1 second.

yaml Copy

sum by (instance, job) (rate(probe_all_duration_seconds_sum{}[10m]))
/
sum by (instance, job) (rate(probe_all_duration_seconds_count{}[10m])) > 1

Example query: check if the probe error rate is higher than 10%.

yaml Copy

1 - (
sum by (instance, job) (rate(probe_all_success_sum{}[10m]))
/
sum by (instance, job) (rate(probe_all_success_count{}[10m]))
) > 0.1

For more examples, refer to the most popular Synthetic Monitoring alerts and find the distinct metrics for the various check types.

To explore the different options and settings for creating alerts, refer to Configure Alert Rules.

Access alert rules from Alerting

Synthetic monitoring alert rules can also be found in Grafana Cloud Alerting within the synthetic_monitoring namespace. Default rules are created inside the default rule group.

You can then use alert labels to configure the alert notifications in Grafana Alerting.

The default Synthetic Monitoring alert rules set the namespace and alert_sensitivity labels. For reference, the default configuration of the low sensitivity alert rule is as follows:

yaml Copy

alert: SyntheticMonitoringCheckFailureAtLowSensitivity
expr: instance_job_severity:probe_success:mean5m{alert_sensitivity="low"} < 75
for: 5m
labels:
  . namespace: synthetic_monitoring
annotations:
  description: check job {{ $labels.job }} instance {{ $labels.instance }} has a success rate of {{ printf "%.1f" $value }}%.
  summary: check success below 75%

Alert notifications

After you have configured your alert rules and enabled your check to trigger alerts, you can set up Alerting notifications to know when a probe fails. Refer to Alerting Notifications for more details.

Link to Synthetic Monitoring dashboards in alert notifications

To quickly access your Synthetic Monitoring check dashboards when an alert fires, add a link in your alert rule’s notification message.

Synthetic Monitoring checks are identified by their job and instance pair, so you must include those labels as query parameters in the dashboard URL.

The following example URL links to the Synthetic Monitoring check dashboard for the specific check that triggered the alert. Replace <GRAFANA STACK URL> with the URL of your Grafana stack:

Copy

<GRAFANA STACK URL>/a/grafana-synthetic-monitoring-app/scene?var-job={{ $labels.job }}&var-instance={{ $labels.instance }}