linux-Grafana_AlertManager通知

分类于

linux-Grafana_AlertManager通知

前篇

Contract points - 通知端点

Alertmanager

  1. 在 Alerting -> Contact points 中增加要通知的端点

    比如: URL: https://aaa.bbb.com:1234

    image-20240908213414983

  2. 会 POST 一下数据到 https://aaa.bbb.com:1234/api/v2/alerts

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    [
        {
            "labels": {
                "alertname": "TestAlert",
                "instance": "Grafana",
                "label001": "value001"
            },
            "annotations": {
                "__dashboardUid__": "adx95cnob9on4e",
                "__panelId__": "1",
                "__value_string__": "[ metric='foo' labels={instance=bar} value=10 ]",
                "annotation_name01": "annotation_content01",
                "description": "my description",
                "runbook_url": "https://xxx.yyy.com",
                "summary": "my summary"
            },
            "startsAt": "2024-09-08T21:27:42.169924125+08:00",
            "endsAt": "0001-01-01T00:00:00Z",
            "generatorURL": "",
            "UpdatedAt": "2024-09-08T21:27:42.169924125+08:00",
            "Timeout": false
        }
    ]

Webhook

Alert rules - 通知规则

  • 示例

    image-20240909111309530

  • 通知结果

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    [
        {
            "labels": {
                "__alert_rule_uid__": "adxc7hzqvwcu8d",
                "__grafana_autogenerated__": "true",
                "__grafana_receiver__": "am_test01",
                "alertname": "test_alert01",
                "app": "feishu",
                "grafana_folder": "rule_feishu"
            },
            "annotations": {
                "__dashboardUid__": "edx97fkx8696oc",
                "__orgId__": "1",
                "__panelId__": "1",
                "__value_string__": "[ var='B' labels={app=feishu} value=10 ], [ var='C' labels={app=feishu} value=1 ]",
                "__values__": "{\"B\":10,\"C\":1}", // 条件计算出的结果
                "description": "Description  001",
                "summary": "Summary 001"
            },
            "startsAt": "2024-09-09T11:05:50+08:00",
            "endsAt": "0001-01-01T00:00:00Z",
            "generatorURL": "http://localhost:8100/alerting/grafana/adxc7hzqvwcu8d/view",
            "UpdatedAt": "2024-09-09T11:05:50.010718027+08:00",
            "Timeout": false
        }
    ]

Notification policies - 通知策略, 标签

符合策略的才通知, 一般用标签去做策略, 然后通知到策略的端点

  1. 给默认策略添加一个子策略, 匹配标签 test_lable01 = test_value01

    image-20240909154639906

  2. 在通知规则里, 添加标签, 然后选择 通知策略, 预览一下就能看符合 标签匹配的 通知策略

    image-20240909155208214

    • 接受到通知的数据里, 就包含这个标签

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      [{
      	"labels": {
      		"__alert_rule_uid__": "ddxciifs00cn4e",
      		"alertname": "test_alert02",
      		"grafana_folder": "rule_feishu",
      		"test_label01": "test_value01" // 自定义的标签
      	},
      	"annotations": {
      		"__dashboardUid__": "bdxcid10r0gsga",
      		"__orgId__": "1",
      		"__panelId__": "1",
      		"__values__": "{\"B\":-1,\"C\":-1}", // 条件计算出的结果, -1 表示没接过
      		"datasource_uid": "bdwqup6apkw00c",
      		"grafana_state_reason": "NoData, KeepLast",
      		"ref_id": "A"
      	},
      	"startsAt": "2024-09-09T15:37:50+08:00",
      	"endsAt": "0001-01-01T00:00:00Z",
      	"generatorURL": "http://localhost:8100/alerting/grafana/ddxciifs00cn4e/view",
      	"UpdatedAt": "2024-09-09T19:37:55.013911142+08:00",
      	"Timeout": false
      }]
      • 如果无数据时, 不想要通知, 可以设置无数据时状态为 Normal (参考: 无数据时不通知)

状态

无数据时不通知

  • 无数据时状态设置为 Normal, 则不会通知

    image-20240909231759788

踩坑

添加规则查询时报类型错误