• Prometheus Consul Blackbox | export 监控实现

    云和安全管理服务专家新钛云服 郭鹏超原创

    前言:

    • blackbox_exporter

    是Prometheus 官方提供的 exporter 之一,主要提供http、dns、tcp、icmp 的监控数据采集。

    • Consul

    主要提供,服务发现,健康检查,等功能,本次集成主要使用到服务发现功能。

    本文主要实现,基于consul_sd_config & consul 的 prometheus 服务发现,实现网路设备ping监控,站点可用行监控,以及证书相关信息监控。

    安装环境:

    • k8s

    • consul

    • Prometheus

    • blackbox_exporter

    1: Consul 安装

    1.1:使用helm 安装 consul

    Bash
    # 添加 consul helm 源
    helm repo add hashicorp https://helm.releases.hashicorp.com
    # 安装consul 
    helm -n consul  install  \
    --set storageClass=alicloud-disk-efficiency  \
    consul hashicorp/consul \
    --version=0.32.1
    

    1.2:查看服务安装状态

    Bash
    [root@xxxxxxxx consul_install]# kubectl -n consul get pods
    NAME                     READY   STATUS    RESTARTS   AGE
    consul-consul-9lxfc      1/1     Running   0          6d1h
    consul-consul-ntqcf      1/1     Running   0          6d1h
    consul-consul-q7c6f      1/1     Running   0          6d1h
    consul-consul-server-0   1/1     Running   0          6d1h
    consul-consul-server-1   1/1     Running   0          6d1h
    consul-consul-server-2   1/1     Running   0          6d1h
    

    1.3:nginx-ingress consul

    • consul_ingress.yml

    Bash
    # consul.xxxxxx.cn  ----->  替换为正确域名
    
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: consul-ingress
      namespace: consul
      annotations:
        kubernetes.io/ingress.class: nginx
        nginx.ingress.kubernetes.io/rewrite-target: /
    spec:
      rules:
      - host: consul.xxxxxx.cn
        http:
          paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: consul-consul-ui
                port: 
                  number: 80
    

    • 执行部署

    Bash
    kubectl apply -f consul_ingress.yml
    

    1.4:访问测试

    2: Blackbox_export

    2.1:blackbox 安装

    • blackbox-exporter-config.yaml

    Bash
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: blackbox-exporter
      labels:
        app: blackbox-exporter
    data:
      blackbox.yml: |-
        modules:
          ## ----------- DNS 检测配置 -----------
          dns_tcp:  
            prober: dns
            dns:
              transport_protocol: "tcp"
              preferred_ip_protocol: "ip4"
              query_name: "kubernetes.default.svc.cluster.local" # 用于检测域名可用的网址
              query_type: "A" 
          ## ----------- TCP 检测模块配置 -----------
          tcp_connect:
            prober: tcp
            timeout: 5s
          ## ----------- ICMP 检测配置 -----------
          ping:
            prober: icmp
            timeout: 5s
            icmp:
              preferred_ip_protocol: "ip4"
          ## ----------- HTTP GET 2xx 检测模块配置 -----------
          http_get_2xx:  
            prober: http
            timeout: 10s
            http:
              method: GET
              preferred_ip_protocol: "ip4"
              valid_http_versions: ["HTTP/1.1","HTTP/2"]
              valid_status_codes: [200]           # 验证的HTTP状态码,默认为2xx
              no_follow_redirects: false          # 是否不跟随重定向
          ## ----------- HTTP GET 3xx 检测模块配置 -----------
          http_get_3xx:  
            prober: http
            timeout: 10s
            http:
              method: GET
              preferred_ip_protocol: "ip4"
              valid_http_versions: ["HTTP/1.1","HTTP/2"]
              valid_status_codes: [301,302,304,305,306,307]  # 验证的HTTP状态码,默认为2xx
              no_follow_redirects: false                     # 是否不跟随重定向
          ## ----------- HTTP POST 监测模块 -----------
          http_post_2xx: 
            prober: http
            timeout: 10s
            http:
              method: POST
              preferred_ip_protocol: "ip4"
              valid_http_versions: ["HTTP/1.1", "HTTP/2"]
              #headers:                             # HTTP头设置
              #  Content-Type: application/json
              #body: '{}'                           # 请求体设置
    

    • blackbox-exporter-deploy.yaml

    Bash
    apiVersion: v1
    kind: Service
    metadata:
      name: blackbox-exporter
      labels:
        k8s-app: blackbox-exporter
    spec:
      type: ClusterIP
      ports:
      - name: http
        port: 9115
        targetPort: 9115
      selector:
        k8s-app: blackbox-exporter
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: blackbox-exporter
      labels:
        k8s-app: blackbox-exporter
    spec:
      replicas: 1
      selector:
        matchLabels:
          k8s-app: blackbox-exporter
      template:
        metadata:
          labels:
            k8s-app: blackbox-exporter
        spec:
          containers:
          - name: blackbox-exporter
            image: prom/blackbox-exporter:v0.19.0
            args:
            - --config.file=/etc/blackbox_exporter/blackbox.yml
            - --web.listen-address=:9115
            - --log.level=info
            ports:
            - name: http
              containerPort: 9115
            resources:
              limits:
                cpu: 3
                memory: 6000Mi
              requests:
                cpu: 100m
                memory: 50Mi
            livenessProbe:
              tcpSocket:
                port: 9115
              initialDelaySeconds: 5
              timeoutSeconds: 5
              periodSeconds: 10
              successThreshold: 1
              failureThreshold: 3
            readinessProbe:
              tcpSocket:
                port: 9115
              initialDelaySeconds: 5
              timeoutSeconds: 5
              periodSeconds: 10
              successThreshold: 1
              failureThreshold: 3
            volumeMounts:
            - name: config
              mountPath: /etc/blackbox_exporter
          volumes:
          - name: config
            configMap:
              name: blackbox-exporter
              defaultMode: 420
    

    • 执行安装

    Bash
    kubectl apply -f blackbox-exporter-deploy.yaml
    kubectl apply -f blackbox-exporter-config.yaml
    

    2.2:nginx ingress blackbox-exporter • blackbox_ingress.yml

    Bash
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: blackbox-ingress
      namespace: monitoring
      annotations:
        kubernetes.io/ingress.class: nginx
        nginx.ingress.kubernetes.io/rewrite-target: /
    spec:
      rules:
      - host: blackbox-devops.lululemon.cn
        http:
          paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: blackbox-exporter
                port:
                  number: 9115
    

    • 执行安装

    Bash
    kubectl apply -f blackbox_ingress.yml
    

    3: rometheus 添加 服务动态发现

    Bash
    #####  http_get_2xx 数据获取
     - job_name: http_get_2xx
          params:
            module:
            - http_get_2xx
          scrape_interval: 2s
          scrape_timeout: 2s
          metrics_path: /probe
          consul_sd_configs:
          # consul 服务地址
          - server: consul-consul-server.consul.svc.cluster.local:8500
            tag_separator: ','
            services:
            - http_get_2xx
          relabel_configs:
            - source_labels: ['__meta_consul_service_address']
              target_label: __param_target
            - source_labels: ['__meta_consul_service_address']
              target_label: instance
            - target_label: __address__
              ## blackbox-export  地址
              replacement: blackbox-exporter.monitoring.svc.cluster.local:9115
     ####### icmp 配置 
         - job_name: blackbox_icmp
          params:
            module:
            - ping
          scrape_interval: 2s
          scrape_timeout: 2s
          metrics_path: /probe
          consul_sd_configs:
          # consul 服务地址
          - server: consul-consul-server.consul.svc.cluster.local:8500
            tag_separator: ','
            services:
            - ping
          relabel_configs:
            - source_labels: ['__meta_consul_service_address']
              target_label: __param_target
            - source_labels: ['__meta_consul_service_address']
              target_label: instance
            - target_label: __address__
              ## blackbox-export  地址
              replacement: blackbox-exporter.monitoring.svc.cluster.local:9115
    

    4:添加 icmp 监控

    4.1:添加监控地址到consul

    • icmp_list

    Bash
    192.168.1.1
    192.168.1.2
    

    • add_consul_service_icmp.sh

    Bash
    #!/usr/bin/env bash
    
    ip_addr=$1
    
    
    if test "$ip_addr";then
            curl -X PUT -d '{
                "id": "icmp_'${ip_addr}'",
                "name": "ping",
                "address": "'${ip_addr}'",
                "port": 443,
                "Meta": {
                  "env": "prod",
                  "team": "network",
                  "project": "network",
                  "owner": "Mike"
                },
                "tags": ["node"],
                "checks": [{"http": "http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval": "15s"}]}' \
                http://consul-consul-server:8500/v1/agent/service/register
    else
            echo "请输入参数"
    fi
    

    • 添加service ping

    Bash
    for i in `cat icmp_list`;do bash add_consul_service_icmp.sh  $i;done
    

    4.2:查看consul 服务

    4.3:删除ping 监控地址脚本

    Bash
    #!/usr/bin/env bash
    ip_addr=$1
    
    curl -X PUT http://consul-consul-server:8500/v1/agent/service/deregister/icmp_${ip_addr}
    

    5: 添加http_get_2xx

    5.1:添加监控域名

    • domain_name_list

    Bash
    wwww.baidu.com
    wwww.1111.com
    wwww.2222.com
    

    • add_consul_service_http_get_2xx.sh

    Bash
    #!/usr/bin/env bash
    
    service_name=$1
    
    
    if test "$service_name";then
            curl -X PUT -d '{
                "id": "http_get_2xx_'${service_name}'",
                "name": "http_get_2xx",
                "address": "https://'${service_name}'",
                "port": 443,
                "Meta": {
                  "env": "prod",
                  "team": "web",
                  "project": "web",
                  "owner": "Devops"
                },
                "tags": ["node"],
                "checks": [{"http": "http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval": "15s"}]}' \
                http://consul-consul-server:8500/v1/agent/service/register
    else
            echo "请输入参数"
    fi
    

    • 添加 service http_get_2xx

    Bash
    for i  in `cat domain_name_list`;do bash  add_consul_service_http_get_2xx.sh  $i;done
    

    5.2:查看consul 服务

    5.3:删除域名监控脚本

    • del_consul_service_http_get_2xx.sh

    Bash
    #!/usr/bin/env bash
    ip_addr=$1
    
    curl -X PUT http://consul-consul-server:8500/v1/agent/service/deregister/http_get_2xx_${ip_addr}
    

    6:查看prometheus 监控

    总结:

    使用上述方案,黑盒监控与自建cmdb 平台很容易进行集成,使其监控自动化,不需要过多的人工干预,可以省去大量的人工成本,grafana 的配置这里就不进行过多介绍,自行通过谷歌完成。

    «
    »
以专业成就每一位客户,让企业IT只为效果和安全买单

以专业成就每一位客户,让企业IT只为效果和安全买单