prometheus operator监控etcd集群

前言

业务中有时需自定义监控，使用prometheus operator可做些自定义监控，但需满足以下条件：

a. 确保有metrics
b. 创建servicemonitor对象，用于prometheus自动发现添加监控项target
c. servicemonitor关联metrics数据接口的service对象
d. 确保service可以正确获取到metrics

针对etcd做监控采集

获取etcd证书

# 想用prometheus访问到etcd监控数据，需要添加证书
cat /etc/kubernetes/manifests/kube-apiserver.yaml
    - --etcd-cafile=/etc/ssl/etcd/ssl/ca.pem
    - --etcd-certfile=/etc/ssl/etcd/ssl/node-master.pem
    - --etcd-keyfile=/etc/ssl/etcd/ssl/node-master-key.pem

# 或者使用此命令查看
cat /etc/etcd.env

测试证书

# 通过curl访问etcd metrics
curl --cert /etc/ssl/etcd/ssl/node-master.pem --key /etc/ssl/etcd/ssl/node-master-key.pem https://192.168.10.2:2379/metrics -k

curl --cert /etc/ssl/etcd/ssl/admin-master.pem --key /etc/ssl/etcd/ssl/admin-master-key.pem https://192.168.1.3:2379/metrics -k

创建etcd证书的secret

1	kubectl create secret generic etcd-ssl --from-file=/etc/ssl/etcd/ssl/ca.pem --from-file=/etc/ssl/etcd/ssl/admin-master.pem --from-file=/etc/ssl/etcd/ssl/admin-master-key.pem -n kuboard

创建servicemonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: etcd
  namespace: kuboard
  labels:
    k8s-app: etcd
spec:
  jobLabel: k8s-app
  endpoints:
  - path: metrics
    port: metrics
    interval: 30s
    scheme: https
    tlsConfig:
      caFile: /etc/prometheus/secrets/etcd-ssl/ca.pem
      certFile: /etc/prometheus/secrets/etcd-ssl/admin-master.pem
      keyFile: /etc/prometheus/secrets/etcd-ssl/admin-master-key.pem
      insecureSkipVerify: true
  selector:
    matchLabels:
      k8s-app: etcd
  namespaceSelector:
    matchNames:
    - kube-system

创建svc及endpoints

apiVersion: v1
kind: Service
metadata:
  name: etcd
  namespace: kube-system
  labels:
    k8s-app: etcd                 ## Kubernetes 会根据该标签和 Endpoints 资源关联
    app.kubernetes.io/name: etcd  ## Prometheus 会根据该标签服务发现到该服务
spec:
  type: ClusterIP
  #clusterIP: None                 ## 设置为 None,不分配 Service IP
  ports:
  - name: port
    port: 2379
    targetPort: 2379
    protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
  name: etcd
  namespace: kube-system
  labels:
    k8s-app: etcd
subsets:
- addresses:                      ## 代理的应用IP地址列表
  - ip: 192.168.1.3
  ports:
    - name: metrics
      port: 2379                   ## 代理的应用端口号
      protocol: TCP

修改prometheus添加secret

kubectl edit -n kuboard prometheus

spec:
  secrets:
  - etcd-ssl
  
# 添加完重启prometheus

查看结果

导入模版

模版id：3070