需求

收集 ES 的指标, 并进行展示和告警;

现状

1. ES 通过 docker compose 安装
2. 所在环境的 K8S 集群有 Prometheus 和 AlertManager 及 Grafana

方案

复用现有的监控体系, 通过: Prometheus 监控 ES.

具体实现为:

采集端 elasticsearch_exporter

可以监控的指标为:

...

展示端 基于Grafana

📚️ Reference:

ElasticSearch dashboard for Grafana | Grafana Labs

告警指标 基于prometheus alertmanager

📚️ Reference:

ElasticSearch：https://awesome-prometheus-alerts.grep.to/rules.html#elasticsearch-1

实施步骤

以下为手动实施步骤

Docker Compose

docker pull quay.io/prometheuscommunity/elasticsearch-exporter:v1.3.0

docker-compose.yml 示例:

🐾 Warning:

exporter 在每次刮削时都会从 ElasticSearch 集群中获取信息，因此过短的刮削间隔会给 ES 主节点带来负载，特别是当你使用 --es.all 和 --es.indices 运行时。我们建议你测量获取/_nodes/stats和/_all/_stats对你的ES集群来说需要多长时间，以确定你的刮削间隔是否太短。

原 ES 的 docker-copmose.yml 示例如下:

version: '3'
services:elasticsearch:image: elasticsearch-plugins:6.8.18...ports:- 9200:9200- 9300:9300restart: always

增加了 elasticsearch_exporter 的yaml如下:

version: '3'
services:elasticsearch:image: elasticsearch-plugins:6.8.18...ports:- 9200:9200- 9300:9300restart: alwayselasticsearch_exporter:image: quay.io/prometheuscommunity/elasticsearch-exporter:v1.3.0command: - '--es.uri=http://elasticsearch:9200'- '--es.all'- '--es.indices'- '--es.indices_settings'- '--es.indices_mappings'- '--es.shards'- '--es.snapshots'- '--es.timeout=30s'      restart: alwaysports:- "9114:9114"    

Prometheus 配置调整

prometheus 配置

Prometheus 增加静态抓取配置:

scrape_configs:- job_name: "es"static_configs:- targets: ["x.x.x.x:9114"]

说明:

x.x.x.x 为 ES Exporter IP, 因为 ES Exporter 通过 docker compose 和 ES部署在同一台机器, 所以这个 IP 也是 ES 的IP.

Prometheus Rules

增加 ES 相关的 Prometheus Rules:

groups:- name: elasticsearchrules:- record: elasticsearch_filesystem_data_used_percentexpr: 100 * (elasticsearch_filesystem_data_size_bytes - elasticsearch_filesystem_data_free_bytes)/ elasticsearch_filesystem_data_size_bytes- record: elasticsearch_filesystem_data_free_percentexpr: 100 - elasticsearch_filesystem_data_used_percent- alert: ElasticsearchTooFewNodesRunningexpr: elasticsearch_cluster_health_number_of_nodes < 3for: 0mlabels:severity: criticalannotations:description: "Missing node in Elasticsearch cluster\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"summary: ElasticSearch running on less than 3 nodes(instance {{ $labels.instance }}, node {{$labels.node}})- alert: ElasticsearchDiskSpaceLowexpr: elasticsearch_filesystem_data_free_percent < 20for: 2mlabels:severity: warningannotations:summary: Elasticsearch disk space low (instance {{ $labels.instance }}, node {{$labels.node}})description: "The disk usage is over 80%\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"- alert: ElasticsearchDiskOutOfSpaceexpr: elasticsearch_filesystem_data_free_percent < 10for: 0mlabels:severity: criticalannotations:summary: Elasticsearch disk out of space (instance {{ $labels.instance }}, node {{$labels.node}})description: "The disk usage is over 90%\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"- alert: ElasticsearchHeapUsageWarningexpr: (elasticsearch_jvm_memory_used_bytes{area="heap"} / elasticsearch_jvm_memory_max_bytes{area="heap"}) * 100 > 80for: 2mlabels:severity: warningannotations:summary: Elasticsearch Heap Usage warning (instance {{ $labels.instance }}, node {{$labels.node}})description: "The heap usage is over 80%\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"- alert: ElasticsearchHeapUsageTooHighexpr: (elasticsearch_jvm_memory_used_bytes{area="heap"} / elasticsearch_jvm_memory_max_bytes{area="heap"}) * 100 > 90for: 2mlabels:severity: criticalannotations:summary: Elasticsearch Heap Usage Too High (instance {{ $labels.instance }}, node {{$labels.node}})description: "The heap usage is over 90%\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"- alert: ElasticsearchClusterRedexpr: elasticsearch_cluster_health_status{color="red"} == 1for: 0mlabels:severity: criticalannotations:summary: Elasticsearch Cluster Red (instance {{ $labels.instance }}, node {{$labels.node}})description: "Elastic Cluster Red status\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"- alert: ElasticsearchClusterYellowexpr: elasticsearch_cluster_health_status{color="yellow"} == 1for: 0mlabels:severity: warningannotations:summary: Elasticsearch Cluster Yellow (instance {{ $labels.instance }}, node {{$labels.node}})description: "Elastic Cluster Yellow status\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"- alert: ElasticsearchHealthyDataNodesexpr: elasticsearch_cluster_health_number_of_data_nodes < 3for: 0mlabels:severity: criticalannotations:summary: Elasticsearch Healthy Data Nodes (instance {{ $labels.instance }}, node {{$labels.node}})description: "Missing data node in Elasticsearch cluster\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"- alert: ElasticsearchRelocatingShardsexpr: elasticsearch_cluster_health_relocating_shards > 0for: 0mlabels:severity: infoannotations:summary: Elasticsearch relocating shards (instance {{ $labels.instance }}, node {{$labels.node}})description: "Elasticsearch is relocating shards\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"- alert: ElasticsearchRelocatingShardsTooLongexpr: elasticsearch_cluster_health_relocating_shards > 0for: 15mlabels:severity: warningannotations:summary: Elasticsearch relocating shards too long (instance {{ $labels.instance }}, node {{$labels.node}})description: "Elasticsearch has been relocating shards for 15min\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"- alert: ElasticsearchInitializingShardsexpr: elasticsearch_cluster_health_initializing_shards > 0for: 0mlabels:severity: infoannotations:summary: Elasticsearch initializing shards (instance {{ $labels.instance }}, node {{$labels.node}})description: "Elasticsearch is initializing shards\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"- alert: ElasticsearchInitializingShardsTooLongexpr: elasticsearch_cluster_health_initializing_shards > 0for: 15mlabels:severity: warningannotations:summary: Elasticsearch initializing shards too long (instance {{ $labels.instance }}, node {{$labels.node}})description: "Elasticsearch has been initializing shards for 15 min\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"- alert: ElasticsearchUnassignedShardsexpr: elasticsearch_cluster_health_unassigned_shards > 0for: 0mlabels:severity: criticalannotations:summary: Elasticsearch unassigned shards (instance {{ $labels.instance }}, node {{$labels.node}})description: "Elasticsearch has unassigned shards\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"- alert: ElasticsearchPendingTasksexpr: elasticsearch_cluster_health_number_of_pending_tasks > 0for: 15mlabels:severity: warningannotations:summary: Elasticsearch pending tasks (instance {{ $labels.instance }}, node {{$labels.node}})description: "Elasticsearch has pending tasks. Cluster works slowly.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"- alert: ElasticsearchNoNewDocumentsexpr: increase(elasticsearch_indices_docs{es_data_node="true"}[10m]) < 1for: 0mlabels:severity: warningannotations:summary: Elasticsearch no new documents (instance {{ $labels.instance }}, node {{$labels.node}})description: "No new documents for 10 min!\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

并重启生效.

🐾Warning:

• ElasticsearchTooFewNodesRunning 告警的条件是 es 集群的node 少于 3个, 对于单节点 ES 会误报, 所以按需开启rule或按需屏蔽(slience).
• ElasticsearchHealthyDataNodes 告警同上.

AlertManager 告警规则及收件人配置

按需调整, 示例如下:

'global':'smtp_smarthost': '''smtp_from': '''smtp_require_tls': false'resolve_timeout': '5m'
'receivers':- 'name': 'es-email''email_configs':- 'to': 'sfw@example.com,sdfwef@example.com''send_resolved': true
'route':'group_by':- 'job''group_interval': '5m''group_wait': '30s''routes':- 'receiver': 'es-email''match':'job': 'es'

并重启生效.

Grafana 配置

导入 json 格式的 Grafana Dashboard: (完整Dashboard可以直接通过 Grafana 搜索获取)

{"__inputs": [],"__requires": [{"type": "grafana","id": "grafana","name": "Grafana","version": "5.4.0"},{"type": "panel","id": "graph","name": "Graph","version": "5.0.0"},{"type": "datasource","id": "prometheus","name": "Prometheus","version": "5.0.0"},{"type": "panel","id": "singlestat","name": "Singlestat","version": "5.0.0"}],"annotations": {"list": [{"builtIn": 1,"datasource": "-- Grafana --","enable": true,"hide": true,"iconColor": "rgba(0, 211, 255, 1)","name": "Annotations & Alerts","type": "dashboard"}]},"editable": true,"gnetId": null,"graphTooltip": 1,"id": null,"iteration": 1549021227642,"links": [],"panels": [{"gridPos": {"h": 1,"w": 24,"x": 0,"y": 0},"id": 90,"title": "Cluster","type": "row"},{"cacheTimeout": null,"colorBackground": true,"colorPostfix": false,"colorPrefix": false,"colorValue": false,"colors": ["#299c46","rgba(237, 129, 40, 0.89)","#d44a3a"],"format": "none","gauge": {"maxValue": 100,"minValue": 0,"show": false,"thresholdLabels": false,"thresholdMarkers": true},"gridPos": {"h": 3,"w": 12,"x": 0,"y": 1},"id": 92,"interval": null,"links": [],"mappingType": 1,"mappingTypes": [{"name": "value to text","value": 1},{"name": "range to text","value": 2}],"maxDataPoints": 100,"nullPointMode": "connected","nullText": null,"postfix": "","postfixFontSize": "50%","prefix": "","prefixFontSize": "50%","rangeMaps": [{"from": "null","text": "N/A","to": "null"}],"sparkline": {"fillColor": "rgba(31, 118, 189, 0.18)","full": false,"lineColor": "rgb(31, 120, 193)","show": false},"tableColumn": "Value","targets": [{"expr": "scalar(elasticsearch_cluster_health_status{color=\"green\",cluster=~\"$cluster\"}) + scalar(elasticsearch_cluster_health_status{color=\"yellow\",cluster=~\"$cluster\"}) * 2 + scalar(elasticsearch_cluster_health_status{color=\"red\",cluster=~\"$cluster\"}) * 3","format": "time_series","instant": false,"intervalFactor": 1,"legendFormat": "","refId": "A"}],
...

📚️ 参考文档

• prometheus-community/elasticsearch_exporter: Elasticsearch stats exporter for Prometheus (github.com)
• ElasticSearch dashboard for Grafana | Grafana Labs
• Awesome Prometheus alerts | Collection of alerting rules (grep.to)

本文由东风微鸣技术博客 EWhisper.cn 编写！