Table of Contents
개요 #
- JMX exporter 적용은 카프카를 재시작해야 함
- 이미 운영중인 서버에서 재시작에 번거로울 경우 kafka exporter 를 적용하는게 좋을듯
- kafka exporter는 카프카를 재시작할 필요는 없음
JMX exporter#
https://github.com/prometheus/jmx_exporter
jar파일다운로드#
https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar
jmx exporter설정파일 생성#
kafka_config.yml
# https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-2_0_0.yml lowercaseOutputName: true rules: # Special cases and very specific rules - pattern : kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value name: kafka_server_$1_$2 type: GAUGE labels: clientId: "$3" topic: "$4" partition: "$5" - pattern : kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value name: kafka_server_$1_$2 type: GAUGE labels: clientId: "$3" broker: "$4:$5" - pattern : kafka.coordinator.(\w+)<type=(.+), name=(.+)><>Value name: kafka_coordinator_$1_$2_$3 type: GAUGE # Generic per-second counters with 0-2 key/value pairs - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count name: kafka_$1_$2_$3_total type: COUNTER labels: "$4": "$5" "$6": "$7" - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count name: kafka_$1_$2_$3_total type: COUNTER labels: "$4": "$5" - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count name: kafka_$1_$2_$3_total type: COUNTER # Generic gauges with 0-2 key/value pairs - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value name: kafka_$1_$2_$3 type: GAUGE labels: "$4": "$5" "$6": "$7" - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value name: kafka_$1_$2_$3 type: GAUGE labels: "$4": "$5" - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value name: kafka_$1_$2_$3 type: GAUGE # Emulate Prometheus 'Summary' metrics for the exported 'Histogram's. # # Note that these are missing the '_sum' metric! - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count name: kafka_$1_$2_$3_count type: COUNTER labels: "$4": "$5" "$6": "$7" - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile name: kafka_$1_$2_$3 type: GAUGE labels: "$4": "$5" "$6": "$7" quantile: "0.$8" - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count name: kafka_$1_$2_$3_count type: COUNTER labels: "$4": "$5" - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile name: kafka_$1_$2_$3 type: GAUGE labels: "$4": "$5" quantile: "0.$6" - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count name: kafka_$1_$2_$3_count type: COUNTER - pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile name: kafka_$1_$2_$3 type: GAUGE labels: quantile: "0.$4"
kafka에 jmx exporter 설정추가#
export KAFKA_OPTS="$KAFKA_OPTS -javaagent:/home/fromm0/apps/jmx_exporter/jmx_prometheus_javaagent-0.12.1-SNAPSHOT.jar=7071:/home/fromm0/apps/jmx_exporter/kafka_config.yml" ... 중략
측정수치확인#
http://localhost:7071/metrics
kafka_exporter#
https://github.com/danielqsj/kafka_exporter
필요한 사항#
- Sarama
- Golang
- Dep
export INSTALL_DIRECTORY=/home/fromm0/apps/go/bin && curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
kafka_exporter 다운로드#
https://github.com/danielqsj/kafka_exporter/releases

실행#
kafka_exporter --kafka.server=server1:9092 \ --kafka.server=server2:9092 \ --kafka.server=server3:9092
kafka_exporter --kafka.server=server1:9092 \ --kafka.server=server2:9092 \ --kafka.server=server3:9092 \ --sasl.enabled \ --sasl.username=username \ --sasl.password=password \ --web.listen-address=:9309
수치확인#
http://localhost:9308/metrics
Prometheus #
Prometheus 다운로드#
https://github.com/prometheus/prometheus/releases
https://github.com/prometheus/prometheus/releases/download/v2.12.0/prometheus-2.12.0.linux-amd64.tar.gz
prometheus.yml 설정#
global: scrape_interval: 10s evaluation_interval: 10s scrape_configs: - job_name: 'kafka' static_configs: - targets: - 'server1:9309' - 'server1:9308' - 'server2:7071' - 'server2:7071' - 'server2:7071'
prometheus 실행#
/home/fromm0/apps/prometheus/bin/prometheus &
prometheus UI접속#
Grafana#
datasource 설정#
- Name : 아무거나 (여기서는 Prometheus-kafka)
- HTTP > Url : http://prometheus_server:9090/
- 나머지는 디폴트값
kafka overview#
kafka exporter overview#
모니터링 URL#
Grafana QL#
sum(kafka_consumergroup_lag) by (consumergroup)
참고 URL#
- https://www.robustperception.io/monitoring-kafka-with-prometheus
- https://github.com/prometheus/jmx_exporter
- https://grafana.com/grafana/dashboards/721
- https://github.com/danielqsj/kafka_exporter
- https://grafana.com/grafana/dashboards/7589
Add new attachment
Only authorized users are allowed to upload new attachments.
G’day (anonymous guest)
My Prefs