개요 #

  • JMX exporter 적용은 카프카를 재시작해야 함
  • 이미 운영중인 서버에서 재시작에 번거로울 경우 kafka exporter 를 적용하는게 좋을듯
  • kafka exporter는 카프카를 재시작할 필요는 없음

JMX exporter#

https://github.com/prometheus/jmx_exporter

jar파일다운로드#

https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar

jmx exporter설정파일 생성#

kafka_config.yml
# https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-2_0_0.yml

lowercaseOutputName: true

rules:
# Special cases and very specific rules
- pattern : kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
  name: kafka_server_$1_$2
  type: GAUGE
  labels:
    clientId: "$3"
    topic: "$4"
    partition: "$5"
- pattern : kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
  name: kafka_server_$1_$2
  type: GAUGE
  labels:
    clientId: "$3"
    broker: "$4:$5"
- pattern : kafka.coordinator.(\w+)<type=(.+), name=(.+)><>Value
  name: kafka_coordinator_$1_$2_$3
  type: GAUGE

# Generic per-second counters with 0-2 key/value pairs
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count
  name: kafka_$1_$2_$3_total
  type: COUNTER
  labels:
    "$4": "$5"
    "$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count
  name: kafka_$1_$2_$3_total
  type: COUNTER
  labels:
    "$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
  name: kafka_$1_$2_$3_total
  type: COUNTER

# Generic gauges with 0-2 key/value pairs
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    "$4": "$5"
    "$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    "$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
  name: kafka_$1_$2_$3
  type: GAUGE

# Emulate Prometheus 'Summary' metrics for the exported 'Histogram's.
#
# Note that these are missing the '_sum' metric!
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count
  name: kafka_$1_$2_$3_count
  type: COUNTER
  labels:
    "$4": "$5"
    "$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    "$4": "$5"
    "$6": "$7"
    quantile: "0.$8"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count
  name: kafka_$1_$2_$3_count
  type: COUNTER
  labels:
    "$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    "$4": "$5"
    quantile: "0.$6"
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count
  name: kafka_$1_$2_$3_count
  type: COUNTER
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    quantile: "0.$4"

kafka에 jmx exporter 설정추가#

export KAFKA_OPTS="$KAFKA_OPTS -javaagent:/home/fromm0/apps/jmx_exporter/jmx_prometheus_javaagent-0.12.1-SNAPSHOT.jar=7071:/home/fromm0/apps/jmx_exporter/kafka_config.yml" 

... 중략

측정수치확인#

http://localhost:7071/metrics 접속

kafka_exporter#

https://github.com/danielqsj/kafka_exporter

필요한 사항#

export INSTALL_DIRECTORY=/home/fromm0/apps/go/bin && curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh

kafka_exporter 다운로드#

https://github.com/danielqsj/kafka_exporter/releases https://github.com/danielqsj/kafka_exporter/releases/download/v1.2.0/kafka_exporter-1.2.0.linux-amd64.tar.gz

실행#

kafka_exporter --kafka.server=server1:9092 \
--kafka.server=server2:9092 \
--kafka.server=server3:9092
kafka_exporter --kafka.server=server1:9092 \
--kafka.server=server2:9092 \
--kafka.server=server3:9092 \
--sasl.enabled \
--sasl.username=username \
--sasl.password=password \
--web.listen-address=:9309

수치확인#

http://localhost:9308/metrics

Prometheus #

Prometheus 다운로드#

https://github.com/prometheus/prometheus/releases https://github.com/prometheus/prometheus/releases/download/v2.12.0/prometheus-2.12.0.linux-amd64.tar.gz

prometheus.yml 설정#

global:
 scrape_interval: 10s
 evaluation_interval: 10s
scrape_configs:
 - job_name: 'kafka'
   static_configs:
    - targets: 
        - 'server1:9309'
        - 'server1:9308'
        - 'server2:7071'
        - 'server2:7071'
        - 'server2:7071'

prometheus 실행#

/home/fromm0/apps/prometheus/bin/prometheus &(info)

prometheus UI접속#

http://localhost:9090/graph

Grafana#

datasource 설정#

  • Name : 아무거나 (여기서는 Prometheus-kafka)
  • HTTP > Url : http://prometheus_server:9090/
  • 나머지는 디폴트값

kafka overview#

kafka exporter overview#

모니터링 URL#

Grafana QL#

sum(kafka_consumergroup_lag) by (consumergroup)

참고 URL#

  • https://www.robustperception.io/monitoring-kafka-with-prometheus
  • https://github.com/prometheus/jmx_exporter
    • https://grafana.com/grafana/dashboards/721
  • https://github.com/danielqsj/kafka_exporter
    • https://grafana.com/grafana/dashboards/7589

Add new attachment

Only authorized users are allowed to upload new attachments.
« This page (revision-3) was last changed on 25-Aug-2019 12:38 by DongGukLee  
G’day (anonymous guest) My Prefs

Referenced by
Monitoring

JSPWiki v2.8.4