Initial commit

2025-11-29 17:56:46 +08:00
commit 96a7ab295d
16 changed files with 4441 additions and 0 deletions
--- a/skills/kafka-observability/SKILL.md
+++ b/skills/kafka-observability/SKILL.md
@@ -0,0 +1,576 @@
+---
+name: kafka-observability
+description: Kafka monitoring and observability expert. Guides Prometheus + Grafana setup, JMX metrics, alerting rules, and dashboard configuration. Activates for kafka monitoring, prometheus, grafana, kafka metrics, jmx exporter, kafka observability, monitoring setup, kafka dashboards, alerting, kafka performance monitoring, metrics collection.
+---
+
+# Kafka Monitoring & Observability
+
+Expert guidance for implementing comprehensive monitoring and observability for Apache Kafka using Prometheus and Grafana.
+
+## When to Use This Skill
+
+I activate when you need help with:
+- **Monitoring setup**: "Set up Kafka monitoring", "configure Prometheus for Kafka", "Grafana dashboards for Kafka"
+- **Metrics collection**: "Kafka JMX metrics", "export Kafka metrics to Prometheus"
+- **Alerting**: "Kafka alerting rules", "alert on under-replicated partitions", "critical Kafka metrics"
+- **Troubleshooting**: "Monitor Kafka performance", "track consumer lag", "broker health monitoring"
+
+## What I Know
+
+### Available Monitoring Components
+
+This plugin provides a complete monitoring stack:
+
+#### 1. **Prometheus JMX Exporter Configuration**
+- **Location**: `plugins/specweave-kafka/monitoring/prometheus/kafka-jmx-exporter.yml`
+- **Purpose**: Export Kafka JMX metrics to Prometheus format
+- **Metrics Exported**:
+  - Broker topic metrics (bytes in/out, messages in, request rate)
+  - Replica manager (under-replicated partitions, ISR shrinks/expands)
+  - Controller metrics (active controller, offline partitions, leader elections)
+  - Request metrics (produce/fetch latency)
+  - Log metrics (flush rate, flush latency)
+  - JVM metrics (heap, GC, threads, file descriptors)
+
+#### 2. **Grafana Dashboards** (5 Dashboards)
+- **Location**: `plugins/specweave-kafka/monitoring/grafana/dashboards/`
+- **Dashboards**:
+  1. **kafka-cluster-overview.json** - Cluster health and throughput
+  2. **kafka-broker-metrics.json** - Per-broker performance
+  3. **kafka-consumer-lag.json** - Consumer lag monitoring
+  4. **kafka-topic-metrics.json** - Topic-level metrics
+  5. **kafka-jvm-metrics.json** - JVM health (heap, GC, threads)
+
+#### 3. **Grafana Provisioning**
+- **Location**: `plugins/specweave-kafka/monitoring/grafana/provisioning/`
+- **Files**:
+  - `dashboards/kafka.yml` - Dashboard provisioning config
+  - `datasources/prometheus.yml` - Prometheus datasource config
+
+## Setup Workflow 1: JMX Exporter (Self-Hosted Kafka)
+
+For Kafka running on VMs or bare metal (non-Kubernetes).
+
+### Step 1: Download JMX Prometheus Agent
+
+```bash
+# Download JMX Prometheus agent JAR
+cd /opt
+wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.20.0/jmx_prometheus_javaagent-0.20.0.jar
+
+# Copy JMX Exporter config
+cp plugins/specweave-kafka/monitoring/prometheus/kafka-jmx-exporter.yml /opt/kafka-jmx-exporter.yml
+```
+
+### Step 2: Configure Kafka Broker
+
+Add JMX exporter to Kafka startup script:
+
+```bash
+# Edit Kafka startup (e.g., /etc/systemd/system/kafka.service)
+[Service]
+Environment="KAFKA_OPTS=-javaagent:/opt/jmx_prometheus_javaagent-0.20.0.jar=7071:/opt/kafka-jmx-exporter.yml"
+```
+
+Or add to `kafka-server-start.sh`:
+
+```bash
+export KAFKA_OPTS="-javaagent:/opt/jmx_prometheus_javaagent-0.20.0.jar=7071:/opt/kafka-jmx-exporter.yml"
+```
+
+### Step 3: Restart Kafka and Verify
+
+```bash
+# Restart Kafka broker
+sudo systemctl restart kafka
+
+# Verify JMX exporter is running (port 7071)
+curl localhost:7071/metrics | grep kafka_server
+
+# Expected output: kafka_server_broker_topic_metrics_bytesin_total{...} 12345
+```
+
+### Step 4: Configure Prometheus Scraping
+
+Add Kafka brokers to Prometheus config:
+
+```yaml
+# prometheus.yml
+scrape_configs:
+  - job_name: 'kafka'
+    static_configs:
+      - targets:
+        - 'kafka-broker-1:7071'
+        - 'kafka-broker-2:7071'
+        - 'kafka-broker-3:7071'
+    scrape_interval: 30s
+```
+
+```bash
+# Reload Prometheus
+sudo systemctl reload prometheus
+
+# OR send SIGHUP
+kill -HUP $(pidof prometheus)
+
+# Verify scraping
+curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.job=="kafka")'
+```
+
+## Setup Workflow 2: Strimzi (Kubernetes)
+
+For Kafka running on Kubernetes with Strimzi Operator.
+
+### Step 1: Create JMX Exporter ConfigMap
+
+```bash
+# Create ConfigMap from JMX exporter config
+kubectl create configmap kafka-metrics \
+  --from-file=kafka-metrics-config.yml=plugins/specweave-kafka/monitoring/prometheus/kafka-jmx-exporter.yml \
+  -n kafka
+```
+
+### Step 2: Configure Kafka CR with Metrics
+
+```yaml
+# kafka-cluster.yaml (add metricsConfig section)
+apiVersion: kafka.strimzi.io/v1beta2
+kind: Kafka
+metadata:
+  name: my-kafka-cluster
+  namespace: kafka
+spec:
+  kafka:
+    version: 3.7.0
+    replicas: 3
+
+    # ... other config ...
+
+    metricsConfig:
+      type: jmxPrometheusExporter
+      valueFrom:
+        configMapKeyRef:
+          name: kafka-metrics
+          key: kafka-metrics-config.yml
+```
+
+```bash
+# Apply updated Kafka CR
+kubectl apply -f kafka-cluster.yaml
+
+# Verify metrics endpoint (wait for rolling restart)
+kubectl exec -it kafka-my-kafka-cluster-0 -n kafka -- curl localhost:9404/metrics | grep kafka_server
+```
+
+### Step 3: Install Prometheus Operator (if not installed)
+
+```bash
+# Add Prometheus Community Helm repo
+helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+helm repo update
+
+# Install kube-prometheus-stack (Prometheus + Grafana + Alertmanager)
+helm install prometheus prometheus-community/kube-prometheus-stack \
+  --namespace monitoring \
+  --create-namespace \
+  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
+  --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false
+```
+
+### Step 4: Create PodMonitor for Kafka
+
+```yaml
+# kafka-podmonitor.yaml
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: kafka-metrics
+  namespace: kafka
+  labels:
+    app: strimzi
+spec:
+  selector:
+    matchLabels:
+      strimzi.io/kind: Kafka
+  podMetricsEndpoints:
+    - port: tcp-prometheus
+      interval: 30s
+```
+
+```bash
+# Apply PodMonitor
+kubectl apply -f kafka-podmonitor.yaml
+
+# Verify Prometheus is scraping Kafka
+kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090
+# Open: http://localhost:9090/targets
+# Should see kafka-metrics/* targets
+```
+
+## Setup Workflow 3: Grafana Dashboards
+
+### Installation (Docker Compose)
+
+If using Docker Compose for local development:
+
+```yaml
+# docker-compose.yml (add to existing Kafka setup)
+version: '3.8'
+services:
+  # ... Kafka services ...
+
+  prometheus:
+    image: prom/prometheus:v2.48.0
+    ports:
+      - "9090:9090"
+    volumes:
+      - ./monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
+      - prometheus-data:/prometheus
+    command:
+      - '--config.file=/etc/prometheus/prometheus.yml'
+      - '--storage.tsdb.path=/prometheus'
+
+  grafana:
+    image: grafana/grafana:10.2.0
+    ports:
+      - "3000:3000"
+    environment:
+      - GF_SECURITY_ADMIN_PASSWORD=admin
+    volumes:
+      - ./monitoring/grafana/provisioning:/etc/grafana/provisioning
+      - ./monitoring/grafana/dashboards:/var/lib/grafana/dashboards
+      - grafana-data:/var/lib/grafana
+
+volumes:
+  prometheus-data:
+  grafana-data:
+```
+
+```bash
+# Start monitoring stack
+docker-compose up -d prometheus grafana
+
+# Access Grafana
+# URL: http://localhost:3000
+# Username: admin
+# Password: admin
+```
+
+### Installation (Kubernetes)
+
+Dashboards are auto-provisioned if using kube-prometheus-stack:
+
+```bash
+# Create ConfigMaps for each dashboard
+for dashboard in plugins/specweave-kafka/monitoring/grafana/dashboards/*.json; do
+  name=$(basename "$dashboard" .json)
+  kubectl create configmap "kafka-dashboard-$name" \
+    --from-file="$dashboard" \
+    -n monitoring \
+    --dry-run=client -o yaml | kubectl apply -f -
+done
+
+# Label ConfigMaps for Grafana auto-discovery
+kubectl label configmap -n monitoring kafka-dashboard-* grafana_dashboard=1
+
+# Grafana will auto-import dashboards (wait 30-60 seconds)
+
+# Access Grafana
+kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80
+# URL: http://localhost:3000
+# Username: admin
+# Password: prom-operator (default kube-prometheus-stack password)
+```
+
+### Manual Dashboard Import
+
+If auto-provisioning doesn't work:
+
+```bash
+# 1. Access Grafana UI
+# 2. Go to: Dashboards → Import
+# 3. Upload JSON files from:
+#    plugins/specweave-kafka/monitoring/grafana/dashboards/
+
+# Or use Grafana API
+for dashboard in plugins/specweave-kafka/monitoring/grafana/dashboards/*.json; do
+  curl -X POST http://admin:admin@localhost:3000/api/dashboards/db \
+    -H "Content-Type: application/json" \
+    -d @"$dashboard"
+done
+```
+
+## Dashboard Overview
+
+### 1. **Kafka Cluster Overview** (`kafka-cluster-overview.json`)
+
+**Purpose**: High-level cluster health
+
+**Key Metrics**:
+- Active Controller Count (should be exactly 1)
+- Under-Replicated Partitions (should be 0) ⚠️  CRITICAL
+- Offline Partitions Count (should be 0) ⚠️  CRITICAL
+- Unclean Leader Elections (should be 0)
+- Cluster Throughput (bytes in/out per second)
+- Request Rate (produce, fetch requests per second)
+- ISR Changes (shrinks/expands)
+- Leader Election Rate
+
+**Use When**: Checking overall cluster health
+
+### 2. **Kafka Broker Metrics** (`kafka-broker-metrics.json`)
+
+**Purpose**: Per-broker performance
+
+**Key Metrics**:
+- Broker CPU Usage (% utilization)
+- Broker Heap Memory Usage
+- Broker Network Throughput (bytes in/out)
+- Request Handler Idle Percentage (low = CPU saturation)
+- File Descriptors (open vs max)
+- Log Flush Latency (p50, p99)
+- JVM GC Collection Count/Time
+
+**Use When**: Investigating broker performance issues
+
+### 3. **Kafka Consumer Lag** (`kafka-consumer-lag.json`)
+
+**Purpose**: Consumer lag monitoring
+
+**Key Metrics**:
+- Consumer Lag per Topic/Partition
+- Total Lag per Consumer Group
+- Offset Commit Rate
+- Current Consumer Offset
+- Log End Offset (producer offset)
+- Consumer Group Members
+
+**Use When**: Troubleshooting slow consumers or lag spikes
+
+### 4. **Kafka Topic Metrics** (`kafka-topic-metrics.json`)
+
+**Purpose**: Topic-level metrics
+
+**Key Metrics**:
+- Messages Produced per Topic
+- Bytes per Topic (in/out)
+- Partition Count per Topic
+- Replication Factor
+- In-Sync Replicas
+- Log Size per Partition
+- Current Offset per Partition
+- Partition Leader Distribution
+
+**Use When**: Analyzing topic throughput and hotspots
+
+### 5. **Kafka JVM Metrics** (`kafka-jvm-metrics.json`)
+
+**Purpose**: JVM health monitoring
+
+**Key Metrics**:
+- Heap Memory Usage (used vs max)
+- Heap Utilization Percentage
+- GC Collection Rate (collections/sec)
+- GC Collection Time (ms/sec)
+- JVM Thread Count
+- Heap Memory by Pool (young gen, old gen, survivor)
+- Off-Heap Memory Usage (metaspace, code cache)
+- GC Pause Time Percentiles (p50, p95, p99)
+
+**Use When**: Investigating memory leaks or GC pauses
+
+## Critical Alerts Configuration
+
+Create Prometheus alerting rules for critical Kafka metrics:
+
+```yaml
+# kafka-alerts.yml
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: kafka-alerts
+  namespace: monitoring
+spec:
+  groups:
+    - name: kafka.rules
+      interval: 30s
+      rules:
+        # CRITICAL: Under-Replicated Partitions
+        - alert: KafkaUnderReplicatedPartitions
+          expr: sum(kafka_server_replica_manager_under_replicated_partitions) > 0
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Kafka has under-replicated partitions"
+            description: "{{ $value }} partitions are under-replicated. Data loss risk!"
+
+        # CRITICAL: Offline Partitions
+        - alert: KafkaOfflinePartitions
+          expr: kafka_controller_offline_partitions_count > 0
+          for: 1m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Kafka has offline partitions"
+            description: "{{ $value }} partitions are offline. Service degradation!"
+
+        # CRITICAL: No Active Controller
+        - alert: KafkaNoActiveController
+          expr: kafka_controller_active_controller_count == 0
+          for: 1m
+          labels:
+            severity: critical
+          annotations:
+            summary: "No active Kafka controller"
+            description: "Cluster has no active controller. Cannot perform administrative operations!"
+
+        # WARNING: High Consumer Lag
+        - alert: KafkaConsumerLagHigh
+          expr: sum by (consumergroup) (kafka_consumergroup_lag) > 10000
+          for: 10m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Consumer group {{ $labels.consumergroup }} has high lag"
+            description: "Lag is {{ $value }} messages. Consumers may be slow."
+
+        # WARNING: High CPU Usage
+        - alert: KafkaBrokerHighCPU
+          expr: os_process_cpu_load{job="kafka"} > 0.8
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Broker {{ $labels.instance }} has high CPU usage"
+            description: "CPU usage is {{ $value | humanizePercentage }}. Consider scaling."
+
+        # WARNING: Low Heap Memory
+        - alert: KafkaBrokerLowHeapMemory
+          expr: jvm_memory_heap_used_bytes{job="kafka"} / jvm_memory_heap_max_bytes{job="kafka"} > 0.9
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Broker {{ $labels.instance }} has low heap memory"
+            description: "Heap usage is {{ $value | humanizePercentage }}. Risk of OOM!"
+
+        # WARNING: High GC Time
+        - alert: KafkaBrokerHighGCTime
+          expr: rate(jvm_gc_collection_time_ms_total{job="kafka"}[5m]) > 500
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Broker {{ $labels.instance }} spending too much time in GC"
+            description: "GC time is {{ $value }}ms/sec. Application pauses likely."
+```
+
+```bash
+# Apply alerts (Kubernetes)
+kubectl apply -f kafka-alerts.yml
+
+# Verify alerts loaded
+kubectl get prometheusrules -n monitoring
+```
+
+## Troubleshooting
+
+### "Prometheus not scraping Kafka metrics"
+
+**Symptoms**: No Kafka metrics in Prometheus
+
+**Fix**:
+```bash
+# 1. Verify JMX exporter is running
+curl http://kafka-broker:7071/metrics
+
+# 2. Check Prometheus targets
+curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.job=="kafka")'
+
+# 3. Check Prometheus logs
+kubectl logs -n monitoring prometheus-kube-prometheus-prometheus-0
+
+# Common issues:
+# - Firewall blocking port 7071
+# - Incorrect scrape config
+# - Kafka broker not running
+```
+
+### "Grafana dashboards not loading"
+
+**Symptoms**: Dashboards show "No data"
+
+**Fix**:
+```bash
+# 1. Verify Prometheus datasource
+# Grafana UI → Configuration → Data Sources → Prometheus → Test
+
+# 2. Check if Kafka metrics exist in Prometheus
+# Prometheus UI → Graph → Enter: kafka_server_broker_topic_metrics_bytesin_total
+
+# 3. Verify dashboard queries match your Prometheus job name
+# Dashboard panels use job="kafka" by default
+# If your job name is different, update dashboard JSON
+```
+
+### "Consumer lag metrics missing"
+
+**Symptoms**: Consumer lag dashboard empty
+
+**Fix**:
+Consumer lag metrics require **Kafka Exporter** (separate from JMX Exporter):
+
+```bash
+# Install Kafka Exporter (Kubernetes)
+helm install kafka-exporter prometheus-community/prometheus-kafka-exporter \
+  --namespace monitoring \
+  --set kafkaServer={kafka-bootstrap:9092}
+
+# Or run as Docker container
+docker run -d -p 9308:9308 \
+  danielqsj/kafka-exporter \
+  --kafka.server=kafka:9092 \
+  --web.listen-address=:9308
+
+# Add to Prometheus scrape config
+scrape_configs:
+  - job_name: 'kafka-exporter'
+    static_configs:
+      - targets: ['kafka-exporter:9308']
+```
+
+## Integration with Other Skills
+
+- **kafka-iac-deployment**: Set up monitoring during Terraform deployment
+- **kafka-kubernetes**: Configure monitoring for Strimzi Kafka on K8s
+- **kafka-architecture**: Use cluster sizing metrics to validate capacity planning
+- **kafka-cli-tools**: Use kcat to generate test traffic and verify metrics
+
+## Quick Reference Commands
+
+```bash
+# Check JMX exporter metrics
+curl http://localhost:7071/metrics | grep -E "(kafka_server|kafka_controller)"
+
+# Prometheus query examples
+curl -g 'http://localhost:9090/api/v1/query?query=kafka_server_replica_manager_under_replicated_partitions'
+
+# Grafana dashboard export
+curl http://admin:admin@localhost:3000/api/dashboards/uid/kafka-cluster-overview | jq .dashboard > backup.json
+
+# Reload Prometheus config
+kill -HUP $(pidof prometheus)
+
+# Check Prometheus targets
+curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.job=="kafka")'
+```
+
+---
+
+**Next Steps After Monitoring Setup**:
+1. Review all 5 Grafana dashboards to familiarize yourself with metrics
+2. Set up alerting (Slack, PagerDuty, email)
+3. Create runbooks for critical alerts (under-replicated partitions, offline partitions, no controller)
+4. Monitor for 7 days to establish baseline metrics
+5. Tune JVM settings based on GC metrics