gh-cskiro-claudex-claude-code-tools/skills/otel-monitoring-setup/data/prometheus-queries.md

# Useful Prometheus Queries (PromQL)

Collection of useful PromQL queries for Claude Code telemetry analysis.

**Note:** All queries use the double prefix: `claude_code_claude_code_*`

---

## Cost Analysis

### Daily Cost Trend
```promql
sum(increase(claude_code_claude_code_cost_usage_USD_total[1d]))
```

### Cost by Model
```promql
sum by (model) (increase(claude_code_claude_code_cost_usage_USD_total[24h]))
```

### Cost per Hour (Rate)
```promql
rate(claude_code_claude_code_cost_usage_USD_total[5m]) * 3600
```

### Average Cost per Session
```promql
sum(increase(claude_code_claude_code_cost_usage_USD_total[24h]))
/
sum(increase(claude_code_claude_code_session_count_total[24h]))
```

### Cumulative Monthly Cost
```promql
sum(increase(claude_code_claude_code_cost_usage_USD_total[30d]))
```

### Cost by Team
```promql
sum by (team) (increase(claude_code_claude_code_cost_usage_USD_total[24h]))
```

### Projected Monthly Cost (based on last 7 days)
```promql
(sum(increase(claude_code_claude_code_cost_usage_USD_total[7d])) / 7) * 30
```

---

## Token Usage

### Total Tokens by Type
```promql
sum by (type) (increase(claude_code_claude_code_token_usage_tokens_total[24h]))
```

### Tokens by Model
```promql
sum by (model) (increase(claude_code_claude_code_token_usage_tokens_total[24h]))
```

### Cache Hit Rate
```promql
sum(increase(claude_code_claude_code_token_usage_tokens_total{type="cache_read"}[24h]))
/
sum(increase(claude_code_claude_code_token_usage_tokens_total{type=~"input|cache_creation|cache_read"}[24h]))
* 100
```

### Input vs Output Token Ratio
```promql
sum(increase(claude_code_claude_code_token_usage_tokens_total{type="input"}[24h]))
/
sum(increase(claude_code_claude_code_token_usage_tokens_total{type="output"}[24h]))
```

### Token Usage Rate (per minute)
```promql
sum by (type) (rate(claude_code_claude_code_token_usage_tokens_total[5m]) * 60)
```

### Total Tokens (All Time)
```promql
sum(claude_code_claude_code_token_usage_tokens_total)
```

---

## Productivity Metrics

### Total Lines of Code Modified
```promql
sum(claude_code_claude_code_lines_of_code_count_total)
```

### LOC by Type (Added, Changed, Deleted)
```promql
sum by (type) (increase(claude_code_claude_code_lines_of_code_count_total[24h]))
```

### LOC per Hour
```promql
rate(claude_code_claude_code_lines_of_code_count_total[5m]) * 3600
```

### Lines per Dollar (Efficiency)
```promql
sum(increase(claude_code_claude_code_lines_of_code_count_total[24h]))
/
sum(increase(claude_code_claude_code_cost_usage_USD_total[24h]))
```

### Commits per Day
```promql
increase(claude_code_claude_code_commit_count_total[24h])
```

### PRs per Week
```promql
increase(claude_code_claude_code_pr_count_total[7d])
```

### LOC per Commit
```promql
sum(increase(claude_code_claude_code_lines_of_code_count_total[24h]))
/
sum(increase(claude_code_claude_code_commit_count_total[24h]))
```

---

## Session Analytics

### Total Sessions
```promql
sum(claude_code_claude_code_session_count_total)
```

### New Sessions (24h)
```promql
increase(claude_code_claude_code_session_count_total[24h])
```

### Active Users (Unique account_uuids)
```promql
count(count by (account_uuid) (claude_code_claude_code_session_count_total))
```

### Average Session Duration
```promql
sum(increase(claude_code_claude_code_active_time_seconds_total[24h]))
/
sum(increase(claude_code_claude_code_session_count_total[24h]))
/ 60
```
*Result in minutes*

### Total Active Hours (24h)
```promql
sum(increase(claude_code_claude_code_active_time_seconds_total[24h])) / 3600
```

### Sessions by Version
```promql
sum by (version) (increase(claude_code_claude_code_session_count_total[24h]))
```

---

## Team Aggregation

### Cost by Team (Last 24h)
```promql
sum by (team) (increase(claude_code_claude_code_cost_usage_USD_total[24h]))
```

### LOC by Team (Last 24h)
```promql
sum by (team) (increase(claude_code_claude_code_lines_of_code_count_total[24h]))
```

### Active Users per Team
```promql
count by (team) (count by (team, account_uuid) (claude_code_claude_code_session_count_total))
```

### Team Efficiency (LOC per Dollar)
```promql
sum by (team) (increase(claude_code_claude_code_lines_of_code_count_total[24h]))
/
sum by (team) (increase(claude_code_claude_code_cost_usage_USD_total[24h]))
```

### Top Spending Teams (Last 7 days)
```promql
topk(5, sum by (team) (increase(claude_code_claude_code_cost_usage_USD_total[7d])))
```

---

## Model Comparison

### Cost by Model (Pie Chart)
```promql
sum by (model) (increase(claude_code_claude_code_cost_usage_USD_total[7d]))
```

### Token Efficiency by Model (Tokens per Dollar)
```promql
sum by (model) (increase(claude_code_claude_code_token_usage_tokens_total[24h]))
/
sum by (model) (increase(claude_code_claude_code_cost_usage_USD_total[24h]))
```

### Most Used Model
```promql
topk(1, sum by (model) (increase(claude_code_claude_code_token_usage_tokens_total[24h])))
```

### Model Usage Distribution (%)
```promql
sum by (model) (increase(claude_code_claude_code_token_usage_tokens_total[24h]))
/
sum(increase(claude_code_claude_code_token_usage_tokens_total[24h]))
* 100
```

---

## Alerting Queries

### High Daily Cost Alert (> $50)
```promql
sum(increase(claude_code_claude_code_cost_usage_USD_total[24h])) > 50
```

### Cost Spike Alert (50% increase compared to yesterday)
```promql
sum(increase(claude_code_claude_code_cost_usage_USD_total[24h]))
/
sum(increase(claude_code_claude_code_cost_usage_USD_total[24h] offset 24h))
> 1.5
```

### No Activity Alert (no sessions in last hour)
```promql
increase(claude_code_claude_code_session_count_total[1h]) == 0
```

### Low Cache Hit Rate Alert (< 20%)
```promql
(
  sum(increase(claude_code_claude_code_token_usage_tokens_total{type="cache_read"}[1h]))
  /
  sum(increase(claude_code_claude_code_token_usage_tokens_total{type=~"input|cache_creation|cache_read"}[1h]))
  * 100
) < 20
```

---

## Forecasting

### Projected Monthly Cost (based on last 7 days)
```promql
(sum(increase(claude_code_claude_code_cost_usage_USD_total[7d])) / 7) * 30
```

### Projected Annual Cost (based on last 30 days)
```promql
(sum(increase(claude_code_claude_code_cost_usage_USD_total[30d])) / 30) * 365
```

### Average Daily Cost (Last 30 days)
```promql
sum(increase(claude_code_claude_code_cost_usage_USD_total[30d])) / 30
```

### Growth Rate (Week over Week)
```promql
(
  sum(increase(claude_code_claude_code_cost_usage_USD_total[7d]))
  -
  sum(increase(claude_code_claude_code_cost_usage_USD_total[7d] offset 7d))
)
/
sum(increase(claude_code_claude_code_cost_usage_USD_total[7d] offset 7d))
* 100
```
*Result as percentage*

---

## Debugging Queries

### Check if Metrics Exist
```promql
claude_code_claude_code_session_count_total
```

### List All Claude Code Metrics
```
# Use Prometheus UI or API
curl -s 'http://localhost:9090/api/v1/label/__name__/values' | jq . | grep claude_code
```

### Check Metric Labels
```promql
# Returns all label combinations
count by (account_uuid, version, team, environment) (claude_code_claude_code_session_count_total)
```

### Latest Value for All Metrics
```promql
# Session count
claude_code_claude_code_session_count_total

# Cost
claude_code_claude_code_cost_usage_USD_total

# Tokens
claude_code_claude_code_token_usage_tokens_total

# LOC
claude_code_claude_code_lines_of_code_count_total
```

### Metrics Cardinality (Number of Time Series)
```promql
count(claude_code_claude_code_token_usage_tokens_total)
```

---

## Recording Rules

Save these as Prometheus recording rules for faster dashboard queries:

```yaml
groups:
  - name: claude_code_aggregations
    interval: 1m
    rules:
      # Daily cost
      - record: claude_code:cost_usd:daily
        expr: sum(increase(claude_code_claude_code_cost_usage_USD_total[24h]))

      # Cost by team
      - record: claude_code:cost_usd:daily:by_team
        expr: sum by (team) (increase(claude_code_claude_code_cost_usage_USD_total[24h]))

      # Cache hit rate
      - record: claude_code:cache_hit_rate:daily
        expr: |
          sum(increase(claude_code_claude_code_token_usage_tokens_total{type="cache_read"}[24h]))
          /
          sum(increase(claude_code_claude_code_token_usage_tokens_total{type=~"input|cache_creation|cache_read"}[24h]))
          * 100

      # LOC efficiency
      - record: claude_code:loc_per_dollar:daily
        expr: |
          sum(increase(claude_code_claude_code_lines_of_code_count_total[24h]))
          /
          sum(increase(claude_code_claude_code_cost_usage_USD_total[24h]))
```

Then use simplified queries:
```promql
# Instead of complex query, just use:
claude_code:cost_usd:daily
claude_code:cost_usd:daily:by_team
```

---

## Visualization Tips

### Time Series Panel
- Use `rate()` for smooth trends
- Set legend to `{{label_name}}` for clarity
- Enable "Lines" draw style with opacity

### Stat Panel
- Use `lastNotNull` for counters
- Use `increase([24h])` for daily totals
- Add thresholds for color coding

### Bar Chart
- Use `sum by (label)` for grouping
- Sort by value descending
- Limit to top 10 with `topk(10, ...)`

### Pie Chart
- Calculate percentages with division
- Use `sum by (label)` for segments
- Limit to top categories

---

## Additional Resources

- **Prometheus Query Docs:** https://prometheus.io/docs/prometheus/latest/querying/basics/
- **PromQL Examples:** https://prometheus.io/docs/prometheus/latest/querying/examples/
- **Grafana Query Editor:** https://grafana.com/docs/grafana/latest/datasources/prometheus/