zhongwei/gh-pythoninthegrass-cli-ninja-cli-ninja

Fork 0

Files

Zhongwei Li 87eb9f0383 Initial commit

2025-11-30 08:48:40 +08:00

11 KiB

Raw Permalink Blame History

jq - JSON Processing Cookbook

Overview

jq is a lightweight command-line JSON processor. It's like sed for JSON data.

Basic Queries

Identity and Pretty Print

# Identity (pretty print)
jq . file.json

# Compact output
jq -c . file.json

# Raw output (no quotes for strings)
jq -r . file.json

# Sort keys
jq -S . file.json

Accessing Fields

# Top-level field
jq '.name' file.json

# Nested field
jq '.user.email' file.json

# Deeply nested
jq '.data.user.profile.settings.theme' file.json

# Optional fields (null if missing)
jq '.user.address?' file.json

Array Access

# First element
jq '.[0]' file.json

# Last element
jq '.[-1]' file.json

# Specific index
jq '.[2]' file.json

# Slice (first 3 elements)
jq '.[:3]' file.json

# Slice (elements 2-5)
jq '.[2:5]' file.json

# All elements
jq '.[]' file.json

Iterating

Array Iteration

# Extract field from each element
jq '.users[].name' file.json

# Multiple fields
jq '.users[] | .name, .email' file.json

# Nested iteration
jq '.departments[].employees[].name' file.json

Object Iteration

# All values
jq '.[]' file.json

# All keys
jq 'keys' file.json

# Key-value pairs
jq 'to_entries' file.json

# Iterate key-value
jq 'to_entries[] | .key + ": " + (.value | tostring)' file.json

Filtering

Select

# Filter by condition
jq '.users[] | select(.active == true)' file.json

# Multiple conditions (AND)
jq '.users[] | select(.active == true and .age > 18)' file.json

# Multiple conditions (OR)
jq '.users[] | select(.active == true or .admin == true)' file.json

# Not
jq '.users[] | select(.active != false)' file.json

# Check field exists
jq '.users[] | select(.email != null)' file.json
jq '.users[] | select(has("email"))' file.json

Comparison Operators

# Equality
jq '.items[] | select(.status == "active")' file.json

# Greater/Less than
jq '.items[] | select(.price > 100)' file.json
jq '.items[] | select(.price <= 50)' file.json

# String contains
jq '.items[] | select(.name | contains("widget"))' file.json

# String starts with
jq '.items[] | select(.name | startswith("A"))' file.json

# String ends with
jq '.items[] | select(.name | endswith(".json"))' file.json

# Regex match
jq '.items[] | select(.email | test("@gmail\\.com$"))' file.json

Transforming

Map

# Extract single field
jq '.users | map(.name)' file.json

# Extract multiple fields
jq '.users | map({name, email})' file.json

# Transform values
jq '.prices | map(. * 1.1)' file.json

# Conditional transformation
jq '.users | map(if .active then .name else empty end)' file.json

Reduce

# Sum
jq '[.items[].price] | add' file.json

# Average
jq '[.items[].price] | add / length' file.json

# Min/Max
jq '[.items[].price] | min' file.json
jq '[.items[].price] | max' file.json

# Count
jq '.items | length' file.json

# Custom reduce
jq 'reduce .items[] as $item (0; . + $item.price)' file.json

Group By

# Group by field
jq 'group_by(.category)' file.json

# Group and count
jq 'group_by(.category) | map({category: .[0].category, count: length})' file.json

# Group and aggregate
jq 'group_by(.category) | map({
  category: .[0].category,
  total: map(.price) | add
})' file.json

Sorting

# Sort array
jq 'sort' file.json

# Sort by field
jq 'sort_by(.name)' file.json

# Reverse sort
jq 'sort_by(.name) | reverse' file.json

# Sort by multiple fields
jq 'sort_by(.category, .name)' file.json

# Sort numbers
jq 'sort_by(.price | tonumber)' file.json

Constructing

Objects

# New object with specific fields
jq '{name: .name, email: .email}' file.json

# Rename fields
jq '{username: .name, mail: .email}' file.json

# Computed fields
jq '{
  name: .name,
  fullName: .firstName + " " + .lastName,
  discountPrice: .price * 0.9
}' file.json

# Nested objects
jq '{
  user: {
    name: .name,
    contact: {email: .email}
  }
}' file.json

Arrays

# Create array from fields
jq '[.name, .email, .age]' file.json

# Array of objects
jq '[.users[] | {name, email}]' file.json

# Flatten nested arrays
jq '.departments[].employees | flatten' file.json

# Unique elements
jq '.tags | unique' file.json

# Array difference
jq '.a - .b' file.json

String Operations

# Concatenation
jq '.firstName + " " + .lastName' file.json

# Interpolation
jq '"Hello, \(.name)!"' file.json

# String length
jq '.name | length' file.json

# Uppercase/Lowercase
jq '.name | ascii_upcase' file.json
jq '.name | ascii_downcase' file.json

# Split string
jq '.path | split("/")' file.json

# Join array
jq '.tags | join(", ")' file.json

# Trim whitespace
jq '.text | gsub("^\\s+|\\s+$"; "")' file.json

# Replace
jq '.text | gsub("old"; "new")' file.json

Type Conversions

# To string
jq '.age | tostring' file.json

# To number
jq '.price | tonumber' file.json

# To array
jq '[.]' file.json

# Type check
jq '.value | type' file.json

# Type test
jq 'if .value | type == "string" then "is string" else "not string" end' file.json

Conditionals

# If-then-else
jq 'if .active then "active" else "inactive" end' file.json

# Multiple conditions
jq 'if .score >= 90 then "A"
    elif .score >= 80 then "B"
    elif .score >= 70 then "C"
    else "F"
    end' file.json

# Ternary-like
jq '.users[] | {name, status: (if .active then "active" else "inactive" end)}' file.json

# Alternative operator
jq '.value // "default"' file.json  # Use "default" if .value is null/false

Combining Data

Merging

# Merge objects
jq '. + {newField: "value"}' file.json

# Merge arrays
jq '.a + .b' file.json

# Recursive merge
jq '. * {user: {active: true}}' file.json

# Merge multiple files
jq -s '.[0] + .[1]' file1.json file2.json

Joining

# Array join
jq '.array1 + .array2' file.json

# Unique elements from multiple arrays
jq '(.array1 + .array2) | unique' file.json

# Intersection
jq '.array1 - (.array1 - .array2)' file.json

Advanced Patterns

Working with Nested Data

# Recursive descent
jq '.. | .name? // empty' file.json

# Find all values for a key
jq '.. | .email? // empty' file.json

# Flatten deeply nested structure
jq '[.. | .users? // empty] | flatten' file.json

Multiple Outputs

# Multiple queries
jq '.name, .email' file.json

# Conditional multiple outputs
jq '.users[] | .name, (if .admin then "ADMIN" else empty end)' file.json

Error Handling

# Try-catch
jq 'try .field catch "not found"' file.json

# Optional field access
jq '.user?.email?' file.json

# Default values
jq '.count // 0' file.json

Custom Functions

# Define and use function
jq 'def double: . * 2; .values[] | double' file.json

# Function with parameters
jq 'def multiply(n): . * n; .values[] | multiply(3)' file.json

# Recursive function
jq 'def factorial: if . <= 1 then 1 else . * ((. - 1) | factorial) end; 5 | factorial'

Practical Examples

API Response Processing

# Extract IDs
curl -s 'https://api.example.com/users' | jq '.data[].id'

# Format for CSV
curl -s 'https://api.example.com/users' | \
  jq -r '.data[] | [.id, .name, .email] | @csv'

# Extract nested data
curl -s 'https://api.example.com/posts' | \
  jq '.posts[] | {
    id,
    title,
    author: .user.name,
    comments: .comments | length
  }'

Configuration Management

# Update config value
jq '.database.host = "localhost"' config.json

# Add new field
jq '. + {newFeature: true}' config.json

# Remove field
jq 'del(.deprecated)' config.json

# Merge configs
jq -s '.[0] * .[1]' base.json override.json > merged.json

Data Analysis

# Count by category
jq 'group_by(.category) | map({category: .[0].category, count: length})' data.json

# Top 10 by price
jq 'sort_by(.price) | reverse | .[:10]' data.json

# Statistics
jq '{
  total: length,
  avgPrice: (map(.price) | add / length),
  maxPrice: (map(.price) | max),
  minPrice: (map(.price) | min)
}' data.json

Testing and Validation

# Check required fields
jq '.users[] | select(has("name") and has("email") | not)' data.json

# Validate email format
jq '.users[] | select(.email | test("^[^@]+@[^@]+$") | not)' data.json

# Find duplicates
jq 'group_by(.id) | map(select(length > 1))' data.json

Log Processing

# Filter log level
jq 'select(.level == "error")' logs.json

# Extract error messages
jq 'select(.level == "error") | .message' logs.json

# Group errors by type
jq -s 'group_by(.error_type) | map({type: .[0].error_type, count: length})' logs.json

# Time range filter
jq 'select(.timestamp >= "2024-01-01" and .timestamp < "2024-02-01")' logs.json

Output Formats

# Raw output (no quotes)
jq -r '.name' file.json

# Compact output
jq -c '.' file.json

# Tab-separated
jq -r '.[] | [.name, .age] | @tsv' file.json

# CSV
jq -r '.[] | [.name, .age, .email] | @csv' file.json

# URL encoding
jq -r '.params | @uri' file.json

# Base64 encoding
jq -r '.data | @base64' file.json

# HTML encoding
jq -r '.content | @html' file.json

# JSON output (default)
jq '.' file.json

Command-Line Options

# Read from stdin
echo '{"name":"test"}' | jq .

# Multiple files
jq . file1.json file2.json

# Slurp (read all files into array)
jq -s . file1.json file2.json

# Null input (generate data)
jq -n '{name: "test", date: now}'

# Exit with status code
jq -e 'select(.status == "ok")' file.json

# Tab indentation
jq --tab . file.json

# No color
jq -M . file.json

# Color (force)
jq -C . file.json

Tips and Tricks

Debugging

# Pretty print
jq . file.json | less

# Inspect structure
jq 'paths' file.json

# Show all keys
jq '[paths | select(length == 1)] | unique' file.json

# Debug filter
jq 'debug | .name' file.json

Performance

# Stream processing (memory efficient)
jq --stream . large.json

# Compact input
jq -c . file.json

# Limit output
jq 'limit(10; .items[])' file.json

Combining with Other Tools

# With curl
curl -s 'https://api.github.com/users/github' | jq '.name'

# With fd/rg
fd -e json | xargs jq '.version'

# With fzf
jq -r '.users[].name' users.json | fzf

# With xargs
jq -r '.files[]' manifest.json | xargs cat

Common Patterns

Filter-Map-Reduce

# Classic pattern
jq '.items[] |           # iterate
    select(.active) |    # filter
    .price |             # map
    add'                 # reduce

Pagination

# Page 1 (items 0-9)
jq '.items[0:10]' data.json

# Page 2 (items 10-19)
jq '.items[10:20]' data.json

# Function for pagination
page() {
  local page=$1
  local size=${2:-10}
  local start=$((page * size))
  local end=$((start + size))
  jq ".items[$start:$end]" data.json
}

Data Migration

# Old format to new format
jq 'map({
  id: .user_id,
  profile: {
    name: .username,
    email: .user_email
  }
})' old_format.json > new_format.json

Error Messages

# Common errors and fixes

# "parse error: Invalid numeric literal"
# → Check JSON syntax, trailing commas

# "Cannot iterate over null"
# → Add optional operator: .field[]?

# "Cannot index string with string"
# → Check data types, .field might be string not object

# "jq: error: syntax error"
# → Check quote escaping in shell

11 KiB Raw Permalink Blame History

jq - JSON Processing Cookbook

Overview

Basic Queries

Identity and Pretty Print

Accessing Fields

Array Access

Iterating

Array Iteration

Object Iteration

Filtering

Select

Comparison Operators

Transforming

Map

Reduce

Group By

Sorting

Constructing

Objects

Arrays

String Operations

Type Conversions

Conditionals

Combining Data

Merging

Joining

Advanced Patterns

Working with Nested Data

Multiple Outputs

Error Handling

Custom Functions

Practical Examples

API Response Processing

Configuration Management

Data Analysis

Testing and Validation

Log Processing

Output Formats

Command-Line Options

Tips and Tricks

Debugging

Performance

Combining with Other Tools

Common Patterns

Filter-Map-Reduce

Pagination

Data Migration

Error Messages

11 KiB

Raw Permalink Blame History