zhongwei/gh-pythoninthegrass-cli-ninja-cli-ninja

Fork 0

Files

Zhongwei Li 87eb9f0383 Initial commit

2025-11-30 08:48:40 +08:00

12 KiB

Raw Blame History

yq - YAML/XML Processing Examples

Overview

yq is a lightweight command-line YAML/XML processor. It's like jq but for YAML and XML.

YAML Processing

Basic Queries

# Pretty print
yq . file.yaml

# Access field
yq '.name' file.yaml

# Nested access
yq '.database.host' file.yaml

# Array access
yq '.items[0]' file.yaml

# All array elements
yq '.items[]' file.yaml

Filtering

# Filter by condition
yq '.users[] | select(.active == true)' file.yaml

# Multiple conditions
yq '.users[] | select(.age > 18 and .active == true)' file.yaml

# String contains
yq '.items[] | select(.name | contains("widget"))' file.yaml

Transforming

# Extract specific fields
yq '.users[] | {name, email}' file.yaml

# Map array
yq '.items | map(.price * 1.1)' file.yaml

# Sort
yq 'sort_by(.name)' file.yaml

# Group by
yq 'group_by(.category)' file.yaml

Updating Values

# Update single field
yq '.database.host = "localhost"' file.yaml

# Update nested field
yq '.server.config.port = 8080' file.yaml

# Update array element
yq '.items[0].price = 99.99' file.yaml

# Add new field
yq '. + {"newField": "value"}' file.yaml

# Delete field
yq 'del(.deprecated)' file.yaml

In-Place Editing

# Edit file in-place
yq -i '.version = "2.0"' file.yaml

# Multiple updates
yq -i '
  .version = "2.0" |
  .updated = now |
  .deprecated = null
' file.yaml

Merging Files

# Merge two YAML files
yq '. *= load("other.yaml")' base.yaml

# Merge with override
yq 'load("base.yaml") * load("override.yaml")'

# Merge multiple files
yq eval-all '. as $item ireduce ({}; . * $item)' file1.yaml file2.yaml file3.yaml

Format Conversion

YAML to JSON

# Convert YAML to JSON
yq -o json . file.yaml

# Compact JSON
yq -o json -I 0 . file.yaml

# Pretty JSON
yq -o json . file.yaml | jq .

JSON to YAML

# Convert JSON to YAML
yq -P . file.json

# From stdin
echo '{"name":"test"}' | yq -P .

# Pipeline
cat file.json | yq -P . > file.yaml

YAML to TOML

# Convert YAML to TOML
yq -o toml . file.yaml

YAML to XML

# Convert YAML to XML
yq -o xml . file.yaml

XML Processing

Reading XML

# Parse XML
yq -p xml . file.xml

# Access element
yq -p xml '.root.element' file.xml

# Access attribute
yq -p xml '.root.+@attr' file.xml

# All elements
yq -p xml '.root.items[]' file.xml

Filtering XML

# Filter by attribute
yq -p xml '.root.item[] | select(.+@id == "123")' file.xml

# Filter by element value
yq -p xml '.root.item[] | select(.name == "test")' file.xml

XML to JSON

# Convert XML to JSON
yq -p xml -o json . file.xml

# Extract and convert
yq -p xml -o json '.root.items' file.xml

XML to YAML

# Convert XML to YAML
yq -p xml . file.xml

# Pretty print
yq -p xml -P . file.xml

Common Use Cases

Docker Compose

# Get service names
yq '.services | keys' docker-compose.yml

# Get image for service
yq '.services.web.image' docker-compose.yml

# Update service port
yq -i '.services.web.ports[0] = "8080:80"' docker-compose.yml

# Add environment variable
yq -i '.services.web.environment.DEBUG = "true"' docker-compose.yml

# Get all exposed ports
yq '.services[].ports[]' docker-compose.yml

Kubernetes

# Get pod name
yq '.metadata.name' pod.yaml

# Get container image
yq '.spec.containers[0].image' pod.yaml

# Update replicas
yq -i '.spec.replicas = 3' deployment.yaml

# Get all container names
yq '.spec.template.spec.containers[].name' deployment.yaml

# Get resource limits
yq '.spec.template.spec.containers[0].resources.limits' deployment.yaml

CI/CD (GitHub Actions, GitLab CI)

# Get job names
yq '.jobs | keys' .github/workflows/ci.yml

# Get steps for job
yq '.jobs.build.steps[].name' .github/workflows/ci.yml

# Update trigger
yq -i '.on.push.branches = ["main", "develop"]' .github/workflows/ci.yml

# Get all used actions
yq '.jobs[].steps[].uses' .github/workflows/ci.yml

Configuration Files

# Application config
yq '.app.port' config.yaml
yq '.database.connection.host' config.yaml

# Update config
yq -i '.app.debug = true' config.yaml
yq -i '.cache.enabled = false' config.yaml

# Merge configs
yq '. *= load("config.local.yaml")' config.yaml

Ansible

# Get hosts
yq '.all.hosts | keys' inventory.yaml

# Get variables
yq '.all.vars' inventory.yaml

# Update host variable
yq -i '.all.hosts.server1.ansible_host = "192.168.1.10"' inventory.yaml

# Get all playbook tasks
yq '.[] | select(.tasks) | .tasks[].name' playbook.yaml

Advanced Patterns

Conditional Updates

# Update if exists
yq '(select(.field) | .field) = "new-value"' file.yaml

# Update based on condition
yq '(.items[] | select(.price > 100) | .discount) = 0.2' file.yaml

# Add field if missing
yq '. + (if has("field") then {} else {"field": "default"} end)' file.yaml

Complex Transformations

# Restructure data
yq '.users | map({
  username: .name,
  contact: {
    email: .email,
    phone: .phone
  }
})' file.yaml

# Aggregate data
yq '.items | group_by(.category) | map({
  category: .[0].category,
  total: map(.price) | add
})' file.yaml

# Pivot data
yq 'reduce .items[] as $item ({}; .[$item.name] = $item.value)' file.yaml

Working with Arrays

# Append to array
yq '.items += ["new-item"]' file.yaml

# Prepend to array
yq '.items = ["new-item"] + .items' file.yaml

# Remove from array
yq 'del(.items[2])' file.yaml

# Filter array
yq '.items = [.items[] | select(.active == true)]' file.yaml

# Unique array
yq '.items | unique' file.yaml

# Flatten array
yq '.items | flatten' file.yaml

Multi-Document YAML

# Process all documents
yq '.' multi-doc.yaml

# Select specific document
yq 'select(document_index == 0)' multi-doc.yaml

# Filter documents
yq 'select(.kind == "Deployment")' multi-doc.yaml

# Update all documents
yq -i '.metadata.labels.app = "myapp"' multi-doc.yaml

String Operations

# Concatenation
yq '.fullName = .firstName + " " + .lastName' file.yaml

# Interpolation
yq '.message = "Hello, " + .name + "!"' file.yaml

# Uppercase
yq '.name | upcase' file.yaml

# Lowercase
yq '.name | downcase' file.yaml

# Split
yq '.path | split("/")' file.yaml

# Join
yq '.items | join(", ")' file.yaml

# Replace
yq '.text | sub("old", "new")' file.yaml

# Trim
yq '.text | trim' file.yaml

Type Checking and Conversion

# Check type
yq '.field | type' file.yaml

# Convert to string
yq '.number | tostring' file.yaml

# Convert to number
yq '.string | tonumber' file.yaml

# Check if exists
yq 'has("field")' file.yaml

# Length
yq '.items | length' file.yaml

Comments

# Preserve comments (default behavior)
yq '.field = "new-value"' file.yaml

# Add headComment
yq '.field headComment="This is a comment"' file.yaml

# Add lineComment
yq '.field lineComment="inline comment"' file.yaml

# Add footComment
yq '.field footComment="bottom comment"' file.yaml

Anchors and Aliases

# Create anchor
yq '.base.&anchor = {"key": "value"}' file.yaml

# Reference anchor
yq '.other = .base.*anchor' file.yaml

# Merge anchors
yq '.combined = .base.*anchor * .override' file.yaml

Validation

# Check if valid YAML
yq . file.yaml > /dev/null && echo "Valid" || echo "Invalid"

# Validate schema (requires schema file)
yq 'load("schema.yaml") | . as $schema | load("data.yaml") | validate($schema)'

# Check required fields
yq 'select(has("requiredField") | not)' file.yaml

# Validate format
yq 'select(.email | test("^[^@]+@[^@]+$") | not)' file.yaml

Output Formatting

# Default YAML output
yq . file.yaml

# Compact (minimal whitespace)
yq -I 0 . file.yaml

# Custom indentation (4 spaces)
yq -I 4 . file.yaml

# No colors
yq -C 0 . file.yaml

# Force colors
yq -C 1 . file.yaml

# Raw output (no quotes)
yq -r '.name' file.yaml

# One line per document
yq -o json . multi-doc.yaml

Scripting Examples

Environment-Specific Configs

#!/bin/bash
ENV=${1:-dev}

yq ". *= load(\"config.$ENV.yaml\")" config.base.yaml > config.yaml

Bulk Updates

#!/bin/bash
for file in *.yaml; do
  yq -i '.metadata.labels.version = "2.0"' "$file"
done

Config Validation

#!/bin/bash
REQUIRED_FIELDS=("name" "version" "description")

for field in "${REQUIRED_FIELDS[@]}"; do
  if ! yq -e "has(\"$field\")" config.yaml > /dev/null; then
    echo "Missing required field: $field"
    exit 1
  fi
done

Secret Management

#!/bin/bash
# Extract secrets from YAML
yq '.secrets' config.yaml > secrets.yaml

# Encrypt secrets (example with age)
yq '.secrets' config.yaml | age -e -o secrets.age

# Decrypt and merge
age -d secrets.age | yq -P . > secrets.yaml
yq '. *= load("secrets.yaml")' config.yaml

Integration Examples

With Git

# Pre-commit hook to sort keys
git diff --cached --name-only | \
  grep '\.yaml$' | \
  xargs -I {} yq -i -P 'sort_keys(..)' {}

With Docker

# Extract image tags
yq '.services[].image' docker-compose.yml | \
  cut -d: -f2 | \
  sort -u

# Update all images to latest
yq -i '(.services[].image) |= sub(":.*", ":latest")' docker-compose.yml

With Kubernetes

# Apply with modifications
yq '.spec.replicas = 3' deployment.yaml | kubectl apply -f -

# Extract all images
kubectl get pods -o yaml | \
  yq '.items[].spec.containers[].image' | \
  sort -u

# Generate manifests
yq eval-all '. as $item ireduce ({}; . * $item)' \
  base/*.yaml overlays/prod/*.yaml | \
  kubectl apply -f -

With CI/CD

# Update version in workflow
VERSION=$(git describe --tags)
yq -i ".env.VERSION = \"$VERSION\"" .github/workflows/deploy.yml

# Extract job matrix
MATRIX=$(yq -o json '.jobs.test.strategy.matrix' .github/workflows/ci.yml)
echo "matrix=$MATRIX" >> $GITHUB_OUTPUT

Performance Tips

Use in-place editing: -i avoids reading/writing twice
Pipe instead of temp files: More efficient for transformations
Specific paths: Access .field.subfield instead of iterating
Limit output: Use select early to filter data
Batch operations: Combine multiple updates in one command

Common Patterns

Configuration Merging

# Base + environment + local
yq eval-all '. as $item ireduce ({}; . * $item)' \
  config.base.yaml \
  config.$ENV.yaml \
  config.local.yaml

Secret Injection

# Replace placeholders with actual values
yq '(.. | select(tag == "!!str")) |= envsubst' config.yaml

Manifest Generation

# Generate K8s manifests from template
yq '(.metadata.name) = "app-" + env(ENV) |
    (.spec.replicas) = env(REPLICAS) |
    (.spec.template.spec.containers[0].image) = env(IMAGE)' \
  template.yaml

Tips and Tricks

Debugging

# Pretty print to see structure
yq -P . file.yaml

# Show paths
yq '.. | path | join(".")' file.yaml

# Find all keys
yq '.. | select(tag == "!!map") | keys' file.yaml

Working with Large Files

# Stream processing
yq -N . large.yaml  # Process without loading entirely

# Extract specific document
yq 'select(document_index == 5)' large-multi-doc.yaml

# Count documents
yq -N '.' multi-doc.yaml | grep -c '^---$'

Shell Aliases

alias yqp='yq -P'           # Pretty print
alias yqj='yq -o json'      # To JSON
alias yqs='yq -I 2'         # 2-space indent
alias yqc='yq -C 1'         # Force colors

Error Handling

# Check if file exists and is valid
if yq . file.yaml &>/dev/null; then
  echo "Valid YAML"
else
  echo "Invalid YAML or file not found"
  exit 1
fi

# Try with fallback
VALUE=$(yq '.field // "default"' file.yaml)

# Conditional processing
if yq -e '.enabled == true' config.yaml > /dev/null; then
  # Process when enabled
  yq '.items[]' config.yaml
fi

Comparison with Other Tools

Task	yq	jq (JSON)	sed/awk
Parse YAML	Native	Need conversion	Complex
Parse JSON	Can do	Native	Possible
Update in-place	`-i` flag	Needs sponge	`-i` flag
Preserve comments	Yes	N/A	Yes
Type safety	Yes	Yes	No
Learning curve	Medium	Medium	Low/High

12 KiB Raw Blame History

yq - YAML/XML Processing Examples

Overview

YAML Processing

Basic Queries

Filtering

Transforming

Updating Values

In-Place Editing

Merging Files

Format Conversion

YAML to JSON

JSON to YAML

YAML to TOML

YAML to XML

XML Processing

Reading XML

Filtering XML

XML to JSON

XML to YAML

Common Use Cases

Docker Compose

Kubernetes

CI/CD (GitHub Actions, GitLab CI)

Configuration Files

Ansible

Advanced Patterns

Conditional Updates

Complex Transformations

Working with Arrays

Multi-Document YAML

String Operations

Type Checking and Conversion

Comments

Anchors and Aliases

Validation

Output Formatting

Scripting Examples

Environment-Specific Configs

Bulk Updates

Config Validation

Secret Management

Integration Examples

With Git

With Docker

With Kubernetes

With CI/CD

Performance Tips

Common Patterns

Configuration Merging

Secret Injection

Manifest Generation

Tips and Tricks

Debugging

Working with Large Files

Shell Aliases

Error Handling

Comparison with Other Tools

12 KiB

Raw Blame History