Initial commit

references/yq-examples.md

@@ -0,0 +1,677 @@
+# yq - YAML/XML Processing Examples
+
+## Overview
+
+`yq` is a lightweight command-line YAML/XML processor. It's like `jq` but for YAML and XML.
+
+## YAML Processing
+
+### Basic Queries
+
+```bash
+# Pretty print
+yq . file.yaml
+
+# Access field
+yq '.name' file.yaml
+
+# Nested access
+yq '.database.host' file.yaml
+
+# Array access
+yq '.items[0]' file.yaml
+
+# All array elements
+yq '.items[]' file.yaml
+```
+
+### Filtering
+
+```bash
+# Filter by condition
+yq '.users[] | select(.active == true)' file.yaml
+
+# Multiple conditions
+yq '.users[] | select(.age > 18 and .active == true)' file.yaml
+
+# String contains
+yq '.items[] | select(.name | contains("widget"))' file.yaml
+```
+
+### Transforming
+
+```bash
+# Extract specific fields
+yq '.users[] | {name, email}' file.yaml
+
+# Map array
+yq '.items | map(.price * 1.1)' file.yaml
+
+# Sort
+yq 'sort_by(.name)' file.yaml
+
+# Group by
+yq 'group_by(.category)' file.yaml
+```
+
+### Updating Values
+
+```bash
+# Update single field
+yq '.database.host = "localhost"' file.yaml
+
+# Update nested field
+yq '.server.config.port = 8080' file.yaml
+
+# Update array element
+yq '.items[0].price = 99.99' file.yaml
+
+# Add new field
+yq '. + {"newField": "value"}' file.yaml
+
+# Delete field
+yq 'del(.deprecated)' file.yaml
+```
+
+### In-Place Editing
+
+```bash
+# Edit file in-place
+yq -i '.version = "2.0"' file.yaml
+
+# Multiple updates
+yq -i '
+  .version = "2.0" |
+  .updated = now |
+  .deprecated = null
+' file.yaml
+```
+
+### Merging Files
+
+```bash
+# Merge two YAML files
+yq '. *= load("other.yaml")' base.yaml
+
+# Merge with override
+yq 'load("base.yaml") * load("override.yaml")'
+
+# Merge multiple files
+yq eval-all '. as $item ireduce ({}; . * $item)' file1.yaml file2.yaml file3.yaml
+```
+
+## Format Conversion
+
+### YAML to JSON
+
+```bash
+# Convert YAML to JSON
+yq -o json . file.yaml
+
+# Compact JSON
+yq -o json -I 0 . file.yaml
+
+# Pretty JSON
+yq -o json . file.yaml | jq .
+```
+
+### JSON to YAML
+
+```bash
+# Convert JSON to YAML
+yq -P . file.json
+
+# From stdin
+echo '{"name":"test"}' | yq -P .
+
+# Pipeline
+cat file.json | yq -P . > file.yaml
+```
+
+### YAML to TOML
+
+```bash
+# Convert YAML to TOML
+yq -o toml . file.yaml
+```
+
+### YAML to XML
+
+```bash
+# Convert YAML to XML
+yq -o xml . file.yaml
+```
+
+## XML Processing
+
+### Reading XML
+
+```bash
+# Parse XML
+yq -p xml . file.xml
+
+# Access element
+yq -p xml '.root.element' file.xml
+
+# Access attribute
+yq -p xml '.root.+@attr' file.xml
+
+# All elements
+yq -p xml '.root.items[]' file.xml
+```
+
+### Filtering XML
+
+```bash
+# Filter by attribute
+yq -p xml '.root.item[] | select(.+@id == "123")' file.xml
+
+# Filter by element value
+yq -p xml '.root.item[] | select(.name == "test")' file.xml
+```
+
+### XML to JSON
+
+```bash
+# Convert XML to JSON
+yq -p xml -o json . file.xml
+
+# Extract and convert
+yq -p xml -o json '.root.items' file.xml
+```
+
+### XML to YAML
+
+```bash
+# Convert XML to YAML
+yq -p xml . file.xml
+
+# Pretty print
+yq -p xml -P . file.xml
+```
+
+## Common Use Cases
+
+### Docker Compose
+
+```bash
+# Get service names
+yq '.services | keys' docker-compose.yml
+
+# Get image for service
+yq '.services.web.image' docker-compose.yml
+
+# Update service port
+yq -i '.services.web.ports[0] = "8080:80"' docker-compose.yml
+
+# Add environment variable
+yq -i '.services.web.environment.DEBUG = "true"' docker-compose.yml
+
+# Get all exposed ports
+yq '.services[].ports[]' docker-compose.yml
+```
+
+### Kubernetes
+
+```bash
+# Get pod name
+yq '.metadata.name' pod.yaml
+
+# Get container image
+yq '.spec.containers[0].image' pod.yaml
+
+# Update replicas
+yq -i '.spec.replicas = 3' deployment.yaml
+
+# Get all container names
+yq '.spec.template.spec.containers[].name' deployment.yaml
+
+# Get resource limits
+yq '.spec.template.spec.containers[0].resources.limits' deployment.yaml
+```
+
+### CI/CD (GitHub Actions, GitLab CI)
+
+```bash
+# Get job names
+yq '.jobs | keys' .github/workflows/ci.yml
+
+# Get steps for job
+yq '.jobs.build.steps[].name' .github/workflows/ci.yml
+
+# Update trigger
+yq -i '.on.push.branches = ["main", "develop"]' .github/workflows/ci.yml
+
+# Get all used actions
+yq '.jobs[].steps[].uses' .github/workflows/ci.yml
+```
+
+### Configuration Files
+
+```bash
+# Application config
+yq '.app.port' config.yaml
+yq '.database.connection.host' config.yaml
+
+# Update config
+yq -i '.app.debug = true' config.yaml
+yq -i '.cache.enabled = false' config.yaml
+
+# Merge configs
+yq '. *= load("config.local.yaml")' config.yaml
+```
+
+### Ansible
+
+```bash
+# Get hosts
+yq '.all.hosts | keys' inventory.yaml
+
+# Get variables
+yq '.all.vars' inventory.yaml
+
+# Update host variable
+yq -i '.all.hosts.server1.ansible_host = "192.168.1.10"' inventory.yaml
+
+# Get all playbook tasks
+yq '.[] | select(.tasks) | .tasks[].name' playbook.yaml
+```
+
+## Advanced Patterns
+
+### Conditional Updates
+
+```bash
+# Update if exists
+yq '(select(.field) | .field) = "new-value"' file.yaml
+
+# Update based on condition
+yq '(.items[] | select(.price > 100) | .discount) = 0.2' file.yaml
+
+# Add field if missing
+yq '. + (if has("field") then {} else {"field": "default"} end)' file.yaml
+```
+
+### Complex Transformations
+
+```bash
+# Restructure data
+yq '.users | map({
+  username: .name,
+  contact: {
+    email: .email,
+    phone: .phone
+  }
+})' file.yaml
+
+# Aggregate data
+yq '.items | group_by(.category) | map({
+  category: .[0].category,
+  total: map(.price) | add
+})' file.yaml
+
+# Pivot data
+yq 'reduce .items[] as $item ({}; .[$item.name] = $item.value)' file.yaml
+```
+
+### Working with Arrays
+
+```bash
+# Append to array
+yq '.items += ["new-item"]' file.yaml
+
+# Prepend to array
+yq '.items = ["new-item"] + .items' file.yaml
+
+# Remove from array
+yq 'del(.items[2])' file.yaml
+
+# Filter array
+yq '.items = [.items[] | select(.active == true)]' file.yaml
+
+# Unique array
+yq '.items | unique' file.yaml
+
+# Flatten array
+yq '.items | flatten' file.yaml
+```
+
+### Multi-Document YAML
+
+```bash
+# Process all documents
+yq '.' multi-doc.yaml
+
+# Select specific document
+yq 'select(document_index == 0)' multi-doc.yaml
+
+# Filter documents
+yq 'select(.kind == "Deployment")' multi-doc.yaml
+
+# Update all documents
+yq -i '.metadata.labels.app = "myapp"' multi-doc.yaml
+```
+
+## String Operations
+
+```bash
+# Concatenation
+yq '.fullName = .firstName + " " + .lastName' file.yaml
+
+# Interpolation
+yq '.message = "Hello, " + .name + "!"' file.yaml
+
+# Uppercase
+yq '.name | upcase' file.yaml
+
+# Lowercase
+yq '.name | downcase' file.yaml
+
+# Split
+yq '.path | split("/")' file.yaml
+
+# Join
+yq '.items | join(", ")' file.yaml
+
+# Replace
+yq '.text | sub("old", "new")' file.yaml
+
+# Trim
+yq '.text | trim' file.yaml
+```
+
+## Type Checking and Conversion
+
+```bash
+# Check type
+yq '.field | type' file.yaml
+
+# Convert to string
+yq '.number | tostring' file.yaml
+
+# Convert to number
+yq '.string | tonumber' file.yaml
+
+# Check if exists
+yq 'has("field")' file.yaml
+
+# Length
+yq '.items | length' file.yaml
+```
+
+## Comments
+
+```bash
+# Preserve comments (default behavior)
+yq '.field = "new-value"' file.yaml
+
+# Add headComment
+yq '.field headComment="This is a comment"' file.yaml
+
+# Add lineComment
+yq '.field lineComment="inline comment"' file.yaml
+
+# Add footComment
+yq '.field footComment="bottom comment"' file.yaml
+```
+
+## Anchors and Aliases
+
+```bash
+# Create anchor
+yq '.base.&anchor = {"key": "value"}' file.yaml
+
+# Reference anchor
+yq '.other = .base.*anchor' file.yaml
+
+# Merge anchors
+yq '.combined = .base.*anchor * .override' file.yaml
+```
+
+## Validation
+
+```bash
+# Check if valid YAML
+yq . file.yaml > /dev/null && echo "Valid" || echo "Invalid"
+
+# Validate schema (requires schema file)
+yq 'load("schema.yaml") | . as $schema | load("data.yaml") | validate($schema)'
+
+# Check required fields
+yq 'select(has("requiredField") | not)' file.yaml
+
+# Validate format
+yq 'select(.email | test("^[^@]+@[^@]+$") | not)' file.yaml
+```
+
+## Output Formatting
+
+```bash
+# Default YAML output
+yq . file.yaml
+
+# Compact (minimal whitespace)
+yq -I 0 . file.yaml
+
+# Custom indentation (4 spaces)
+yq -I 4 . file.yaml
+
+# No colors
+yq -C 0 . file.yaml
+
+# Force colors
+yq -C 1 . file.yaml
+
+# Raw output (no quotes)
+yq -r '.name' file.yaml
+
+# One line per document
+yq -o json . multi-doc.yaml
+```
+
+## Scripting Examples
+
+### Environment-Specific Configs
+
+```bash
+#!/bin/bash
+ENV=${1:-dev}
+
+yq ". *= load(\"config.$ENV.yaml\")" config.base.yaml > config.yaml
+```
+
+### Bulk Updates
+
+```bash
+#!/bin/bash
+for file in *.yaml; do
+  yq -i '.metadata.labels.version = "2.0"' "$file"
+done
+```
+
+### Config Validation
+
+```bash
+#!/bin/bash
+REQUIRED_FIELDS=("name" "version" "description")
+
+for field in "${REQUIRED_FIELDS[@]}"; do
+  if ! yq -e "has(\"$field\")" config.yaml > /dev/null; then
+    echo "Missing required field: $field"
+    exit 1
+  fi
+done
+```
+
+### Secret Management
+
+```bash
+#!/bin/bash
+# Extract secrets from YAML
+yq '.secrets' config.yaml > secrets.yaml
+
+# Encrypt secrets (example with age)
+yq '.secrets' config.yaml | age -e -o secrets.age
+
+# Decrypt and merge
+age -d secrets.age | yq -P . > secrets.yaml
+yq '. *= load("secrets.yaml")' config.yaml
+```
+
+## Integration Examples
+
+### With Git
+
+```bash
+# Pre-commit hook to sort keys
+git diff --cached --name-only | \
+  grep '\.yaml$' | \
+  xargs -I {} yq -i -P 'sort_keys(..)' {}
+```
+
+### With Docker
+
+```bash
+# Extract image tags
+yq '.services[].image' docker-compose.yml | \
+  cut -d: -f2 | \
+  sort -u
+
+# Update all images to latest
+yq -i '(.services[].image) |= sub(":.*", ":latest")' docker-compose.yml
+```
+
+### With Kubernetes
+
+```bash
+# Apply with modifications
+yq '.spec.replicas = 3' deployment.yaml | kubectl apply -f -
+
+# Extract all images
+kubectl get pods -o yaml | \
+  yq '.items[].spec.containers[].image' | \
+  sort -u
+
+# Generate manifests
+yq eval-all '. as $item ireduce ({}; . * $item)' \
+  base/*.yaml overlays/prod/*.yaml | \
+  kubectl apply -f -
+```
+
+### With CI/CD
+
+```bash
+# Update version in workflow
+VERSION=$(git describe --tags)
+yq -i ".env.VERSION = \"$VERSION\"" .github/workflows/deploy.yml
+
+# Extract job matrix
+MATRIX=$(yq -o json '.jobs.test.strategy.matrix' .github/workflows/ci.yml)
+echo "matrix=$MATRIX" >> $GITHUB_OUTPUT
+```
+
+## Performance Tips
+
+1. **Use in-place editing**: `-i` avoids reading/writing twice
+2. **Pipe instead of temp files**: More efficient for transformations
+3. **Specific paths**: Access `.field.subfield` instead of iterating
+4. **Limit output**: Use `select` early to filter data
+5. **Batch operations**: Combine multiple updates in one command
+
+## Common Patterns
+
+### Configuration Merging
+
+```bash
+# Base + environment + local
+yq eval-all '. as $item ireduce ({}; . * $item)' \
+  config.base.yaml \
+  config.$ENV.yaml \
+  config.local.yaml
+```
+
+### Secret Injection
+
+```bash
+# Replace placeholders with actual values
+yq '(.. | select(tag == "!!str")) |= envsubst' config.yaml
+```
+
+### Manifest Generation
+
+```bash
+# Generate K8s manifests from template
+yq '(.metadata.name) = "app-" + env(ENV) |
+    (.spec.replicas) = env(REPLICAS) |
+    (.spec.template.spec.containers[0].image) = env(IMAGE)' \
+  template.yaml
+```
+
+## Tips and Tricks
+
+### Debugging
+
+```bash
+# Pretty print to see structure
+yq -P . file.yaml
+
+# Show paths
+yq '.. | path | join(".")' file.yaml
+
+# Find all keys
+yq '.. | select(tag == "!!map") | keys' file.yaml
+```
+
+### Working with Large Files
+
+```bash
+# Stream processing
+yq -N . large.yaml  # Process without loading entirely
+
+# Extract specific document
+yq 'select(document_index == 5)' large-multi-doc.yaml
+
+# Count documents
+yq -N '.' multi-doc.yaml | grep -c '^---$'
+```
+
+### Shell Aliases
+
+```bash
+alias yqp='yq -P'           # Pretty print
+alias yqj='yq -o json'      # To JSON
+alias yqs='yq -I 2'         # 2-space indent
+alias yqc='yq -C 1'         # Force colors
+```
+
+## Error Handling
+
+```bash
+# Check if file exists and is valid
+if yq . file.yaml &>/dev/null; then
+  echo "Valid YAML"
+else
+  echo "Invalid YAML or file not found"
+  exit 1
+fi
+
+# Try with fallback
+VALUE=$(yq '.field // "default"' file.yaml)
+
+# Conditional processing
+if yq -e '.enabled == true' config.yaml > /dev/null; then
+  # Process when enabled
+  yq '.items[]' config.yaml
+fi
+```
+
+## Comparison with Other Tools
+
+| Task | yq | jq (JSON) | sed/awk |
+|------|----|-----------| --------|
+| Parse YAML | Native | Need conversion | Complex |
+| Parse JSON | Can do | Native | Possible |
+| Update in-place | `-i` flag | Needs sponge | `-i` flag |
+| Preserve comments | Yes | N/A | Yes |
+| Type safety | Yes | Yes | No |
+| Learning curve | Medium | Medium | Low/High |