Initial commit
This commit is contained in:
Zhongwei Li
394
skills/aws-cost-operations/references/operations-patterns.md
Normal file
394
skills/aws-cost-operations/references/operations-patterns.md
Normal file
@@ -0,0 +1,394 @@
|
||||
# AWS Cost & Operations Patterns
|
||||
|
||||
Comprehensive patterns and best practices for AWS cost optimization, monitoring, and operational excellence.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Cost Optimization Patterns](#cost-optimization-patterns)
|
||||
- [Monitoring Patterns](#monitoring-patterns)
|
||||
- [Observability Patterns](#observability-patterns)
|
||||
- [Security and Audit Patterns](#security-and-audit-patterns)
|
||||
- [Troubleshooting Workflows](#troubleshooting-workflows)
|
||||
|
||||
## Cost Optimization Patterns
|
||||
|
||||
### Pattern 1: Cost Estimation Before Deployment
|
||||
|
||||
**When**: Before deploying any new infrastructure
|
||||
|
||||
**MCP Server**: AWS Pricing MCP
|
||||
|
||||
**Steps**:
|
||||
1. List all resources to be deployed
|
||||
2. Query pricing for each resource type
|
||||
3. Calculate monthly costs based on expected usage
|
||||
4. Compare pricing across regions
|
||||
5. Document cost estimates in architecture docs
|
||||
|
||||
**Example**:
|
||||
```
|
||||
Resource: Lambda Function
|
||||
- Invocations: 1,000,000/month
|
||||
- Duration: 3 seconds avg
|
||||
- Memory: 512 MB
|
||||
- Region: us-east-1
|
||||
Estimated cost: $X/month
|
||||
```
|
||||
|
||||
### Pattern 2: Monthly Cost Review
|
||||
|
||||
**When**: First week of every month
|
||||
|
||||
**MCP Servers**: Cost Explorer MCP, Billing and Cost Management MCP
|
||||
|
||||
**Steps**:
|
||||
1. Review total spending vs. budget
|
||||
2. Analyze cost by service (top 5 services)
|
||||
3. Identify cost anomalies (>20% increase)
|
||||
4. Review cost by environment (dev/staging/prod)
|
||||
5. Check cost allocation tag coverage
|
||||
6. Generate cost optimization recommendations
|
||||
|
||||
**Key Metrics**:
|
||||
- Month-over-month cost change
|
||||
- Cost per environment
|
||||
- Cost per application/project
|
||||
- Untagged resource costs
|
||||
|
||||
### Pattern 3: Right-Sizing Resources
|
||||
|
||||
**When**: Quarterly or when utilization alerts trigger
|
||||
|
||||
**MCP Servers**: CloudWatch MCP, Cost Explorer MCP
|
||||
|
||||
**Steps**:
|
||||
1. Query CloudWatch for resource utilization metrics
|
||||
2. Identify over-provisioned resources (< 40% utilization)
|
||||
3. Identify under-provisioned resources (> 80% utilization)
|
||||
4. Calculate potential savings from right-sizing
|
||||
5. Plan and execute right-sizing changes
|
||||
6. Monitor post-change performance
|
||||
|
||||
**Common Right-Sizing Scenarios**:
|
||||
- EC2 instances with low CPU utilization
|
||||
- RDS instances with excess capacity
|
||||
- DynamoDB tables with low read/write usage
|
||||
- Lambda functions with excessive memory allocation
|
||||
|
||||
### Pattern 4: Unused Resource Cleanup
|
||||
|
||||
**When**: Monthly or triggered by cost anomalies
|
||||
|
||||
**MCP Servers**: Cost Explorer MCP, CloudTrail MCP
|
||||
|
||||
**Steps**:
|
||||
1. Identify resources with zero usage
|
||||
2. Query CloudTrail for last access time
|
||||
3. Tag resources for deletion review
|
||||
4. Notify resource owners
|
||||
5. Delete confirmed unused resources
|
||||
6. Track cost savings
|
||||
|
||||
**Common Unused Resources**:
|
||||
- Unattached EBS volumes
|
||||
- Old EBS snapshots
|
||||
- Idle Load Balancers
|
||||
- Unused Elastic IPs
|
||||
- Old AMIs and snapshots
|
||||
- Stopped EC2 instances (long-term)
|
||||
|
||||
## Monitoring Patterns
|
||||
|
||||
### Pattern 1: Critical Service Monitoring
|
||||
|
||||
**When**: All production services
|
||||
|
||||
**MCP Server**: CloudWatch MCP
|
||||
|
||||
**Metrics to Monitor**:
|
||||
- **Availability**: Service uptime, health checks
|
||||
- **Performance**: Latency, response time
|
||||
- **Errors**: Error rate, failed requests
|
||||
- **Saturation**: CPU, memory, disk, network utilization
|
||||
|
||||
**Alarm Thresholds** (adjust based on SLAs):
|
||||
- Error rate: > 1% for 2 consecutive periods
|
||||
- Latency: p99 > 1 second for 5 minutes
|
||||
- CPU: > 80% for 10 minutes
|
||||
- Memory: > 85% for 5 minutes
|
||||
|
||||
### Pattern 2: Lambda Function Monitoring
|
||||
|
||||
**MCP Server**: CloudWatch MCP
|
||||
|
||||
**Key Metrics**:
|
||||
```
|
||||
- Invocations (Count)
|
||||
- Errors (Count, %)
|
||||
- Duration (Average, p99)
|
||||
- Throttles (Count)
|
||||
- ConcurrentExecutions (Max)
|
||||
- IteratorAge (for stream processing)
|
||||
```
|
||||
|
||||
**Recommended Alarms**:
|
||||
- Error rate > 1%
|
||||
- Duration > 80% of timeout
|
||||
- Throttles > 0
|
||||
- ConcurrentExecutions > 80% of reserved
|
||||
|
||||
### Pattern 3: API Gateway Monitoring
|
||||
|
||||
**MCP Server**: CloudWatch MCP
|
||||
|
||||
**Key Metrics**:
|
||||
```
|
||||
- Count (Total requests)
|
||||
- 4XXError, 5XXError
|
||||
- Latency (p50, p95, p99)
|
||||
- IntegrationLatency
|
||||
- CacheHitCount, CacheMissCount
|
||||
```
|
||||
|
||||
**Recommended Alarms**:
|
||||
- 5XX error rate > 0.5%
|
||||
- 4XX error rate > 5%
|
||||
- Latency p99 > 2 seconds
|
||||
- Integration latency spike
|
||||
|
||||
### Pattern 4: Database Monitoring
|
||||
|
||||
**MCP Server**: CloudWatch MCP
|
||||
|
||||
**RDS Metrics**:
|
||||
```
|
||||
- CPUUtilization
|
||||
- DatabaseConnections
|
||||
- FreeableMemory
|
||||
- ReadLatency, WriteLatency
|
||||
- ReadIOPS, WriteIOPS
|
||||
- FreeStorageSpace
|
||||
```
|
||||
|
||||
**DynamoDB Metrics**:
|
||||
```
|
||||
- ConsumedReadCapacityUnits
|
||||
- ConsumedWriteCapacityUnits
|
||||
- UserErrors
|
||||
- SystemErrors
|
||||
- ThrottledRequests
|
||||
```
|
||||
|
||||
**Recommended Alarms**:
|
||||
- RDS CPU > 80% for 10 minutes
|
||||
- RDS connections > 80% of max
|
||||
- RDS free storage < 10 GB
|
||||
- DynamoDB throttled requests > 0
|
||||
- DynamoDB user errors spike
|
||||
|
||||
## Observability Patterns
|
||||
|
||||
### Pattern 1: Distributed Tracing Setup
|
||||
|
||||
**MCP Server**: CloudWatch Application Signals MCP
|
||||
|
||||
**Components**:
|
||||
1. **Service Map**: Visualize service dependencies
|
||||
2. **Traces**: Track requests across services
|
||||
3. **Metrics**: Monitor latency and errors per service
|
||||
4. **SLOs**: Define and track service level objectives
|
||||
|
||||
**Implementation**:
|
||||
- Enable X-Ray tracing on Lambda functions
|
||||
- Add X-Ray SDK to application code
|
||||
- Configure sampling rules
|
||||
- Create service lens dashboards
|
||||
|
||||
### Pattern 2: Log Aggregation and Analysis
|
||||
|
||||
**MCP Server**: CloudWatch MCP
|
||||
|
||||
**Log Strategy**:
|
||||
1. **Centralize Logs**: Send all application logs to CloudWatch Logs
|
||||
2. **Structure Logs**: Use JSON format for structured logging
|
||||
3. **Log Insights**: Use CloudWatch Logs Insights for queries
|
||||
4. **Retention**: Set appropriate retention periods
|
||||
|
||||
**Example Log Insights Queries**:
|
||||
```
|
||||
# Find errors in last hour
|
||||
fields @timestamp, @message
|
||||
| filter @message like /ERROR/
|
||||
| sort @timestamp desc
|
||||
| limit 100
|
||||
|
||||
# Count errors by type
|
||||
stats count() by error_type
|
||||
| sort count desc
|
||||
|
||||
# Calculate p99 latency
|
||||
stats percentile(duration, 99) by service_name
|
||||
```
|
||||
|
||||
### Pattern 3: Custom Metrics
|
||||
|
||||
**MCP Server**: CloudWatch MCP
|
||||
|
||||
**When to Use Custom Metrics**:
|
||||
- Business-specific KPIs (orders/minute, revenue/hour)
|
||||
- Application-specific metrics (cache hit rate, queue depth)
|
||||
- Performance metrics not provided by AWS
|
||||
|
||||
**Best Practices**:
|
||||
- Use consistent namespace: `CompanyName/ApplicationName`
|
||||
- Include relevant dimensions (environment, region, version)
|
||||
- Publish metrics at appropriate intervals
|
||||
- Use metric filters for log-derived metrics
|
||||
|
||||
## Security and Audit Patterns
|
||||
|
||||
### Pattern 1: API Activity Auditing
|
||||
|
||||
**MCP Server**: CloudTrail MCP
|
||||
|
||||
**Regular Audit Queries**:
|
||||
```
|
||||
# Find all IAM changes
|
||||
eventName: CreateUser, DeleteUser, AttachUserPolicy, etc.
|
||||
Time: Last 24 hours
|
||||
|
||||
# Track S3 bucket deletions
|
||||
eventName: DeleteBucket
|
||||
Time: Last 7 days
|
||||
|
||||
# Find failed login attempts
|
||||
eventName: ConsoleLogin
|
||||
errorCode: Failure
|
||||
|
||||
# Monitor privileged actions
|
||||
userIdentity.arn: *admin* OR *root*
|
||||
```
|
||||
|
||||
**Audit Schedule**:
|
||||
- Daily: Review privileged user actions
|
||||
- Weekly: Audit IAM changes and security group modifications
|
||||
- Monthly: Comprehensive security review
|
||||
|
||||
### Pattern 2: Security Posture Assessment
|
||||
|
||||
**MCP Server**: Well-Architected Security Assessment Tool MCP
|
||||
|
||||
**Assessment Areas**:
|
||||
1. **Identity and Access Management**
|
||||
- Least privilege implementation
|
||||
- MFA enforcement
|
||||
- Role-based access control
|
||||
- Service control policies
|
||||
|
||||
2. **Detective Controls**
|
||||
- CloudTrail enabled in all regions
|
||||
- GuardDuty findings review
|
||||
- Config rule compliance
|
||||
- Security Hub findings
|
||||
|
||||
3. **Infrastructure Protection**
|
||||
- VPC security groups review
|
||||
- Network ACLs configuration
|
||||
- AWS WAF rules
|
||||
- Security group ingress rules
|
||||
|
||||
4. **Data Protection**
|
||||
- Encryption at rest (S3, EBS, RDS)
|
||||
- Encryption in transit (TLS/SSL)
|
||||
- KMS key usage and rotation
|
||||
- Secrets Manager utilization
|
||||
|
||||
5. **Incident Response**
|
||||
- IR playbooks documented
|
||||
- Automated response procedures
|
||||
- Contact information current
|
||||
- Regular IR drills
|
||||
|
||||
**Assessment Frequency**:
|
||||
- Quarterly: Full Well-Architected review
|
||||
- Monthly: High-priority findings review
|
||||
- Weekly: Critical security findings
|
||||
|
||||
### Pattern 3: Compliance Monitoring
|
||||
|
||||
**MCP Servers**: CloudTrail MCP, CloudWatch MCP
|
||||
|
||||
**Compliance Requirements**:
|
||||
- Data residency (ensure data stays in approved regions)
|
||||
- Access logging (all access logged and retained)
|
||||
- Encryption requirements (data encrypted at rest and in transit)
|
||||
- Change management (all changes tracked in CloudTrail)
|
||||
|
||||
**Compliance Dashboards**:
|
||||
- Encryption coverage by service
|
||||
- CloudTrail logging status
|
||||
- Failed login attempts
|
||||
- Privileged access usage
|
||||
- Non-compliant resources
|
||||
|
||||
## Troubleshooting Workflows
|
||||
|
||||
### Workflow 1: High Lambda Error Rate
|
||||
|
||||
**MCP Servers**: CloudWatch MCP, CloudWatch Application Signals MCP
|
||||
|
||||
**Steps**:
|
||||
1. Query CloudWatch for Lambda error metrics
|
||||
2. Check error logs in CloudWatch Logs
|
||||
3. Identify error patterns (timeout, memory, permission)
|
||||
4. Check Lambda configuration (memory, timeout, permissions)
|
||||
5. Review recent code deployments
|
||||
6. Check downstream service health
|
||||
7. Implement fix and monitor
|
||||
|
||||
### Workflow 2: Increased Latency
|
||||
|
||||
**MCP Servers**: CloudWatch MCP, CloudWatch Application Signals MCP
|
||||
|
||||
**Steps**:
|
||||
1. Identify latency spike in CloudWatch metrics
|
||||
2. Check service map for slow dependencies
|
||||
3. Query distributed traces for slow requests
|
||||
4. Check database query performance
|
||||
5. Review API Gateway integration latency
|
||||
6. Check Lambda cold starts
|
||||
7. Identify bottleneck and optimize
|
||||
|
||||
### Workflow 3: Cost Spike Investigation
|
||||
|
||||
**MCP Servers**: Cost Explorer MCP, CloudWatch MCP, CloudTrail MCP
|
||||
|
||||
**Steps**:
|
||||
1. Use Cost Explorer to identify service causing spike
|
||||
2. Check CloudWatch metrics for usage increase
|
||||
3. Review CloudTrail for recent resource creation
|
||||
4. Identify root cause (misconfiguration, runaway process, attack)
|
||||
5. Implement cost controls (budgets, alarms, service quotas)
|
||||
6. Clean up unnecessary resources
|
||||
|
||||
### Workflow 4: Security Incident Response
|
||||
|
||||
**MCP Servers**: CloudTrail MCP, GuardDuty (via CloudWatch), Well-Architected Assessment MCP
|
||||
|
||||
**Steps**:
|
||||
1. Identify security event in GuardDuty or CloudWatch
|
||||
2. Query CloudTrail for related API activity
|
||||
3. Determine scope and impact
|
||||
4. Isolate affected resources
|
||||
5. Revoke compromised credentials
|
||||
6. Implement remediation
|
||||
7. Conduct post-incident review
|
||||
8. Update security controls
|
||||
|
||||
## Summary
|
||||
|
||||
- **Cost Optimization**: Use Pricing, Cost Explorer, and Billing MCPs for proactive cost management
|
||||
- **Monitoring**: Set up comprehensive CloudWatch alarms for all critical services
|
||||
- **Observability**: Implement distributed tracing and structured logging
|
||||
- **Security**: Regular CloudTrail audits and Well-Architected assessments
|
||||
- **Proactive**: Don't wait for incidents - monitor and optimize continuously
|
||||
Reference in New Issue
Block a user