Initial commit
This commit is contained in:
Zhongwei Li
290
skills/web-search-fallback/INTEGRATION.md
Normal file
290
skills/web-search-fallback/INTEGRATION.md
Normal file
@@ -0,0 +1,290 @@
|
||||
# Web Search Fallback Integration Guide
|
||||
|
||||
## Quick Start
|
||||
|
||||
This skill provides robust web search capabilities when the built-in WebSearch tool fails or hits limits.
|
||||
|
||||
## Integration in Agents
|
||||
|
||||
### Basic Fallback Pattern
|
||||
|
||||
```bash
|
||||
# Try WebSearch first, fallback if it fails
|
||||
search_query="your search terms"
|
||||
|
||||
# Attempt with WebSearch
|
||||
if result=$(WebSearch "$search_query"); then
|
||||
echo "$result"
|
||||
else
|
||||
# Fallback to bash+curl method
|
||||
result=$(python3 lib/web_search_fallback.py "$search_query" -n 10 -t json)
|
||||
echo "$result"
|
||||
fi
|
||||
```
|
||||
|
||||
### Advanced Integration with Error Detection
|
||||
|
||||
```python
|
||||
# In Python-based agents
|
||||
from lib.web_search_fallback import WebSearchFallback
|
||||
|
||||
def search_with_fallback(query, num_results=10):
|
||||
try:
|
||||
# Try primary WebSearch
|
||||
return web_search(query)
|
||||
except (APILimitError, ValidationError, ToolError) as e:
|
||||
# Use fallback
|
||||
print(f"WebSearch failed: {e}, using fallback")
|
||||
searcher = WebSearchFallback()
|
||||
return searcher.search(query, num_results=num_results)
|
||||
```
|
||||
|
||||
### Orchestrator Integration
|
||||
|
||||
The orchestrator can automatically delegate to this skill when:
|
||||
|
||||
```yaml
|
||||
trigger_conditions:
|
||||
- WebSearch returns error code
|
||||
- User mentions "search fallback"
|
||||
- Pattern database shows WebSearch failures > 3 in last hour
|
||||
- Bulk search operations (> 20 queries)
|
||||
```
|
||||
|
||||
## Usage Patterns
|
||||
|
||||
### 1. Rate Limit Mitigation
|
||||
|
||||
```bash
|
||||
# For bulk searches, use fallback with delays
|
||||
for query in "${queries[@]}"; do
|
||||
python3 lib/web_search_fallback.py "$query" -n 5
|
||||
sleep 2 # Prevent rate limiting
|
||||
done
|
||||
```
|
||||
|
||||
### 2. Cross-Platform Compatibility
|
||||
|
||||
```bash
|
||||
# Detect platform and use appropriate method
|
||||
if [[ "$OSTYPE" == "msys" ]] || [[ "$OSTYPE" == "cygwin" ]]; then
|
||||
# Windows - use Python
|
||||
python3 lib/web_search_fallback.py "$query"
|
||||
else
|
||||
# Unix-like - use bash or Python
|
||||
bash lib/web_search_fallback.sh "$query"
|
||||
fi
|
||||
```
|
||||
|
||||
### 3. Result Parsing
|
||||
|
||||
```bash
|
||||
# Extract only titles
|
||||
titles=$(python3 lib/web_search_fallback.py "$query" -t titles)
|
||||
|
||||
# Get JSON for programmatic use
|
||||
json_results=$(python3 lib/web_search_fallback.py "$query" -t json)
|
||||
|
||||
# Parse JSON with jq if available
|
||||
echo "$json_results" | jq '.[] | .title'
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Errors and Solutions
|
||||
|
||||
| Error | Cause | Solution |
|
||||
|-------|-------|----------|
|
||||
| Connection timeout | Network issues | Retry with exponential backoff |
|
||||
| Empty results | Query too specific | Broaden search terms |
|
||||
| HTML parsing fails | Website structure changed | Try alternative search engine |
|
||||
| Cache permission denied | Directory permissions | Create cache dir with proper permissions |
|
||||
|
||||
### Graceful Degradation
|
||||
|
||||
```bash
|
||||
# Multiple fallback levels
|
||||
search_result=""
|
||||
|
||||
# Level 1: WebSearch API
|
||||
if ! search_result=$(WebSearch "$query" 2>/dev/null); then
|
||||
# Level 2: DuckDuckGo
|
||||
if ! search_result=$(python3 lib/web_search_fallback.py "$query" -e duckduckgo 2>/dev/null); then
|
||||
# Level 3: Searx
|
||||
if ! search_result=$(python3 lib/web_search_fallback.py "$query" -e searx 2>/dev/null); then
|
||||
# Level 4: Return error message
|
||||
search_result="All search methods failed. Please try again later."
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "$search_result"
|
||||
```
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Caching Strategy
|
||||
|
||||
```bash
|
||||
# Use cache for repeated queries
|
||||
python3 lib/web_search_fallback.py "$query" # First query cached
|
||||
|
||||
# Subsequent queries use cache (60 min TTL)
|
||||
python3 lib/web_search_fallback.py "$query" # Returns instantly
|
||||
|
||||
# Force fresh results when needed
|
||||
python3 lib/web_search_fallback.py "$query" --no-cache
|
||||
```
|
||||
|
||||
### Parallel Searches
|
||||
|
||||
```bash
|
||||
# Run multiple searches in parallel
|
||||
search_terms=("term1" "term2" "term3")
|
||||
|
||||
for term in "${search_terms[@]}"; do
|
||||
python3 lib/web_search_fallback.py "$term" -n 5 &
|
||||
done
|
||||
wait # Wait for all searches to complete
|
||||
```
|
||||
|
||||
## Agent-Specific Examples
|
||||
|
||||
### For research-analyzer Agent
|
||||
|
||||
```bash
|
||||
# Comprehensive research with fallback
|
||||
research_topic="quantum computing applications"
|
||||
|
||||
# Get multiple perspectives
|
||||
ddg_results=$(python3 lib/web_search_fallback.py "$research_topic" -e duckduckgo -n 15)
|
||||
searx_results=$(python3 lib/web_search_fallback.py "$research_topic" -e searx -n 10)
|
||||
|
||||
# Combine and deduplicate results
|
||||
echo "$ddg_results" > /tmp/research_results.txt
|
||||
echo "$searx_results" >> /tmp/research_results.txt
|
||||
```
|
||||
|
||||
### For background-task-manager Agent
|
||||
|
||||
```bash
|
||||
# Non-blocking search in background
|
||||
{
|
||||
python3 lib/web_search_fallback.py "$query" -n 20 > search_results.txt
|
||||
echo "Search completed: $(wc -l < search_results.txt) results found"
|
||||
} &
|
||||
|
||||
# Continue with other tasks while search runs
|
||||
echo "Search running in background..."
|
||||
```
|
||||
|
||||
## Testing the Integration
|
||||
|
||||
### Unit Test
|
||||
|
||||
```bash
|
||||
# Test fallback functionality
|
||||
test_query="test search fallback"
|
||||
|
||||
# Test Python implementation
|
||||
python3 lib/web_search_fallback.py "$test_query" -n 1 -v
|
||||
|
||||
# Test bash implementation
|
||||
bash lib/web_search_fallback.sh "$test_query" -n 1
|
||||
|
||||
# Test cache functionality
|
||||
python3 lib/web_search_fallback.py "$test_query" # Creates cache
|
||||
python3 lib/web_search_fallback.py "$test_query" # Uses cache
|
||||
|
||||
# Verify cache file exists
|
||||
ls -la .claude-patterns/search-cache/
|
||||
```
|
||||
|
||||
### Integration Test
|
||||
|
||||
```bash
|
||||
# Simulate WebSearch failure and fallback
|
||||
function test_search_with_fallback() {
|
||||
local query="$1"
|
||||
|
||||
# Simulate WebSearch failure
|
||||
if false; then # Always fails
|
||||
echo "WebSearch result"
|
||||
else
|
||||
echo "WebSearch failed, using fallback..." >&2
|
||||
python3 lib/web_search_fallback.py "$query" -n 3 -t titles
|
||||
fi
|
||||
}
|
||||
|
||||
test_search_with_fallback "integration test"
|
||||
```
|
||||
|
||||
## Monitoring and Logging
|
||||
|
||||
### Track Fallback Usage
|
||||
|
||||
```python
|
||||
# In pattern_storage.py integration
|
||||
pattern = {
|
||||
"task_type": "web_search",
|
||||
"method_used": "fallback",
|
||||
"search_engine": "duckduckgo",
|
||||
"success": True,
|
||||
"response_time": 2.3,
|
||||
"cached": False,
|
||||
"timestamp": "2024-01-01T10:00:00"
|
||||
}
|
||||
```
|
||||
|
||||
### Success Metrics
|
||||
|
||||
Monitor these metrics in the pattern database:
|
||||
- Fallback trigger frequency
|
||||
- Success rate by search engine
|
||||
- Average response time
|
||||
- Cache hit rate
|
||||
- Error types and frequencies
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Always try WebSearch first** - It's the primary tool
|
||||
2. **Use caching wisely** - Enable for repeated queries, disable for fresh data
|
||||
3. **Handle errors gracefully** - Multiple fallback levels
|
||||
4. **Respect rate limits** - Add delays for bulk operations
|
||||
5. **Parse results appropriately** - Use JSON for structured data
|
||||
6. **Log fallback usage** - Track patterns for optimization
|
||||
7. **Test regularly** - HTML structures may change
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Debug Mode
|
||||
|
||||
```bash
|
||||
# Enable verbose output for debugging
|
||||
python3 lib/web_search_fallback.py "debug query" -v
|
||||
|
||||
# Check cache status
|
||||
ls -la .claude-patterns/search-cache/
|
||||
find .claude-patterns/search-cache/ -type f -mmin -60 # Files < 60 min old
|
||||
|
||||
# Test specific search engine
|
||||
python3 lib/web_search_fallback.py "test" -e duckduckgo -v
|
||||
python3 lib/web_search_fallback.py "test" -e searx -v
|
||||
```
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **No results returned**
|
||||
- Check internet connectivity
|
||||
- Verify search engine is accessible
|
||||
- Try different search terms
|
||||
|
||||
2. **Cache not working**
|
||||
- Check directory permissions
|
||||
- Verify disk space available
|
||||
- Clear old cache files
|
||||
|
||||
3. **Parsing errors**
|
||||
- HTML structure may have changed
|
||||
- Update parsing patterns in script
|
||||
- Try alternative search engine
|
||||
Reference in New Issue
Block a user