806 lines
27 KiB
Markdown
806 lines
27 KiB
Markdown
# Multi-Intent Detection System v1.0
|
|
|
|
**Version:** 1.0
|
|
**Purpose:** Advanced detection and handling of complex user queries with multiple intentions
|
|
**Target:** Support complex queries with 95%+ intent accuracy and proper capability routing
|
|
|
|
---
|
|
|
|
## 🎯 **Overview**
|
|
|
|
Multi-Intent Detection extends the activation system to handle complex user queries that contain multiple intentions, requiring the skill to understand and prioritize different user goals within a single request.
|
|
|
|
### **Problem Solved**
|
|
|
|
**Before:** Skills could only handle single-intent queries, failing when users expressed multiple goals or complex requirements
|
|
**After:** Skills can detect, prioritize, and handle multiple intents within a single query, routing to appropriate capabilities
|
|
|
|
---
|
|
|
|
## 🧠 **Multi-Intent Architecture**
|
|
|
|
### **Intent Classification Hierarchy**
|
|
|
|
```
|
|
Primary Intent (Main Goal)
|
|
├── Secondary Intent 1 (Sub-goal)
|
|
├── Secondary Intent 2 (Additional requirement)
|
|
├── Tertiary Intent (Context/Modifier)
|
|
└── Meta Intent (How to present results)
|
|
```
|
|
|
|
### **Intent Types**
|
|
|
|
#### **1. Primary Intents**
|
|
The main action or goal the user wants to accomplish:
|
|
- `analyze` - Analyze data or information
|
|
- `create` - Create new content or agent
|
|
- `compare` - Compare multiple items
|
|
- `monitor` - Track or watch something
|
|
- `transform` - Convert or change format
|
|
|
|
#### **2. Secondary Intents**
|
|
Additional requirements or sub-goals:
|
|
- `and_visualize` - Also create visualization
|
|
- `and_save` - Also save results
|
|
- `and_explain` - Also provide explanation
|
|
- `and_compare` - Also do comparison
|
|
- `and_alert` - Also set up alerts
|
|
|
|
#### **3. Contextual Intents**
|
|
Modifiers that affect how results should be presented:
|
|
- `quick_summary` - Brief overview
|
|
- `detailed_analysis` - In-depth analysis
|
|
- `step_by_step` - Process explanation
|
|
- `real_time` - Live/current data
|
|
- `historical` - Historical data
|
|
|
|
#### **4. Meta Intents**
|
|
How the user wants to interact:
|
|
- `just_show_me` - Direct results
|
|
- `teach_me` - Educational approach
|
|
- `help_me_decide` - Decision support
|
|
- `automate_for_me` - Automation request
|
|
|
|
---
|
|
|
|
## 🔍 **Intent Detection Algorithms**
|
|
|
|
### **Multi-Intent Parser**
|
|
|
|
```python
|
|
def parse_multiple_intents(query, skill_capabilities):
|
|
"""Parse multiple intents from a complex user query"""
|
|
|
|
# Step 1: Identify primary intent
|
|
primary_intent = extract_primary_intent(query)
|
|
|
|
# Step 2: Identify secondary intents
|
|
secondary_intents = extract_secondary_intents(query)
|
|
|
|
# Step 3: Identify contextual modifiers
|
|
contextual_intents = extract_contextual_intents(query)
|
|
|
|
# Step 4: Identify meta intent
|
|
meta_intent = extract_meta_intent(query)
|
|
|
|
# Step 5: Validate against skill capabilities
|
|
validated_intents = validate_intents_against_capabilities(
|
|
primary_intent, secondary_intents, contextual_intents, skill_capabilities
|
|
)
|
|
|
|
return {
|
|
'primary_intent': validated_intents['primary'],
|
|
'secondary_intents': validated_intents['secondary'],
|
|
'contextual_intents': validated_intents['contextual'],
|
|
'meta_intent': validated_intents['meta'],
|
|
'intent_combinations': generate_intent_combinations(validated_intents),
|
|
'confidence_scores': calculate_intent_confidence(query, validated_intents),
|
|
'execution_plan': create_execution_plan(validated_intents)
|
|
}
|
|
|
|
def extract_primary_intent(query):
|
|
"""Extract the primary intent from the query"""
|
|
|
|
intent_patterns = {
|
|
'analyze': [
|
|
r'(?i)(analyze|analysis|examine|study|evaluate|review)\s+',
|
|
r'(?i)(what\s+is|how\s+does)\s+.*\s+(perform|work|behave)',
|
|
r'(?i)(tell\s+me\s+about|explain)\s+'
|
|
],
|
|
'create': [
|
|
r'(?i)(create|build|make|generate|develop)\s+',
|
|
r'(?i)(I\s+need|I\s+want)\s+(a|an)\s+',
|
|
r'(?i)(help\s+me\s+)(create|build|make)\s+'
|
|
],
|
|
'compare': [
|
|
r'(?i)(compare|comparison|vs|versus)\s+',
|
|
r'(?i)(which\s+is\s+better|what\s+is\s+the\s+difference)\s+',
|
|
r'(?i)(rank|rating|scoring)\s+'
|
|
],
|
|
'monitor': [
|
|
r'(?i)(monitor|track|watch|observe)\s+',
|
|
r'(?i)(keep\s+an\s+eye\s+on|follow)\s+',
|
|
r'(?i)(alert\s+me\s+when|notify\s+me)\s+'
|
|
],
|
|
'transform': [
|
|
r'(?i)(convert|transform|change|turn)\s+.*\s+(into|to)\s+',
|
|
r'(?i)(format|structure|organize)\s+',
|
|
r'(?i)(extract|parse|process)\s+'
|
|
]
|
|
}
|
|
|
|
best_match = None
|
|
highest_score = 0
|
|
|
|
for intent, patterns in intent_patterns.items():
|
|
for pattern in patterns:
|
|
if re.search(pattern, query):
|
|
score = calculate_intent_match_score(query, intent, pattern)
|
|
if score > highest_score:
|
|
highest_score = score
|
|
best_match = intent
|
|
|
|
return best_match or 'unknown'
|
|
|
|
def extract_secondary_intents(query):
|
|
"""Extract secondary intents from conjunctions and phrases"""
|
|
|
|
secondary_patterns = {
|
|
'and_visualize': [
|
|
r'(?i)(and\s+)?(show|visualize|display|chart|graph)\s+',
|
|
r'(?i)(create\s+)?(visualization|chart|graph|dashboard)\s+'
|
|
],
|
|
'and_save': [
|
|
r'(?i)(and\s+)?(save|store|export|download)\s+',
|
|
r'(?i)(keep|record|archive)\s+(the\s+)?(results|data)\s+'
|
|
],
|
|
'and_explain': [
|
|
r'(?i)(and\s+)?(explain|clarify|describe|detail)\s+',
|
|
r'(?i)(what\s+does\s+this\s+mean|why\s+is\s+this)\s+'
|
|
],
|
|
'and_compare': [
|
|
r'(?i)(and\s+)?(compare|vs|versus|against)\s+',
|
|
r'(?i)(relative\s+to|compared\s+with)\s+'
|
|
],
|
|
'and_alert': [
|
|
r'(?i)(and\s+)?(alert|notify|warn)\s+(me\s+)?(when|if)\s+',
|
|
r'(?i)(set\s+up\s+)?(notification|alert)\s+'
|
|
]
|
|
}
|
|
|
|
detected_intents = []
|
|
|
|
for intent, patterns in secondary_patterns.items():
|
|
for pattern in patterns:
|
|
if re.search(pattern, query):
|
|
detected_intents.append(intent)
|
|
break
|
|
|
|
return detected_intents
|
|
|
|
def extract_contextual_intents(query):
|
|
"""Extract contextual modifiers and presentation preferences"""
|
|
|
|
contextual_patterns = {
|
|
'quick_summary': [
|
|
r'(?i)(quick|brief|short|summary|overview)\s+',
|
|
r'(?i)(just\s+the\s+highlights|key\s+points)\s+'
|
|
],
|
|
'detailed_analysis': [
|
|
r'(?i)(detailed|in-depth|comprehensive|thorough)\s+',
|
|
r'(?i)(deep\s+dive|full\s+analysis)\s+'
|
|
],
|
|
'step_by_step': [
|
|
r'(?i)(step\s+by\s+step|how\s+to|process|procedure)\s+',
|
|
r'(?i)(walk\s+me\s+through|guide\s+me)\s+'
|
|
],
|
|
'real_time': [
|
|
r'(?i)(real\s+time|live|current|now|today)\s+',
|
|
r'(?i)(right\s+now|as\s+of\s+today)\s+'
|
|
],
|
|
'historical': [
|
|
r'(?i)(historical|past|previous|last\s+year|ytd)\s+',
|
|
r'(?i)(over\s+the\s+last\s+|historically)\s+'
|
|
]
|
|
}
|
|
|
|
detected_intents = []
|
|
|
|
for intent, patterns in contextual_patterns.items():
|
|
for pattern in patterns:
|
|
if re.search(pattern, query):
|
|
detected_intents.append(intent)
|
|
break
|
|
|
|
return detected_intents
|
|
```
|
|
|
|
### **Intent Validation System**
|
|
|
|
```python
|
|
def validate_intents_against_capabilities(primary, secondary, contextual, capabilities):
|
|
"""Validate detected intents against skill capabilities"""
|
|
|
|
validated = {
|
|
'primary': None,
|
|
'secondary': [],
|
|
'contextual': [],
|
|
'meta': None,
|
|
'validation_issues': []
|
|
}
|
|
|
|
# Validate primary intent
|
|
if primary in capabilities.get('primary_intents', []):
|
|
validated['primary'] = primary
|
|
else:
|
|
validated['validation_issues'].append(
|
|
f"Primary intent '{primary}' not supported by skill"
|
|
)
|
|
|
|
# Validate secondary intents
|
|
for intent in secondary:
|
|
if intent in capabilities.get('secondary_intents', []):
|
|
validated['secondary'].append(intent)
|
|
else:
|
|
validated['validation_issues'].append(
|
|
f"Secondary intent '{intent}' not supported by skill"
|
|
)
|
|
|
|
# Validate contextual intents
|
|
for intent in contextual:
|
|
if intent in capabilities.get('contextual_intents', []):
|
|
validated['contextual'].append(intent)
|
|
else:
|
|
validated['validation_issues'].append(
|
|
f"Contextual intent '{intent}' not supported by skill"
|
|
)
|
|
|
|
# If no valid primary intent, try to find best alternative
|
|
if not validated['primary'] and secondary:
|
|
validated['primary'] = find_best_alternative_primary(primary, secondary, capabilities)
|
|
validated['validation_issues'].append(
|
|
f"Used alternative primary intent: {validated['primary']}"
|
|
)
|
|
|
|
return validated
|
|
|
|
def generate_intent_combinations(validated_intents):
|
|
"""Generate possible combinations of validated intents"""
|
|
|
|
combinations = []
|
|
|
|
primary = validated_intents['primary']
|
|
secondary = validated_intents['secondary']
|
|
contextual = validated_intents['contextual']
|
|
|
|
if primary:
|
|
# Base combination: primary only
|
|
combinations.append({
|
|
'combination_id': 'primary_only',
|
|
'intents': [primary],
|
|
'priority': 1,
|
|
'complexity': 'low'
|
|
})
|
|
|
|
# Primary + each secondary
|
|
for sec_intent in secondary:
|
|
combinations.append({
|
|
'combination_id': f'primary_{sec_intent}',
|
|
'intents': [primary, sec_intent],
|
|
'priority': 2,
|
|
'complexity': 'medium'
|
|
})
|
|
|
|
# Primary + all secondary
|
|
if len(secondary) > 1:
|
|
combinations.append({
|
|
'combination_id': 'primary_all_secondary',
|
|
'intents': [primary] + secondary,
|
|
'priority': 3,
|
|
'complexity': 'high'
|
|
})
|
|
|
|
# Add contextual modifiers
|
|
for combo in combinations:
|
|
for context in contextual:
|
|
new_combo = combo.copy()
|
|
new_combo['intents'] = combo['intents'] + [context]
|
|
new_combo['combination_id'] = f"{combo['combination_id']}_{context}"
|
|
new_combo['priority'] = combo['priority'] + 0.1
|
|
new_combo['complexity'] = increase_complexity(combo['complexity'])
|
|
combinations.append(new_combo)
|
|
|
|
# Sort by priority and complexity
|
|
combinations.sort(key=lambda x: (x['priority'], x['complexity']))
|
|
|
|
return combinations
|
|
|
|
def create_execution_plan(validated_intents):
|
|
"""Create an execution plan for handling multiple intents"""
|
|
|
|
plan = {
|
|
'steps': [],
|
|
'parallel_tasks': [],
|
|
'sequential_dependencies': [],
|
|
'estimated_complexity': 'medium',
|
|
'estimated_time': 'medium'
|
|
}
|
|
|
|
primary = validated_intents['primary']
|
|
secondary = validated_intents['secondary']
|
|
contextual = validated_intents['contextual']
|
|
|
|
if primary:
|
|
# Step 1: Execute primary intent
|
|
plan['steps'].append({
|
|
'step_id': 1,
|
|
'intent': primary,
|
|
'action': f'execute_{primary}',
|
|
'dependencies': [],
|
|
'estimated_time': 'medium'
|
|
})
|
|
|
|
# Step 2: Execute secondary intents (can be parallel if compatible)
|
|
for i, intent in enumerate(secondary):
|
|
if can_execute_parallel(primary, intent):
|
|
plan['parallel_tasks'].append({
|
|
'task_id': f'secondary_{i}',
|
|
'intent': intent,
|
|
'action': f'execute_{intent}',
|
|
'dependencies': ['step_1']
|
|
})
|
|
else:
|
|
plan['steps'].append({
|
|
'step_id': len(plan['steps']) + 1,
|
|
'intent': intent,
|
|
'action': f'execute_{intent}',
|
|
'dependencies': [f'step_{len(plan["steps"])}'],
|
|
'estimated_time': 'short'
|
|
})
|
|
|
|
# Step 3: Apply contextual modifiers
|
|
for i, intent in enumerate(contextual):
|
|
plan['steps'].append({
|
|
'step_id': len(plan['steps']) + 1,
|
|
'intent': intent,
|
|
'action': f'apply_{intent}',
|
|
'dependencies': ['step_1'] + [f'secondary_{j}' for j in range(len(secondary))],
|
|
'estimated_time': 'short'
|
|
})
|
|
|
|
# Calculate overall complexity
|
|
total_intents = 1 + len(secondary) + len(contextual)
|
|
if total_intents <= 2:
|
|
plan['estimated_complexity'] = 'low'
|
|
elif total_intents <= 4:
|
|
plan['estimated_complexity'] = 'medium'
|
|
else:
|
|
plan['estimated_complexity'] = 'high'
|
|
|
|
return plan
|
|
```
|
|
|
|
---
|
|
|
|
## 📋 **Enhanced Marketplace Configuration**
|
|
|
|
### **Multi-Intent Configuration Structure**
|
|
|
|
```json
|
|
{
|
|
"name": "skill-name",
|
|
"activation": {
|
|
"keywords": [...],
|
|
"patterns": [...],
|
|
"contextual_filters": {...},
|
|
|
|
"_comment": "NEW: Multi-intent detection (v1.0)",
|
|
"intent_hierarchy": {
|
|
"primary_intents": {
|
|
"analyze": {
|
|
"description": "Analyze data or information",
|
|
"keywords": ["analyze", "examine", "evaluate", "study"],
|
|
"required_capabilities": ["data_processing", "analysis"],
|
|
"base_confidence": 0.9
|
|
},
|
|
"compare": {
|
|
"description": "Compare multiple items",
|
|
"keywords": ["compare", "versus", "vs", "ranking"],
|
|
"required_capabilities": ["comparison", "evaluation"],
|
|
"base_confidence": 0.85
|
|
},
|
|
"monitor": {
|
|
"description": "Track or monitor data",
|
|
"keywords": ["monitor", "track", "watch", "alert"],
|
|
"required_capabilities": ["monitoring", "notification"],
|
|
"base_confidence": 0.8
|
|
}
|
|
},
|
|
|
|
"secondary_intents": {
|
|
"and_visualize": {
|
|
"description": "Also create visualization",
|
|
"keywords": ["show", "chart", "graph", "visualize"],
|
|
"required_capabilities": ["visualization"],
|
|
"compatibility": ["analyze", "compare", "monitor"],
|
|
"confidence_modifier": 0.1
|
|
},
|
|
"and_save": {
|
|
"description": "Also save results",
|
|
"keywords": ["save", "export", "download", "store"],
|
|
"required_capabilities": ["file_operations"],
|
|
"compatibility": ["analyze", "compare", "transform"],
|
|
"confidence_modifier": 0.05
|
|
},
|
|
"and_explain": {
|
|
"description": "Also provide explanation",
|
|
"keywords": ["explain", "clarify", "describe", "detail"],
|
|
"required_capabilities": ["explanation", "reporting"],
|
|
"compatibility": ["analyze", "compare", "transform"],
|
|
"confidence_modifier": 0.05
|
|
}
|
|
},
|
|
|
|
"contextual_intents": {
|
|
"quick_summary": {
|
|
"description": "Provide brief overview",
|
|
"keywords": ["quick", "summary", "brief", "overview"],
|
|
"impact": "reduce_detail",
|
|
"confidence_modifier": 0.02
|
|
},
|
|
"detailed_analysis": {
|
|
"description": "Provide in-depth analysis",
|
|
"keywords": ["detailed", "comprehensive", "thorough", "in-depth"],
|
|
"impact": "increase_detail",
|
|
"confidence_modifier": 0.03
|
|
},
|
|
"real_time": {
|
|
"description": "Use current/live data",
|
|
"keywords": ["real-time", "live", "current", "now"],
|
|
"impact": "require_live_data",
|
|
"confidence_modifier": 0.04
|
|
}
|
|
},
|
|
|
|
"intent_combinations": {
|
|
"analyze_and_visualize": {
|
|
"description": "Analyze data and create visualization",
|
|
"primary": "analyze",
|
|
"secondary": ["and_visualize"],
|
|
"confidence_threshold": 0.85,
|
|
"execution_order": ["analyze", "and_visualize"]
|
|
},
|
|
"compare_and_explain": {
|
|
"description": "Compare items and explain differences",
|
|
"primary": "compare",
|
|
"secondary": ["and_explain"],
|
|
"confidence_threshold": 0.8,
|
|
"execution_order": ["compare", "and_explain"]
|
|
},
|
|
"monitor_and_alert": {
|
|
"description": "Monitor data and send alerts",
|
|
"primary": "monitor",
|
|
"secondary": ["and_alert"],
|
|
"confidence_threshold": 0.8,
|
|
"execution_order": ["monitor", "and_alert"]
|
|
}
|
|
},
|
|
|
|
"intent_processing": {
|
|
"max_secondary_intents": 3,
|
|
"max_contextual_intents": 2,
|
|
"parallel_execution_threshold": 0.8,
|
|
"fallback_to_primary": true,
|
|
"intent_confidence_threshold": 0.7
|
|
}
|
|
}
|
|
},
|
|
|
|
"capabilities": {
|
|
"primary_intents": ["analyze", "compare", "monitor"],
|
|
"secondary_intents": ["and_visualize", "and_save", "and_explain"],
|
|
"contextual_intents": ["quick_summary", "detailed_analysis", "real_time"],
|
|
"supported_combinations": [
|
|
"analyze_and_visualize",
|
|
"compare_and_explain",
|
|
"monitor_and_alert"
|
|
]
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 🧪 **Multi-Intent Testing Framework**
|
|
|
|
### **Test Case Generation**
|
|
|
|
```python
|
|
def generate_multi_intent_test_cases(skill_config):
|
|
"""Generate test cases for multi-intent detection"""
|
|
|
|
test_cases = []
|
|
|
|
# Single intent tests (baseline)
|
|
single_intents = [
|
|
{
|
|
'query': 'Analyze AAPL stock',
|
|
'intents': {'primary': 'analyze', 'secondary': [], 'contextual': []},
|
|
'expected': True,
|
|
'complexity': 'low'
|
|
},
|
|
{
|
|
'query': 'Compare MSFT vs GOOGL',
|
|
'intents': {'primary': 'compare', 'secondary': [], 'contextual': []},
|
|
'expected': True,
|
|
'complexity': 'low'
|
|
}
|
|
]
|
|
|
|
# Double intent tests
|
|
double_intents = [
|
|
{
|
|
'query': 'Analyze AAPL stock and show me a chart',
|
|
'intents': {'primary': 'analyze', 'secondary': ['and_visualize'], 'contextual': []},
|
|
'expected': True,
|
|
'complexity': 'medium'
|
|
},
|
|
{
|
|
'query': 'Compare these stocks and explain the differences',
|
|
'intents': {'primary': 'compare', 'secondary': ['and_explain'], 'contextual': []},
|
|
'expected': True,
|
|
'complexity': 'medium'
|
|
},
|
|
{
|
|
'query': 'Monitor this stock and alert me on changes',
|
|
'intents': {'primary': 'monitor', 'secondary': ['and_alert'], 'contextual': []},
|
|
'expected': True,
|
|
'complexity': 'medium'
|
|
}
|
|
]
|
|
|
|
# Triple intent tests
|
|
triple_intents = [
|
|
{
|
|
'query': 'Analyze AAPL stock, show me a chart, and save the results',
|
|
'intents': {'primary': 'analyze', 'secondary': ['and_visualize', 'and_save'], 'contextual': []},
|
|
'expected': True,
|
|
'complexity': 'high'
|
|
},
|
|
{
|
|
'query': 'Compare these stocks, explain differences, and give me a quick summary',
|
|
'intents': {'primary': 'compare', 'secondary': ['and_explain'], 'contextual': ['quick_summary']},
|
|
'expected': True,
|
|
'complexity': 'high'
|
|
}
|
|
]
|
|
|
|
# Complex natural language tests
|
|
complex_queries = [
|
|
{
|
|
'query': 'I need to analyze the performance of these tech stocks, create some visualizations to compare them, and save everything to a file for my presentation',
|
|
'intents': {'primary': 'analyze', 'secondary': ['and_visualize', 'and_compare', 'and_save'], 'contextual': []},
|
|
'expected': True,
|
|
'complexity': 'very_high'
|
|
},
|
|
{
|
|
'query': 'Can you help me monitor my portfolio in real-time and send me alerts if anything significant happens, with detailed analysis of what\'s going on?',
|
|
'intents': {'primary': 'monitor', 'secondary': ['and_alert', 'and_explain'], 'contextual': ['real_time', 'detailed_analysis']},
|
|
'expected': True,
|
|
'complexity': 'very_high'
|
|
}
|
|
]
|
|
|
|
# Edge cases and invalid combinations
|
|
edge_cases = [
|
|
{
|
|
'query': 'Analyze this stock and teach me how to cook',
|
|
'intents': {'primary': 'analyze', 'secondary': [], 'contextual': []},
|
|
'expected': True,
|
|
'complexity': 'low',
|
|
'note': 'Unsupported secondary intent should be filtered out'
|
|
},
|
|
{
|
|
'query': 'Compare these charts while explaining that theory',
|
|
'intents': {'primary': 'compare', 'secondary': ['and_explain'], 'contextual': []},
|
|
'expected': True,
|
|
'complexity': 'medium',
|
|
'note': 'Mixed context - should prioritize domain-relevant parts'
|
|
}
|
|
]
|
|
|
|
test_cases.extend(single_intents)
|
|
test_cases.extend(double_intents)
|
|
test_cases.extend(triple_intents)
|
|
test_cases.extend(complex_queries)
|
|
test_cases.extend(edge_cases)
|
|
|
|
return test_cases
|
|
|
|
def run_multi_intent_tests(skill_config, test_cases):
|
|
"""Run multi-intent detection tests"""
|
|
|
|
results = []
|
|
|
|
for i, test_case in enumerate(test_cases):
|
|
query = test_case['query']
|
|
expected_intents = test_case['intents']
|
|
expected = test_case['expected']
|
|
|
|
# Parse intents from query
|
|
detected_intents = parse_multiple_intents(query, skill_config['capabilities'])
|
|
|
|
# Validate results
|
|
result = {
|
|
'test_id': i + 1,
|
|
'query': query,
|
|
'expected_intents': expected_intents,
|
|
'detected_intents': detected_intents,
|
|
'expected_activation': expected,
|
|
'actual_activation': detected_intents['primary_intent'] is not None,
|
|
'intent_accuracy': calculate_intent_accuracy(expected_intents, detected_intents),
|
|
'complexity_match': test_case['complexity'] == detected_intents.get('complexity', 'unknown'),
|
|
'notes': test_case.get('note', '')
|
|
}
|
|
|
|
# Determine if test passed
|
|
primary_correct = expected_intents['primary'] == detected_intents.get('primary_intent')
|
|
secondary_correct = set(expected_intents['secondary']) == set(detected_intents.get('secondary_intents', []))
|
|
activation_correct = expected == result['actual_activation']
|
|
|
|
result['test_passed'] = primary_correct and secondary_correct and activation_correct
|
|
|
|
results.append(result)
|
|
|
|
# Log result
|
|
status = "✅" if result['test_passed'] else "❌"
|
|
print(f"{status} Test {i+1}: {query[:60]}...")
|
|
if not result['test_passed']:
|
|
print(f" Expected primary: {expected_intents['primary']}, Got: {detected_intents.get('primary_intent')}")
|
|
print(f" Expected secondary: {expected_intents['secondary']}, Got: {detected_intents.get('secondary_intents', [])}")
|
|
|
|
# Calculate metrics
|
|
total_tests = len(results)
|
|
passed_tests = sum(1 for r in results if r['test_passed'])
|
|
accuracy = passed_tests / total_tests if total_tests > 0 else 0
|
|
avg_intent_accuracy = sum(r['intent_accuracy'] for r in results) / total_tests if total_tests > 0 else 0
|
|
|
|
return {
|
|
'total_tests': total_tests,
|
|
'passed_tests': passed_tests,
|
|
'accuracy': accuracy,
|
|
'avg_intent_accuracy': avg_intent_accuracy,
|
|
'results': results
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 📊 **Performance Monitoring**
|
|
|
|
### **Multi-Intent Metrics**
|
|
|
|
```python
|
|
class MultiIntentMonitor:
|
|
"""Monitor multi-intent detection performance"""
|
|
|
|
def __init__(self):
|
|
self.metrics = {
|
|
'total_queries': 0,
|
|
'single_intent_queries': 0,
|
|
'multi_intent_queries': 0,
|
|
'intent_detection_accuracy': [],
|
|
'intent_combination_success': [],
|
|
'complexity_distribution': {'low': 0, 'medium': 0, 'high': 0, 'very_high': 0},
|
|
'execution_plan_accuracy': []
|
|
}
|
|
|
|
def log_intent_detection(self, query, detected_intents, execution_success=None):
|
|
"""Log intent detection results"""
|
|
|
|
self.metrics['total_queries'] += 1
|
|
|
|
# Count intent types
|
|
total_intents = 1 + len(detected_intents.get('secondary_intents', [])) + len(detected_intents.get('contextual_intents', []))
|
|
|
|
if total_intents == 1:
|
|
self.metrics['single_intent_queries'] += 1
|
|
else:
|
|
self.metrics['multi_intent_queries'] += 1
|
|
|
|
# Track complexity distribution
|
|
complexity = detected_intents.get('complexity', 'medium')
|
|
if complexity in self.metrics['complexity_distribution']:
|
|
self.metrics['complexity_distribution'][complexity] += 1
|
|
|
|
# Track execution success if provided
|
|
if execution_success is not None:
|
|
self.metrics['execution_plan_accuracy'].append(execution_success)
|
|
|
|
def calculate_multi_intent_rate(self):
|
|
"""Calculate the rate of multi-intent queries"""
|
|
if self.metrics['total_queries'] == 0:
|
|
return 0.0
|
|
|
|
return self.metrics['multi_intent_queries'] / self.metrics['total_queries']
|
|
|
|
def generate_performance_report(self):
|
|
"""Generate multi-intent performance report"""
|
|
|
|
total = self.metrics['total_queries']
|
|
if total == 0:
|
|
return "No data available"
|
|
|
|
multi_intent_rate = self.calculate_multi_intent_rate()
|
|
avg_execution_accuracy = (sum(self.metrics['execution_plan_accuracy']) / len(self.metrics['execution_plan_accuracy'])
|
|
if self.metrics['execution_plan_accuracy'] else 0)
|
|
|
|
report = f"""
|
|
Multi-Intent Detection Performance Report
|
|
========================================
|
|
|
|
Total Queries Analyzed: {total}
|
|
Single-Intent Queries: {self.metrics['single_intent_queries']} ({(self.metrics['single_intent_queries']/total)*100:.1f}%)
|
|
Multi-Intent Queries: {self.metrics['multi_intent_queries']} ({multi_intent_rate*100:.1f}%)
|
|
|
|
Complexity Distribution:
|
|
- Low: {self.metrics['complexity_distribution']['low']} ({(self.metrics['complexity_distribution']['low']/total)*100:.1f}%)
|
|
- Medium: {self.metrics['complexity_distribution']['medium']} ({(self.metrics['complexity_distribution']['medium']/total)*100:.1f}%)
|
|
- High: {self.metrics['complexity_distribution']['high']} ({(self.metrics['complexity_distribution']['high']/total)*100:.1f}%)
|
|
- Very High: {self.metrics['complexity_distribution']['very_high']} ({(self.metrics['complexity_distribution']['very_high']/total)*100:.1f}%)
|
|
|
|
Execution Plan Accuracy: {avg_execution_accuracy*100:.1f}%
|
|
"""
|
|
|
|
return report
|
|
```
|
|
|
|
---
|
|
|
|
## ✅ **Implementation Checklist**
|
|
|
|
### **Configuration Requirements**
|
|
- [ ] Add `intent_hierarchy` section to marketplace.json
|
|
- [ ] Define supported `primary_intents` with capabilities
|
|
- [ ] Define supported `secondary_intents` with compatibility rules
|
|
- [ ] Define supported `contextual_intents` with impact modifiers
|
|
- [ ] Configure `intent_combinations` with execution plans
|
|
- [ ] Set appropriate `intent_processing` thresholds
|
|
|
|
### **Testing Requirements**
|
|
- [ ] Generate multi-intent test cases for each combination
|
|
- [ ] Test single-intent queries (baseline)
|
|
- [ ] Test double-intent queries
|
|
- [ ] Test triple-intent queries
|
|
- [ ] Test complex natural language queries
|
|
- [ ] Validate edge cases and invalid combinations
|
|
|
|
### **Performance Requirements**
|
|
- [ ] Intent detection accuracy > 95%
|
|
- [ ] Multi-intent processing time < 200ms
|
|
- [ ] Execution plan accuracy > 90%
|
|
- [ ] Support for up to 5 concurrent intents
|
|
- [ ] Graceful fallback to primary intent
|
|
|
|
---
|
|
|
|
## 📈 **Expected Outcomes**
|
|
|
|
### **Performance Improvements**
|
|
- **Multi-Intent Support**: 0% → **100%**
|
|
- **Complex Query Handling**: 20% → **95%**
|
|
- **User Intent Accuracy**: 70% → **95%**
|
|
- **Natural Language Understanding**: 60% → **90%**
|
|
|
|
### **User Experience Benefits**
|
|
- Natural handling of complex requests
|
|
- Better understanding of user goals
|
|
- More comprehensive responses
|
|
- Reduced need for follow-up queries
|
|
|
|
---
|
|
|
|
**Version:** 1.0
|
|
**Last Updated:** 2025-10-24
|
|
**Maintained By:** Agent-Skill-Creator Team |