Initial commit

skills/incident-response/ir-velociraptor/references/artifact-development.md

@@ -0,0 +1,627 @@
+# Velociraptor Artifact Development Guide
+
+Guide to creating custom VQL artifacts for specific investigation and threat hunting scenarios.
+
+## Table of Contents
+- [Artifact Structure](#artifact-structure)
+- [Parameter Types](#parameter-types)
+- [Source Types](#source-types)
+- [Best Practices](#best-practices)
+- [Common Patterns](#common-patterns)
+- [Testing Artifacts](#testing-artifacts)
+
+## Artifact Structure
+
+Velociraptor artifacts are YAML files with a defined structure:
+
+```yaml
+name: Category.Subcategory.ArtifactName
+description: |
+  Detailed description of what this artifact collects and why.
+  Include use cases and expected output.
+
+author: Your Name <email@domain.com>
+
+type: CLIENT  # CLIENT, SERVER, or CLIENT_EVENT
+
+parameters:
+  - name: ParameterName
+    default: "default_value"
+    type: string
+    description: Parameter description
+
+precondition: |
+  SELECT OS FROM info() WHERE OS = 'windows'
+
+sources:
+  - name: SourceName
+    query: |
+      SELECT * FROM plugin()
+      WHERE condition
+
+reports:
+  - type: CLIENT
+    template: |
+      # Report Title
+      {{ .Description }}
+
+      {{ range .Rows }}
+      - {{ .Column }}
+      {{ end }}
+```
+
+### Required Fields
+
+- **name**: Unique artifact identifier in dot notation
+- **description**: What the artifact does and when to use it
+- **sources**: At least one VQL query source
+
+### Optional Fields
+
+- **author**: Creator information
+- **type**: Artifact type (CLIENT, SERVER, CLIENT_EVENT)
+- **parameters**: User-configurable inputs
+- **precondition**: Check before running (OS, software presence)
+- **reports**: Output formatting templates
+- **references**: External documentation links
+
+## Parameter Types
+
+### String Parameters
+
+```yaml
+parameters:
+  - name: SearchPath
+    default: "C:/Windows/System32/"
+    type: string
+    description: Directory path to search
+```
+
+### Integer Parameters
+
+```yaml
+parameters:
+  - name: DaysBack
+    default: 7
+    type: int
+    description: Number of days to look back
+```
+
+### Boolean Parameters
+
+```yaml
+parameters:
+  - name: IncludeSystem
+    default: Y
+    type: bool
+    description: Include system files
+```
+
+### Regex Parameters
+
+```yaml
+parameters:
+  - name: ProcessPattern
+    default: "(?i)(powershell|cmd)"
+    type: regex
+    description: Process name pattern to match
+```
+
+### Choice Parameters
+
+```yaml
+parameters:
+  - name: LogLevel
+    default: "INFO"
+    type: choices
+    choices:
+      - DEBUG
+      - INFO
+      - WARNING
+      - ERROR
+    description: Logging verbosity
+```
+
+### CSV Parameters
+
+```yaml
+parameters:
+  - name: IOCList
+    default: |
+      evil.com
+      malicious.net
+    type: csv
+    description: List of IOC domains
+```
+
+## Source Types
+
+### Query Sources
+
+Standard VQL query that collects data:
+
+```yaml
+sources:
+  - name: ProcessCollection
+    query: |
+      SELECT Pid, Name, CommandLine, Username
+      FROM pslist()
+      WHERE Name =~ ProcessPattern
+```
+
+### Event Sources
+
+Continuous monitoring queries for CLIENT_EVENT artifacts:
+
+```yaml
+sources:
+  - name: ProcessCreation
+    query: |
+      SELECT * FROM watch_evtx(
+        filename="C:/Windows/System32/winevt/Logs/Security.evtx"
+      )
+      WHERE System.EventID.Value = 4688
+```
+
+### Multiple Sources
+
+Artifacts can have multiple sources for different data collection:
+
+```yaml
+sources:
+  - name: Processes
+    query: |
+      SELECT * FROM pslist()
+
+  - name: NetworkConnections
+    query: |
+      SELECT * FROM netstat()
+
+  - name: LoadedDLLs
+    query: |
+      SELECT * FROM modules()
+```
+
+## Best Practices
+
+### 1. Use Preconditions
+
+Prevent artifact execution on incompatible systems:
+
+```yaml
+# Windows-only artifact
+precondition: |
+  SELECT OS FROM info() WHERE OS = 'windows'
+
+# Requires specific tool
+precondition: |
+  SELECT * FROM stat(filename="C:/Tools/sysinternals/psexec.exe")
+
+# Version check
+precondition: |
+  SELECT * FROM info() WHERE OS = 'windows' AND OSVersion =~ '10'
+```
+
+### 2. Parameterize Paths and Patterns
+
+Make artifacts flexible and reusable:
+
+```yaml
+parameters:
+  - name: TargetPath
+    default: "C:/Users/**/AppData/**"
+    type: string
+
+  - name: FilePattern
+    default: "*.exe"
+    type: string
+
+sources:
+  - query: |
+      SELECT * FROM glob(globs=TargetPath + "/" + FilePattern)
+```
+
+### 3. Use LET for Query Composition
+
+Break complex queries into manageable parts:
+
+```yaml
+sources:
+  - query: |
+      -- Define reusable subqueries
+      LET SuspiciousProcesses = SELECT Pid, Name, CommandLine
+      FROM pslist()
+      WHERE CommandLine =~ "(?i)(bypass|hidden)"
+
+      LET NetworkConnections = SELECT Pid, Raddr.IP AS RemoteIP
+      FROM netstat()
+      WHERE Status = "ESTABLISHED"
+
+      -- Join and correlate
+      SELECT sp.Name,
+             sp.CommandLine,
+             nc.RemoteIP
+      FROM SuspiciousProcesses sp
+      JOIN NetworkConnections nc ON sp.Pid = nc.Pid
+```
+
+### 4. Add Error Handling
+
+Handle missing data gracefully:
+
+```yaml
+sources:
+  - query: |
+      SELECT * FROM foreach(
+        row={
+          SELECT FullPath FROM glob(globs=SearchPath)
+        },
+        query={
+          SELECT FullPath,
+                 hash(path=FullPath, accessor="file").SHA256 AS SHA256
+          FROM scope()
+          WHERE log(message="Processing: " + FullPath)
+        },
+        workers=5
+      )
+      WHERE SHA256  -- Filter out hash failures
+```
+
+### 5. Include Documentation
+
+Add inline comments and comprehensive descriptions:
+
+```yaml
+description: |
+  ## Overview
+  This artifact hunts for suspicious scheduled tasks.
+
+  ## Use Cases
+  - Persistence mechanism detection
+  - Lateral movement artifact collection
+  - Threat hunting campaigns
+
+  ## Output
+  Returns task name, actions, triggers, and creation time.
+
+  ## References
+  - MITRE ATT&CK T1053.005 (Scheduled Task/Job)
+```
+
+## Common Patterns
+
+### Pattern: File Collection with Hashing
+
+```yaml
+name: Custom.Windows.FileCollection
+description: Collect files matching patterns with hashes
+
+parameters:
+  - name: GlobPatterns
+    default: |
+      C:/Users/**/AppData/**/*.exe
+      C:/Windows/Temp/**/*.dll
+    type: csv
+
+sources:
+  - query: |
+      SELECT FullPath,
+             Size,
+             timestamp(epoch=Mtime) AS Modified,
+             timestamp(epoch=Btime) AS Created,
+             hash(path=FullPath, accessor="file") AS Hashes
+      FROM foreach(
+        row={
+          SELECT * FROM parse_csv(filename=GlobPatterns, accessor="data")
+        },
+        query={
+          SELECT * FROM glob(globs=_value)
+        }
+      )
+      WHERE NOT IsDir
+```
+
+### Pattern: Event Log Analysis
+
+```yaml
+name: Custom.Windows.EventLogHunt
+description: Hunt for specific event IDs with context
+
+parameters:
+  - name: LogFile
+    default: "C:/Windows/System32/winevt/Logs/Security.evtx"
+    type: string
+
+  - name: EventIDs
+    default: "4624,4625,4672"
+    type: csv
+
+sources:
+  - query: |
+      LET EventIDList = SELECT parse_string_with_regex(
+        string=EventIDs,
+        regex="(\\d+)"
+      ).g1 AS EventID FROM scope()
+
+      SELECT timestamp(epoch=System.TimeCreated.SystemTime) AS EventTime,
+             System.EventID.Value AS EventID,
+             System.Computer AS Computer,
+             EventData
+      FROM parse_evtx(filename=LogFile)
+      WHERE str(str=System.EventID.Value) IN EventIDList.EventID
+      ORDER BY EventTime DESC
+```
+
+### Pattern: Process Tree Analysis
+
+```yaml
+name: Custom.Windows.ProcessTree
+description: Build process tree from a starting PID
+
+parameters:
+  - name: RootPID
+    default: 0
+    type: int
+    description: Starting process PID (0 for all)
+
+sources:
+  - query: |
+      LET ProcessList = SELECT Pid, Ppid, Name, CommandLine, Username, CreateTime
+      FROM pslist()
+
+      LET RECURSIVE GetChildren(ParentPID) = SELECT *
+      FROM ProcessList
+      WHERE Ppid = ParentPID
+
+      LET RECURSIVE BuildTree(Level, ParentPID) = SELECT
+        Level,
+        Pid,
+        Ppid,
+        Name,
+        CommandLine,
+        Username,
+        CreateTime
+      FROM GetChildren(ParentPID=ParentPID)
+      UNION ALL
+      SELECT * FROM BuildTree(Level=Level+1, ParentPID=Pid)
+
+      SELECT * FROM if(
+        condition=RootPID > 0,
+        then={
+          SELECT * FROM BuildTree(Level=0, ParentPID=RootPID)
+        },
+        else={
+          SELECT 0 AS Level, * FROM ProcessList
+        }
+      )
+      ORDER BY CreateTime
+```
+
+### Pattern: Network IOC Matching
+
+```yaml
+name: Custom.Windows.NetworkIOCMatch
+description: Match network connections against IOC list
+
+parameters:
+  - name: IOCList
+    default: |
+      IP,Description
+      192.0.2.1,C2 Server
+      198.51.100.50,Malicious Host
+    type: csv
+
+sources:
+  - query: |
+      LET IOCs = SELECT IP, Description
+      FROM parse_csv(filename=IOCList, accessor="data")
+
+      LET Connections = SELECT
+        Raddr.IP AS RemoteIP,
+        Raddr.Port AS RemotePort,
+        Pid,
+        process_tracker_get(id=Pid).Name AS ProcessName,
+        process_tracker_get(id=Pid).CommandLine AS CommandLine
+      FROM netstat()
+      WHERE Status = "ESTABLISHED"
+
+      SELECT c.RemoteIP,
+             c.RemotePort,
+             c.ProcessName,
+             c.CommandLine,
+             i.Description AS IOCMatch
+      FROM Connections c
+      JOIN IOCs i ON c.RemoteIP = i.IP
+```
+
+### Pattern: Registry Timeline
+
+```yaml
+name: Custom.Windows.RegistryTimeline
+description: Timeline registry modifications in specific keys
+
+parameters:
+  - name: RegistryPaths
+    default: |
+      HKEY_LOCAL_MACHINE/SOFTWARE/Microsoft/Windows/CurrentVersion/Run/**
+      HKEY_CURRENT_USER/SOFTWARE/Microsoft/Windows/CurrentVersion/Run/**
+    type: csv
+
+  - name: DaysBack
+    default: 7
+    type: int
+
+sources:
+  - query: |
+      LET StartTime = timestamp(epoch=now() - DaysBack * 86400)
+
+      SELECT timestamp(epoch=Key.Mtime) AS Modified,
+             Key.FullPath AS RegistryPath,
+             ValueName,
+             ValueData.value AS Value
+      FROM foreach(
+        row={
+          SELECT * FROM parse_csv(filename=RegistryPaths, accessor="data")
+        },
+        query={
+          SELECT * FROM read_reg_key(globs=_value)
+        }
+      )
+      WHERE Key.Mtime > StartTime
+      ORDER BY Modified DESC
+```
+
+## Testing Artifacts
+
+### 1. Local Testing with GUI
+
+```bash
+# Start Velociraptor in GUI mode
+velociraptor gui
+
+# Navigate to: View Artifacts → Add Artifact
+# Paste your artifact YAML and click Save
+# Run artifact via Collected Artifacts → New Collection
+```
+
+### 2. Command Line Testing
+
+```bash
+# Test artifact syntax
+velociraptor artifacts show Custom.Artifact.Name
+
+# Run artifact locally
+velociraptor artifacts collect Custom.Artifact.Name \
+  --args ParameterName=value \
+  --format json
+
+# Run with output file
+velociraptor artifacts collect Custom.Artifact.Name \
+  --output results.json
+```
+
+### 3. Notebook Testing
+
+Use VQL notebooks for interactive development:
+
+```sql
+-- Test query components in isolation
+SELECT * FROM pslist() WHERE Name =~ "powershell" LIMIT 10
+
+-- Test parameter substitution
+LET ProcessPattern = "(?i)(powershell|cmd)"
+SELECT * FROM pslist() WHERE Name =~ ProcessPattern
+
+-- Test full artifact query
+/* Paste your artifact query here */
+```
+
+### 4. Validation Checklist
+
+Before deploying artifacts:
+
+- [ ] Artifact name follows convention: Category.Subcategory.Name
+- [ ] Description includes use cases and expected output
+- [ ] Parameters have sensible defaults
+- [ ] Precondition prevents incompatible execution
+- [ ] Query tested in notebook mode
+- [ ] Error handling for missing data
+- [ ] Performance acceptable on test system
+- [ ] Output format is useful and parseable
+- [ ] Documentation includes MITRE ATT&CK mapping if applicable
+
+## Performance Considerations
+
+### Limit Scope
+
+```yaml
+# BAD: Scans entire filesystem
+SELECT * FROM glob(globs="C:/**/*.exe")
+
+# GOOD: Targeted scope
+SELECT * FROM glob(globs=[
+  "C:/Users/**/AppData/**/*.exe",
+  "C:/Windows/Temp/**/*.exe"
+])
+```
+
+### Use Workers for Parallel Processing
+
+```yaml
+sources:
+  - query: |
+      SELECT * FROM foreach(
+        row={SELECT * FROM glob(globs=SearchPath)},
+        query={
+          SELECT FullPath,
+                 hash(path=FullPath, accessor="file").SHA256 AS SHA256
+          FROM scope()
+        },
+        workers=10  -- Process 10 files concurrently
+      )
+```
+
+### Rate Limiting
+
+```yaml
+sources:
+  - query: |
+      SELECT * FROM foreach(
+        row={SELECT * FROM glob(globs="C:/**")},
+        query={
+          SELECT * FROM scope()
+          WHERE rate(query_name="my_query", ops_per_sec=100)
+        }
+      )
+```
+
+## MITRE ATT&CK Mapping
+
+Map artifacts to MITRE ATT&CK techniques:
+
+```yaml
+name: Custom.Windows.PersistenceHunt
+description: |
+  Hunt for persistence mechanisms.
+
+  MITRE ATT&CK Techniques:
+  - T1547.001: Registry Run Keys / Startup Folder
+  - T1053.005: Scheduled Task/Job
+  - T1543.003: Windows Service
+  - T1546.003: Windows Management Instrumentation Event Subscription
+
+references:
+  - https://attack.mitre.org/techniques/T1547/001/
+  - https://attack.mitre.org/techniques/T1053/005/
+```
+
+## Artifact Distribution
+
+### Export Artifacts
+
+```bash
+# Export single artifact
+velociraptor artifacts show Custom.Artifact.Name > artifact.yaml
+
+# Export all custom artifacts
+velociraptor artifacts list --filter Custom > all_artifacts.yaml
+```
+
+### Import Artifacts
+
+```bash
+# Via command line
+velociraptor --config server.config.yaml artifacts import artifact.yaml
+
+# Via GUI
+# Navigate to: View Artifacts → Upload Artifact Pack
+```
+
+### Share via Artifact Exchange
+
+Contribute artifacts to the community:
+
+1. Test thoroughly across different systems
+2. Document clearly with examples
+3. Add MITRE ATT&CK mappings
+4. Submit to: https://docs.velociraptor.app/exchange/