Initial commit

commands/ingest-add-klaviyo.md

@@ -0,0 +1,160 @@
+---
+name: ingest-add-klaviyo
+description: Generate complete Klaviyo ingestion workflow with all data sources using exact templates
+---
+
+# Add Klaviyo Ingestion
+
+## ⚠️ CRITICAL: This command generates ALL files at ONCE using exact templates
+
+I'll create a complete Klaviyo ingestion setup based on proven templates from `docs/sources/klaviyo.md`.
+
+---
+
+## What I'll Generate
+
+### MANDATORY: All files created in ONE response
+
+I will generate ALL of the following files in a SINGLE response using multiple Write tool calls:
+
+### Workflow Files
+1. **`ingestion/klaviyo_ingest_inc.dig`** - Incremental ingestion workflow
+2. **`ingestion/klaviyo_ingest_hist.dig`** - Historical backfill workflow
+
+### Configuration Files (in `ingestion/config/`)
+3. **`klaviyo_datasources.yml`** - Datasource definitions for all objects
+4. **`klaviyo_profiles_load.yml`** - Profiles configuration
+5. **`klaviyo_events_load.yml`** - Events configuration
+6. **`klaviyo_campaigns_load.yml`** - Campaigns configuration
+7. **`klaviyo_lists_load.yml`** - Lists configuration
+8. **`klaviyo_email_templates_load.yml`** - Email templates configuration
+9. **`klaviyo_metrics_load.yml`** - Metrics configuration
+
+### Credentials Template
+10. Updated `credentials_ingestion.json` with Klaviyo credentials section
+
+**Total: 10 files created in ONE response**
+
+---
+
+## Prerequisites
+
+Please provide the following information:
+
+### Required
+1. **Klaviyo API Key**: Your Klaviyo private API key (will be stored as secret)
+2. **TD Authentication ID**: Treasure Data authentication ID for Klaviyo connector (e.g., `klaviyo_auth_default`)
+3. **Default Start Date**: Initial historical load start date
+   - Format: `YYYY-MM-DDTHH:MM:SS.000000`
+   - Example: `2023-09-01T00:00:00.000000`
+
+### Optional
+4. **Target Database**: Default is `client_src` (leave blank to use default)
+
+---
+
+## Process I'll Follow
+
+### Step 1: Read Klaviyo Documentation (MANDATORY)
+I will READ these documentation files BEFORE generating ANY code:
+- `docs/sources/klaviyo.md` - Klaviyo exact templates
+- `docs/patterns/workflow-patterns.md` - Workflow patterns
+- `docs/patterns/logging-patterns.md` - Logging templates
+- `docs/patterns/timestamp-formats.md` - Klaviyo timestamp format (`.000000`)
+- `docs/patterns/incremental-patterns.md` - Dual field names for campaigns
+
+### Step 2: Generate ALL 10 Files in ONE Response
+Using multiple Write tool calls in a SINGLE message:
+- Write workflow files (2 files)
+- Write datasource config (1 file)
+- Write load configs (6 files)
+- Write credentials template update (1 file)
+
+### Step 3: Copy Exact Templates
+Templates will be copied character-for-character from documentation:
+- Klaviyo-specific timestamp format: `.000000` (6 decimals, NO Z)
+- Dual incremental fields for campaigns: `updated_at` in table, `updated` in API
+- Events with NO incremental field parameter
+- Exact SQL logging blocks
+- Exact error handling blocks
+
+### Step 4: Verify Quality Gates
+Before delivering, I will verify:
+✅ All 10 files created
+✅ Klaviyo timestamp format: `.000000` (6 decimals, NO Z)
+✅ Campaigns dual field names correct
+✅ Events config has NO incremental_field parameter
+✅ All logging blocks present (start, success, error)
+✅ All error handling blocks present
+✅ Parallel processing with limit: 3
+✅ COALESCE fallback to historical table
+
+---
+
+## Klaviyo-Specific Configuration
+
+### Objects Included
+1. **Profiles**: Customer profiles (incremental: `updated`)
+2. **Events**: Customer events (NO incremental field)
+3. **Campaigns**: Email campaigns (incremental: `updated_at` in table, `updated` in API)
+4. **Lists**: Email lists (incremental: `updated`)
+5. **Email Templates**: Campaign templates (incremental: `updated`)
+6. **Metrics**: Event metrics (incremental: `updated`)
+
+### Key Features
+- **Dual incremental fields**: Campaigns use different field names in table vs API
+- **Events handling**: No incremental parameter in config
+- **Timestamp format**: `.000000` (6 decimals, NO Z suffix)
+- **Parallel processing**: Limit of 3 for API rate limits
+- **Fallback logic**: COALESCE from incremental → historical → default
+
+---
+
+## After Generation
+
+### 1. Upload Credentials
+```bash
+# Navigate to your ingestion directory
+cd ingestion/
+td wf secrets --project ingestion --set @credentials_ingestion.json
+```
+
+### 2. Test Syntax
+```bash
+td wf check klaviyo_ingest_inc.dig
+td wf check klaviyo_ingest_hist.dig
+```
+
+### 3. Run Historical Backfill (First Time)
+```bash
+td wf run klaviyo_ingest_hist.dig
+```
+
+### 4. Run Incremental (Ongoing)
+```bash
+td wf run klaviyo_ingest_inc.dig
+```
+
+### 5. Monitor Ingestion
+```sql
+SELECT * FROM client_src.ingestion_log
+WHERE source_name LIKE 'klaviyo%'
+ORDER BY time DESC
+LIMIT 20
+```
+
+---
+
+## Production-Ready Guarantee
+
+All generated code will:
+- ✅ Follow exact Klaviyo templates from `docs/sources/klaviyo.md`
+- ✅ Use correct timestamp format (`.000000`)
+- ✅ Handle dual incremental fields correctly
+- ✅ Include all 6 data objects
+- ✅ Include comprehensive logging and error handling
+- ✅ Work the first time without modifications
+
+---
+
+**Ready to proceed? Provide the required information (API key, TD auth ID, start date) and I'll generate all 10 files in ONE response using exact templates from documentation.**