161 lines
5.1 KiB
Markdown
161 lines
5.1 KiB
Markdown
---
|
|
name: ingest-add-klaviyo
|
|
description: Generate complete Klaviyo ingestion workflow with all data sources using exact templates
|
|
---
|
|
|
|
# Add Klaviyo Ingestion
|
|
|
|
## ⚠️ CRITICAL: This command generates ALL files at ONCE using exact templates
|
|
|
|
I'll create a complete Klaviyo ingestion setup based on proven templates from `docs/sources/klaviyo.md`.
|
|
|
|
---
|
|
|
|
## What I'll Generate
|
|
|
|
### MANDATORY: All files created in ONE response
|
|
|
|
I will generate ALL of the following files in a SINGLE response using multiple Write tool calls:
|
|
|
|
### Workflow Files
|
|
1. **`ingestion/klaviyo_ingest_inc.dig`** - Incremental ingestion workflow
|
|
2. **`ingestion/klaviyo_ingest_hist.dig`** - Historical backfill workflow
|
|
|
|
### Configuration Files (in `ingestion/config/`)
|
|
3. **`klaviyo_datasources.yml`** - Datasource definitions for all objects
|
|
4. **`klaviyo_profiles_load.yml`** - Profiles configuration
|
|
5. **`klaviyo_events_load.yml`** - Events configuration
|
|
6. **`klaviyo_campaigns_load.yml`** - Campaigns configuration
|
|
7. **`klaviyo_lists_load.yml`** - Lists configuration
|
|
8. **`klaviyo_email_templates_load.yml`** - Email templates configuration
|
|
9. **`klaviyo_metrics_load.yml`** - Metrics configuration
|
|
|
|
### Credentials Template
|
|
10. Updated `credentials_ingestion.json` with Klaviyo credentials section
|
|
|
|
**Total: 10 files created in ONE response**
|
|
|
|
---
|
|
|
|
## Prerequisites
|
|
|
|
Please provide the following information:
|
|
|
|
### Required
|
|
1. **Klaviyo API Key**: Your Klaviyo private API key (will be stored as secret)
|
|
2. **TD Authentication ID**: Treasure Data authentication ID for Klaviyo connector (e.g., `klaviyo_auth_default`)
|
|
3. **Default Start Date**: Initial historical load start date
|
|
- Format: `YYYY-MM-DDTHH:MM:SS.000000`
|
|
- Example: `2023-09-01T00:00:00.000000`
|
|
|
|
### Optional
|
|
4. **Target Database**: Default is `client_src` (leave blank to use default)
|
|
|
|
---
|
|
|
|
## Process I'll Follow
|
|
|
|
### Step 1: Read Klaviyo Documentation (MANDATORY)
|
|
I will READ these documentation files BEFORE generating ANY code:
|
|
- `docs/sources/klaviyo.md` - Klaviyo exact templates
|
|
- `docs/patterns/workflow-patterns.md` - Workflow patterns
|
|
- `docs/patterns/logging-patterns.md` - Logging templates
|
|
- `docs/patterns/timestamp-formats.md` - Klaviyo timestamp format (`.000000`)
|
|
- `docs/patterns/incremental-patterns.md` - Dual field names for campaigns
|
|
|
|
### Step 2: Generate ALL 10 Files in ONE Response
|
|
Using multiple Write tool calls in a SINGLE message:
|
|
- Write workflow files (2 files)
|
|
- Write datasource config (1 file)
|
|
- Write load configs (6 files)
|
|
- Write credentials template update (1 file)
|
|
|
|
### Step 3: Copy Exact Templates
|
|
Templates will be copied character-for-character from documentation:
|
|
- Klaviyo-specific timestamp format: `.000000` (6 decimals, NO Z)
|
|
- Dual incremental fields for campaigns: `updated_at` in table, `updated` in API
|
|
- Events with NO incremental field parameter
|
|
- Exact SQL logging blocks
|
|
- Exact error handling blocks
|
|
|
|
### Step 4: Verify Quality Gates
|
|
Before delivering, I will verify:
|
|
✅ All 10 files created
|
|
✅ Klaviyo timestamp format: `.000000` (6 decimals, NO Z)
|
|
✅ Campaigns dual field names correct
|
|
✅ Events config has NO incremental_field parameter
|
|
✅ All logging blocks present (start, success, error)
|
|
✅ All error handling blocks present
|
|
✅ Parallel processing with limit: 3
|
|
✅ COALESCE fallback to historical table
|
|
|
|
---
|
|
|
|
## Klaviyo-Specific Configuration
|
|
|
|
### Objects Included
|
|
1. **Profiles**: Customer profiles (incremental: `updated`)
|
|
2. **Events**: Customer events (NO incremental field)
|
|
3. **Campaigns**: Email campaigns (incremental: `updated_at` in table, `updated` in API)
|
|
4. **Lists**: Email lists (incremental: `updated`)
|
|
5. **Email Templates**: Campaign templates (incremental: `updated`)
|
|
6. **Metrics**: Event metrics (incremental: `updated`)
|
|
|
|
### Key Features
|
|
- **Dual incremental fields**: Campaigns use different field names in table vs API
|
|
- **Events handling**: No incremental parameter in config
|
|
- **Timestamp format**: `.000000` (6 decimals, NO Z suffix)
|
|
- **Parallel processing**: Limit of 3 for API rate limits
|
|
- **Fallback logic**: COALESCE from incremental → historical → default
|
|
|
|
---
|
|
|
|
## After Generation
|
|
|
|
### 1. Upload Credentials
|
|
```bash
|
|
# Navigate to your ingestion directory
|
|
cd ingestion/
|
|
td wf secrets --project ingestion --set @credentials_ingestion.json
|
|
```
|
|
|
|
### 2. Test Syntax
|
|
```bash
|
|
td wf check klaviyo_ingest_inc.dig
|
|
td wf check klaviyo_ingest_hist.dig
|
|
```
|
|
|
|
### 3. Run Historical Backfill (First Time)
|
|
```bash
|
|
td wf run klaviyo_ingest_hist.dig
|
|
```
|
|
|
|
### 4. Run Incremental (Ongoing)
|
|
```bash
|
|
td wf run klaviyo_ingest_inc.dig
|
|
```
|
|
|
|
### 5. Monitor Ingestion
|
|
```sql
|
|
SELECT * FROM client_src.ingestion_log
|
|
WHERE source_name LIKE 'klaviyo%'
|
|
ORDER BY time DESC
|
|
LIMIT 20
|
|
```
|
|
|
|
---
|
|
|
|
## Production-Ready Guarantee
|
|
|
|
All generated code will:
|
|
- ✅ Follow exact Klaviyo templates from `docs/sources/klaviyo.md`
|
|
- ✅ Use correct timestamp format (`.000000`)
|
|
- ✅ Handle dual incremental fields correctly
|
|
- ✅ Include all 6 data objects
|
|
- ✅ Include comprehensive logging and error handling
|
|
- ✅ Work the first time without modifications
|
|
|
|
---
|
|
|
|
**Ready to proceed? Provide the required information (API key, TD auth ID, start date) and I'll generate all 10 files in ONE response using exact templates from documentation.**
|