gh-treasure-data-aps-claude-tools-plugins-cdp-ingestion/commands/ingest-new.md

name, description

name	description
ingest-new	Create a complete ingestion workflow for a new data source

STOP - READ THIS FIRST

You are about to create a CDP ingestion workflow. You MUST collect configuration parameters interactively using the AskUserQuestion tool.

DO NOT ask all questions at once. DO NOT use markdown lists. DO NOT explain what you're going to do.

EXECUTE the AskUserQuestion tool calls below IN ORDER.

---

EXECUTION SEQUENCE - FOLLOW EXACTLY

ACTION 1: Ask Data Source Question

USE the AskUserQuestion tool RIGHT NOW to ask question 1.

DO NOT PROCEED until you execute this tool call:

AskUserQuestion with:
- Question: "What data source are you ingesting from?"
- Header: "Data Source"
- Options:
  * Klaviyo (API-based connector)
  * Shopify (E-commerce platform)
  * Salesforce (CRM system)
  * Custom API (REST-based)

STOP. EXECUTE THIS TOOL NOW. DO NOT READ FURTHER UNTIL COMPLETE.

---

ACTION 2: Ask Ingestion Mode Question

CHECKPOINT: Did you get an answer to question 1? If NO, go back to ACTION 1.

NOW ask question 2 using AskUserQuestion tool:

AskUserQuestion with:
- Question: "What ingestion mode do you need?"
- Header: "Mode"
- Options:
  * Both (historical + incremental) - Recommended for complete setup
  * Incremental only - Ongoing sync only
  * Historical only - One-time backfill

STOP. EXECUTE THIS TOOL NOW. DO NOT READ FURTHER UNTIL COMPLETE.

---

ACTION 3: Ask Tables/Objects

CHECKPOINT: Did you get an answer to question 2? If NO, go back to ACTION 2.

This is a free-text question. Tell the user:

"Please provide the table or object names to ingest (comma-separated)."

Example: orders, customers, products

WAIT for user response. DO NOT PROCEED.

---

ACTION 4: Ask Incremental Field (CONDITIONAL)

CHECKPOINT:

• Did user select "Incremental only" OR "Both" in question 2?
• YES → Ask this question
• NO → Skip to ACTION 6

NOW ask question 4 using AskUserQuestion tool:

AskUserQuestion with:
- Question: "What field tracks record updates?"
- Header: "Incremental Field"
- Options:
  * updated_at (Timestamp field)
  * modified_date (Date field)
  * last_modified_time (Datetime field)

STOP. EXECUTE THIS TOOL NOW. DO NOT READ FURTHER UNTIL COMPLETE.

---

ACTION 5: Ask Start Date (CONDITIONAL)

CHECKPOINT:

• Did user select "Incremental only" OR "Both" in question 2?
• YES → Ask this question
• NO → Skip to ACTION 6

This is a free-text question. Tell the user:

"What is the initial load start date?"

Format: YYYY-MM-DDTHH:mm:ss.000000 Example: 2024-01-01T00:00:00.000000

WAIT for user response. DO NOT PROCEED.

---

ACTION 6: Ask Target Database

CHECKPOINT: Did you complete all previous questions? If NO, go back.

NOW ask question 6 using AskUserQuestion tool:

AskUserQuestion with:
- Question: "Which target database should data be loaded into?"
- Header: "Target DB"
- Options:
  * mck_src (Standard client database)
  * Custom database (Specify custom name)

STOP. EXECUTE THIS TOOL NOW. DO NOT READ FURTHER UNTIL COMPLETE.

---

ACTION 7: Ask Authentication

CHECKPOINT: Did you get an answer to question 6? If NO, go back to ACTION 6.

NOW ask question 7 using AskUserQuestion tool:

AskUserQuestion with:
- Question: "What type of authentication is required?"
- Header: "Auth Type"
- Options:
  * API Key (Single key authentication)
  * OAuth Token (Token-based auth)
  * Service Account JSON (Google/Cloud auth)
  * Username & Password (Basic auth)

STOP. EXECUTE THIS TOOL NOW. DO NOT READ FURTHER UNTIL COMPLETE.

---

ACTION 8: Show Configuration Summary

CHECKPOINT: Have you collected ALL required parameters? If NO, go back and complete missing questions.

NOW display the configuration summary:

📋 Configuration Summary:

Source: {source_name}
Connector Type: {connector_type}
Ingestion Mode: {mode}
Tables/Objects: {objects}
Target Database: {target_database}
[If applicable] Incremental Field: {field}
[If applicable] Start Date: {date}
Authentication: {auth_type}

ASK: "Does this configuration look correct? Type 'yes' to proceed with generation."

WAIT for user confirmation. DO NOT PROCEED until user types "yes".

---

WORKFLOW GENERATION (ONLY AFTER USER CONFIRMS)

CHECKPOINT: Did user confirm with "yes"? If NO, STOP and wait.

Step 1: Read Documentation Templates

Read these files in this EXACT order:

1. docs/sources/template-new-source.md
2. docs/patterns/workflow-patterns.md
3. docs/patterns/logging-patterns.md
4. docs/patterns/timestamp-formats.md
5. docs/patterns/incremental-patterns.md

Check if source-specific template exists:

• docs/sources/{source-name}.md (e.g., docs/sources/klaviyo.md)

Step 2: Generate Files (ALL IN ONE RESPONSE)

Use multiple Write tool calls in a SINGLE message to create:

For "Incremental only" mode:

1. ingestion/{source}_ingest_inc.dig
2. ingestion/config/{source}_datasources.yml
3. ingestion/config/{source}_{object1}_load.yml
4. ingestion/config/{source}_{object2}_load.yml (if multiple objects)

For "Historical only" mode:

1. ingestion/{source}_ingest_hist.dig
2. ingestion/config/{source}_datasources.yml
3. ingestion/config/{source}_{object}_load.yml

For "Both" mode:

1. ingestion/{source}_ingest_hist.dig
2. ingestion/{source}_ingest_inc.dig
3. ingestion/config/{source}_datasources.yml
4. ingestion/config/{source}_{object}_load.yml (per object)

Step 3: Template Rules (MANDATORY)

• Copy templates EXACTLY character-for-character
• NO simplification, NO optimization, NO improvements
• ONLY replace placeholders: {source_name}, {object_name}, {database}, {connector_type}
• Keep ALL logging blocks
• Keep ALL error handling blocks
• Keep ALL timestamp functions

Step 4: Quality Verification

Before showing output to user, verify:

• ✅ All template sections present
• ✅ All logging blocks included (start, success, error)
• ✅ All error handling blocks present
• ✅ Timestamp format matches connector type
• ✅ Incremental field handling correct
• ✅ No deviations from template

---

Post-Generation Instructions

After successfully creating all files, show the user:

Next Steps:

1. Upload credentials:

   cd ingestion
   td wf secrets --project ingestion --set @credentials_ingestion.json
   

2. Test workflow syntax:

   td wf check {source}_ingest_inc.dig
   

3. Deploy to Treasure Data:

   td wf push ingestion
   

4. Run the workflow:

   td wf start ingestion {source}_ingest_inc --session now
   

5. Monitor ingestion log:

   SELECT * FROM {target_database}.ingestion_log
   WHERE source_name = '{source}'
   ORDER BY time DESC
   LIMIT 10
   

---

ERROR RECOVERY

IF you did NOT use AskUserQuestion tool for each question:

• Print: "❌ ERROR: I failed to follow the interactive collection process."
• Print: "🔄 Restarting from ACTION 1..."
• GO BACK to ACTION 1 and start over

IF user says "skip questions" or "just ask all at once":

• Print: "❌ Cannot skip interactive collection - this ensures accuracy and prevents errors."
• Print: "✅ I'll collect parameters one at a time to ensure we get the configuration right."
• PROCEED with ACTION 1

---

NOW BEGIN: Execute ACTION 1 immediately. Use AskUserQuestion tool for the first question.