gh-lyndonkl-claude/skills/chain-estimation-decision-storytelling/resources/examples/build-vs-buy-analytics-platform.md

# Decision: Build Custom Analytics Platform vs. Buy SaaS Solution

**Date:** 2024-01-15
**Decision-maker:** CTO + VP Product
**Audience:** Executive team
**Stakes:** Medium ($500k-$1.5M over 3 years)

---

## 1. Decision Context

**What we're deciding:**
Should we build a custom analytics platform in-house or purchase a SaaS analytics solution?

**Why this matters:**
- Current analytics are manual and time-consuming (20 hours/week analyst time)
- Product team needs real-time insights to inform roadmap decisions
- Sales needs usage data to identify expansion opportunities
- Engineering wants to reduce operational burden of maintaining custom tools

**Alternatives:**
1. **Build custom**: Develop in-house analytics platform with our exact requirements
2. **Buy SaaS**: Purchase enterprise analytics platform (e.g., Amplitude, Mixpanel)
3. **Hybrid**: Use SaaS for standard metrics, build custom for proprietary analysis

**Key uncertainties:**
- Development cost and timeline (historical variance ±40%)
- Feature completeness of SaaS solution (will it meet all needs?)
- Usage growth rate (affects SaaS costs which scale with volume)
- Long-term flexibility needs (will we outgrow SaaS or need custom features?)

**Constraints:**
- Budget: $150k available in current year, $50k/year ongoing
- Timeline: Need solution operational within 6 months
- Requirements: Must support 100M events/month, 50+ team members, custom dashboards
- Strategic: Prefer minimal vendor lock-in, prioritize time-to-value

**Audience:** Executive team (need bottom-line recommendation + risks)

---

## 2. Estimation

### Alternative 1: Build Custom

**Costs:**
- **Initial development**: $200k-$400k (most likely $300k)
  - Base estimate: 6 engineer-months × $50k loaded cost = $300k
  - Range reflects scope uncertainty and potential technical challenges
  - Source: Similar internal projects averaged $280k ±$85k (30% std dev)

- **Annual operational costs**: $40k-$60k per year (most likely $50k)
  - Infrastructure: $15k-$25k (based on 100M events/month)
  - Maintenance: 0.5 engineer FTE = $25k-$35k per year
  - Source: Current analytics tools cost $45k/year to maintain

- **Opportunity cost**: $150k
  - Engineering team would otherwise work on core product features
  - Estimated value of deferred features: $150k in potential revenue impact

**Benefits:**
- **Cost savings**: $0 subscription fees (vs $120k/year for SaaS)
- **Perfect fit**: 100% feature match to our specific needs
- **Flexibility**: Full control to add custom analysis
- **Strategic value**: Build analytics competency, own our data

**Probabilities:**
- **Best case (20%)**: On-time delivery at $250k, perfect execution
  - Prerequisites: Clear requirements, no scope creep, experienced team available

- **Base case (50%)**: Moderate delays and cost overruns to $350k over 8 months
  - Typical scenario based on historical performance

- **Worst case (30%)**: Significant delays to $500k over 12 months, some features cut
  - Risk factors: Key engineer departure, underestimated complexity, changing requirements

**Key assumptions:**
- Engineering team has capacity (currently 70% utilized)
- No major technical unknowns in data pipeline
- Requirements are stable (< 10% scope change)
- Infrastructure costs scale linearly with events

### Alternative 2: Buy SaaS

**Costs:**
- **Initial implementation**: $15k-$25k (most likely $20k)
  - Setup and integration: 2-3 weeks consulting
  - Data migration and testing
  - Team training
  - Source: Vendor quote + reference customer feedback

- **Annual subscription**: $100k-$140k per year (most likely $120k)
  - Base: $80k for 100M events/month
  - Users: $2k per user × 20 power users = $40k
  - Growth buffer: Assume 20% event growth per year
  - Source: Vendor pricing confirmed, escalates with usage

- **Switching cost** (if we change vendors later): $50k-$75k
  - Data export and migration
  - Re-implementing integrations
  - Team retraining

**Benefits:**
- **Faster time-to-value**: 2 months vs. 8 months for build
  - 6-month head start = earlier insights = better decisions sooner
  - Estimated value: $75k (half of opportunity cost avoided)

- **Proven reliability**: 99.9% uptime SLA
  - Reduces operational risk
  - Frees engineering for core product

- **Feature velocity**: Continuous improvements from vendor
  - New capabilities quarterly (ML-powered insights, predictive analytics)
  - Estimated value: $30k/year in avoided feature development

- **Lower risk**: Predictable costs, no schedule risk
  - High confidence in timeline and total cost

**Probabilities:**
- **Best case (40%)**: Perfect fit, seamless implementation, $100k/year steady state
  - Vendor delivers on promises, usage grows slower than expected

- **Base case (45%)**: Good fit with minor gaps, standard implementation, $120k/year
  - 85% of needs met out-of-box, workarounds for remaining 15%

- **Worst case (15%)**: Poor fit requiring workarounds or supplemental tools, $150k/year
  - Missing critical features, need to maintain some custom tooling

**Key assumptions:**
- SaaS vendor is stable and continues product development
- Event volume growth is 20% per year (manageable)
- Vendor lock-in is acceptable (switching cost is reasonable)
- Security and compliance requirements are met by vendor

### Alternative 3: Hybrid

**Costs:**
- **Initial investment**: $100k-$150k (most likely $125k)
  - SaaS implementation: $20k
  - Custom integrations and proprietary metrics: $100k-$130k development

- **Annual costs**: $80k-$100k per year (most likely $90k)
  - SaaS subscription (smaller tier): $60k-$70k
  - Maintenance of custom components: $20k-$30k

**Benefits:**
- **Balanced approach**: Standard analytics from SaaS, custom analysis in-house
- **Reduced risk**: Less development than full build, more control than pure SaaS
- **Flexibility**: Can shift balance over time based on needs

**Probabilities:**
- **Base case (60%)**: Works reasonably well, $125k + $90k/year
- **Integration complexity (40%)**: More overhead than expected, $150k + $100k/year

**Key assumptions:**
- Clean separation between standard and custom analytics
- SaaS provides good API for custom integrations
- Maintaining two systems doesn't create excessive complexity

---

## 3. Decision Analysis

### Expected Value Calculation (3-Year NPV)

**Discount rate:** 10% (company's cost of capital)

#### Alternative 1: Build Custom

**Year 0 (Initial):**
- Best case (20%): -$250k development - $150k opportunity cost = -$400k
- Base case (50%): -$350k development - $150k opportunity cost = -$500k
- Worst case (30%): -$500k development - $150k opportunity cost = -$650k

**Expected Year 0:** ($-400k × 0.20) + ($-500k × 0.50) + ($-650k × 0.30) = -$525k

**Years 1-3 (Operational):**
- Annual cost: $50k/year
- PV of 3 years at 10%: $50k × 2.49 = $124k

**Total Expected NPV (Build):** -$525k - $124k = **-$649k**

*Note: Costs are negative because this is an investment. Focus is on minimizing cost since benefits (analytics capability) are equivalent across alternatives.*

#### Alternative 2: Buy SaaS

**Year 0 (Initial):**
- Implementation: $20k
- No opportunity cost (fast implementation)

**Years 1-3 (Operational):**
- Best case (40%): $100k/year × 2.49 = $249k
- Base case (45%): $120k/year × 2.49 = $299k
- Worst case (15%): $150k/year × 2.49 = $374k

**Expected annual cost:** ($100k × 0.40) + ($120k × 0.45) + ($150k × 0.15) = $116.5k/year
**PV of 3 years:** $116.5k × 2.49 = $290k

**Total Expected NPV (Buy):** -$20k - $290k = **-$310k**

**Benefit adjustment for faster time-to-value:** +$75k (6-month head start)
**Adjusted NPV (Buy):** -$310k + $75k = **-$235k**

#### Alternative 3: Hybrid

**Year 0 (Initial):**
- Development + implementation: $125k
- Partial opportunity cost: $75k (half the custom build time)

**Years 1-3 (Operational):**
- Expected annual: $90k/year × 2.49 = $224k

**Total Expected NPV (Hybrid):** -$125k - $75k - $224k = **-$424k**

### Comparison Summary

| Alternative | Expected 3-Year Cost | Risk Profile | Time to Value |
|-------------|---------------------|--------------|---------------|
| Build Custom | $649k              | **High** (30% worst case) | 8 months |
| Buy SaaS     | $235k              | **Low** (predictable) | 2 months |
| Hybrid       | $424k              | **Medium** | 5 months |

**Cost difference:** Buy SaaS saves **$414k** vs. Build Custom over 3 years

### Sensitivity Analysis

**What if development cost for Build is 20% lower ($240k base instead of $300k)?**
- Build NPV: -$577k (still $342k worse than Buy)
- **Conclusion still holds**

**What if SaaS costs grow 40% per year instead of 20%?**
- Year 3 SaaS cost: $230k (vs. $145k base case)
- Buy NPV: -$325k (still $324k better than Build)
- **Conclusion still holds**

**What if we need to switch SaaS vendors in Year 3?**
- Additional switching cost: $65k
- Buy NPV: -$300k (still $349k better than Build)
- **Conclusion still holds**

**Break-even analysis:**
At what annual SaaS cost does Build become cheaper?
- Build 3-year cost: $649k
- Buy 3-year cost: $20k + (X × 2.49) - $75k = $649k
- Solve: X = $282k/year

**Interpretation:** SaaS would need to cost $282k/year (2.4x current estimate) for Build to break even. Very unlikely.

### Robustness Check

**Conclusion is robust if:**
- Development cost < $600k (currently $300k base, $500k worst case ✓)
- SaaS annual cost < $280k (currently $120k base, $150k worst case ✓)
- Time-to-value benefit > $0 (6-month head start valuable ✓)

**Conclusion changes if:**
- SaaS vendor goes out of business (low probability, large incumbents)
- Regulatory requirements force on-premise solution (not currently foreseen)
- Custom analytics become core competitive differentiator (possible but unlikely)

---

## 4. Recommendation

### **Recommended option: Buy SaaS Solution**

**Reasoning:**

Buy SaaS dominates Build Custom on three dimensions:

1. **Lower expected cost**: $235k vs. $649k over 3 years (saves $414k)
2. **Lower risk**: Predictable subscription vs. 30% chance of 2x cost overrun on build
3. **Faster time-to-value**: 2 months vs. 8 months (6-month head start enables better decisions sooner)

The cost advantage is substantial ($414k savings) and robust to reasonable assumption changes. Even if SaaS costs double or we need to switch vendors, Buy still saves $300k+.

The risk profile strongly favors Buy. Historical data shows 30% of similar build projects experience 2x cost overruns. SaaS has predictable costs with 99.9% uptime SLA.

Time-to-value matters: getting analytics operational 6 months sooner means better product decisions sooner, worth approximately $75k in avoided opportunity cost.

**Key factors:**
1. **Cost**: $414k lower expected cost over 3 years
2. **Risk**: Predictable vs. high uncertainty (30% worst case for Build)
3. **Speed**: 2 months vs. 8 months to operational
4. **Strategic fit**: Analytics are important but not core competitive differentiator

**Tradeoffs accepted:**
- **Vendor dependency**: Accepting switching cost of $65k if we change vendors
  - Mitigation: Choose stable, market-leading vendor (Amplitude or Mixpanel)

- **Some feature gaps**: SaaS may not support 100% of custom analysis needs
  - Mitigation: 85% coverage out-of-box, workarounds for remaining 15%
  - Can supplement with lightweight custom tools if needed ($20k-$30k vs. $300k+ full build)

- **Less flexibility**: Can't customize as freely as in-house solution
  - Mitigation: Most SaaS platforms offer extensive APIs and integrations
  - True custom needs can be addressed incrementally

**Why not Hybrid?**
Hybrid ($424k) is $189k more expensive than Buy with minimal additional benefit. The complexity of maintaining two systems outweighs the incremental flexibility.

---

## 5. Risks and Mitigations

### Risk 1: SaaS doesn't meet all requirements

**Probability:** Medium (15% worst case scenario)

**Impact:** Need workarounds or supplemental tools

**Mitigation:**
- Conduct thorough vendor evaluation with 2-week pilot
- Map all requirements to vendor capabilities before committing
- Budget $30k for lightweight custom supplements if needed
- Still cheaper than full Build even with supplements

### Risk 2: Vendor lock-in / price increases

**Probability:** Low-Medium (vendors typically increase 5-10%/year)

**Impact:** Higher ongoing costs

**Mitigation:**
- Negotiate multi-year contract with price protection
- Maintain data export capability (ensure vendor supports data portability)
- Budget includes 20% annual growth buffer
- Switching cost is manageable ($65k) if needed

### Risk 3: Usage growth exceeds estimates

**Probability:** Low (current trajectory is 15%/year, estimated 20%)

**Impact:** Higher subscription costs

**Mitigation:**
- Monitor usage monthly against plan
- Optimize event instrumentation to reduce unnecessary events
- Renegotiate tier if growth is faster than expected
- Even at 2x usage growth, still cheaper than Build

### Risk 4: Security or compliance issues

**Probability:** Very Low (vendor is SOC 2 Type II certified)

**Impact:** Cannot use vendor, forced to build

**Mitigation:**
- Verify vendor security certifications before contract
- Review data handling and privacy policies
- Include compliance requirements in vendor evaluation
- This risk applies to any vendor; not specific to this decision

---

## 6. Next Steps

**If approved:**

1. **Vendor evaluation** (2 weeks) - VP Product + Data Lead
   - Demo top 3 vendors (Amplitude, Mixpanel, Heap)
   - Map requirements to capabilities
   - Validate pricing and terms
   - Decision by: Feb 1

2. **Pilot implementation** (2 weeks) - Engineering Lead
   - 2-week pilot with selected vendor
   - Instrument 3 key product flows
   - Validate data accuracy and latency
   - Go/no-go decision by: Feb 15

3. **Full rollout** (4 weeks) - Data Team + Engineering
   - Instrument all product events
   - Migrate existing dashboards
   - Train team on new platform
   - Launch by: March 15

**Success metrics:**
- **Time to value**: Analytics operational within 2 months (by March 15)
- **Cost**: Stay within $20k implementation + $120k annual budget
- **Adoption**: 50+ team members using platform within 30 days of launch
- **Value delivery**: Reduce manual analytics time from 20 hours/week to <5 hours/week

**Decision review:**
- **6-month review** (Sept 2024): Validate cost and value delivered
  - Key question: Are we getting value proportional to cost?
  - Metrics: Usage stats, time savings, decisions influenced by data

- **Annual review** (Jan 2025): Assess whether to continue, renegotiate, or reconsider build
  - Key indicators: Usage growth trend, missing features impact, pricing changes

**What would change our mind:**
- If vendor quality degrades significantly (downtime, bugs, poor support)
- If pricing increases >30% beyond projections
- If we identify analytics as core competitive differentiator (requires custom innovation)
- If regulatory requirements force on-premise solution

---

## 7. Appendix: Assumptions Log

**Development estimates:**
- Based on: 3 similar internal projects (API platform, reporting tool, data pipeline)
- Historical variance: ±30% from initial estimate
- Team composition: 2-3 senior engineers for 3-4 months
- Scope: Event ingestion, storage, query engine, dashboarding UI

**SaaS pricing:**
- Based on: Vendor quotes for 100M events/month, 50 users
- Confirmed with: 2 reference customers at similar scale
- Growth assumption: 20% annual event growth (aligned with product roadmap)
- User assumption: 20 power users (product, sales, exec) need full access

**Opportunity cost:**
- Based on: Engineering team would otherwise work on product features
- Estimated value: Product features could drive $150k additional revenue
- Source: Product roadmap prioritization (deferred features)

**Time-to-value benefit:**
- Based on: 6-month head start with SaaS (2 months vs. 8 months)
- Estimated value: Better decisions sooner = avoided mistakes + seized opportunities
- Conservative estimate: 50% of opportunity cost = $75k

**Discount rate:**
- Company cost of capital: 10%
- Used to calculate present value of multi-year costs

---

## Self-Assessment (Rubric Scores)

**Estimation Quality:** 4/5
- Comprehensive estimation with ranges and probabilities
- Justification provided for estimates with sources
- Could improve: More rigorous data collection from reference customers

**Probability Calibration:** 4/5
- Probabilities justified with base rates (historical project performance)
- Well-calibrated ranges
- Could improve: External validation of probability estimates

**Decision Analysis Rigor:** 5/5
- Sound expected value calculation with NPV
- Appropriate decision criteria
- Multiple scenarios tested

**Sensitivity Analysis:** 5/5
- Comprehensive one-way sensitivity on key variables
- Break-even analysis performed
- Conditions that change conclusion clearly stated

**Alternative Comparison:** 4/5
- Three alternatives analyzed fairly
- Could improve: Consider more creative alternatives (e.g., open-source + custom)

**Assumption Transparency:** 5/5
- All key assumptions stated explicitly with justification
- Alternative assumptions tested in sensitivity analysis

**Narrative Clarity:** 4/5
- Clear structure and logical flow
- Could improve: More compelling framing for exec audience

**Audience Tailoring:** 4/5
- Appropriate detail for executive audience
- Could improve: Add one-page executive summary

**Risk Acknowledgment:** 5/5
- Comprehensive risk analysis with probabilities and mitigations
- Downside scenarios quantified
- "What would change our mind" conditions stated

**Actionability:** 5/5
- Clear recommendation with specific next steps
- Owners and timeline defined
- Success metrics and review cadence specified

**Average Score:** 4.5/5 (Exceeds standard for medium-stakes decision)

---

**Analysis completed:** January 15, 2024
**Analyst:** [Name]
**Reviewed by:** CTO
**Status:** Ready for executive decision