273 lines
8.9 KiB
Markdown
273 lines
8.9 KiB
Markdown
# EDA Example: Customer Churn Analysis
|
||
|
||
Complete exploratory data analysis for telecom customer churn dataset.
|
||
|
||
## Task
|
||
|
||
Explore customer churn dataset to understand:
|
||
- What factors correlate with churn?
|
||
- Are there data quality issues?
|
||
- What features should we engineer for predictive model?
|
||
|
||
## Dataset
|
||
|
||
- **Rows**: 7,043 customers
|
||
- **Target**: `Churn` (Yes/No)
|
||
- **Features**: 20 columns (demographics, account info, usage patterns)
|
||
|
||
## EDA Scaffold Applied
|
||
|
||
### 1. Data Overview
|
||
|
||
```python
|
||
import pandas as pd
|
||
import numpy as np
|
||
import matplotlib.pyplot as plt
|
||
import seaborn as sns
|
||
|
||
df = pd.read_csv('telecom_churn.csv')
|
||
|
||
print(f"Shape: {df.shape}")
|
||
# Output: (7043, 21)
|
||
|
||
print(f"Columns: {df.columns.tolist()}")
|
||
# ['customerID', 'gender', 'SeniorCitizen', 'Partner', 'Dependents',
|
||
# 'tenure', 'PhoneService', 'MultipleLines', 'InternetService',
|
||
# 'OnlineSecurity', 'OnlineBackup', 'DeviceProtection', 'TechSupport',
|
||
# 'StreamingTV', 'StreamingMovies', 'Contract', 'PaperlessBilling',
|
||
# 'PaymentMethod', 'MonthlyCharges', 'TotalCharges', 'Churn']
|
||
|
||
print(df.dtypes)
|
||
# customerID object
|
||
# gender object
|
||
# SeniorCitizen int64
|
||
# tenure int64
|
||
# MonthlyCharges float64
|
||
# TotalCharges object ← Should be numeric!
|
||
# Churn object
|
||
|
||
print(df.head())
|
||
print(df.describe())
|
||
```
|
||
|
||
**Findings**:
|
||
- TotalCharges is object type (should be numeric) - needs fixing
|
||
- Churn is target variable (26.5% churn rate)
|
||
|
||
### 2. Data Quality Checks
|
||
|
||
```python
|
||
# Missing values
|
||
missing = df.isnull().sum()
|
||
missing_pct = (missing / len(df)) * 100
|
||
print(missing_pct[missing_pct > 0])
|
||
# No missing values marked as NaN
|
||
|
||
# But TotalCharges is object - check for empty strings
|
||
print((df['TotalCharges'] == ' ').sum())
|
||
# Output: 11 rows have space instead of number
|
||
|
||
# Fix: Convert TotalCharges to numeric
|
||
df['TotalCharges'] = pd.to_numeric(df['TotalCharges'], errors='coerce')
|
||
print(df['TotalCharges'].isnull().sum())
|
||
# Output: 11 (now properly marked as missing)
|
||
|
||
# Strategy: Drop 11 rows (< 0.2% of data)
|
||
df = df.dropna()
|
||
|
||
# Duplicates
|
||
print(f"Duplicates: {df.duplicated().sum()}")
|
||
# Output: 0
|
||
|
||
# Data consistency checks
|
||
print("Tenure vs TotalCharges consistency:")
|
||
print(df[['tenure', 'MonthlyCharges', 'TotalCharges']].head())
|
||
# tenure=1, Monthly=$29, Total=$29 ✓
|
||
# tenure=34, Monthly=$57, Total=$1889 ≈ $57*34 ✓
|
||
```
|
||
|
||
**Findings**:
|
||
- 11 rows (0.16%) with missing TotalCharges - dropped
|
||
- No duplicates
|
||
- TotalCharges ≈ MonthlyCharges × tenure (consistent)
|
||
|
||
### 3. Univariate Analysis
|
||
|
||
```python
|
||
# Target variable
|
||
print(df['Churn'].value_counts(normalize=True))
|
||
# No 73.5%
|
||
# Yes 26.5%
|
||
|
||
# Imbalanced but not severely (>20% minority class is workable)
|
||
|
||
# Numeric variables
|
||
numeric_cols = ['tenure', 'MonthlyCharges', 'TotalCharges']
|
||
for col in numeric_cols:
|
||
print(f"\n{col}:")
|
||
print(f" Mean: {df[col].mean():.2f}, Median: {df[col].median():.2f}")
|
||
print(f" Std: {df[col].std():.2f}, Range: [{df[col].min()}, {df[col].max()}]")
|
||
|
||
# Histogram
|
||
df[col].hist(bins=50, edgecolor='black')
|
||
plt.title(f'{col} Distribution')
|
||
plt.xlabel(col)
|
||
plt.show()
|
||
|
||
# Check outliers
|
||
Q1, Q3 = df[col].quantile([0.25, 0.75])
|
||
IQR = Q3 - Q1
|
||
outliers = ((df[col] < (Q1 - 1.5*IQR)) | (df[col] > (Q3 + 1.5*IQR))).sum()
|
||
print(f" Outliers: {outliers} ({outliers/len(df)*100:.1f}%)")
|
||
```
|
||
|
||
**Findings**:
|
||
- **tenure**: Right-skewed (mean=32, median=29). Many new customers (0-12 months).
|
||
- **MonthlyCharges**: Bimodal distribution (peaks at ~$20 and ~$80). Suggests customer segments.
|
||
- **TotalCharges**: Right-skewed (correlated with tenure). Few outliers (2.3%).
|
||
|
||
```python
|
||
# Categorical variables
|
||
cat_cols = ['gender', 'SeniorCitizen', 'Partner', 'Dependents', 'Contract', 'PaymentMethod']
|
||
for col in cat_cols:
|
||
print(f"\n{col}: {df[col].nunique()} unique values")
|
||
print(df[col].value_counts())
|
||
|
||
# Bar plot
|
||
df[col].value_counts().plot(kind='bar')
|
||
plt.title(f'{col} Distribution')
|
||
plt.xticks(rotation=45)
|
||
plt.show()
|
||
```
|
||
|
||
**Findings**:
|
||
- **gender**: Balanced (50/50 male/female)
|
||
- **SeniorCitizen**: 16% are senior citizens
|
||
- **Contract**: 55% month-to-month, 24% one-year, 21% two-year
|
||
- **PaymentMethod**: Electronic check most common (34%)
|
||
|
||
### 4. Bivariate Analysis (Churn vs Features)
|
||
|
||
```python
|
||
# Churn rate by categorical variables
|
||
for col in cat_cols:
|
||
churn_rate = df.groupby(col)['Churn'].apply(lambda x: (x=='Yes').mean())
|
||
print(f"\n{col} vs Churn:")
|
||
print(churn_rate.sort_values(ascending=False))
|
||
|
||
# Stacked bar chart
|
||
pd.crosstab(df[col], df['Churn'], normalize='index').plot(kind='bar', stacked=True)
|
||
plt.title(f'Churn Rate by {col}')
|
||
plt.ylabel('Proportion')
|
||
plt.show()
|
||
```
|
||
|
||
**Key Findings**:
|
||
- **Contract**: Month-to-month churn=42.7%, One-year=11.3%, Two-year=2.8% (Strong signal!)
|
||
- **SeniorCitizen**: Seniors churn=41.7%, Non-seniors=23.6%
|
||
- **PaymentMethod**: Electronic check=45.3% churn, others~15-18%
|
||
- **tenure**: Customers with tenure<12 months churn=47.5%, >60 months=7.9%
|
||
|
||
```python
|
||
# Numeric variables vs Churn
|
||
for col in numeric_cols:
|
||
plt.figure(figsize=(10, 4))
|
||
|
||
# Box plot
|
||
plt.subplot(1, 2, 1)
|
||
df.boxplot(column=col, by='Churn')
|
||
plt.title(f'{col} by Churn')
|
||
|
||
# Histogram (overlay)
|
||
plt.subplot(1, 2, 2)
|
||
df[df['Churn']=='No'][col].hist(bins=30, alpha=0.5, label='No Churn', density=True)
|
||
df[df['Churn']=='Yes'][col].hist(bins=30, alpha=0.5, label='Churn', density=True)
|
||
plt.legend()
|
||
plt.xlabel(col)
|
||
plt.title(f'{col} Distribution by Churn')
|
||
plt.show()
|
||
```
|
||
|
||
**Key Findings**:
|
||
- **tenure**: Churned customers have lower tenure (mean=18 vs 38 months)
|
||
- **MonthlyCharges**: Churned customers pay MORE ($74 vs $61/month)
|
||
- **TotalCharges**: Churned customers have lower total (correlated with tenure)
|
||
|
||
```python
|
||
# Correlation matrix
|
||
numeric_df = df[['tenure', 'MonthlyCharges', 'TotalCharges', 'SeniorCitizen']].copy()
|
||
numeric_df['Churn_binary'] = (df['Churn'] == 'Yes').astype(int)
|
||
|
||
corr = numeric_df.corr()
|
||
plt.figure(figsize=(8, 6))
|
||
sns.heatmap(corr, annot=True, cmap='coolwarm', center=0)
|
||
plt.title('Correlation Matrix')
|
||
plt.show()
|
||
```
|
||
|
||
**Key Findings**:
|
||
- tenure ↔ TotalCharges: 0.83 (strong positive correlation - expected)
|
||
- Churn ↔ tenure: -0.35 (negative: longer tenure → less churn)
|
||
- Churn ↔ MonthlyCharges: +0.19 (positive: higher charges → more churn)
|
||
- Churn ↔ TotalCharges: -0.20 (negative: driven by tenure)
|
||
|
||
### 5. Insights & Recommendations
|
||
|
||
```python
|
||
print("\n=== KEY FINDINGS ===")
|
||
print("1. Data Quality:")
|
||
print(" - 11 rows (<0.2%) dropped due to missing TotalCharges")
|
||
print(" - No other quality issues. Data is clean.")
|
||
print("")
|
||
print("2. Churn Patterns:")
|
||
print(" - Overall churn rate: 26.5% (slightly imbalanced)")
|
||
print(" - Strongest predictor: Contract type (month-to-month 42.7% vs two-year 2.8%)")
|
||
print(" - High-risk segment: New customers (<12mo tenure) with high monthly charges")
|
||
print(" - Low churn: Long-term customers (>60mo) on two-year contracts")
|
||
print("")
|
||
print("3. Feature Importance:")
|
||
print(" - **High signal**: Contract, tenure, PaymentMethod, SeniorCitizen")
|
||
print(" - **Medium signal**: MonthlyCharges, InternetService")
|
||
print(" - **Low signal**: gender, PhoneService (balanced across churn/no-churn)")
|
||
print("")
|
||
print("\n=== RECOMMENDED ACTIONS ===")
|
||
print("1. Feature Engineering:")
|
||
print(" - Create 'tenure_bucket' (0-12mo, 12-24mo, 24-60mo, >60mo)")
|
||
print(" - Create 'high_charges' flag (MonthlyCharges > $70)")
|
||
print(" - Interaction: tenure × Contract (captures switching cost)")
|
||
print(" - Payment risk score (Electronic check is risky)")
|
||
print("")
|
||
print("2. Model Strategy:")
|
||
print(" - Use all categorical features (one-hot encode)")
|
||
print(" - Baseline: Predict churn for month-to-month + new customers")
|
||
print(" - Advanced: Random Forest or Gradient Boosting (handle interactions)")
|
||
print(" - Validate with stratified 5-fold CV (preserve 26.5% churn rate)")
|
||
print("")
|
||
print("3. Business Insights:")
|
||
print(" - **Retention program**: Target month-to-month customers < 12mo tenure")
|
||
print(" - **Contract incentives**: Offer discounts for one/two-year contracts")
|
||
print(" - **Payment method**: Encourage auto-pay (reduce electronic check)")
|
||
print(" - **Early warning**: Monitor customers with high MonthlyCharges + short tenure")
|
||
```
|
||
|
||
### 6. Self-Assessment
|
||
|
||
Using rubric:
|
||
|
||
- **Clarity** (5/5): Systematic exploration, clear findings at each stage
|
||
- **Completeness** (5/5): Data quality, univariate, bivariate, insights all covered
|
||
- **Rigor** (5/5): Proper statistical analysis, visualizations, quantified relationships
|
||
- **Actionability** (5/5): Specific feature engineering and business recommendations
|
||
|
||
**Average**: 5.0/5 ✓
|
||
|
||
This EDA provides solid foundation for predictive modeling and business action.
|
||
|
||
## Next Steps
|
||
|
||
1. **Feature engineering**: Implement recommended features
|
||
2. **Baseline model**: Logistic regression with top 5 features
|
||
3. **Advanced models**: Random Forest, XGBoost with feature interactions
|
||
4. **Evaluation**: F1-score, precision/recall curves, AUC-ROC
|
||
5. **Deployment**: Real-time churn scoring API
|