Initial commit

skills/statistical-analysis/references/test_selection_guide.md

@@ -0,0 +1,129 @@
+# Statistical Test Selection Guide
+
+This guide provides a decision tree for selecting appropriate statistical tests based on research questions, data types, and study designs.
+
+## Decision Tree for Test Selection
+
+### 1. Comparing Groups
+
+#### Two Independent Groups
+- **Continuous outcome, normally distributed**: Independent samples t-test
+- **Continuous outcome, non-normal**: Mann-Whitney U test (Wilcoxon rank-sum test)
+- **Binary outcome**: Chi-square test or Fisher's exact test (if expected counts < 5)
+- **Ordinal outcome**: Mann-Whitney U test
+
+#### Two Paired/Dependent Groups
+- **Continuous outcome, normally distributed**: Paired t-test
+- **Continuous outcome, non-normal**: Wilcoxon signed-rank test
+- **Binary outcome**: McNemar's test
+- **Ordinal outcome**: Wilcoxon signed-rank test
+
+#### Three or More Independent Groups
+- **Continuous outcome, normally distributed, equal variances**: One-way ANOVA
+- **Continuous outcome, normally distributed, unequal variances**: Welch's ANOVA
+- **Continuous outcome, non-normal**: Kruskal-Wallis H test
+- **Binary/categorical outcome**: Chi-square test
+- **Ordinal outcome**: Kruskal-Wallis H test
+
+#### Three or More Paired/Dependent Groups
+- **Continuous outcome, normally distributed**: Repeated measures ANOVA
+- **Continuous outcome, non-normal**: Friedman test
+- **Binary outcome**: Cochran's Q test
+
+#### Multiple Factors (Factorial Designs)
+- **Continuous outcome**: Two-way ANOVA (or higher-way ANOVA)
+- **With covariates**: ANCOVA
+- **Mixed within and between factors**: Mixed ANOVA
+
+### 2. Relationships Between Variables
+
+#### Two Continuous Variables
+- **Linear relationship, bivariate normal**: Pearson correlation
+- **Monotonic relationship or non-normal**: Spearman rank correlation
+- **Rank-based data**: Spearman or Kendall's tau
+
+#### One Continuous Outcome, One or More Predictors
+- **Single continuous predictor**: Simple linear regression
+- **Multiple continuous/categorical predictors**: Multiple linear regression
+- **Categorical predictors**: ANOVA/ANCOVA framework
+- **Non-linear relationships**: Polynomial regression or generalized additive models (GAM)
+
+#### Binary Outcome
+- **Single predictor**: Logistic regression
+- **Multiple predictors**: Multiple logistic regression
+- **Rare events**: Exact logistic regression or Firth's method
+
+#### Count Outcome
+- **Poisson-distributed**: Poisson regression
+- **Overdispersed counts**: Negative binomial regression
+- **Zero-inflated**: Zero-inflated Poisson/negative binomial
+
+#### Time-to-Event Outcome
+- **Comparing survival curves**: Log-rank test
+- **Modeling with covariates**: Cox proportional hazards regression
+- **Parametric survival models**: Weibull, exponential, log-normal
+
+### 3. Agreement and Reliability
+
+#### Inter-Rater Reliability
+- **Categorical ratings, 2 raters**: Cohen's kappa
+- **Categorical ratings, >2 raters**: Fleiss' kappa or Krippendorff's alpha
+- **Continuous ratings**: Intraclass correlation coefficient (ICC)
+
+#### Test-Retest Reliability
+- **Continuous measurements**: ICC or Pearson correlation
+- **Internal consistency**: Cronbach's alpha
+
+#### Agreement Between Methods
+- **Continuous measurements**: Bland-Altman analysis
+- **Categorical classifications**: Cohen's kappa
+
+### 4. Categorical Data Analysis
+
+#### Contingency Tables
+- **2x2 table**: Chi-square test or Fisher's exact test
+- **Larger than 2x2**: Chi-square test
+- **Ordered categories**: Cochran-Armitage trend test
+- **Paired categories**: McNemar's test (2x2) or McNemar-Bowker test (larger)
+
+### 5. Bayesian Alternatives
+
+Any of the above tests can be performed using Bayesian methods:
+- **Group comparisons**: Bayesian t-test, Bayesian ANOVA
+- **Correlations**: Bayesian correlation
+- **Regression**: Bayesian linear/logistic regression
+
+**Advantages of Bayesian approaches:**
+- Provides probability of hypotheses given data
+- Naturally incorporates prior information
+- Provides credible intervals instead of confidence intervals
+- No p-value interpretation issues
+
+## Key Considerations
+
+### Sample Size
+- Small samples (n < 30): Consider non-parametric tests or exact methods
+- Very large samples: Even small effects may be statistically significant; focus on effect sizes
+
+### Multiple Comparisons
+- When conducting multiple tests, adjust for multiple comparisons using:
+  - Bonferroni correction (conservative)
+  - Holm-Bonferroni (less conservative)
+  - False Discovery Rate (FDR) control (Benjamini-Hochberg)
+  - Tukey HSD for post-hoc ANOVA comparisons
+
+### Missing Data
+- Complete case analysis (listwise deletion)
+- Multiple imputation
+- Maximum likelihood methods
+- Ensure missing data mechanism is understood (MCAR, MAR, MNAR)
+
+### Effect Sizes
+- Always report effect sizes alongside p-values
+- See `effect_sizes_and_power.md` for guidance
+
+### Study Design Considerations
+- Randomized controlled trials: Standard parametric/non-parametric tests
+- Observational studies: Consider confounding and use regression/matching
+- Clustered/nested data: Use mixed-effects models or GEE
+- Time series: Use time series methods (ARIMA, etc.)