# Evidence Hierarchy and Quality Assessment

## Traditional Evidence Hierarchy (Medical/Clinical)

### Level 1: Systematic Reviews and Meta-Analyses
**Description:** Comprehensive synthesis of all available evidence on a question.

**Strengths:**
- Combines multiple studies for greater power
- Reduces impact of single-study anomalies
- Can identify patterns across studies
- Quantifies overall effect size

**Weaknesses:**
- Quality depends on included studies ("garbage in, garbage out")
- Publication bias can distort findings
- Heterogeneity may make pooling inappropriate
- Can mask important differences between studies

**Critical evaluation:**
- Was search comprehensive (multiple databases, grey literature)?
- Were inclusion criteria appropriate and prespecified?
- Was study quality assessed?
- Was heterogeneity explored?
- Was publication bias assessed (funnel plots, fail-safe N)?
- Were appropriate statistical methods used?

### Level 2: Randomized Controlled Trials (RCTs)
**Description:** Experimental studies with random assignment to conditions.

**Strengths:**
- Gold standard for establishing causation
- Controls for known and unknown confounders
- Minimizes selection bias
- Enables causal inference

**Weaknesses:**
- May not be ethical or feasible
- Artificial settings may limit generalizability
- Often short-term with selected populations
- Expensive and time-consuming

**Critical evaluation:**
- Was randomization adequate (sequence generation, allocation concealment)?
- Was blinding implemented (participants, providers, assessors)?
- Was sample size adequate (power analysis)?
- Was intention-to-treat analysis used?
- Was attrition rate acceptable and balanced?
- Are results generalizable?

### Level 3: Cohort Studies
**Description:** Observational studies following groups over time.

**Types:**
- **Prospective:** Follow forward from exposure to outcome
- **Retrospective:** Look backward at existing data

**Strengths:**
- Can study multiple outcomes
- Establishes temporal sequence
- Can calculate incidence and relative risk
- More feasible than RCTs for many questions

**Weaknesses:**
- Susceptible to confounding
- Selection bias possible
- Attrition can bias results
- Cannot prove causation definitively

**Critical evaluation:**
- Were cohorts comparable at baseline?
- Was exposure measured reliably?
- Was follow-up adequate and complete?
- Were potential confounders measured and controlled?
- Was outcome assessment blinded to exposure?

### Level 4: Case-Control Studies
**Description:** Compare people with outcome (cases) to those without (controls), looking back at exposures.

**Strengths:**
- Efficient for rare outcomes
- Relatively quick and inexpensive
- Can study multiple exposures
- Useful for generating hypotheses

**Weaknesses:**
- Cannot calculate incidence
- Susceptible to recall bias
- Selection of controls is challenging
- Cannot prove causation

**Critical evaluation:**
- Were cases and controls defined clearly?
- Were controls appropriate (same source population)?
- Was matching appropriate?
- How was exposure ascertained (records vs. recall)?
- Were potential confounders controlled?
- Could recall bias explain findings?

### Level 5: Cross-Sectional Studies
**Description:** Snapshot observation at single point in time.

**Strengths:**
- Quick and inexpensive
- Can assess prevalence
- Useful for hypothesis generation
- Can study multiple outcomes and exposures

**Weaknesses:**
- Cannot establish temporal sequence
- Cannot determine causation
- Prevalence-incidence bias
- Survival bias

**Critical evaluation:**
- Was sample representative?
- Were measures validated?
- Could reverse causation explain findings?
- Are confounders acknowledged?

### Level 6: Case Series and Case Reports
**Description:** Description of observations in clinical practice.

**Strengths:**
- Can identify new diseases or effects
- Hypothesis-generating
- Details rare phenomena
- Quick to report

**Weaknesses:**
- No control group
- No statistical inference possible
- Highly susceptible to bias
- Cannot establish causation or frequency

**Use:** Primarily for hypothesis generation and clinical description.

### Level 7: Expert Opinion
**Description:** Statements by recognized authorities.

**Strengths:**
- Synthesizes experience
- Useful when no research available
- May integrate multiple sources

**Weaknesses:**
- Subjective and potentially biased
- May not reflect current evidence
- Appeal to authority fallacy risk
- Individual expertise varies

**Use:** Lowest level of evidence; should be supported by data when possible.

## Nuances and Limitations of Traditional Hierarchy

### When Lower-Level Evidence Can Be Strong
1. **Well-designed observational studies** with:
   - Large effects (hard to confound)
   - Dose-response relationships
   - Consistent findings across contexts
   - Biological plausibility
   - No plausible confounders

2. **Multiple converging lines of evidence** from different study types

3. **Natural experiments** approximating randomization

### When Higher-Level Evidence Can Be Weak
1. **Poor-quality RCTs** with:
   - Inadequate randomization
   - High attrition
   - No blinding when feasible
   - Conflicts of interest

2. **Biased meta-analyses**:
   - Publication bias
   - Selective inclusion
   - Inappropriate pooling
   - Poor search strategy

3. **Not addressing the right question**:
   - Wrong population
   - Wrong comparison
   - Wrong outcome
   - Too artificial to generalize

## Alternative: GRADE System

GRADE (Grading of Recommendations Assessment, Development and Evaluation) assesses evidence quality across four levels:

### High Quality
**Definition:** Very confident that true effect is close to estimated effect.

**Characteristics:**
- Well-conducted RCTs
- Overwhelming evidence from observational studies
- Large, consistent effects
- No serious limitations

### Moderate Quality
**Definition:** Moderately confident; true effect likely close to estimated, but could be substantially different.

**Downgrades from high:**
- Some risk of bias
- Inconsistency across studies
- Indirectness (different populations/interventions)
- Imprecision (wide confidence intervals)
- Publication bias suspected

### Low Quality
**Definition:** Limited confidence; true effect may be substantially different.

**Downgrades:**
- Serious limitations in above factors
- Observational studies without special strengths

### Very Low Quality
**Definition:** Very limited confidence; true effect likely substantially different.

**Characteristics:**
- Very serious limitations
- Expert opinion
- Multiple serious flaws

## Study Quality Assessment Criteria

### Internal Validity (Bias Control)
**Questions:**
- Was randomization adequate?
- Was allocation concealed?
- Were groups similar at baseline?
- Was blinding implemented?
- Was attrition minimal and balanced?
- Was intention-to-treat used?
- Were all outcomes reported?

### External Validity (Generalizability)
**Questions:**
- Is sample representative of target population?
- Are inclusion/exclusion criteria too restrictive?
- Is setting realistic?
- Are results applicable to other populations?
- Are effects consistent across subgroups?

### Statistical Conclusion Validity
**Questions:**
- Was sample size adequate (power)?
- Were statistical tests appropriate?
- Were assumptions checked?
- Were effect sizes and confidence intervals reported?
- Were multiple comparisons addressed?
- Was analysis prespecified?

### Construct Validity (Measurement)
**Questions:**
- Were measures validated and reliable?
- Was outcome defined clearly and appropriately?
- Were assessors blinded?
- Were exposures measured accurately?
- Was timing of measurement appropriate?

## Critical Appraisal Tools

### For Different Study Types

**RCTs:**
- Cochrane Risk of Bias Tool
- Jadad Scale
- PEDro Scale (for trials in physical therapy)

**Observational Studies:**
- Newcastle-Ottawa Scale
- ROBINS-I (Risk of Bias in Non-randomized Studies)

**Diagnostic Studies:**
- QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies)

**Systematic Reviews:**
- AMSTAR-2 (A Measurement Tool to Assess Systematic Reviews)

**All Study Types:**
- CASP Checklists (Critical Appraisal Skills Programme)

## Domain-Specific Considerations

### Basic Science Research
**Hierarchy differs:**
1. Multiple convergent lines of evidence
2. Mechanistic understanding
3. Reproducible experiments
4. Established theoretical framework

**Key considerations:**
- Replication essential
- Mechanistic plausibility
- Consistency across model systems
- Convergence of methods

### Psychological Research
**Additional concerns:**
- Replication crisis
- Publication bias particularly problematic
- Small effect sizes often expected
- Cultural context matters
- Measures often indirect (self-report)

**Strong evidence includes:**
- Preregistered studies
- Large samples
- Multiple measures
- Behavioral (not just self-report) outcomes
- Cross-cultural replication

### Epidemiology
**Causal inference frameworks:**
- Bradford Hill criteria
- Rothman's causal pies
- Directed Acyclic Graphs (DAGs)

**Strong observational evidence:**
- Dose-response relationships
- Temporal consistency
- Biological plausibility
- Specificity
- Consistency across populations
- Large effects unlikely due to confounding

### Social Sciences
**Challenges:**
- Complex interventions
- Context-dependent effects
- Measurement challenges
- Ethical constraints on RCTs

**Strengthening evidence:**
- Mixed methods
- Natural experiments
- Instrumental variables
- Regression discontinuity designs
- Multiple operationalizations

## Synthesizing Evidence Across Studies

### Consistency
**Strong evidence:**
- Multiple studies, different investigators
- Different populations and settings
- Different research designs converge
- Different measurement methods

**Weak evidence:**
- Single study
- Only one research group
- Conflicting results
- Publication bias evident

### Biological/Theoretical Plausibility
**Strengthens evidence:**
- Known mechanism
- Consistent with other knowledge
- Dose-response relationship
- Coherent with animal/in vitro data

**Weakens evidence:**
- No plausible mechanism
- Contradicts established knowledge
- Biological implausibility

### Temporality
**Essential for causation:**
- Cause must precede effect
- Cross-sectional studies cannot establish
- Reverse causation must be ruled out

### Specificity
**Moderate indicator:**
- Specific cause → specific effect strengthens causation
- But lack of specificity doesn't rule out causation
- Most causes have multiple effects

### Strength of Association
**Strong evidence:**
- Large effects unlikely to be due to confounding
- Dose-response relationships
- All-or-none effects

**Caution:**
- Small effects may still be real
- Large effects can still be confounded

## Red Flags in Evidence Quality

### Study Design Red Flags
- No control group
- Self-selected participants
- No randomization when feasible
- No blinding when feasible
- Very small sample
- Inappropriate statistical tests

### Reporting Red Flags
- Selective outcome reporting
- No study registration/protocol
- Missing methodological details
- No conflicts of interest statement
- Cherry-picked citations
- Results don't match methods

### Interpretation Red Flags
- Causal language from correlational data
- Claiming "proof"
- Ignoring limitations
- Overgeneralizing
- Spinning negative results
- Post hoc rationalization

### Context Red Flags
- Industry funding without independence
- Single study in isolation
- Contradicts preponderance of evidence
- No replication
- Published in predatory journal
- Press release before peer review

## Practical Decision Framework

### When Evaluating Evidence, Ask:

1. **What type of study is this?** (Design)
2. **How well was it conducted?** (Quality)
3. **What does it actually show?** (Results)
4. **How likely is bias?** (Internal validity)
5. **Does it apply to my question?** (External validity)
6. **How does it fit with other evidence?** (Context)
7. **Are the conclusions justified?** (Interpretation)
8. **What are the limitations?** (Uncertainty)

### Making Decisions with Imperfect Evidence

**High-quality evidence:**
- Strong confidence in acting on findings
- Reasonable to change practice/policy

**Moderate-quality evidence:**
- Provisional conclusions
- Consider in conjunction with other factors
- May warrant action depending on stakes

**Low-quality evidence:**
- Weak confidence
- Hypothesis-generating
- Insufficient for major decisions alone
- Consider cost/benefit of waiting for better evidence

**Very low-quality evidence:**
- Very uncertain
- Should not drive decisions alone
- Useful for identifying gaps and research needs

### When Evidence is Conflicting

**Strategies:**
1. Weight by study quality
2. Look for systematic differences (population, methods)
3. Consider publication bias
4. Update with most recent, rigorous evidence
5. Conduct/await systematic review
6. Consider if question is well-formed

## Communicating Evidence Strength

**Avoid:**
- Absolute certainty ("proves")
- False balance (equal weight to unequal evidence)
- Ignoring uncertainty
- Cherry-picking studies

**Better:**
- Quantify uncertainty
- Describe strength of evidence
- Acknowledge limitations
- Present range of evidence
- Distinguish established from emerging findings
- Be clear about what is/isn't known