Initial commit

skills/scientific-critical-thinking/references/scientific_method.md

@@ -0,0 +1,169 @@
+# Scientific Method Core Principles
+
+## Fundamental Principles
+
+### 1. Empiricism
+- Knowledge derives from observable, measurable evidence
+- Claims must be testable through observation or experiment
+- Subjective experience alone is insufficient for scientific conclusions
+
+### 2. Falsifiability (Popper's Criterion)
+- A hypothesis must be capable of being proven false
+- Unfalsifiable claims are not scientific (e.g., "invisible, undetectable forces")
+- Good hypotheses make specific, testable predictions
+
+### 3. Reproducibility
+- Results must be replicable by independent researchers
+- Methods must be described with sufficient detail for replication
+- Single studies are rarely definitive; replication strengthens confidence
+
+### 4. Parsimony (Occam's Razor)
+- Prefer simpler explanations over complex ones when both fit the data
+- Don't multiply entities unnecessarily
+- Extraordinary claims require extraordinary evidence
+
+### 5. Systematic Observation
+- Use standardized, rigorous methods
+- Control for confounding variables
+- Minimize observer bias through blinding and protocols
+
+## The Scientific Process
+
+### 1. Question Formation
+- Identify a specific, answerable question
+- Ensure the question is within the scope of scientific inquiry
+- Consider whether current methods can address the question
+
+### 2. Literature Review
+- Survey existing knowledge
+- Identify gaps and contradictions
+- Build on previous work rather than reinventing
+
+### 3. Hypothesis Development
+- State a clear, testable prediction
+- Define variables operationally
+- Specify the expected relationship between variables
+
+### 4. Experimental Design
+- Choose appropriate methodology
+- Identify independent and dependent variables
+- Control confounding variables
+- Select appropriate sample size and population
+- Plan statistical analyses in advance
+
+### 5. Data Collection
+- Follow protocols consistently
+- Record all observations, including unexpected results
+- Maintain detailed lab notebooks or data logs
+- Use validated measurement instruments
+
+### 6. Analysis
+- Apply appropriate statistical methods
+- Test assumptions of statistical tests
+- Consider effect size, not just significance
+- Look for alternative explanations
+
+### 7. Interpretation
+- Distinguish between correlation and causation
+- Acknowledge limitations
+- Consider alternative interpretations
+- Avoid overgeneralizing beyond the data
+
+### 8. Communication
+- Report methods transparently
+- Include negative results
+- Acknowledge conflicts of interest
+- Make data and code available when possible
+
+## Critical Evaluation Criteria
+
+### When Reviewing Scientific Work, Ask:
+
+**Validity Questions:**
+- Does the study measure what it claims to measure?
+- Are the methods appropriate for the research question?
+- Were controls adequate?
+- Could confounding variables explain the results?
+
+**Reliability Questions:**
+- Are measurements consistent?
+- Would the study produce similar results if repeated?
+- Are inter-rater reliability and measurement precision reported?
+
+**Generalizability Questions:**
+- Is the sample representative of the target population?
+- Are the conditions realistic or artificial?
+- Do the results apply beyond the specific context?
+
+**Statistical Questions:**
+- Is the sample size adequate for the analysis?
+- Are the statistical tests appropriate?
+- Are effect sizes reported alongside p-values?
+- Were multiple comparisons corrected?
+
+**Logical Questions:**
+- Do the conclusions follow from the data?
+- Are alternative explanations considered?
+- Are causal claims supported by the study design?
+- Are limitations acknowledged?
+
+## Red Flags in Scientific Claims
+
+1. **Cherry-picking data** - Highlighting only supporting evidence
+2. **Moving goalposts** - Changing predictions after seeing results
+3. **Ad hoc hypotheses** - Adding explanations to rescue a failed prediction
+4. **Appeal to authority** - "Expert X says" without evidence
+5. **Anecdotal evidence** - Relying on personal stories over systematic data
+6. **Correlation implies causation** - Confusing association with causality
+7. **Post hoc rationalization** - Explaining results after the fact without prediction
+8. **Ignoring base rates** - Not considering prior probability
+9. **Confirmation bias** - Seeking only evidence that supports beliefs
+10. **Publication bias** - Only positive results get published
+
+## Standards for Causal Inference
+
+### Bradford Hill Criteria (adapted)
+1. **Strength** - Strong associations are more likely causal
+2. **Consistency** - Repeated observations by different researchers
+3. **Specificity** - Specific outcomes from specific causes
+4. **Temporality** - Cause precedes effect (essential)
+5. **Biological gradient** - Dose-response relationship
+6. **Plausibility** - Coherent with existing knowledge
+7. **Coherence** - Consistent with other evidence
+8. **Experiment** - Experimental evidence supports causation
+9. **Analogy** - Similar cause-effect relationships exist
+
+### Establishing Causation Requires:
+- Temporal precedence (cause before effect)
+- Covariation (cause and effect correlate)
+- Elimination of alternative explanations
+- Ideally: experimental manipulation showing cause produces effect
+
+## Peer Review and Scientific Consensus
+
+### Understanding Peer Review
+- Filters obvious errors but isn't perfect
+- Reviewers can miss problems or have biases
+- Published ≠ proven; it means "passed initial scrutiny"
+- Retraction mechanisms exist for flawed papers
+
+### Scientific Consensus
+- Emerges from convergence of multiple independent lines of evidence
+- Consensus can change with new evidence
+- Individual studies rarely overturn consensus
+- Consider the weight of evidence, not individual papers
+
+## Open Science Principles
+
+### Transparency Practices
+- Preregistration of hypotheses and methods
+- Open data sharing
+- Open-source code
+- Preprints for rapid dissemination
+- Registered reports (peer review before data collection)
+
+### Why Transparency Matters
+- Reduces publication bias
+- Enables verification
+- Prevents p-hacking and HARKing (Hypothesizing After Results are Known)
+- Accelerates scientific progress