The p-value alone cannot settle this. Ask about the design, independent units, effect and interval, analysis choices, and supporting evidence.

Causal: credibility of the treatment effect

• Design: randomize, or justify how confounding will be controlled; prespecify the effect and analysis.
• Check: plan sensitivity analyses for missing outcomes and plausible assumption violations.
• Confirm: replicate in new animals; compare effect sizes and uncertainty.

Prediction: performance in new cases

• Split: reserve test data before exploration; keep each animal's measurements in one partition.
• Train: fit preprocessing, select features and tune models within training data only.
• Evaluate: check held-out error and calibration; use new sites or times when relevant.

• The experimental unit is independently assigned and treated: here the mouse; in other designs, a litter, cage or well.
• Technical repeats estimate measurement noise; they do not add independent biological sources.
• Fix: summarize within animal, or model cells within animals. Plan enough independent animals; a mixed model cannot create replication.

• Randomize treatment allocation; balance or randomize processing order, cage position and measurement. Convenience allocation can favor particular animals.
• Blind outcome assessment where feasible: use coded samples and a separate allocation list. Expectations can influence scoring, measurement and exclusions.
• Conceal the upcoming allocation; pre-specify exclusions before unblinding.
• Some meta-research links absent or unreported safeguards with larger effects (Macleod et al. 2008; Hirst et al. 2014). Report what was done, by whom and when.

set.seed(2026)
sample(rep(c("Treated","Control"), each = 10))

Example: 20 individually assigned mice, 10 per arm. Save the seed, list and assignment process; use blocks when needed.

Illustrative recovery rates: the treated group has higher recovery within each sex, but lower recovery overall because the group proportions differ. These associations alone do not establish a treatment effect.

• Usual suspects: processing batch or day, sex, litter, cage, age, site, operator, instrument.
• Fixes: randomize and block treatment across batches; record covariates. Pre-specify adjustment using the causal question, not every available variable.

• In this simulation the drug reduces mean volume by 150 mm³, from 500 to 350. We generate new mice under the same assumptions.
• Two sources of noise: biological variation between animals and measurement error.
• With more mice per group, the estimated difference varies less from one experiment to the next.

• This population is skewed. As n grows, the distribution of sample means becomes more nearly normal: the central limit theorem.
• One study gives one estimate. A statistical model lets us estimate its sampling uncertainty; independence and other assumptions matter.

Δ = μ_treated − μ_control.
For this power calculation: H₀: Δ = 0; H_a: Δ = −150 mm³.
350 − 500 = −150: treated tumors are smaller on average.

• Choose α before examining outcomes. For this test, it determines c and both decision regions.
• At fixed effect, n and SD, lower α moves c farther left and raises β. At fixed α, more mice improve power.
• The one-sided test looks for Δ < 0. Rejecting H₀ does not establish exactly −150; non-rejection does not establish zero.

Δ = μ_treated − μ_control.
H₀: Δ = 0; H_a: Δ = −150 mm³ for planning.

Four ingredients

1. The planning effect: Δ = 350 − 500 = −150 mm³, a 30% reduction
2. The noise: SD 120 mm³ between mice, from a pilot or the literature
3. The prespecified rule: one-sided α = 0.05
4. n per group: vary it while holding the other inputs fixed

n <- 3; df <- 2*n - 2
pt(qt(0.05, df), df,
   ncp = -150/(120*sqrt(2/n)))
# power = 0.356

What to bring to the consultation

• The scientific question in one sentence
• The primary outcome and how it is measured
• A sketch of the design: units, groups, time points, batches
• The smallest effect that would matter, and a source for the SD
• Constraints: animals, budget, time, ethics

Using AI tools without getting burned

• Useful for: drafting code and suggesting methods or assumptions to verify
• Your responsibility: the question, design, assumptions and interpretation
• Never paste identifiable patient data
• Verify every number it gives you; keep prompts in the analysis log

• Summary bars hide the distribution and influential observations.
• Plot each observation and its group mean. Formal uncertainty intervals come next.
• Show pairing or nesting. Investigate unusual values and record any changes (Weissgerber et al. 2015).

Random allocation supports a treatment comparison. Generalizing to other strains, laboratories or patients needs additional evidence.

• Each line is one full experiment, 10 vs 10 mice, and its 95% CI for the true difference.
• 95% describes the procedure: under its assumptions, about 95% of intervals from repeated studies contain the true effect.
• The number of misses varies between runs. Here, larger n narrows the intervals while long-run coverage remains about 95%.
• Read one CI as a range of effects compatible with the data and model, not a 95% probability for this fixed effect.

Δ = μ_treated − μ_control (mean tumor volume, mm³).
H₀: Δ = 0 versus H_a: Δ ≠ 0 (two-sided).

PitfallWhat a p-value is not

• The probability that the null hypothesis is true
• The probability that you are wrong
• The probability the result will replicate
• A measure of how big or how important the effect is
• A sharp boundary: 0.049 and 0.051 give very similar evidence

“P-values do not measure the probability that the studied hypothesis is true.” American Statistical Association, 2016

• A. Includes zero and a meaningful benefit. The data are inconclusive about benefit; they do not establish no effect.
• B. Excludes zero and supports a benefit larger than the pre-specified meaningful effect.
• C. Excludes zero, but the estimated benefit is smaller than the meaningful threshold.
• D. Wide enough to include benefit and harm: the estimate is too imprecise for this decision.

The independent units are 10 patients. Pairing retains the link between each person’s measurements.

Question and hypothesis

d = extra sleep on drug 2 − drug 1.
H₀: μ_d = 0 vs. H_a: μ_d ≠ 0.

Estimated difference

+1.58 hours

95% CI: +0.70 to +2.46 hours
Paired t-test: t(9) = 4.06; p = 0.0028.

Drug 2 is associated with a larger mean increase in sleep in these paired data.

Check the design and differences

Independent patients; approximately normal differences for this small sample. Consider order and carryover in a crossover design.

One animal = one observation. Use the animal-level outcome, not each cell as an independent replicate.

Overall ANOVA

H₀: μ_0.5 = μ₁ = μ₂
H_a: at least one mean differs.
F(2, 27) = 67.07; p = 3.36 × 10⁻¹¹.

ANOVA alone does not identify which doses differ or how much.

Specific contrast: high − low dose

Δ = μ₂ − μ_0.5
H₀: Δ = 0 vs. H_a: Δ ≠ 0.

+18.16 units

95% CI: +14.94 to +21.38 units.

Illustrative unadjusted contrast. Prespecify comparisons; correct for multiplicity across a family of tests.

Assumptions: independent animals; approximately normal errors with common variance. Check residuals and unequal spread.

Design: 24 participants randomized, 12 per group; one day-14 result each.

H₀: π_T = π_C
H_a: π_T ≠ π_C (two-sided).
π = probability of culture clearance.

Estimated increase: 50 percentage points
95% CI: 12.2 to 73.4 points.
Fisher’s exact p = 0.027.

Under H₀, expected clearances are only 4 per group. For this small sample, plan to use Fisher’s exact test; the chi-square approximation may be unreliable.

Design: 40 participants randomized, 20 per group. Baseline BP is measured before treatment.

ANCOVA = linear regression
Week-8 BP ~ treatment + baseline BP.

H₀: β_T = 0 versus H_a: β_T ≠ 0.
β_T = adjusted treatment − control difference in mean BP.

Adjusted difference: -7.1 mmHg
95% CI: -10.7 to -3.5.
Two-sided p < 0.001.

At the same baseline BP, the fitted treatment mean is 7.1 mmHg lower. Baseline adjustment can improve precision.

Kaplan–Meier curves

Drops occur at observed relapse times. A tick marks follow-up ending without an observed relapse.

Log-rank comparison

H₀: S_M(t) = S_N(t) for all t.
Hₐ: the remission curves differ.

χ²(1) = 3.40; p = 0.065

Suggestive separation, with substantial uncertainty. At α = 0.05, we do not reject H₀.

Censoring means “not observed after this time,” not “never relapses.”

Hazard: instantaneous relapse rate among those still in remission.

Proportional hazards model

h_M(t) = h_N(t) × exp(β)

H₀: β = 0 (HR = 1).
Hₐ: β ≠ 0 (HR ≠ 1).

Estimate: 60% lower relapse hazard with maintenance, assuming a constant hazard ratio.

Uncertainty: the interval includes HR = 1. The data remain compatible with no difference.

Two-sided Wald p = 0.074.
Group-only model; Efron handling of tied event times.

Hover or tap to inspect. With plot focused, ← / → select adjacent displayed variants; ↑ / ↓ jump 100. Last details stay visible.

m = – variant tests

Full SBP file, including known regions:

Bonferroni: reject H_0j if
p_j < 0.05 / m ≈ –.
With valid null p-values, P(any false rejection) ≤ 5%, even with dependence.

Why dependent? Linkage disequilibrium (LD) correlates nearby variants and their tests. Many significant variants may mark one locus.

Common GWAS cutoff: 5 × 10⁻⁸.
This differs from 0.05 / m for this file.

Repeated looks change the procedure

“Add three more mice if p > 0.05; stop if p < 0.05” gives repeated chances to cross the threshold under the null.

Choose before collecting confirmation data

• A fixed sample size with one planned final test, or
• A valid sequential design with planned looks, stopping boundaries and appropriate inference.

P-hacking and double dipping

P-hacking: choose analyses or reported results because their p-values are favorable, without accounting for the search.

Double dipping: use data to construct a feature or group, then assess it using the same information as if construction and assessment were independent.

Account for the selection rule

Selective inference: tests and CIs account for how the outcome or model was selected.

Return to the pilot: day-21 tumor volume had the smallest p = 0.01 among 20 outcome/timepoint tests.

Simpler familywise safeguard: for these 20 valid tests, Bonferroni gives 20 × 0.01 = 0.20. This is not a conditional selective p-value or a correction for the gene score.

In the methods and results

• n per group, and what n counts: animals, litters, wells, cells
• How allocation was randomized and who was blinded, where applicable
• The pre-specified primary outcome and analysis; any deviations
• Missing data and exclusions, with reasons and pre-specified rules
• Effect with CI; numerical p when relevant; method, assumptions and rationale
• Which error bar, in every legend
• Software and version; code and data where possible

Checklists that editors use

Many reporting requirements begin with decisions made before data collection.

Eight ALS treatments put to the test again

Scott et al. (2008) retested eight compounds previously reported to help SOD1 ALS mice, using improved study designs. They found no survival benefit at the previously reported doses and routes, including for riluzole. This result concerns those regimens in that mouse model.

Sex, litter, transgene copy number and unrelated deaths can distort survival comparisons. Design and model limitations can create misleading efficacy signals. Better design makes the evidence more informative.

About 9 per group. Even for an effect this large, n = 3 gives only about a one-in-three chance of seeing it.

• The hit rate is the power. With 3 per group it is about 36%, the same number the calculation gave.
• If we select only significant results, their effects are exaggerated on average. Low power warns about this selection; it does not tell us whether one result is true.

• Power depends on effect size, variability, α, n and design. Better measurement or blocking can reduce unexplained variation.
• With other inputs fixed, halving the target effect requires roughly four times as many independent units.
• For 80% power here: power.t.test(delta = 150, sd = 120, power = 0.8, alternative = "one.sided") gives n ≈ 8.68; round up to 9 per group.
• Approximation: n ≈ 12.37 / d² per group for 80% power and one-sided α = 0.05. Use the t calculation for small n; examine uncertain SD values.

• When measurements include transient variation, extreme values tend to be less extreme on repeat measurement, on average.
• Where it bites: selecting patients with extreme screening values, uncontrolled pre/post studies, or choosing “responders” from noisy measurements.
• Fix: a concurrent randomized control selected the same way. Pre-specify the between-arm comparison; baseline-adjusted analysis (ANCOVA) is often efficient.

Assumptions for this research example

Among significant results, what fraction correspond to real effects?

Expected: 20 true and 45 false discoveries.
20 / (20 + 45) ≈ 31% under this model.