Methods / E02 / November 7, 2026
The cost of checking every metric
A frozen protocol and reproducible pilot: four synthetic metrics, six planned looks, and 480,000 independent simulated experiments.
Repeated checking changes the error rate
Across all-null cells satisfying the model assumptions, naive checking falsely rejected at least one metric in 38.24–49.63% of experiments. Fixed-horizon correction estimates were 4.11–5.19%; planned-look correction estimates were 2.73–3.36%. These are ranges across cells, not pooled estimates.
The table includes every all-null cell. FWER is the probability of at least one true null rejection by time 60. Time points, metrics and looks are dependent measurements within an experiment; they are not additional independent replicates.
| Cell | Serial φ | Metric ρ | Observation rule | Naive FWER (95% MC interval) | Fixed FWER (95% MC interval) | Planned FWER (95% MC interval) |
|---|---|---|---|---|---|---|
| 0 | 0.0 | 0.0 | complete | 49.63% (48.94–50.32) | 5.08% (4.78–5.39) | 3.36% (3.12–3.62) |
| 2 | 0.0 | 0.0 | mcar | 48.88% (48.18–49.57) | 5.00% (4.71–5.31) | 3.36% (3.11–3.61) |
| 4 | 0.0 | 0.0 | informative | 89.54% (89.10–89.95) | 47.40% (46.70–48.09) | 26.32% (25.71–26.93) |
| 6 | 0.0 | 0.6 | complete | 38.98% (38.30–39.65) | 4.16% (3.89–4.44) | 2.78% (2.56–3.02) |
| 8 | 0.0 | 0.6 | mcar | 44.38% (43.69–45.07) | 4.63% (4.34–4.92) | 3.08% (2.84–3.32) |
| 10 | 0.0 | 0.6 | informative | 78.08% (77.50–78.64) | 37.22% (36.55–37.89) | 21.83% (21.26–22.40) |
| 12 | 0.6 | 0.0 | complete | 47.79% (47.10–48.48) | 5.19% (4.89–5.50) | 3.29% (3.05–3.54) |
| 14 | 0.6 | 0.0 | mcar | 47.65% (46.96–48.34) | 4.57% (4.29–4.87) | 2.99% (2.76–3.23) |
| 16 | 0.6 | 0.0 | informative | 61.64% (60.96–62.31) | 13.87% (13.40–14.36) | 5.78% (5.46–6.11) |
| 18 | 0.6 | 0.6 | complete | 38.24% (37.56–38.91) | 4.11% (3.84–4.39) | 2.73% (2.51–2.96) |
| 20 | 0.6 | 0.6 | mcar | 39.56% (38.88–40.23) | 4.21% (3.94–4.50) | 2.79% (2.57–3.02) |
| 22 | 0.6 | 0.6 | informative | 50.16% (49.46–50.85) | 10.87% (10.45–11.31) | 4.79% (4.50–5.09) |
Informative rows violate the assumptions. They drop negative readings more often than positive ones. The independent-noise negative control raises corrected-method FWER to 47.40% and 26.32%. A multiple-testing correction cannot repair invalid individual tests.
Intervals show finite simulation uncertainty only. They are pointwise 95% Wilson intervals, not simultaneous intervals across this table. Near 5%, half-width is about 0.30 percentage points. The methods share each experiment, so their estimates are paired.
The detection and delay tradeoff
Cell 1 has independent complete observations; cell 15 has serial correlation φ=0.6 and 30% MCAR missingness, with metric correlation ρ=0. A constant mean shift of 0.5 known marginal standard deviations affects only M1. This is a synthetic effect size.
| Cell | Method | M1 detection (95% MC interval) | Mean first detection time among detected (approx. 95% MC interval) | Mean time score (nondetection = 61) |
|---|---|---|---|---|
| 1 | Naive checking | 98.09% (97.89–98.27) | 22.41 (22.23–22.59); n=19,618 | 23.15 |
| 1 | Fixed horizon | 91.58% (91.19–91.96) | 60.00 (60.00–60.00); n=18,316 | 60.08 |
| 1 | Planned looks | 81.49% (80.94–82.02) | 35.78 (35.56–36.01); n=16,297 | 40.45 |
| 15 | Naive checking | 57.24% (56.55–57.92) | 31.35 (31.04–31.66); n=11,448 | 44.03 |
| 15 | Fixed horizon | 26.39% (25.78–27.00) | 60.00 (60.00–60.00); n=5,277 | 60.74 |
| 15 | Planned looks | 15.15% (14.66–15.65) | 40.10 (39.53–40.66); n=3,030 | 57.83 |
Conditional delay includes only detected experiments. The final column includes all experiments and codes nondetection as 61. Its interpretation is a descriptive capped score, not eventual detection time. Naive checking's apparent detection advantage accompanies uncontrolled familywise error.
All 72 method/cell rows retain endpoint simultaneous coverage, joint coverage over actual decision times, partial-alternative FWER, detection counts and delay uncertainty. The full report explains how coverage and delay differ.
What the calculations support
Each experiment has four stationary Gaussian AR(1) metrics and 60 time points. Serial and metric correlations each vary between 0 and 0.6. Means are either all zero or (0.5, 0, 0, 0). Observation rules are complete, 30% independent missingness, or informative dropout. The 24 cells each contain 20,000 independent experiments.
- Naive checking: nominal two-sided 5% tests for every metric at times 10, 20, 30, 40, 50 and 60.
- Fixed horizon: Bonferroni correction across four metrics at time 60 only.
- Planned looks: Bonferroni correction across four metrics and six prespecified looks.
For an outcome-independent observation mask, each observed sum has its exact known AR covariance variance. Gaussian standardization makes each marginal null test valid. A finite union bound controls the corrected policies' probability of any false rejection. Dependence between those valid tests does not break the bound.
The premise fails with outcome-dependent dropout, estimated covariance, arbitrary serial dependence or data-dependent family selection. No e-process, arbitrary-time confidence sequence or new theorem is claimed. See the frozen derivation and proceed/narrow/stop rules.
Decision: narrow. No predeclared simulation diagnostic triggered in the valid cells. Independent agent mathematical review passed within this scope; a separately scoped empirical design and qualified external methods review remain future work. Simulations alone cannot establish a missing assumption or validate a wearable policy.
Inspect the complete record
- Independent agent mathematical review · Review scope and reviewed hashes
- Frozen protocol, estimands, seeds and mathematical derivation
- Full original report and reproducibility commands
- All results as CSV · Structured results, environment and diagnostics
- Python benchmark · Calibration, edge and refusal tests
- Protocol-freeze receipt · Executed run · Replay receipt · Checksums
- Existing methodological source history
Python 3.12.14, NumPy 2.3.5, SciPy 1.17.0; PCG64 with fixed per-cell seeds. No new article retrieval, device dataset or dependency installation was used. The local protocol freeze is not public registration.