Inference calibration and limitations

This page records empirical null-size measurements, deliberate departures from textbook estimators, and regimes where factrix p-values are not calibrated. It is evidence for the contracts in Statistical methods, not a promise that one simulation design covers every user dataset.

Sizes below are rejection rates at a nominal 5% unless stated otherwise. Characterisation tests rerun selected cells at lower replication counts to detect material drift; the larger research sweeps remain documented here.

Coverage contract

Every registered metric that can publish a non-null p_value must appear on this page with either a measured size result or an explicit gap. A source-based test enforces names, not a claim that every path is calibrated.

Family	Covered p-value metrics
Series means and spreads	ic, quantile_spread, quantile_spread_vw, k_spread, fm_beta
Predictive and panel regressions	predictive_beta, spanning_alpha, common_asymmetry, common_quantile_spread, pooled_beta, common_beta
Event inference	caar, bmp_z, corrado_rank, event_hit_rate, event_ic
Direction, order, and trend	directional_hit_rate, monotonicity, positive_rate, ic_trend

HAC family calibration

The same overlapping input reaches different references depending on the rank of the restriction. The ranges are intentionally kept beside the path rather than repeated throughout the method guide.

Path	Measured null size	Interpretation
Scalar series mean with NEWEY_WEST	3.9–7.3% over T ∈ {60,120,240,500}, h ∈ {1,5,21}; 5.4–8.1% on AR(0.6) at h=1	Calibrated to mildly liberal on the measured grid; conservative when h is declared on a non-overlapping series
NON_OVERLAPPING mean, including ic, caar, and spread metrics	4.5–5.4% on measured overlap nulls	Removes mechanical overlap; does not remove persistence beyond the stride
predictive_beta, h=1	4.3–5.5% at regressor persistence ρ=0; 6.2–8.3% in the strongest measured Stambaugh cells	Amihud-Hurvich path is characterised at one-period horizons
predictive_beta, h>1	7.5–14.5%	Known oversized overlapping-regression HAC regime
Rank-one common_* contrasts	3.3–8.0% on non-persistent common-factor nulls; 10.5–19.5% in the re-measured persistent N=50, T=60, h=5, φ=0.9 cell (600 draws)	Scalar reference fixes the ordinary overlap case; persistence remains an oversized, incompletely detected warning regime
spanning_alpha rank-one contrast	5.7–11.7% on the measured overlapping-sum spread null	Uses the scalar reference, but the upper measured cells remain mildly oversized
Date-disjoint joint period slice tests	8–9% analytic and 10–15% bootstrap for K=5 on 50–90 periods per slice	Known short-slice over-rejection; prefer pairwise contrasts
Date-disjoint pairwise period bootstrap	4.0–7.0% on the MA(h-1) grid; 5.5–6.5% family-wise rejection on iid K=3,5 panels	Uses overlap-floored blocks and a genuine two-sample bootstrap-t root
pooled_beta Driscoll-Kraay	2.6–8.7% on its persistent-regressor, cross-sectionally correlated, overlapping-return panel grid	The scalar HAR reference reduced the former path's 6.3–22.5% range; the low-overlap-dependence cells can be conservative

The scalar recipe combines three factrix-specific choices: a 3(h - 1) overlap floor, variance scale T/(T-L-1), and effective degrees of freedom bounded by T/h - 1. These move together. Applying only the wider lag floor while keeping an ordinary reference made rank-one regressions worse in the measured cells.

Producing tests include tests/stats/test_hac_overlap_size.py, test_overlap_floor_size.py, test_scalar_wald_overlap_size.py, test_driscoll_kraay_size.py, and tests/test_stambaugh_bias.py. Run python tests/stats/test_driscoll_kraay_size.py from the repository root to regenerate the complete 1,000-replication Driscoll-Kraay grid.

Single-restriction Wald contrasts

common_asymmetry, common_quantile_spread, and spanning_alpha each test one linear restriction. Treating them like a multi-restriction Wald matrix left the two common-factor metrics 10–34% oversized at h > 1. Moving the bandwidth, finite-sample scale, and reference together to the scalar recipe put every non-persistent common-factor cell at or below 8.0%.

The correction is not a persistence cure. The persistent cell was re-measured at a nominal 5% with one AR(0.9) factor broadcast to 50 assets and forward returns formed as independent overlapping h=5 sums of iid shocks. Over 600 replications (seed = 20260907 + rep), at T=60 the rejection rates were 10.5% ± 1.3 percentage points for common_asymmetry and 19.5% ± 1.6 percentage points for common_quantile_spread. This is a measured cell, not a ceiling for other persistent null constructions.

serial_correlation_detected is a partial diagnostic rather than a complete classifier for that regime. On the same N=50, h=5, φ=0.9 construction it fired on 57.8% of draws at T=60 and 79.2% at T=120; absence of the warning therefore does not establish calibrated inference. Increasing a raw lag override does not repair the reference.

Shanken correction on fm_beta

On a true-null panel, the measured uncorrected versus Shanken rejection rates were 7.3/7.3%, 3.3/3.3%, 4.3/4.0%, and 6.0/6.0% over (T,h)={(120,1),(120,5),(240,1),(240,5)}. Under the null the estimated premium approaches zero and the multiplicative correction approaches one, so near-equality is expected. This table characterises size; it does not measure power or the omitted additive term in the simplified single-factor variance.

Persistence beyond the overlap horizon

The persistence screen reads lag-1 autocorrelation on the series after the overlap_periods stride. Reading the raw series would flag the MA(h - 1) structure that the stride or kernel was designed to handle.

On a directly constructed persistent per-period signal, the measured ranges were:

Lag-1 persistence	Plain t	NEWEY_WEST	STATIONARY_BOOTSTRAP
0	7–9%	7–9%	7–9%
0.6	32–34%	13–17%	12–19%
0.85	55–61%	32–34%	20–32%

No member is calibrated in the strongest rows. Switching from analytic to bootstrap inference changes the number but does not resolve the model failure. Use a raised hurdle, redesign the sample, or fit a model that explicitly handles the persistence. A screen estimated close to the 10-period floor is itself noisy; always read n_obs.

Andrews-Monahan AR(1) prewhitening is measured behind a private flag. It substantially reduces moderate-persistence excess on univariate mean paths but does not rescue near-unit-root input and is not implemented for every vector or regression kernel. Keeping it private avoids implying a library-wide correction that has not been calibrated.

Spread and rank-based paths

Metric or path	Measured result	Main limit
Spread-series STATIONARY_BOOTSTRAP	4.8–9.0% on the iid-factor grid; 3.7–10.0% on the persistent-factor overlap grid	Long-horizon cells can be mildly liberal while NW becomes conservative
monotonicity stationary-bootstrap MR test	2.0–7.3% over T ∈ {120,240}, h ∈ {1,5}, factor persistence φ ∈ {0,0.9}	Calibrated to conservative on the measured grid
positive_rate exact binomial	2.3–4.7% in most iid cells; one IC-pipeline cell measured 8.0% at n=240. On AR null series at T=200, size was 5.2% (phi=0), 11.5% (phi=0.3), 14.8% (phi=0.4), 20.2% (phi=0.5), 25.2% (phi=0.6) and 54.8% (phi=0.9) over 400 draws. With the hit-series threshold, the screen fired on 0.5% / 44.8% / 78.5% / 96.8% / 99.8% / 100% of draws at phi=0 / 0.3 / 0.4 / 0.5 / 0.6 / 0.9.	Exact-binomial discreteness is conservative under independent trials but does not protect against serially dependent hit indicators. serial_correlation_detected identifies persistence left after the overlap stride; the p-value is returned under the project-wide advisory contract but is not calibrated in that regime. The screen reads the dichotomised hit sequence, whose lag-1 autocorrelation is (2/pi) arcsin(phi) for a zero-mean Gaussian AR(phi) input — measured 0.194 at phi=0.3 and 0.263 at phi=0.4 against a predicted 0.194 / 0.262. POSITIVE_RATE_HIT_AUTOCORR = (2/pi) arcsin(PERSISTENT_SERIES_AUTOCORR) ≈ 0.194 restores the population warning boundary to phi=0.3 on that calibration grid. It is a metric-specific warning threshold, not a claim that arbitrary non-Gaussian factor series have a recoverable latent Gaussian correlation. The exact-binomial estimator and p-value are unchanged.
directional_hit_rate Pesaran-Timmermann with period deflation	3.0–7.7% on the measured panel grid	Same-period dependence must remain visible in the event-period metadata
ic_trend strided Hamed-Rao Mann-Kendall	1.7–5.3% over T ∈ {60,120,240}, h ∈ {1,5}	Strong residual persistence and unit-root input remain warning regimes
common_beta calendar-time cross-asset t	2.0–7.3% on the measured common-factor panel grid	A hand-built beta table uses a flagged iid reference; a failed producer estimate withholds inference instead of substituting iid

The positive_rate threshold uses the Gaussian sign-correlation identity from Hurt (1973). Its measured trigger rates, rather than that identity alone, define the supported calibration boundary.

Producing tests include test_spread_bootstrap_size.py, test_monotonicity_size.py, test_positive_rate_size.py, test_directional_hit_rate_size.py, test_ic_trend_size.py, and test_common_beta_size.py under tests/stats/.

Event-study paths

caar and corrado_rank infer on event periods; bmp_z infers on valid events and uses distinct event periods to assess clustering and sample reliability. Their n_obs values are therefore not interchangeable.

Path	Characterisation	Limit
caar non-overlapping event-period t	Falls in the 4.5–5.4% non-overlap band on the measured null	Event-induced variance motivates the standardised read
bmp_z with design-effect deflation	About 7% on the sign-aligned shared-shock null; around 10% at 8 effective event periods, around 7% at 15, clearing near 30	Residual size follows distinct event periods, not raw events; heavy skew at long horizons can bias standardised abnormal returns
corrado_rank	6% on the same sign-aligned null	Rank robustness does not create power from few event periods
event_hit_rate	7% on the sign-aligned null	Uses the same-period dependence adjustment; still thin with few event periods
event_ic	Covered by the shared clustering characterisation and dedicated clustered-inference tests	A pooled event correlation remains sensitive to the effective event-period count

event_skewness has no calibrated test

The withdrawn D'Agostino path failed on both non-normal signed returns and sign-aligned same-period shocks. On two panel nulls with only about 1.5 events per event period, it rejected 19.0% and 23.3%; on a sign-aligned clustered null it rejected 30.3%. Applying the event design-effect deflator barely moved the first failure and drove the last to 0.0%, so no single adjustment calibrated both. event_skewness is descriptive and publishes no p-value.

Joint period test on short slices: known over-rejection

For date-disjoint slices, slice_period_joint_test with K ≥ 3 and fewer than roughly 150 periods per slice is not a reliably sized omnibus test. At K=5, measured size is 8–9% for the analytic path and 10–15% for the bootstrap path at the short end. The bootstrap inherits noisy per-slice variance estimates; it is not automatically safer because the sample is short.

The pairwise bootstrap uses a genuine two-sample bootstrap-t root: observed and resampled contrasts each use their own block-based SE, and the same joint studentized draws feed raw p-values and Romano-Wolf. On iid per-period IC at T=60, B=499, 200 replications, family-wise rejection measured 5.5% at K=3 and 6.5% at K=5 (raw-pair rejection 6.2% and 6.5%).

The overlap floor matters separately. On two independent MA(h-1) null series with T=240, B=499, 200 replications, rejection measured 4.0%, 4.5%, and 7.0% for h=1, 5, 21; the pre-fix h=21 route measured 15% because its Politis-White block could be shorter than the known horizon. The analytic Welch-HAC path was not changed by this bootstrap fix: its narrow bandwidth remains uncalibrated on that MA(h-1) regime (16.7% at h=21 in the review grid), so it is not a substitute for the corrected default there.

Uneven evaluation grids

The main size tables use a constant stride. Scalar series-mean paths were also checked on one uneven (20, 20, 40)-spaced grid: NON_OVERLAPPING measured 5.8% and NEWEY_WEST 5.5%. Regression HAC, multi-restriction Wald, ADF, rolling, event-window, and adjacent-period paths have not been comprehensively recalibrated on uneven grids. uneven_evaluation_grid is therefore a design warning, not a claim that every method is invalid or valid there.