Helpers for shaping a raw panel before evaluate. The canonical entry point, compute_forward_return, attaches a forward_return column to a raw (date, asset_id, price) panel — the output (date, asset_id, factor, forward_return) panel is the canonical input to evaluate.

The surrounding helpers cover the rest of the documented preprocessing pipeline and are independently usable on a canonical panel: return cleaning (winsorize_forward_return, compute_abnormal_return), factor normalization (mad_winsorize, cross_sectional_zscore), and orthogonalization against base factors (orthogonalize_factor).

Column adaptation

Use adapt when the input is already a long panel but carries vendor- or project-specific column names such as trade_date, ticker, or close_adj. It renames those columns to factrix's canonical date, asset_id, price, and optional OHLCV names before compute_forward_return. It does not reshape data, construct factors, or compute returns.

adapt preserves Polars eager/lazy inputs, converts pandas input to Polars, and leaves unrelated columns such as factors, industries, market caps, or regime labels unchanged. It never fills observations. compute_forward_return maps non-finite numeric values to null at its ingestion gate and drops rows whose required entry or exit price is unavailable. A missing price must not be forward-filled: doing so invents a tradable level and changes the effective holding period. Repair genuine source errors upstream under an explicit column and staleness policy.

factrix.adapt.adapt

adapt(data: AdaptInput, *, date: str = 'date', asset_id: str = 'asset_id', price: str = 'close', open: str | None = None, high: str | None = None, low: str | None = None, volume: str | None = None) -> DataFrame | LazyFrame

Rename user columns to factrix canonical names.

Type-preserving for polars inputs: a pl.LazyFrame stays lazy (rename / cast happen inside the lazy chain, no implicit .collect()), a pl.DataFrame stays eager. pd.DataFrame is converted to pl.DataFrame (pandas has no lazy equivalent). Only renames columns that differ from the canonical name; all other columns pass through unchanged.

Parameters:

Name	Type	Description	Default
data	AdaptInput	Input frame — pl.DataFrame, pl.LazyFrame, or pd.DataFrame.	required
date	str	User's date column name. Renamed to date — the panel's ordering and alignment key; every factrix horizon, window and lag counts periods on its distinct-date grid.	'date'
asset_id	str	User's asset identifier column name. Renamed to asset_id — the cross-sectional key every metric groups by.	'asset_id'
price	str	User's price column name. Renamed to price — the level series compute_forward_return differences into forward_return, and the volatility source the event-study metrics (caar, event_horizon) prefer when present.	'close'
open	str | None	User's opening-price column name. Renamed to open. Not read by factrix; passed through for downstream use.	None
high	str | None	User's period-high price column name. Renamed to high. Not read by factrix; passed through for downstream use.	None
low	str | None	User's period-low price column name. Renamed to low. Not read by factrix; passed through for downstream use.	None
volume	str | None	User's traded-volume column name. Renamed to volume. Not read by factrix; passed through for downstream use.	None

Returns:

Type	Description
DataFrame | LazyFrame	Same polars type as input (pl.DataFrame → pl.DataFrame,
DataFrame | LazyFrame	pl.LazyFrame → pl.LazyFrame) with canonical column
DataFrame | LazyFrame	names. pd.DataFrame input returns pl.DataFrame.

Raises:

Type	Description
TypeError	If data is none of pl.DataFrame, pl.LazyFrame, pd.DataFrame.
ValueError	If any specified source column does not exist, one source is assigned to multiple canonical names, or a rename would overwrite an existing canonical column.

The panel contract

Every public entry point — evaluate, evaluate_horizons, inspect_data, compute_forward_return — puts its input through one structural gate before anything else happens. Three things it enforces, each of which was previously left to individual producers and therefore missed by whichever path forgot it:

• date must be Date or Datetime. Only column names were validated, so a String date flowed through sort / shift / over / group_by as text and was ordered lexicographically. With ISO-8601 strings that accidentally works, which is what makes it dangerous — it passes every test and every demo. With MM/DD/YYYY the panel is silently reordered and every forward return is paired with the wrong neighbour. Parse first, e.g. pl.col("date").str.to_datetime("%m/%d/%Y").
• (date, asset_id) must be unique. Entry and exit prices are looked up by (asset_id, period_index). A duplicated key makes that lookup ambiguous and can fan out the joins, so factrix rejects it instead of guessing which price represents the asset-period.
• NaN and ±Inf become null. Applied to inputs, not to computed outputs. This makes the whole class structurally unrepresentable downstream rather than patching instances of it: polars ranks NaN as larger than every real value, pl.corr(method="spearman") silently ranks it, and a +Inf denominator makes finite / inf evaluate to 0.0 — a finite fabricated −100% return that an output-side is_finite() filter cannot catch.

The first two raise UserInputError; the third is a silent, deliberate normalization. _finite_expr remains in place inside the producers as defence in depth.

Forward return

compute_forward_return accepts forward_periods as a positive int. 0, negative values, floats, strings, and bool values raise UserInputError.

The horizon is measured on the panel's own period grid — the distinct sorted date values present in the panel, indexed 0, 1, 2, … Each asset's forward return pairs its row at period index i + 1 with its row at i + 1 + forward_periods. This used to be a positional shift within each asset, which equals a time horizon only on a complete per-asset panel: with one asset missing 20 periods mid-sample, a "5-period" return silently spanned 25 real periods, contaminating both the return and the overlap stamped in _overlap_periods that every downstream HAC inference reads. Suspensions, halts, delist-relist and staggered entry are ordinary in regional equity data, and sparse event panels are ragged by construction.

A ragged grid raises ragged_period_grid. The pairing is correct for every asset either way; the point of the warning is that an asset with a gap has no observation to pair at the exit period and so contributes fewer rows than a complete one. Reindex onto a common grid if the horizons must be comparable across names.

Rows whose computed return is not finite are dropped. If the horizon is too long for the panel, or price data leaves no finite forward returns after filtering, the function raises UserInputError instead of returning an empty panel.

The returned rows are the forward-return sample, not a complete price history. Event offsets and MFE/MAE windows may need the dropped tail even when their event row remains in the sample. Preserve the raw panel and pass it as evaluate(panel, price_data=raw, ...); see Full price data for event paths.

winsorize_forward_return clips forward_return by per-date quantiles. Its bounds must satisfy 0 <= lower <= upper <= 1; invalid ordering, out-of-range values, non-numeric values, and bool bounds raise UserInputError.

Evaluating on a coarser grid

A factor is often observed on a finer grid than it is traded on: the price grid supplies the return horizon, while the evaluation happens on a caller-chosen rebalance grid. Two separate quantities then live on the panel, and compute_forward_return stamps both:

Stamp	Name on EvaluationResult	Meaning
_forward_periods	forward_periods	The return horizon — the forward_periods the return was built with, in periods of the price grid. It names the hypothesis: the (factor, forward_periods, *params) identity that compare / bhy use, and the axis expand_over may name. Never changes with the evaluation grid.
_overlap_periods	overlap_periods	The overlap of adjacent observations on the evaluation grid — what inference consumes: the HAC bandwidth and effective degrees of freedom, the non-overlapping stride, and the stride-scaled sample floors. Bookkeeping only; it does not join the identity, because the same horizon evaluated on two grids is one hypothesis estimated twice.

On the full grid the two coincide. That is why sub-sampling a panel by hand after compute_forward_return goes wrong: the overlap_periods stamp still says "horizon", so a 60-period return evaluated every 60 periods is treated as 60-fold overlapping and the stride-scaled floor (50 × 60 periods for ic()) rejects a healthy 24-period panel with insufficient_ic_periods. The short-circuit message says so and points here.

Pass the evaluation grid as dates= instead. The return is still computed on the full grid at forward_periods, only rows on those dates are kept, and the overlap is derived on the full period index:

overlap_periods = 1 + max_i #{ j in dates : 0 < idx(j) − idx(i) < h }

The row at period i covers (i + 1, i + 1 + h], so two kept rows overlap exactly when they are fewer than h periods apart. Stride 60 at h = 60 gives 1; stride 20 gives 3; the full grid gives h.

import factrix as fx
from factrix.metrics import ic, quantile_spread
from factrix.preprocess import compute_forward_return

raw = fx.datasets.make_cs_panel(n_assets=30, n_dates=1500)      # fine grid
grid = raw["date"].unique().sort()
rebalance_dates = grid.gather_every(60)                        # every 60th period

panel = compute_forward_return(raw, forward_periods=60, dates=rebalance_dates)
result = fx.evaluate(
    panel, metrics={"ic": ic(), "spread": quantile_spread()}, factor_cols=["factor"]
)["factor"]

result.forward_periods   # 60 — the hypothesis
result.overlap_periods   # 1  — derived: adjacent evaluations share no future period
result.metrics["ic"].n_obs        # 24, the sampled periods (no stride applied)
result.metrics["spread"].n_obs    # 24

Three details of the derivation are deliberate:

• Every date must be on the grid; nothing is snapped. Snapping backward would read a factor observed after the entry period (look-ahead); snapping forward would silently move the entry period. A value off the grid raises UserInputError. A Date value names a Datetime period unambiguously and a Datetime in another time unit is the same instant; any other dtype mismatch is rejected rather than parsed.
• The maximum, not a typical count. The evaluation grid may be spaced unevenly on the period grid. Under-stating the overlap by one leaves dependence the non-overlapping stride does not remove and the stride t-test over-rejects (measured 8.3% at a nominal 5% on the test grid in tests/stats/test_uneven_grid_overlap_size.py, 21.7% with no stride); over-stating it by one only thins the strided sample (size intact, some power lost). The Newey-West path is insensitive either way (5.5%).
• Derived inside compute_forward_return, on the full index. The is_finite filter removes any period on which every asset's return is non-finite; an index rebuilt from the returned panel's own dates would compress those gaps and under-count the overlap — the dangerous direction.

The unit of forward_return does not change with the grid: it is the return per period of the horizon (the / forward_periods normalisation), not per evaluation period. Every p-value on the affected paths is scale-invariant, and a per-evaluation-period rate is not defined on an unevenly spaced grid.

evaluate_horizons(..., dates=) forwards the same grid to every horizon of a sweep; each result carries its own derived overlap_periods while the (factor, forward_periods) identity is unaffected.

factrix.preprocess.compute_forward_return

compute_forward_return(data: DataFrame, forward_periods: int = 5, *, dates: Series | Iterable[object] | None = None, overwrite: bool = False) -> DataFrame

Step 1: Compute per-period forward return per asset.

forward_return = (price[t+1+forward_periods] / price[t+1] - 1) / forward_periods

Entry at t+1 (next bar after density), exit at t+1+forward_periods.

WHY t+1 entry: The density at t is computed using data up to and including price[t]. Using price[t] as both density input and entry price assumes you can trade at the same price used to generate the density — unrealistic in practice. Entry at t+1 enforces a strict causal boundary: density → wait → trade → measure.

This also keeps the return window cleanly separated from the estimation window in event studies (BMP test), eliminating the need for ad-hoc shift corrections.

Dividing by forward_periods normalizes returns to a per-period basis, making different forward_periods directly comparable on a scale basis (see Notes for the scope boundary).

Parameters:

Name	Type	Description	Default
data	DataFrame	Must contain date, asset_id, price, with one row per (date, asset_id) and a temporal date column. Both are enforced (see Raises); neither used to be.	required
forward_periods	int	Holding horizon in periods of the panel's own grid — the distinct sorted date values present in the panel, not calendar time and not row position within an asset (default 5). On a daily panel this is 5 trading days; on a weekly panel, 5 weeks; on 1-min bars, 5 minutes. Which frequency those periods represent is the caller's responsibility.	5
dates	Series | Iterable[object] | None	Optional evaluation grid — the dates to keep, as a polars Series or a sequence. Every value must be a member of the panel's distinct-date grid (matched exactly, never snapped; see Raises). The return is still computed on the full grid at forward_periods; only rows on these dates are returned, and the overlap_periods stamp is derived from their spacing on the full period grid (see Notes) instead of being set to the horizon. None (default) keeps every date and stamps overlap_periods = forward_periods.	None
overwrite	bool	Allow recomputation when data already carries a forward_return column. False (default) raises rather than silently overwrite — the function is not idempotent: the previous call already dropped the last forward_periods + 1 rows per asset, so recomputing on the result drops a further tail. To change the horizon, recompute from the original (pre-forward-return) panel; overwrite=True recomputes in place anyway, accepting the additional truncation.	False

Raises:

Type	Description
UserInputError	forward_periods is not a positive int; data carries duplicate (date, asset_id) rows or a non-temporal date column; data already has a forward_return column and overwrite is False; the horizon / price data leaves no finite forward returns after filtering; or dates is empty, has a dtype the panel's date cannot match exactly, or contains a value that is not on the panel's distinct-date grid.

Returns:

Type	Description
DataFrame	Input DataFrame with forward_return column appended and two
DataFrame	reserved stamp columns — _forward_periods (the return horizon,
DataFrame	the hypothesis) and _overlap_periods (the overlap of adjacent
DataFrame	observations on the evaluation grid, the quantity inference consumes;
DataFrame	equal to the horizon unless dates was given). They are the
DataFrame	single source of truth factrix.evaluate reads (it strips both
DataFrame	before dispatch, so they never reach a metric or to_frame). Rows
DataFrame	where forward return is not finite (tail nulls, NaN, +inf, -inf) are
DataFrame	dropped; with dates, rows off the evaluation grid are dropped
DataFrame	too.

Notes

The horizon is measured on the panel's period grid. The distinct sorted date values in the panel are indexed 0, 1, 2, ..., and each asset's forward return pairs its row at period index i + 1 with its row at i + 1 + forward_periods. This used to be a positional shift within each asset, which equals a time horizon only on a complete per-asset panel: with asset A missing 20 periods mid-sample, a "5-period" return silently spanned 25 real periods, contaminating both the return and the overlap horizon stamped in _forward_periods that every downstream HAC inference reads. Suspensions, halts, delist-relist and staggered entry are ordinary in regional equity data, and sparse event panels are ragged by construction.

A ragged grid (an asset missing periods that others have) raises WarningCode.RAGGED_PERIOD_GRID: the pairing is now correct for every asset, but an asset with a gap simply has no observation at i + 1 + h, so it contributes fewer rows than a complete one. Reindex onto a common grid if the horizons must be comparable across names.

Evaluating on a coarser grid. A factor is often observed on a finer grid than it is traded on — the price grid supplies the return horizon, while the evaluation happens on a caller-chosen rebalance grid. Two separate quantities then matter. forward_periods is the economic horizon and names the hypothesis; it is the same on any grid. overlap_periods is how many adjacent evaluation observations share future periods, which is what the HAC bandwidth and effective degrees of freedom, the non-overlapping stride and the stride-scaled sample floors consume. On the full grid the two coincide, so a panel sub-sampled by hand after this function keeps a stale overlap_periods = forward_periods stamp: a 60-period return evaluated every 60 periods is treated as 60-fold overlapping and the stride-scaled floor rejects it outright. Pass the evaluation grid as dates instead and the overlap is derived here, on the full grid, as::

overlap_periods = 1 + max_i #{ j in dates : 0 < idx(j) - idx(i) < h }

with idx the period index on the panel's full distinct-date grid. The row at i covers (i + 1, i + 1 + h], so two kept rows overlap exactly when they are fewer than h periods apart. Stride 60 at h = 60 gives 1; stride 20 gives 3; the full grid gives h. The maximum, not a typical count, is used because the evaluation grid may be spaced unevenly on the period grid: under-stating the overlap by one leaves dependence the stride does not remove and the stride t-test over-rejects (12-13% at a nominal 5% in simulation), while over-stating it by one only thins the strided sample. The derivation must run here rather than on the returned panel: the is_finite filter below removes any period on which every asset's return is non-finite, so an index rebuilt from the output's own dates would compress the gaps and under-count.

The unit of forward_return does not change with the grid: it is the return per period of the horizon (the / forward_periods below), not per evaluation period. Every p-value on the affected paths is scale-invariant, and a per-evaluation-period rate is not defined on an unevenly spaced grid.

Non-finite prices are blanked before the division, not after. The filter used to be applied to the quotient: with a +Inf denominator, finite / inf is 0.0, so the result was a perfectly finite fabricated -100% return that sailed straight through is_finite(). (An inf numerator gives inf and was correctly dropped — the leak was asymmetric, and so easy to miss.)

The / forward_periods per-period normalization is a scale choice with three caveats the caller should know:

1. Arithmetic, not summed-log-return. This is the arithmetic per-period mean of a simple return, not the academic-standard direct long-horizon regression of summed log returns on the predictor (the latter is linear-additive across horizons by construction).
2. Compounding bias. Compounding at the arithmetic mean is an upward-biased estimator of cumulative wealth; the bias grows with forward_periods and per-bar return variance. Negligible for rank-based information coefficient (IC); not negligible for signed-return mean and t-tests at large forward_periods.
3. Scale, not inference. / forward_periods aligns the scale across horizons — it does not address the inference problem. Overlap is handled by heteroskedasticity-and-autocorrelation-consistent (HAC) (see :class:factrix.inference.NeweyWest); across-horizon selection is handled by a declared multiple-testing family in :func:factrix.multi_factor.bhy (BHY controls FDR, not FWER). The three concerns (scale, overlap, cross-horizon selection) are addressed at separate layers; overlap and across-horizon dependence share a common source in the persistent regressor, but each requires its own tool.

References

• Fama & French (1988). "Dividend Yields and Expected Stock Returns." Journal of Financial Economics, 22(1), 3–25. Direct summed-log-return long-horizon regression — the academic-standard alternative to factrix's ÷N.
• Jacquier, Kane & Marcus (2003). "Geometric or Arithmetic Mean: A Reconsideration." Financial Analysts Journal, 59(6), 46–53. Compounding bias of the arithmetic mean and the unbiased horizon-weighted blend.
• Boudoukh, Richardson & Whitelaw (2008). "The Myth of Long-Horizon Predictability." Review of Financial Studies, 21(4), 1577–1605. Documents that across-horizon regression statistics share information through the persistent regressor — separate from any per-period scaling choice, and the reason inference across horizons is not addressed by normalization.

Examples:

>>> import factrix as fx
>>> from factrix.preprocess import compute_forward_return
>>> raw = fx.datasets.make_cs_panel(n_assets=20, n_dates=120)
>>> panel = compute_forward_return(raw, forward_periods=5)
>>> "forward_return" in panel.columns
True
>>> panel["forward_return"].null_count() == 0
True

The output panel is the canonical input to fx.evaluate:

>>> from factrix.metrics import ic
>>> results = fx.evaluate(
...     panel, metrics={"ic": ic()}, factor_cols=["factor"], forward_periods=5
... )
>>> isinstance(results, dict) and "factor" in results
True

A 60-period return evaluated every 60 periods: the horizon stays 60 (the hypothesis), while the derived overlap is 1, so inference treats the sampled observations as non-overlapping:

>>> raw = fx.datasets.make_cs_panel(n_assets=30, n_dates=1500)
>>> grid = raw["date"].unique().sort()
>>> sampled = compute_forward_return(
...     raw, forward_periods=60, dates=grid.gather_every(60)
... )
>>> sampled["date"].n_unique()
24
>>> result = fx.evaluate(sampled, metrics={"ic": ic()}, factor_cols=["factor"])
>>> (result["factor"].forward_periods, result["factor"].overlap_periods)
(60, 1)

factrix.preprocess.winsorize_forward_return

winsorize_forward_return(data: DataFrame, lower: float = 0.01, upper: float = 0.99) -> DataFrame

Step 2: Per-date percentile clip on forward returns.

Parameters:

Name	Type	Description	Default
lower	float	Lower quantile bound (default 0.01 = 1st percentile). Must satisfy 0 <= lower <= upper <= 1.	0.01
upper	float	Upper quantile bound (default 0.99 = 99th percentile). Must satisfy 0 <= lower <= upper <= 1. Set to (0.0, 1.0) to disable.	0.99

Raises:

Type	Description
UserInputError	lower / upper are not numeric quantile bounds satisfying 0 <= lower <= upper <= 1.

Returns:

Type	Description
DataFrame	DataFrame with forward_return clipped in-place.

Notes

Quantile interpolation. The per-date bounds are computed with interpolation="linear" — the numpy / pandas / alphalens default — rather than polars' own default of "nearest". With "nearest" the bound snaps to an actually observed value, so on a small cross-section the winsorisation can be a complete no-op: for [0, 1, ..., 8, 100] at upper=0.95 the nearest-rank quantile is 100 itself and the outlier survives untouched, while the linear quantile is 58.6 and clips it. Linear interpolation makes the clip level a continuous function of the requested percentile, which is what a per-date winsoriser needs in order to behave consistently as the cross-section size changes.

Examples:

>>> import factrix as fx
>>> from factrix.preprocess import (
...     compute_forward_return,
...     winsorize_forward_return,
... )
>>> raw = fx.datasets.make_cs_panel(n_assets=20, n_dates=120)
>>> panel = compute_forward_return(raw, forward_periods=5)
>>> clipped = winsorize_forward_return(panel, lower=0.01, upper=0.99)
>>> clipped.height == panel.height
True
>>> clipped["forward_return"].max() <= panel["forward_return"].max()
True

factrix.preprocess.compute_abnormal_return

compute_abnormal_return(data: DataFrame) -> DataFrame

Step 3: Cross-sectional abnormal return.

abnormal_return = forward_return - mean(forward_return) per date

Returns:

Type	Description
DataFrame	DataFrame with abnormal_return column appended.

Examples:

>>> import factrix as fx
>>> from factrix.preprocess import (
...     compute_abnormal_return,
...     compute_forward_return,
... )
>>> raw = fx.datasets.make_cs_panel(n_assets=20, n_dates=120)
>>> panel = compute_forward_return(raw, forward_periods=5)
>>> adjusted = compute_abnormal_return(panel)
>>> "abnormal_return" in adjusted.columns
True

Factor normalization

Both normalizers scale by the median absolute deviation (MAD). Three places where factrix departs from the most common implementation, and why:

Finite-sample MAD constant. The standard scale is 1.4826 × MAD, where 1.4826 = 1/Φ⁻¹(0.75) is the asymptotic consistency constant — this is what R's mad() and scipy.stats.median_abs_deviation ship. factrix multiplies in the Croux-Rousseeuw (1992) finite-sample factor b_n as well (tabulated for n ≤ 9, n/(n−0.8) above), keyed on the per-date count of finite values, because this library targets cross-sections of 5–20 names where the asymptotic constant is materially biased. Measured on a standard-normal factor (true σ = 1), E[1.4826 × MAD] is 0.82 at n=5, 0.91 at n=10 and 0.96 at n=20, against 0.99–1.01 at every size with b_n. The bias matters twice over: the output of a function called "z-score" was not unit-scale, and because the bias is n-dependent, thin dates in an unbalanced panel (delistings, staggered entry, sparse event triggers) were handed systematically larger z-scores, so anything pooling or weighting by z over-weighted the noisiest cross-sections. Fewer than three finite values on a date leaves no estimable robust scale at all: the date comes back null rather than as a fabricated 0.0.

Centring. The textbook z-score subtracts the mean. cross_sectional_zscore defaults to center="median" to stay robust to the outliers the MAD scale is chosen for — with the consequence that the output is not mean-zero on a skewed factor, so w ∝ z carries a net long or short leg (measured: +0.32 net exposure on a 2-of-20 sparse trigger column). Pass center="mean" when the weights must be dollar-neutral by construction; the scale stays the robust MAD either way.

Regime switches are announced. When the per-date MAD collapses to zero (more than 50% ties at the median) the scale falls back to the non-robust per-date standard deviation. That is a data-driven change of estimator, so it raises a UserWarning carrying WarningCode.ZERO_MAD_STD_FALLBACK rather than mixing robust and non-robust dates into one column silently. On a sparse {0, R} trigger column mad_winsorize skips the clip entirely (SPARSE_WINSORIZE_SKIPPED): that column's standard deviation is produced by the triggers, so a 3-std band shrinks with the trigger rate and destroyed 58% of a unit event's magnitude at one trigger in fifty names.

Non-finite input. NaN and ±Inf are blanked to null on output by both functions, not clipped into the band. A non-finite tick is a data error, not an extreme value; winsorizing it produced a plausible finite number that survived downstream finite-observation filters and put that asset at the top of the date's ranking.

cross_sectional_zscore names its output column after its input: factor → factor_zscore, momentum → momentum_zscore, so several factors can be standardized in one panel without colliding.

factrix.preprocess.mad_winsorize

mad_winsorize(data: DataFrame, factor_col: str = 'factor', n_mad: float = 3.0, *, center: Center = 'median') -> DataFrame

Step 4: Per-date MAD-based winsorization on factor values.

Clips factor values to [centre +- n_mad * b_n * 1.4826 * MAD] within each cross-section, where b_n is the Croux-Rousseeuw (1992) finite-sample correction for the per-date finite count.

Parameters:

Name	Type	Description	Default
factor_col	str	Column to winsorize in place.	'factor'
n_mad	float	Number of MAD units for clipping (default 3.0). Must be a finite number >= 0; 0 disables the step.	3.0
center	Center	Where the clip band is centred — "median" (default, robust) or "mean". The MAD itself is always taken about the median.	'median'

Raises:

Type	Description
UserInputError	n_mad is not a finite number >= 0, or center is not "median" / "mean".

Returns:

Type	Description
DataFrame	DataFrame with factor_col clipped in-place. Non-finite inputs
DataFrame	come back null.

Notes

Small-sample MAD scaling. 1.4826 is the asymptotic consistency constant; at finite n it under-estimates sigma by 18% at n=5 and 9% at n=10, which makes n_mad=3.0 an effective 2.46 sigma clip on a five-name cross-section — over-winsorizing exactly the tail a factor's signal lives in. The Croux-Rousseeuw (1992) factors (tabulated for n <= 9, n/(n-0.8) above) restore n_mad to its nominal sigma meaning at every cross-section size, so an unbalanced panel is clipped consistently across thin and wide dates.

Zero-MAD dates. More than 50% ties on a date drives the MAD to exactly 0. Two sub-regimes are handled differently, and both raise a UserWarning:

• A sparse {0, R} trigger column (median 0, MAD 0) is left unwinsorized (WarningCode.SPARSE_WINSORIZE_SKIPPED). Its standard deviation is produced by the triggers, so 3 * std collapses as the trigger rate falls — at one trigger in fifty names a unit event was clipped to 0.42, destroying 58% of the magnitude the metric downstream is trying to measure.
• Any other zero-MAD date (a bucketed or binary factor) falls back to the per-date sample standard deviation (ddof=1), which keeps the factor finite and rank-preserving (WarningCode.ZERO_MAD_STD_FALLBACK).

A cross-section with no dispersion at all has scale = 0 and is left untouched by the clip. A date with fewer than MIN_SCALE_ASSETS_HARD finite values has no estimable robust scale and is left untouched too (WarningCode.INSUFFICIENT_SCALE_ASSETS).

Non-finite input. NaN / +-Inf values are excluded from the per-date statistics and blanked to null in the output, matching :func:cross_sectional_zscore. Clipping them into the band (the previous behaviour) turned a hard data error into a plausible finite extreme that survived every downstream finite-observation filter and put the asset at the top of that date's ranking.

References

• Croux, C. & Rousseeuw, P. J. (1992). "Time-Efficient Algorithms for Two Highly Robust Estimators of Scale." Finite-sample correction factors for the MAD.

Examples:

>>> import factrix as fx
>>> from factrix.preprocess import mad_winsorize
>>> raw = fx.datasets.make_cs_panel(n_assets=20, n_dates=120)
>>> clipped = mad_winsorize(raw, n_mad=3.0)
>>> clipped.columns == raw.columns
True
>>> clipped.height == raw.height
True

factrix.preprocess.cross_sectional_zscore

cross_sectional_zscore(data: DataFrame, factor_col: str = 'factor', *, center: Center = 'median') -> DataFrame

Step 5: MAD-robust z-score within each cross-section (date).

z = (x - centre(x)) / (b_n * 1.4826 * MAD(x))

Parameters:

Name	Type	Description	Default
factor_col	str	Column to standardize.	'factor'
center	Center	Where the score is centred. "median" (default) — robust to outliers, but not mean-zero. For any skewed factor (sparse event triggers, log-scaled fundamentals) mean(z) != 0, so weights w ~ z carry a persistent net long or short leg and the book is charged market beta the caller did not intend. On a 2-of-20 sparse trigger column the net exposure is +0.32. "mean" — subtracts the per-date mean instead, so the output is exactly mean-zero and w ~ z is dollar-neutral by construction, at the cost of a centre that a single outlier can move. The scale stays the robust MAD either way.	'median'

Returns:

Type	Description
DataFrame	DataFrame with a f"{factor_col}_zscore" column appended (so
DataFrame	standardizing several factors in one panel no longer collides on a
DataFrame	single hard-coded name).

Notes

Small-sample MAD scaling. The scale carries the Croux-Rousseeuw (1992) finite-sample factor b_n on top of the asymptotic 1.4826. Without it the output of a function called "z-score" is not unit-scale at small n — its expected cross-sectional standard deviation is 1.67 at n=5 and 1.18 at n=10 — and, worse, the inflation is n-dependent, so on an unbalanced panel thin dates produce systematically larger z and every pooled or z-weighted statistic over-weights the noisiest cross-sections. b_n removes the bias in the scale estimator (E[b_n · 1.4826 · MAD] = σ to within 1% at every n >= 3), which brings the expected sd(z) to 1.38 at n=5 and 1.08 at n=10; the residual excess is Jensen curvature in E[σ̂ / scale] and no fixed constant removes it. The constant targets the normal-consistent scale: on heavier tails it overstates the distribution's own MAD-implied scale (about +12% at n=3 on a t(5), under +1% by n=50). See :func:mad_winsorize.

Zero-MAD fallback. With more than 50% ties on a date the MAD is exactly 0, so the naive ratio is +-inf for every non-median value (and 0/0 = NaN at the median). We fall back to the per-date sample standard deviation (ddof=1) as the scale, which keeps the z finite and rank-preserving — and raise a UserWarning carrying WarningCode.ZERO_MAD_STD_FALLBACK, because robust and non-robust dates otherwise end up mixed into one column and one downstream statistic with no way to tell them apart.

Thin cross-sections. A date with fewer than MIN_SCALE_ASSETS_HARD finite values yields null, not 0.0. At n=1 the old chain fell through to 0.0 — an "average" fabricated from a single observation — and at n=2 it returned +-0.6745 whatever the two values were, a constant carrying no information but indistinguishable downstream from a real score. WarningCode.INSUFFICIENT_SCALE_ASSETS is raised. A fully constant cross-section of at least that size does have a genuine "no spread" reading and still maps to 0.0.

Missing values stay missing. Per the library convention, producers never impute and consumers retain finite observations. A null, NaN or infinite input yields a null z-score.

Examples:

>>> import factrix as fx
>>> from factrix.preprocess import cross_sectional_zscore
>>> raw = fx.datasets.make_cs_panel(n_assets=20, n_dates=120)
>>> standardized = cross_sectional_zscore(raw)
>>> "factor_zscore" in standardized.columns
True

Orthogonalization

orthogonalize_factor runs a per-date cross-sectional OLS and returns the residual. Two guards that the textbook formulation leaves to the analyst:

A degrees-of-freedom floor, not a solvability floor. The regression is only fitted on dates leaving at least min_residual_df residual degrees of freedom (n_assets − n_base − 1, default 10 — the N ≥ K + 10 form of the Fama-MacBeth convention). Raw R² is mechanically ≈ K/(N − 1) even when the true R² is zero, so on a six-name book regressed on four exposures the old one-df floor reported mean_r_squared = 0.79 (adjusted: −0.03) after stripping 83% of the factor's variance — the factor reads as redundant when it is orthogonal by construction. Dates below the floor keep their original values, are counted in n_dates_insufficient_df, and raise insufficient_regression_df. mean_adj_r_squared is reported alongside the raw figure. Pass min_residual_df=1 to restore the old behaviour.

Rank deficiency is detected, not assumed away. An intercept is always prepended, so a full industry dummy set is exactly singular. np.linalg.lstsq does not raise on that — it returns the minimum-norm solution — so the residual is still exact (the projection onto the column space is unique) but mean_betas was an arbitrary point in the solution space. Rank is read from the singular values lstsq already computes, at matrix_rank's tolerance so near-collinear designs are caught too; deficient dates are excluded from mean_betas and raise rank_deficient_design. Drop one dummy category as the reference level.

Residual scale. The residual's dispersion is sqrt(1 − R²) times the input's, and R² varies by date, so the output scale varies by date. Rank-based metrics are unaffected; any magnitude-based use is otherwise quietly on a time-varying scale. Pass restandardize=True to rescale each date's residual back to the input's per-date dispersion. Both dispersions use exactly the finite rows fitted by that date's regression; a row excluded for a non-finite factor or base value, or because it has no matching base row, affects neither scale estimate.

factrix.preprocess.orthogonalize_factor

orthogonalize_factor(factor_df: DataFrame, base_factors: DataFrame, factor_col: str = 'factor', base_cols: list[str] | None = None, *, min_residual_df: int = MIN_ORTHOGONALIZE_RESIDUAL_ASSETS, restandardize: bool = False) -> OrthogonalizeResult

Orthogonalize factor against base factors via per-date ordinary least squares (OLS).

Parameters:

Name	Type	Description	Default
factor_df	DataFrame	Panel with date, asset_id, {factor_col}. factor_col should already be z-scored (Step 5 output).	required
base_factors	DataFrame	Panel with date, asset_id and base factor columns. Industry dummies must be pre-encoded as 0/1 columns with one category dropped as the reference level: an intercept is always prepended, so a full dummy set is exactly singular (the classic dummy trap). See the rank-deficiency note below for what happens if it is not.	required
factor_col	str	Column name of the factor to orthogonalize.	'factor'
base_cols	list[str] | None	List of column names in base_factors to regress on. If None, uses all columns except date and asset_id.	None
min_residual_df	int	Minimum residual degrees of freedom (n_finite - len(base_cols) - 1) a date must leave to be regressed. Defaults to :data:~factrix._types.MIN_ORTHOGONALIZE_RESIDUAL_ASSETS (10), the N >= K + 10 form of the Fama-MacBeth convention; pass 4 * len(base_cols) for the N >= 5K form, or 1 to restore the old behaviour of fitting anything that is arithmetically solvable.	MIN_ORTHOGONALIZE_RESIDUAL_ASSETS
restandardize	bool	Rescale the residual to the per-date dispersion of the input factor on the same finite rows used to fit that date's regression. Rows excluded because the factor or any base value is non-finite, or because no matching base row exists, contribute to neither scale estimate. False (default) returns the raw residual, whose scale is sqrt(1 - R²) times the input — and since R² varies by date, so does the output scale. Rank metrics are unaffected; any magnitude-based use (weights w ~ f, a spread in factor units) is otherwise quietly running on a time-varying scale.	False

Returns:

Type	Description
OrthogonalizeResult	OrthogonalizeResult with: data (factor_df with factor_col
OrthogonalizeResult	replaced by the residual and factor_pre_ortho preserving the
OrthogonalizeResult	original value), mean_betas (average beta per base factor
OrthogonalizeResult	across full-rank dates), mean_r_squared and
OrthogonalizeResult	mean_adj_r_squared, plus the per-reason skip counts and
OrthogonalizeResult	warning_codes.

Raises:

Type	Description
ValueError	base_factors carries duplicate (date, asset_id) keys, which would fan the inner join out and duplicate rows.

Notes

Minimum residual degrees of freedom. A cross-sectional OLS needs residual df to mean anything. The old floor was len(base_cols) + 2 rows — a single residual df — and raw R² is mechanically ≈ K/(N - 1) even when the true R² is zero, so a regional book of six names regressed on size / value / momentum plus an industry dummy passed the guard, fitted noise, and came back with mean_r_squared = 0.79 (adjusted: -0.03) after removing 83% of the factor's variance — the user concludes the factor is redundant when it is orthogonal by construction. Dates below min_residual_df are now skipped, counted in n_dates_insufficient_df, and flagged with WarningCode.INSUFFICIENT_REGRESSION_DF. mean_adj_r_squared is reported alongside the raw figure for the same reason.

Rank deficiency. np.linalg.lstsq does not raise on a rank-deficient design — it returns the minimum-norm solution — so the except LinAlgError branch never fired for the dummy trap. The residual stays correct either way (the projection onto the column space is unique), but mean_betas was an arbitrary point in the solution space, reported without qualification: on a 4-dummy panel the full set gave {ind0: 0.008, ind1: -0.007, ...} and dropping one category gave {ind1: -0.015, ...}, with identical residuals. Rank is now read from the singular values lstsq already computes (the matrix_rank tolerance, so near-collinear designs are caught too, not only exactly singular ones); deficient dates are counted in n_dates_rank_deficient, excluded from mean_betas, and flagged with WarningCode.RANK_DEFICIENT_DESIGN.

Residual scale. See restandardize above; the default leaves the residual on its own per-date scale rather than the input's.

Non-finite rows. A single null / NaN / ±Inf in factor_col or in any base column used to make np.linalg.lstsq return all-NaN betas, which turned every asset on that date into NaN while the date still counted as orthogonalised. The regression now runs on the finite rows only. Rows that were not finite come back null, not with their original value: the residual is undefined for them, and keeping the raw value would mix two scales (raw factor and residual) inside one column — a silent, unrecoverable contamination. Their count is reported as n_rows_non_finite and logged.

Skipped dates. A date that leaves fewer than min_residual_df residual degrees of freedom (or one where lstsq raises) cannot support the regression; the original values are kept for that date and it is counted in n_dates_skipped. Such dates are excluded from coverage, mean_betas and both R² figures.

Duplicate keys. base_factors must be unique on (date, asset_id). A duplicated key silently fans the inner join out (panel height doubles, coverage exceeds 1.0), so it is rejected up front rather than de-duplicated with a guessed rule.

Examples:

>>> import factrix as fx
>>> import polars as pl
>>> from factrix.preprocess import (
...     cross_sectional_zscore,
...     orthogonalize_factor,
... )
>>> raw = fx.datasets.make_cs_panel(n_assets=20, n_dates=120)
>>> factor_df = cross_sectional_zscore(raw).select(
...     "date", "asset_id", pl.col("factor_zscore").alias("factor")
... )
>>> base = raw.with_columns(
...     pl.col("price").rank().over("date").alias("size")
... ).select("date", "asset_id", "size")
>>> result = orthogonalize_factor(factor_df, base, base_cols=["size"])
>>> "factor_pre_ortho" in result.data.columns
True
>>> isinstance(result.mean_r_squared, float)
True

factrix.preprocess.OrthogonalizeResult dataclass

OrthogonalizeResult(data: DataFrame, mean_betas: dict[str, float] = dict(), mean_r_squared: float = 0.0, mean_adj_r_squared: float = 0.0, n_dates: int = 0, coverage: float = 0.0, n_base: int = 0, n_rows_non_finite: int = 0, n_dates_skipped: int = 0, n_dates_insufficient_df: int = 0, n_dates_rank_deficient: int = 0, restandardized: bool = False, warning_codes: tuple[str, ...] = ())

Result of factor orthogonalization with attribution info.

Attributes:

Name	Type	Description
data	DataFrame	factor_df with factor_col replaced by the residual and factor_pre_ortho preserving the original value.
mean_betas	dict[str, float]	Average beta per base factor across regressed dates whose design matrix had full column rank. Empty when every regressed date was rank deficient — the betas are not identified there, so reporting an average of arbitrary minimum-norm solutions would be worse than reporting nothing (see n_dates_rank_deficient).
mean_r_squared	float	Average raw R² across regressed dates.
mean_adj_r_squared	float	Average degrees-of-freedom-adjusted R², 1 - (1 - R²)(N - 1)/(N - K - 1). Raw R² is mechanically ≈ K/(N - 1) even when the true R² is 0, so on a thin cross-section the adjusted figure is the one that answers "is the factor actually spanned by the base set?".
n_dates	int	Number of per-date cross-sections seen after the join.
coverage	float	Fraction of input rows that carry an actual residual — rows dropped by the join, rows on a skipped date, and rows with non-finite inputs are all excluded.
n_base	int	Number of base factor columns regressed on.
n_rows_non_finite	int	Rows whose factor or base values were null / NaN / ±Inf on an otherwise regressed date. These rows are excluded from the regression and their residual is null in data.
n_dates_skipped	int	Dates left un-orthogonalised, for any reason. Original factor values are kept for those dates.
n_dates_insufficient_df	int	Subset of n_dates_skipped skipped because the cross-section left fewer than min_residual_df residual degrees of freedom.
n_dates_rank_deficient	int	Dates that were regressed but whose design matrix was rank deficient. The residual is still exact (the projection is unique) but the betas are not identified, so those dates are excluded from mean_betas.
restandardized	bool	Whether the residual was rescaled to the per-date dispersion of the input factor.
warning_codes	tuple[str, ...]	:class:~factrix._codes.WarningCode values raised by this run, as strings — the structured twin of the UserWarning echoes, for callers that inspect the result rather than trap warnings.