factrix.metrics.directional_hit_rate ¶

Directional hit rate — small-N robust sibling of positive_rate.

Notes

Pipeline. Pool sign agreement between predicted (factor) and realised (forward_return) direction over a non-overlapping subsample; Pesaran-Timmermann (1992) market-timing test against H₀: predicted and realised direction are independent.

Input. DataFrame with date, asset_id, factor, forward_return.

The small-N robust counterpart of :func:~factrix.metrics.positive_rate.positive_rate (and a directional sibling of :func:~factrix.metrics.event_quality.event_hit_rate). positive_rate runs a naive two-sided binomial against p = 0.5 on a single pre-aggregated per-date series, implicitly assuming each call is an independent Bernoulli draw with a fixed 0.5 success rate. The Pesaran-Timmermann test instead conditions on the marginal up/down frequencies of both the prediction and the realisation, so a factor that is simply long a persistently-rising market is not credited with skill. This makes it the appropriate directional test for small, sign-imbalanced samples (the n_assets < 30 allocation regime). The headline value remains a hit rate: the fraction of correctly signed calls.

factrix.metrics.directional_hit_rate.directional_hit_rate ¶

directional_hit_rate(data: DataFrame, forward_periods: int = 5, factor_col: str = 'factor', return_col: str = 'forward_return') -> MetricResult

Directional hit rate via the Pesaran-Timmermann (1992) test.

Proportion of observations whose predicted direction sign(factor) matches the realised direction sign(forward_return), tested for predictive ability against the PT independence null.

Parameters:

Name	Type	Description	Default
`data`	`DataFrame`	Panel with `date, asset_id`, `factor_col` and `return_col`. Pooled across all `(date, asset)` observations on the non-overlapping subsample.	required
`forward_periods`	`int`	Sampling stride for non-overlapping dates; match the forward-return horizon so overlapping windows do not inflate the test.	`5`
`factor_col`	`str`	Prediction column (default `"factor"`).	`'factor'`
`return_col`	`str`	Realisation column (default `"forward_return"`).	`'forward_return'`

Returns:

Type	Description
`MetricResult`	MetricResult with value = directional hit rate `P̂` (0.0-1.0),
`MetricResult`	`stat` = the PT statistic `S_n` (deflated for within-date
`MetricResult`	cross-sectional correlation on a panel — see Notes), one-sided
`MetricResult`	p-value. `metadata["kolari_pynnonen_applied"]` records whether
`MetricResult`	the deflation fired and `stat_uncorrected` carries the raw
`MetricResult`	`S_n` when it did.

Notes

On the non-overlapping subsample, drop observations where either sign is zero (direction undefined), leaving \(n\) pooled pairs. With \(\hat P_x = \Pr(\text{factor} > 0)\), \(\hat P_y = \Pr(\text{return} > 0)\) and the realised hit rate \(\hat P = \frac1n \sum_t \mathbb{1}[\operatorname{sign}(x_t) = \operatorname{sign}(y_t)]\), the rate expected under independence is

\[P_* = \hat P_x \hat P_y + (1 - \hat P_x)(1 - \hat P_y).\]

The Pesaran-Timmermann statistic

\[S_n = \frac{\hat P - P_*}{\sqrt{\widehat{\operatorname{var}}(\hat P) - \widehat{\operatorname{var}}(P_*)}}\]

is asymptotically \(N(0, 1)\) under \(H_0\), with \(\widehat{\operatorname{var}}(\hat P) = P_*(1 - P_*)/n\) and \(\widehat{\operatorname{var}}(P_*) = \frac1n (2\hat P_y - 1)^2 \hat P_x (1 - \hat P_x) + \frac1n (2\hat P_x - 1)^2 \hat P_y (1 - \hat P_y) + \frac{4}{n^2} \hat P_x \hat P_y (1 - \hat P_x) (1 - \hat P_y)\).

Cross-sectional correlation. Pooling every (date, asset) trial as independent over-states the effective sample on a panel — same-date returns share shocks, so the trials are correlated and \(\widehat{\operatorname{var}}(\hat P)\) understates the true variance. \(S_n\) is therefore deflated by the Kolari-Pynnönen (2010) factor \(\sqrt{(1 - \hat r)/(1 + (\bar m - 1)\hat r)}\), where \(\hat r\) is the within-date intraclass correlation of the sign-hit indicator and \(\bar m\) the mean assets-per-date — the same correction :func:~factrix.metrics.caar.bmp_z applies to clustered SARs. A single-asset series has one trial per date, so \(\hat r\) is undefined and \(S_n\) is left exact (the canonical PT setting).

The test is one-sided: a large positive \(S_n\) signals genuine directional skill, so \(p = \Pr(Z > S_n)\). A factor expected to be negatively related to forward returns scores poorly here — flip its sign before testing. Degenerate samples (all predictions or all realisations one-signed, or a non-positive variance estimate) short-circuit: \(P_*\) is then 1 and the statistic is undefined. This is why return_col must be sign-symmetric around zero — a risk-adjusted return (e.g. forward_return / forward_realized_vol) qualifies, but an always-positive magnitude target (realised volatility, turnover) collapses P_* to 1 and hits this short-circuit; use :func:~factrix.metrics.ic.ic or :func:~factrix.metrics.monotonicity.monotonicity for those instead.

The sample floor is on the pairs axis — the \(n\) pooled (date, asset) directional trials, not the period count. Below MIN_DIRECTIONAL_PAIRS_HARD the metric short-circuits; between the HARD and MIN_DIRECTIONAL_PAIRS_WARN floors it returns the hit rate but flags WarningCode.FEW_DIRECTIONAL_PAIRS because the normal approximation to \(S_n\) is power-thin on few pooled trials.

References

Pesaran, M. H., & Timmermann, A. (1992). A simple nonparametric test of predictive performance. Journal of Business & Economic Statistics, 10(4), 461-465.

Examples:

>>> import factrix as fx
>>> from factrix.preprocess import compute_forward_return
>>> from factrix.metrics.directional_hit_rate import directional_hit_rate
>>> panel = compute_forward_return(
...     fx.datasets.make_cs_panel(n_assets=80, n_dates=180, seed=0),
...     forward_periods=5,
... )
>>> result = directional_hit_rate(panel, forward_periods=5)
>>> result.name == ""
True

Use cases¶

Small-N robust sibling of positive_rate

directional_hit_rate is the (*, DENSE, *) directional counterpart of positive_rate. Where positive_rate runs a naive binomial against \(p = 0.5\) on a single (date, value) series, this metric consumes the raw panel (date, asset_id, factor, forward_return) and tests whether sign(factor) predicts sign(forward_return). The headline value is itself a hit rate — the fraction of correctly-signed calls.
Pesaran-Timmermann conditions on the marginals

The Pesaran-Timmermann (1992) statistic compares the realised hit rate to the rate expected under directional independence, derived from the marginal up/down frequencies of both the prediction and the realisation. A factor that is simply long a persistently-rising market is therefore not credited with skill — the correct framing for small, sign-imbalanced samples (the n_assets < 30 allocation regime).
One-sided, non-overlap sampled

The test is one-sided: a large positive \(S_n\) signals genuine directional skill, so a sign-inverted predictor scores poorly (flip its sign before testing). Observations are sub-sampled at stride forward_periods so overlapping forward-return windows do not inflate the statistic; degenerate samples (one-signed predictions or realisations) short-circuit to NaN. inspect_data() also blocks obviously one-sided factor signs up front, so bulk discovery does not advertise directional_hit_rate as clean when runtime would hit degenerate_directional_variance.

Choosing a function¶

Goal	Function
Sign-significance of a `(date, value)` series vs \(p = 0.5\)	`positive_rate`
Directional skill of a factor in small, sign-imbalanced samples	`directional_hit_rate`

Worked example — directional skill on a panel¶

factor → directional_hit_rate

import factrix as fx
from factrix.metrics.directional_hit_rate import directional_hit_rate
from factrix.preprocess import compute_forward_return

raw   = fx.datasets.make_cs_panel(
    n_assets=25, n_dates=120, ic_target=0.06, seed=2024,
)
panel = compute_forward_return(raw, forward_periods=5)

out = directional_hit_rate(panel, forward_periods=5)
print(out.value, out.stat, out.p_value, out.metadata["method"])
# 0.57  2.74  0.0031  Pesaran-Timmermann (1992)   (approximate)

factrix.metrics.directional_hit_rate ¶

factrix.metrics.directional_hit_rate.directional_hit_rate ¶

Use cases¶

Choosing a function¶

Worked example — directional skill on a panel¶

See also¶