Quantile Measures

Quantile and interval scoring measures.

This module provides functions for: 1. Pinball cost function for quantile and expectile forecasts 2. Mean Interval Score (MIS) and related measures

References

  • Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359-378.

greybox.quantile_measures.mis(actual: ndarray, lower: ndarray, upper: ndarray, level: float = 0.95, na_rm: bool = True) → float[source]

Mean Interval Score (MIS).

The MIS evaluates the quality of prediction intervals. It rewards narrow intervals and penalizes misses (when actual is outside the interval).

Formula:
MIS = (upper - lower)

  • (2/alpha) * abs(lower - actual), for actual < lower

  • (2/alpha) * abs(actual - upper), for actual > upper

where alpha = 1 - level

Parameters:

  • actual (np.ndarray) – Actual (observed) values.

  • lower (np.ndarray) – Lower bound of prediction interval.

  • upper (np.ndarray) – Upper bound of prediction interval.

  • level (float, default=0.95) – Confidence level of the interval (e.g., 0.95 for 95% interval).

  • na_rm (bool, default=True) – Remove NA values.

Returns:

Mean Interval Score.

Return type:

float

Examples

>>> actual = np.array([1, 2, 3, 4, 5])
>>> lower = np.array([0.5, 1.5, 2.5, 3.5, 4.5])
>>> upper = np.array([1.5, 2.5, 3.5, 4.5, 5.5])
>>> mis(actual, lower, upper, level=0.95)

References

Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359-378.

greybox.quantile_measures.pinball(holdout: ndarray, forecast: ndarray, level: float, loss: int = 1, na_rm: bool = True) → float[source]

Pinball cost function.

The pinball function measures the quality of quantile or expectile forecasts. It is used in quantile regression and forecast accuracy evaluation.

For quantiles (loss=1):

pinball = (1-level) * sum(abs(e) * I(e <= 0)) + level * sum(abs(e) * I(e > 0))

where e = holdout - forecast.

For expectiles (loss=2):

Uses squared errors instead of absolute errors.

Parameters:

  • holdout (np.ndarray) – Actual values.

  • forecast (np.ndarray) – Forecasted values (quantile or expectile).

  • level (float) – The level associated with the forecast (e.g., 0.5 for median, 0.95 for 95th percentile).

  • loss (int, default=1) – Loss function type: - 1: L1 loss (for quantiles) - 2: L2 loss (for expectiles)

  • na_rm (bool, default=True) – Remove NA values.

Returns:

Pinball cost value.

Return type:

float

Examples

>>> holdout = np.array([1, 2, 3, 4, 5])
>>> forecast = np.array([1.1, 2.0, 3.2, 3.9, 5.1])
>>> pinball(holdout, forecast, level=0.5)  # Median pinball
>>> pinball(holdout, forecast, level=0.975)  # Upper quantile
>>> pinball(holdout, forecast, level=0.025)  # Lower quantile

References

Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359-378.

greybox.quantile_measures.rmis(actual: ndarray, lower: ndarray, upper: ndarray, benchmark_lower: ndarray, benchmark_upper: ndarray, level: float = 0.95, na_rm: bool = True) → float[source]

Relative Mean Interval Score (rMIS).

The rMIS compares the MIS of a forecast to a benchmark forecast.

Formula:

rMIS = MIS(forecast) / MIS(benchmark)

Parameters:

  • actual (np.ndarray) – Actual (observed) values.

  • lower (np.ndarray) – Lower bound of prediction interval.

  • upper (np.ndarray) – Upper bound of prediction interval.

  • benchmark_lower (np.ndarray) – Lower bound of benchmark prediction interval.

  • benchmark_upper (np.ndarray) – Upper bound of benchmark prediction interval.

  • level (float, default=0.95) – Confidence level of the interval.

  • na_rm (bool, default=True) – Remove NA values.

Returns:

Relative Mean Interval Score.

Return type:

float

Examples

>>> actual = np.array([1, 2, 3, 4, 5])
>>> lower = np.array([0.5, 1.5, 2.5, 3.5, 4.5])
>>> upper = np.array([1.5, 2.5, 3.5, 4.5, 5.5])
>>> benchmark_lower = np.array([0.0, 1.0, 2.0, 3.0, 4.0])
>>> benchmark_upper = np.array([2.0, 3.0, 4.0, 5.0, 6.0])
>>> rmis(actual, lower, upper, benchmark_lower, benchmark_upper)

References

See Gneiting & Raftery (2007).

greybox.quantile_measures.smis(actual: ndarray, lower: ndarray, upper: ndarray, scale: float | floating, level: float = 0.95, na_rm: bool = True) → float[source]

Scaled Mean Interval Score (sMIS).

The sMIS scales the MIS by a scale parameter.

Formula:

sMIS = MIS / scale

Parameters:

  • actual (np.ndarray) – Actual (observed) values.

  • lower (np.ndarray) – Lower bound of prediction interval.

  • upper (np.ndarray) – Upper bound of prediction interval.

  • scale (float) – Scale parameter.

  • level (float, default=0.95) – Confidence level of the interval.

  • na_rm (bool, default=True) – Remove NA values.

Returns:

Scaled Mean Interval Score.

Return type:

float

Examples

>>> actual = np.array([1, 2, 3, 4, 5])
>>> lower = np.array([0.5, 1.5, 2.5, 3.5, 4.5])
>>> upper = np.array([1.5, 2.5, 3.5, 4.5, 5.5])
>>> smis(actual, lower, upper, scale=3.0)

References

See Gneiting & Raftery (2007).