stormlog.infer.quantiles

Latency quantiles, how much data they need, and what failures do to them.

A quantile from n i.i.d. observations has a distribution-free confidence interval made of two order statistics (Le Boudec, Performance Evaluation of Computer and Communication Systems, Theorem 2.1): [X(j), X(k)] covers the p-quantile with probability B(k-1) - B(j-1), where B is the Binomial(n, p) distribution function. The ranks come from n and p alone, never from the data.

sufficient is the project-wide rule: the equal-tailed interval (each tail at most alpha/2) exists with at least margin order statistics above its upper rank, 5 by default. It certifies that statement and nothing more: not a precision in milliseconds, and not coverage under the dependence queueing creates between requests. n_min_exists is the smallest n for which any interval exists at all (Le Boudec’s own tables: 6 for the median, 59 for the 95th percentile).

Two estimands describe a case’s latency:

  • successful: the latency of requests that succeeded.

  • failure_penalized: every offered request, with each one that did not succeed ranked worst. That is a policy penalty, not an observed latency. The p-quantile is the successful values’ quantile at level p / (1 - f), where f is the share that did not succeed; above level 1 it falls in the failure mass and has no value. That is the estimand’s definition, not an identity with interpolating over a sample padded with infinite values: it stays finite at level 1, and otherwise differs from that by less than one gap between order statistics. When a successful request has no value the level cannot be computed, and the estimate has none.

Functions

penalized_quantile(successful, failures, p)

A quantile over every offered request, failures ranked worst.

quantile(values, p)

The p-quantile by linear interpolation between order statistics.

quantile_interval(values, p[, rule, penalized])

The rule's order-statistic interval, or None when it does not exist.

quantile_minimum_n(p[, confidence, margin, ...])

The smallest n for which the rule's interval exists.

successful_quantile(values, p[, rule])

A quantile of the successful requests' latencies.

Classes

OrderStatisticInterval(lower_rank, ...)

[X(lower_rank), X(upper_rank)] with 1-based ranks.

QuantileEstimate(estimand, p, value_ms, n, ...)

One quantile of one latency metric, with how far it can be trusted.

SufficiencyRule([confidence, margin, tails])

Which order-statistic interval must exist for a quantile to count.
class stormlog.infer.quantiles.OrderStatisticInterval(lower_rank, upper_rank, lower_ms, upper_ms, coverage)[source]

Bases: object

[X(lower_rank), X(upper_rank)] with 1-based ranks.

coverage is what the ranks achieve for i.i.d. continuous data. upper_ms is None when the upper rank falls among penalized failures.

Parameters:

  • lower_rank (int)

  • upper_rank (int)

  • lower_ms (float)

  • upper_ms (float | None)

  • coverage (float)

lower_rank: int
upper_rank: int
lower_ms: float
upper_ms: float | None
coverage: float
property width_ms: float | None
class stormlog.infer.quantiles.QuantileEstimate(estimand, p, value_ms, n, sufficient, n_min, n_min_exists, interval, penalized=False, observed_lower_bound_ms=None, reason=None)[source]

Bases: object

One quantile of one latency metric, with how far it can be trusted.

Parameters:

  • estimand (Literal['successful', 'failure_penalized'])

  • p (float)

  • value_ms (float | None)

  • n (int)

  • sufficient (bool)

  • n_min (int)

  • n_min_exists (int)

  • interval (OrderStatisticInterval | None)

  • penalized (bool)

  • observed_lower_bound_ms (float | None)

  • reason (str | None)

estimand: Literal['successful', 'failure_penalized']
p: float
value_ms: float | None
n: int
sufficient: bool
n_min: int
n_min_exists: int
interval: OrderStatisticInterval | None
penalized: bool = False
observed_lower_bound_ms: float | None = None
reason: str | None = None
class stormlog.infer.quantiles.SufficiencyRule(confidence=0.95, margin=5, tails='symmetric')[source]

Bases: object

Which order-statistic interval must exist for a quantile to count.

Parameters:

  • confidence (float)

  • margin (int)

  • tails (Literal['symmetric', 'narrowest'])

confidence: float = 0.95
margin: int = 5
tails: Literal['symmetric', 'narrowest'] = 'symmetric'
to_record()[source]
Return type:

dict[str, object]

stormlog.infer.quantiles.penalized_quantile(successful, failures, p, rule=SufficiencyRule(confidence=0.95, margin=5, tails='symmetric'), *, timeout_elapsed_ms=None, successful_missing=0)[source]

A quantile over every offered request, failures ranked worst.

successful_missing counts successful requests with no value. They belong to the offered count but cannot be ranked, so the estimate is then undefined (reason successful_values_missing) rather than taken over a smaller cohort that inflates the share that failed.

timeout_elapsed_ms gives the elapsed time of each failure when every failure was a timeout; only then does a quantile in the failure mass get an observed lower bound: the quantile had each timeout ended when it was abandoned. A request cancelled after 1 ms is no evidence of a 60 s latency.

Parameters:

  • successful (Sequence[float])

  • failures (int)

  • p (float)

  • rule (SufficiencyRule)

  • timeout_elapsed_ms (Sequence[float] | None)

  • successful_missing (int)

Return type:

QuantileEstimate

stormlog.infer.quantiles.quantile(values, p)[source]

The p-quantile by linear interpolation between order statistics.

The same rule as report_stats.percentile, for any level in [0, 1].

Parameters:

  • values (Sequence[float])

  • p (float)

Return type:

float | None

stormlog.infer.quantiles.quantile_interval(values, p, rule=SufficiencyRule(confidence=0.95, margin=5, tails='symmetric'), *, penalized=0)[source]

The rule’s order-statistic interval, or None when it does not exist.

penalized failures rank above every value, so an upper rank among them leaves the upper bound unknown.

Parameters:

  • values (Sequence[float])

  • p (float)

  • rule (SufficiencyRule)

  • penalized (int)

Return type:

OrderStatisticInterval | None

stormlog.infer.quantiles.quantile_minimum_n(p, confidence=0.95, *, margin=5, tails='symmetric')[source]

The smallest n for which the rule’s interval exists.

Symmetric, margin 5 (the default): 20, 230, 1,164 and 11,665 for p50, p95, p99 and p99.9. Narrowest, margin 0: Le Boudec’s 6, 59, 299, 2,995.

Parameters:

  • p (float)

  • confidence (float)

  • margin (int)

  • tails (Literal['symmetric', 'narrowest'])

Return type:

int

stormlog.infer.quantiles.successful_quantile(values, p, rule=SufficiencyRule(confidence=0.95, margin=5, tails='symmetric'))[source]

A quantile of the successful requests’ latencies.

Parameters:
Return type:

QuantileEstimate