Slowdowns over time

Not every slowdown comes from a rollout. A query slows down as a table grows, or a dependency gets slower over a few days. Hud compares every endpoint, queue and custom flow with its own normal over days and weeks, and opens an issue when one drifts well past it.

The rules

RuleComparesChecksApplies to
1 Week DegradationThe past week with the week beforeEvery hourEndpoints, queues and custom flows
1 Day DegradationThe past day with the week before itEvery hourEndpoints, queues and custom flows
Latency SpikeThe last 10 minutes with the day beforeEvery 10 minutesEndpoints

Each rule compares P90 duration. Functions in the same flow that slowed down the same way are listed on the issue, so you can see where the slowdown started.

Defaults

By default, Hud opens an issue when an endpoint, queue or custom flow meets all three settings for its rule:

RuleSlower by at leastCallsP90 durationNext issue for the same endpoint
1 Week Degradation70%At least 350At least 150 msAfter 7 days
1 Day Degradation100%At least 100At least 150 msAfter 7 days
Latency Spike70%At least 50At least 150 msAfter 7 days

Latency Spike starts turned off, so turn it on from the Rules page if you want it. While an issue keeps recurring, Hud alerts on it again every 3 days by default. To change any of these for a service or an endpoint, open the rule on the Rules page. Detection conditions decide when Hud opens an issue, and alert conditions decide when it alerts you.

Slower than a target you set

To flag an endpoint whenever it goes over a fixed duration, set an SLO instead. See SLO rules.


Did this page help you?