Cardinality Control for Rate Limit Metrics

Rate limit metrics tempt you into the single worst label choice in observability: the API key. It answers every question you have during an incident, and it multiplies your time series by the number of customers — which is how a limiter’s instrumentation ends up costing more than the limiter. This page is the cardinality discipline behind the counters described in metrics and instrumentation.

The problem in concrete numbers

Start with a reasonable-looking counter:

text
ratelimit_decisions_total{key, route, outcome, plan, region}

With 200,000 active API keys, 40 route patterns, 2 outcomes, 4 plans, and 3 regions, the upper bound is 200,000 × 40 × 2 × 4 × 3 = 192 million series. Even at a 5% realised density that is nearly 10 million active series — enough to require a dedicated metrics cluster, and enough that a single query over a day will time out.

Remove the key label and bucket the identity into a class:

text
ratelimit_decisions_total{key_class, route, outcome, plan}

With 5 key classes: 5 × 40 × 2 × 4 = 1,600 series. Same dashboards, same alerts, four orders of magnitude less storage — and the per-key detail moves to logs and exemplars, where it belongs.

Series count with a per-key label versus a key class Labelling by API key multiplies the series count by the number of customers into the millions, while bucketing identities into a handful of classes keeps the same dashboards at a few thousand series. One label decides the size of your metrics bill label: key 200,000 x 40 x 2 x 4 x 3 192,000,000 possible series queries time out retention becomes a budget line label: key_class 5 x 40 x 2 x 4 1,600 series same dashboards and alerts per-key detail lives in logs metrics answer “is something wrong?”; logs and traces answer “for whom?” Route labels are the usual source of a surprise A label taken from the matched route template stays in the dozens, a label taken from the concrete request path grows with every identifier a client sends, and a normalised fallback bounds the unknown cases. Two ways to label a route, and a fallback matched template dozens of values stable over time safe to keep concrete path unbounded values grows with traffic the usual blow-up normalised fallback unknown routes bucketed bounded by construction worth adding a routing change can silently turn the safe form into the unsafe one

Label budget

Label Cardinality Keep? Notes
outcome 3 Yes allowed, limited, degraded — the core signal
key_class 4–6 Yes anonymous, free, pro, enterprise, internal
route 20–60 Yes, as patterns /v1/users/:id, never a concrete path
plan 3–6 Yes Only if it is not already implied by key_class
algorithm 2–4 Optional Useful during a migration, drop afterwards
region 2–5 Yes if multi-region Otherwise omit
key / account_id 10⁵–10⁶ No Use exemplars and logs
ip 10⁶+ No Never a metric label
user_agent 10⁴+ No Unbounded, attacker-controlled
path (raw) Unbounded No Query strings explode it
status_code 5–10 Yes Cheap and useful

The rule to apply mechanically: a label may only take values from a set you can enumerate at design time. Anything an attacker or a customer can invent is not a label.

Step-by-step: instrumenting affordably

python
# metrics.py — a closed label set, an exemplar for detail, and one place to enforce it.
from prometheus_client import Counter, Histogram

KEY_CLASSES = ("anonymous", "free", "pro", "enterprise", "internal")

decisions = Counter(
    "ratelimit_decisions_total",
    "Rate limit decisions by outcome",
    ["outcome", "key_class", "route", "plan"],      # all four are enumerable
)
wait = Histogram(
    "ratelimit_retry_after_seconds",
    "Advertised wait on rejection",
    ["key_class", "route"],
    buckets=(0.5, 1, 2, 5, 10, 30, 60, 300),
)

def classify(identity, account) -> str:
    """Map an unbounded identity to a bounded class. The whole trick is here."""
    if identity is None:
        return "anonymous"
    if account and account.internal:
        return "internal"
    plan = (account.plan if account else "free")
    return plan if plan in KEY_CLASSES else "free"

def record(outcome, identity, account, route_pattern, retry_after=None, trace_id=None):
    labels = {
        "outcome": outcome,                          # allowed | limited | degraded
        "key_class": classify(identity, account),
        "route": route_pattern,                      # "/v1/users/:id", never the real path
        "plan": (account.plan if account else "none"),
    }
    # The account id rides along as an exemplar: queryable from a graph, not a series.
    decisions.labels(**labels).inc(exemplar={"trace_id": trace_id or "", "account": str(account.id) if account else ""})
    if retry_after is not None:
        wait.labels(key_class=labels["key_class"], route=route_pattern).observe(retry_after)

    if outcome == "limited":
        # Full detail goes to logs, sampled, where cardinality costs nothing.
        log.info("rate_limited", extra={
            "account_id": getattr(account, "id", None),
            "identity_hash": hash_identity(identity),
            "route": route_pattern,
            "retry_after": retry_after,
            "trace_id": trace_id,
        })
yaml
# prometheus.yml — a safety net so an accidental label cannot land in storage.
scrape_configs:
  - job_name: api
    metric_relabel_configs:
      # Drop any limiter series that somehow carries a per-identity label.
      - source_labels: [__name__, key]
        regex: 'ratelimit_.*;.+'
        action: drop
      - source_labels: [__name__, ip]
        regex: 'ratelimit_.*;.+'
        action: drop
      # Guard against raw paths sneaking in as routes: patterns contain a colon or are static.
      - source_labels: [__name__, route]
        regex: 'ratelimit_.*;.*\?.*'
        action: drop
Which signal answers which question about a rate limit Metrics with bounded labels answer whether something is wrong and where, exemplars link a graph to a single trace, and sampled logs carry the per-customer detail that would be ruinous as a metric label. Three signals, three questions metrics is something wrong? bounded labels only cheap, always on exemplars show me one example a trace id per bucket graph to trace in one click logs which customer? full detail, sampled cardinality is free here the mistake is asking the metrics layer to answer the third question

Gotchas and edge cases

  • Routes taken from the request path. /v1/users/8134 and /v1/users/8135 are different label values. Use the router’s matched pattern; most frameworks expose it.
  • Anonymous traffic keyed by address. Attractive during an attack, ruinous in storage: an attacker rotating addresses mints series. Use a class label and put the address in logs.
  • Enum labels that grow. A reason label starting with three values tends to reach thirty as engineers add cases. Cap the set and map anything unexpected to other.
  • Per-key histograms. Histograms multiply by bucket count, so a per-key histogram is a per-key metric times ten. Keep histograms on the class dimension only.
  • Deleted labels that linger. Removing a label does not remove existing series until retention passes; expect the improvement to appear gradually.
  • Cardinality from error strings. An exception message as a label value is unbounded by construction. Map exceptions to a small set of codes first.
  • Forgetting the degraded outcome. allowed and limited are not exhaustive: a store outage produces neither. Add degraded or the fail-open path is invisible.

Verification and testing

bash
# 1. What is actually being stored? Count series per limiter metric.
curl -s 'http://prometheus:9090/api/v1/query' \
  --data-urlencode 'query=count by (__name__)({__name__=~"ratelimit_.*"})' | jq -r \
  '.data.result[] | "\(.metric.__name__): \(.value[1])"'
# ratelimit_decisions_total: 1584        <- matches the budget
# ratelimit_retry_after_seconds_bucket: 3200

# 2. Which label is driving the count? Find the widest one.
for l in outcome key_class route plan; do
  n=$(curl -s 'http://prometheus:9090/api/v1/query' \
    --data-urlencode "query=count(count by ($l) (ratelimit_decisions_total))" |
    jq -r '.data.result[0].value[1]')
  echo "$l: $n distinct values"
done
# route: 41   <- expected; a number in the thousands means raw paths are leaking in

# 3. Fail the build if a limiter metric declares a forbidden label.
grep -rn 'ratelimit_[a-z_]*' src/ --include='*.py' -A3 |
  grep -E '"(key|account_id|ip|user_agent|path)"' && \
  { echo "forbidden high-cardinality label"; exit 1; } || echo "label set OK"

Check two is the diagnostic worth keeping in a runbook: when the series count grows unexpectedly, the widest label is nearly always a route label that has started receiving concrete paths after a routing change.

Frequently Asked Questions

How do I find which customer is being rate limited without a key label?

From logs and traces. Log every rejection with the account identifier, sampled if volume demands it, and attach a trace exemplar to the metric so a spike on a graph links straight to one example request. Metrics tell you a spike exists; the log tells you whose it is.

What is a safe series budget for limiter metrics?

A few thousand for the whole limiter. Multiply your label cardinalities before shipping: outcomes times key classes times route patterns times plans should land in the low thousands. If it does not, one label is unbounded and needs to become a class.

Are exemplars a replacement for labels?

For the "show me an example" question, yes — an exemplar attaches a trace identifier to a bucket without creating a series. They are not aggregatable, so you cannot chart per-customer rejection rates from them; that is a logs or data-warehouse query, and it is the right place for it.

Should anonymous traffic be labelled by address?

No. Addresses are effectively unbounded and attacker-controlled, so labelling by them lets a botnet inflate your metrics bill as a side effect of an attack. Use an anonymous class in metrics and keep addresses in logs where high cardinality costs nothing.