High Cardinality: Why Your Metrics Bill Exploded

High cardinality means a label or field has a very large number of distinct values. In a time-series system, every unique combination of label values is a separate stored series, so adding one unbounded label such as a user id or request id multiplies your series count without limit. The result is memory pressure, slow queries and a bill that scales with distinct values rather than with traffic.

Key takeaways

• Every unique combination of label values is its own time series. Labels multiply, they do not add.
• High cardinality is desirable in a database index and destructive in a metric label. The word means the same thing; the consequence is inverted.
• The offenders are always unbounded identifiers: user id, request id, session id, email, full URL path, container id.
• Classic Prometheus histograms cost 14 series per label combination. Native histograms cost one, and have been stable since v3.8.0.
• Billing is by distinct series, not by traffic. A feature that adds no load can multiply your invoice.

Three meanings, and the one that costs money

Cardinality just means "how many distinct values". The word arrives in monitoring from mathematics, passes through databases, and reverses its moral sign on the way, which is why advice about it seems to contradict itself depending on who is talking.

The database sense is worth holding onto because it explains the trap. MySQL documents cardinality as an index statistic and states the benefit plainly: "The higher the cardinality, the greater the chance that MySQL uses the index when doing joins." An index on a column with two possible values is nearly useless; an index on user_id is excellent. An engineer carrying that instinct into a metrics pipeline will reach for user_id as a label and do real damage, because a time-series database is not filtering rows, it is creating one stored object per distinct combination.

How labels multiply

Prometheus states the cost model without euphemism in its own naming documentation: "Each labelset is an additional time series that has RAM, CPU, disk, and network costs." The important word is each. Labels do not add, they multiply, and the arithmetic runs away faster than intuition expects.

Take an ordinary HTTP request counter on a modest service:

http_requests_total{
  method   = "GET",      # 5 values:  GET POST PUT DELETE PATCH
  status   = "200",      # 12 values: the codes you actually return
  endpoint = "/v1/plans",# 40 values: your route table
  service  = "billing",  # 20 values: your services
  region   = "eu-west",  # 3 values:  your regions
}

5 x 12 x 40 x 20 x 3  =  144,000 time series

# from ONE metric, with no identifiers, nothing careless.

One hundred and forty-four thousand series from a single counter with five well-behaved labels. Now add the label somebody always wants:

# "Can we break it down per customer?"
+ customer_id = 500 values

144,000 x 500  =  72,000,000 time series

# The traffic has not changed. Not one extra request.
# The number of stored objects went up 500x.

This is the whole problem in two lines. The system is doing exactly the same work; the observability layer is now tracking seventy-two million distinct things. And customer_id is one of the tamer choices, because it is at least bounded by your customer count. A request_id is unbounded by construction: it grows forever, one new series per request, and never stops.

Prometheus makes the same point with its own exporter. Discussing whether node_exporter should expose per-user statistics, the documentation observes that "you would quickly reach a double digit number of millions with 10,000 users on 10,000 nodes. This is too much for the current implementation of Prometheus." The guidance that follows is blunter than most teams expect: "The vast majority of your metrics should have no labels."

Histograms multiply again

A classic Prometheus histogram is not one series. It is one series per bucket, plus an infinity bucket, plus a sum and a count. The Go client library's default buckets are {.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10}, so eleven buckets, plus +Inf, plus _sum, plus _count: fourteen series per label combination. Apply that to the 144,000 above and you get 2,016,000 series from one histogram.

That multiplier is the single most cited number in cardinality writing, and as of 2026 it is also the most out of date. Native histograms store the sum, the count and a dynamic set of buckets in a single sample, and they have been a stable feature since Prometheus v3.8.0. Fourteen series becomes one, with more bucket resolution rather than less. If you are still reading advice that treats the 14x multiplier as unavoidable, that advice predates the fix.

The labels that always cause it

Cardinality incidents are strikingly unoriginal. The offender is almost always an identifier that grows with usage rather than with your architecture.

The pattern behind every row: a label is an identity, not a value. If the thing you want to record varies per event, it is a value or an attribute, and it belongs in a log line or a trace span. Labels are for the small, fixed set of dimensions you slice by.

Prometheus publishes a rule of thumb for this, and it is stricter than most teams realise: keep the distinct values of a label "below 10" and, when they exceed 100, actively reconsider the design. Read it carefully though, because the limit is per metric, not per server. It is a bound on how many series one metric produces, not a budget for your whole estate.

What actually breaks

Cardinality problems rarely announce themselves as cardinality problems. They arrive as four separate-looking symptoms.

• Memory. Prometheus keeps the current block of incoming samples in memory and is very efficient per sample, storing "an average of only 1-2 bytes per sample". The per-sample cost is not the problem; the per-series overhead is, and it scales with distinct series rather than with data volume. Measure it on your own instance by watching prometheus_tsdb_head_series against process memory, because there is no published bytes-per-series figure to plan against.
• Query latency and failure. A query touching a large series set can trip --query.max-samples, which defaults to 50000000. The documentation is explicit that "queries will fail if they try to load more samples than this into memory". Dashboards start timing out at the two-minute default before anything else looks wrong.
• Ingestion rejection. Managed backends enforce hard caps. Mimir defaults -ingester.max-global-series-per-user to 150,000 in-memory series per tenant across the cluster before replication. The 144,000-series counter from the example above is 96% of that default, from one metric.
• The invoice. Covered below, and usually the symptom that gets noticed first.

What it costs, with real numbers

Every major vendor prices on distinct series or distinct metrics, which is why cardinality shows up on a finance dashboard before it shows up on an engineering one. These figures are from vendor documentation at the time of writing; pricing moves, so treat them as shape rather than as a quote.

Grafana Cloud

Billing is per thousand active series, where active means "a time series that has received new data points within the previous 20 minutes". On the Pro plan, 10,000 active series are included in the platform fee and additional series are charged per thousand. Running the 144,000-series example gives roughly $890 per month for a single counter, before anything else you monitor. One useful piece of mercy in the model: billing is at the 95th percentile, which "forgives the top five percent of usage time in each monthly billing period, which is roughly the top 36 hours", so a brief spike does not set the bill.

Datadog

Datadog bills custom metrics, counting ingested and indexed custom metrics as two separate allotments. The Pro plan includes 100 ingested and 100 indexed custom metrics per host, Enterprise includes 200 and 200, and ingested metrics above the allotment are charged at $0.10 per 100. Two details do real damage in practice. A HISTOGRAM or DISTRIBUTION counts as five custom metrics by default, and rising to ten once you enable percentiles. And Metrics without Limits does not reduce ingestion: "every metric datapoint your services send to Datadog counts toward ingestion, independent of Metrics without Limits configuration."

New Relic

New Relic behaves differently depending on which door you came through, and the difference matters because most readers of this article are on the second one. Through the Metric API, exceeding cardinality limits degrades rollups rather than rejecting data. Through Prometheus remote write, the documentation states a limit on unique Count and Summary time series per account per five minute interval and is unambiguous about the consequence: "Time series received above this limit are dropped. This limit is enforced prior to and in addition to standard metric limits." Note that New Relic's own pages state that limit inconsistently, in one place as a range and elsewhere as a flat figure, so confirm the number for your account rather than trusting either.

The structural point across all three: your bill tracks distinct values, not traffic. A feature flag that adds a label can multiply your invoice while serving exactly the same number of requests.

Finding it before finance does

Prometheus ships the diagnostics. The TSDB status endpoint at /api/v1/status/tsdb returns the top label names by distinct values and the series counts by metric name, which is usually enough to identify the offender in under a minute. On the command line:

# Top offenders straight from a running server
curl -s localhost:9090/api/v1/status/tsdb | jq .

# Analyse a block on disk. --limit defaults to 20.
promtool tsdb analyze /path/to/data --limit 20
promtool tsdb analyze /path/to/data --extended

# Lint metrics and optionally analyse cardinality
cat metrics.txt | promtool check metrics --extended

# Series count right now, the number to alert on
prometheus_tsdb_head_series

Set a scrape-level guard too. The scrape limits (sample_limit, label_limit, target_limit and friends) all default to 0, meaning unlimited. Note that target_limit carries an explicit "This is an experimental feature, this behaviour could change in the future" caveat. And sample_limit is a circuit breaker rather than a filter: crossing it fails the entire scrape, so you lose everything from that target rather than the excess. Set extra_scrape_metrics: true to expose scrape_sample_limit as a series and alert when a target approaches it, instead of finding out when the scrape dies.

Fixing it

In rough order of how much they return for the effort.

Do not emit it

Always the cheapest fix and always the one skipped. Before adding a label, ask what query needs it and whether anyone will group by it. Labels get added speculatively and then cost money forever.

Drop it at ingestion with relabelling

Prometheus is direct about the intent here. Target relabelling is "a powerful tool to dynamically rewrite the label set of a target before it gets scraped", while metric relabelling "is applied to samples as the last step before ingestion", and the docs name this exact use case: "One use for this is to exclude time series that are too expensive to ingest."

scrape_configs:
  - job_name: api
    metric_relabel_configs:
      # Remove one runaway label from every sample
      - regex: 'request_id|session_id'
        action: labeldrop

      # Drop an entire metric that is not worth its series count
      - source_labels: [__name__]
        regex: 'app_debug_.*'
        action: drop

      # Collapse ids in paths: /orders/1042 -> /orders/:id
      - source_labels: [endpoint]
        regex: '(/orders/)[0-9]+'
        target_label: endpoint
        replacement: '${1}:id'

# Shipping to Mimir, Thanos or Grafana Cloud? The control point is
# write_relabel_configs under remote_write, which runs on the way out.

Two cautions the documentation supplies. On labeldrop and labelkeep: "Care must be taken with labeldrop and labelkeep to ensure that metrics are still uniquely labeled once the labels are removed." Strip the label that distinguishes two series and they collide. And metric relabelling "does not apply to automatically generated timeseries such as up", which quietly defeats attempts to filter those.

Switch classic histograms to native histograms

Fourteen series to one, stable since v3.8.0, with better bucket resolution rather than worse. For anyone still on classic histograms this is the largest single reduction available and it costs no information.

Move the data to the right signal

Most cardinality problems are a signal-selection mistake. Per-request identity belongs in traces, per-event detail belongs in logs, and metrics should carry the small number of dimensions you aggregate by. Exemplars bridge the gap, letting a metric point at a specific trace without becoming a label: they require --enable-feature=exemplar-storage, and the buffer is sized in the config file under storage.exemplars with max_exemplars defaulting to 100000.

Set a cardinality limit in the SDK

OpenTelemetry specifies a cardinality limit as a stable part of the spec, with a default of 2000 when nothing is configured, confirmed in the Go and Java SDKs. Overflowing measurements are aggregated into a single series marked otel.metric.overflow, which turns a silent explosion into a visible signal you can alert on.

Aggregate away what nobody queries

Recording rules pre-aggregate expensive queries into cheaper series. Grafana Cloud automates the same idea with Adaptive Metrics, which recommends aggregating metrics nothing queries. Read its constraints before enabling it: Grafana states that "using aggregations in alerting or recording rules is unsupported behavior and can lead to unexpected issues", and that it "is not cost efficient to aggregate metrics of fewer than 100 time series".

How much is too much

There is no published number, and anyone offering one without qualification is guessing. The most-quoted figure comes from Brian Brazil, writing in 2019 about Prometheus 2.x: "A Prometheus 2.x can handle somewhere north of ten millions series over a time window, which is rather generous, but unwise label choices can eat that surprisingly quickly." That is expert judgement about a version two majors behind current, so use it as an order of magnitude and nothing more.

The number that actually governs you is whichever limit you meet first, and it is usually your backend's rather than Prometheus's. A default Mimir tenant caps at 150,000 in-memory series. A Grafana Cloud bill becomes a conversation somewhere well below a million. The useful practice is not finding the ceiling but watching the trend: alert on prometheus_tsdb_head_series growth, and treat a step change after a deploy as a defect, because that is exactly what it is.

Where this stops being our topic

Worth being straight about the boundary. Pulsetic is an external uptime and performance monitor. It is not a time-series database, it does not ingest your metrics, and nothing above is a description of a Pulsetic feature. If your Prometheus is falling over, the fixes are in your Prometheus.

The connection worth drawing is narrower and, we think, more useful. External checks answer "is it up, is it fast, from where" with a series count fixed by how many checks you configure, not by how many users you have. That property is the reason a lot of teams keep availability and response time on a separate, boring system from the one carrying their cardinality risk: when the metrics pipeline is the thing that broke, you want the answer to "is the site up" to come from somewhere else entirely.

That separation also survives the failure mode this article is about. A backend dropping series above a quota, or a query timing out on a dashboard, does not stop an external check from telling you the API is returning errors. It is the same argument as the one for keeping a status page off your own infrastructure, and the same reason service degradation is worth measuring from outside the system that is degrading.

See how Pulsetic's website monitoring catches this from the outside, across 15+ locations.