Prometheus labels describe dimensions of a metric, such as an operation type or a service instance. They make aggregation and filtering powerful, but each distinct combination of label values creates a separate time series. Unbounded values can turn a useful metric into an expensive and difficult-to-query collection of individual events.
Good instrumentation begins with the question an operator needs to answer. This guide explains how to choose useful dimensions without encoding user identities, arbitrary URLs, or confidential information into a monitoring system that was not designed to be a general event archive.
Start with an operational question
Define what the metric should reveal: request volume by route, error ratio by service, queue depth, or task duration. Choose an appropriate metric type and unit before adding labels. A metric that mixes durations and counts cannot become meaningful through more detailed dimensions.
Use stable naming conventions with recognizable units and quantity meaning. Prometheus documents conventions such as base units and clear suffixes. Follow the relevant client library’s behavior for counters and histogram families rather than inventing names that conflict with its generated series.
Write a sample query during the design review. If no one can explain how the proposed labels contribute to an action, they may be unnecessary. Metrics should support decisions, not merely preserve every detail the application can observe.
Calculate the combination space
The cost comes from combinations, not just the number of label names. A metric with several bounded dimensions can still produce many series when their values multiply across service instances. Count expected combinations and include infrastructure labels added during collection.
For example, route, method, outcome, and instance dimensions have a combined footprint. A histogram adds further series associated with its representation. Estimate the complete metric family rather than reviewing one visible sample and assuming that is the total storage cost.
Not every theoretical combination necessarily appears, but churn matters too. New ephemeral values can create series over time even if few exist simultaneously. Review deployment behavior, retention, and how identifiers change during restarts or scaling.
Avoid unbounded identifiers
User IDs, email addresses, request IDs, session identifiers, and arbitrary object IDs usually make poor metric labels. Their value spaces grow with activity and can expose private data. Even a hashed identifier can retain unbounded cardinality and may remain linkable to a person or account.
Use logs or traces with appropriate controls for event-level correlation, rather than putting every identifier into a metric. Metrics answer aggregate questions; event stores can preserve specific occurrences under a different access and retention model.
Do not place secrets or entire exception messages in labels. A failed authentication value or raw upstream error can leak information to anyone with metrics access. Choose controlled outcome categories and keep detailed diagnostic data in a protected channel when it is actually needed.
Normalize route labels deliberately
A raw URL path can contain resource IDs, names, or other unlimited values. Prefer the application’s recognized route template, such as a route pattern with an object placeholder, when the framework supports it. This preserves useful route-level visibility without creating one series per object.
Handle unmatched routes with a bounded category. Attackers or ordinary clients can send arbitrary paths, so a fallback that logs the raw path as a label defeats the normalization design. Decide how redirects, errors, and requests rejected before routing are classified.
Review query strings separately. They can contain search terms, tokens, or tracking data. They rarely belong in metric dimensions. If the product needs a breakdown by a query option, map approved values into a controlled set rather than recording the entire string.
Keep label meaning stable
Use consistent label names and categories across services where comparison is meaningful. Renaming a value or changing its interpretation can break dashboards and alerts even when the metric continues to scrape successfully. Treat schema changes as operational changes with a migration plan.
Do not use one metric family for fundamentally different quantities distinguished only by a label. Aggregation should remain sensible when dimensions are removed. Mixing unrelated meanings makes sums and averages misleading, which can turn a correct query engine into an incorrect dashboard.
Document whether categories represent attempts, completed operations, or current states. A success label applied before a downstream commit tells a different story from one applied after confirmed completion. Define the boundary that matches the operator’s question.
Validate series growth in a test environment
Exercise normal routes, error paths, and arbitrary synthetic inputs. Inspect exported metrics and verify that labels remain within the approved sets. A route-normalization test should include unmatched paths and identifiers that differ only in their resource value.
Observe series counts and monitoring resource use under representative traffic. Compare before and after instrumentation changes. A small application benchmark does not establish the monitoring impact of a large deployment with many instances and retention periods.
Check privacy at the same time. Search controlled exports for synthetic secrets or personal-data markers used in tests. The result should show the appropriate bounded outcome, not the supplied sensitive string. Avoid using real customer values to perform this verification.
Plan alerts and fallback behavior
Add monitoring for unexpected series growth and scrape problems where appropriate. Alert ownership should include the application team responsible for instrumentation, not only the platform team that stores the data. Removing a bad label at its source is often more effective than scaling storage indefinitely.
If a rollout causes excessive growth, have a tested rollback or instrumentation disable path. Collection-side filtering can be an emergency measure, but it may also remove data required by alerts. Review downstream queries and record the temporary visibility loss.
A practical example is an API metric labeled with full request paths. Traffic against many object IDs creates a growing set of series. Replacing the label with the approved route template should reduce growth while preserving route-level latency and error analysis. Verify both the series footprint and dashboard meaning after the change.
Frequently asked questions
Are labels themselves a problem?
No. Bounded dimensions are central to useful Prometheus metrics. The risk is uncontrolled combinations, unstable meaning, and information that does not belong in metrics.
Does hashing user IDs fix cardinality?
No. It changes the representation but not the number of distinct values. It also does not automatically remove privacy concerns.
Where should I check naming guidance?
Read the Prometheus metric and label naming guidance. For keeping request-level diagnostics in a separately reviewed telemetry path, see our OpenTelemetry tracing guide.
