Metrics explorer: infrastructure and custom metrics
This page is about the numbers your services and hosts send over OTLP or an SDK: request counts, queue depths, latency distributions, anything with a name, a unit and attributes. It covers finding a metric, asking a question of it, reducing what it costs, defining a number once, and what an agent can and cannot do with it.
It is not the page for product-analytics numbers such as an activation rate. Those are canonical metrics: one agreed definition per question, proposed by agents and certified by people, behind the per-project analytics gate.
Find a metric
Bucker keeps a catalogue of the metrics a project sends, refreshed hourly. For each one it holds the kind, the unit to
show, the origin (sdk, otlp or derived), how many series and points a day it has, what reads it (alert rules,
widgets, saved queries) and when it was last queried. A metric nothing reads and nobody has queried in 30 days is marked
an orphan, which is the list to start from when you want to reduce volume.
| To do this | Use |
|---|---|
| Search by name fragment, kind, origin, owner or "nothing reads it" | metrics_search, or GET /orgs/:orgSlug/projects/:projectSlug/metrics/catalogue |
| Everything about one metric: metadata, series, top attribute keys, readers, newest point, exemplar trace ids | metrics_context_get, or GET …/metrics/catalogue/entry |
| Change the description, the unit to display or the owning team | metrics_update, or PUT …/metrics/metadata |
Searching reads the catalogue only and scans no telemetry. Reading one metric's newest point and exemplars needs
event:read, because those are your telemetry rather than configuration. Changing the owner is the one metadata edit that
asks for approval, because the owner decides who hears about a cardinality problem. Metric names, units and descriptions are
text the sender chose, so an agent receives them marked untrusted.
Ask a question
metrics_query (or POST …/metrics/query) takes one closed JSON spec. Up to six lettered queries, each with a metric, a
read, filters, a group-by, a top-N and an optional time shift, then formulas over their ids.
{
"v": 1,
"range": { "from": "-24h" },
"interval": "5m",
"queries": [
{ "id": "a", "metric": "http.server.errors", "read": "rate" },
{ "id": "b", "metric": "http.server.requests", "read": "rate" },
{ "id": "c", "metric": "http.server.requests", "read": "rate", "timeshift": "7d" }
],
"formulas": [
{ "id": "error_ratio", "expr": "a / b" },
{ "id": "traffic_vs_last_week", "expr": "b / c" }
],
"fill": null
}
The reads are avg, sum, min, max, count, rate, increase, p50, p75, p90, p95, p99 and mean.
Formulas use + - * / over query ids, plus interval_seconds and window_seconds. A series comes back as a dense array:
values[i] is the bucket at axis.from + i * axis.stepSeconds.
Every answer states what it is made of:
- The normalized spec and its digest, so a person can paste the echoed spec and get the same numbers.
- The tier it read: raw points, 5-minute rollups or hourly rollups, with the watermark and whether the window is covered. A point-level filter forces a raw read.
- The scan price before it runs:
lowon rollups,mediumon raw points up to 24 hours,highon raw points beyond that, where a scan cap can truncate andtruncatedsays so. PassmaxScanRowsas a ceiling: if the estimate is above it, nothing runs and nothing is charged, and the answer is the estimate.
Group labels are attribute values chosen by the sender, so an agent receives them scrubbed and marked untrusted.
Rollup tiers are off until an operator turns them on
The 5-minute and hourly rollups are written only while pillar.metric_rollups is on, and the hourly cardinality profile
reads the metric_hours pillar. With the flag off a query answers from raw points and says tier: raw, so a long window
is more expensive but not wrong. Which switches exist, and what turns each on, is in the
pillar flag runbook (docs/runbooks/pillar-flags.md, for the operators of a deployment).
More reads
| Question | Tool | What you get |
|---|---|---|
| What is different about the bad window? | metrics_compare |
Attribute values ranked by their share of the change against the same hours a week earlier, with how much no value explains and which other metrics changed too. A correlation ranking, not a cause. |
| Which host or pod is unlike its peers? | metrics_outliers_get |
Members of a group (3 to 50) whose deviation from the peer median stays above a threshold for at least three consecutive hours, from hourly rollups only. Fewer than three peers or 24 aligned hours reads not_enough_data. |
| Where is this series going? | forecast_get |
p10, p50 and p90 bands and the projected time a bound you name is crossed. The seasonal default is the only forecast an alert rule reads; an enabled second model appears beside it. See forecast licences. |
| Pages of evidence | notebooks_create, notebooks_update |
An agent's notebook is a private draft of up to 50 cells until a person approves publishing it to the project. |
Reduce what a metric costs
A tag rule names a metric, exactly or by name prefix (at least three characters), and a deny list or allow list of at most 20 attribute keys. A project can have 50.
| Scope | What it removes | What it saves | Reversible? |
|---|---|---|---|
ingest |
The keys, before the series identity is computed, so series that differed only by those keys collapse as they are written | The units billed, because points are metered at ingest | No, for data written while it runs. Disabling a rule keeps nothing back. |
rollup |
The keys from the hourly rollup only; raw points keep them | Hourly rows and long-window scans, never units | Hours already rolled up change only when rebuilt |
An ingest rule is not applied to cumulative counters, because a running total split across two exports would be stored as a
partial sum that reads as a counter reset. Rules never remove reserved keys (otel.metric.overflow, anything under bucker.
or otel.scope.), and a rule that would remove more than a quarter of a project's points is refused.
A rule an agent proposes through metrics_update lands disabled. Turning it on is a second act that must cite the
cardinality preview from the last 15 minutes. Removing a rule is not an agent's: a person disables it in settings, because
widening again raises cost. Only enabled rules apply, and the most specific match wins. List a project's rules, enabled or
not, with GET …/metrics/tag-rules (the dashboard's Tag rules view).
Span metrics are the other way to get a number without paying for a custom metric. span_metrics_create proposes a rule that
counts spans before sampling, so the series stays exact under sampling, and it lands in shadow: points are written and none is
served until a person moves it live with span_metric_update. Identifier-class group keys such as user.* and session.id
are refused. The span_metrics pillar gates it and is off by default.
Define a number once
A derived metric is a named query of up to four reads and one formula, such as checkout.error_ratio = a / b. Nothing is
stored or billed but the definition: every value is computed from the raw metrics when it is read, so a definition cannot
disagree with the data under it.
Definitions are versioned and each version is immutable. Anything that reads one names the version
(derived:checkout.error_ratio@2), so a reader never changes because a newer version exists.
| State | Meaning |
|---|---|
| proposed | A person or an agent wrote it. Only drafts use it. An edit is a new version. |
| approved | A person certified it. Alert rules and shared widgets can pin only an approved version. |
| archived | A person retired it. Refused while something still reads that version. |
The dashboard's derived-metrics view lists definitions with GET …/derived-metrics and what reads a version with
GET …/derived-metrics/:name/dependents. If the metric a definition reads disappears, the definition is marked as drifted
rather than returning zeros.
Certifying is a person's act. An agent, a service or an OAuth client is refused. The person needs admin on the project,
gives a reason, and confirms with a security key or a one-time code. That confirmation is minted by
POST …/derived-metrics/:name/versions/:v/certify/challenge, is tied to the name, the version, the digest of the stored
definition and the reason, works once and expires after five minutes. POST …/certify then flips the status, writes one audit
row (derived_metric.certify, with the method used) and one signed metric_certified receipt in the same transaction. If the
definition changed between the challenge and the call, it is refused with 409 and nothing is certified.
What is on, and what is off
| Capability | State | What turns it on |
|---|---|---|
| Catalogue, query, compare, notebooks | Available. Queries read raw points | A token with event:read for reads and metrics:write for writes |
| 5-minute and hourly rollups | Off | An operator enables pillar.metric_rollups |
| Hourly cardinality profile | Off | An operator enables pillar.metric_hours |
| Span metrics | Off | An operator enables pillar.span_metrics; a person approves each rule |
| Cold storage of old metric days | Off | An operator enables pillar.cold_tier after the bucket settings in the cold-tier runbook (docs/runbooks/cold-tier.md) are checked |
| A columnar store for metrics | Not available | Needs a managed ClickHouse account and a provider; today the only store is Postgres |
The enabled-pillar rules are the same everywhere: an absent or unreadable flag means off, and only turning one on is gated.
The metrics:write and apm:write scopes are inside the 41-scope MCP ceiling that shipped in the 2026-11-18 release. That opening was an owner
decision on 2026-11-16 that lifted the SG-059 holds without a security review of the OAuth authorization server, and
none is recorded; see the changelog. Every metric write still goes through preview, approval and a
signed receipt.
Related
- Canonical metrics: product-analytics definitions, certified by people.
- OTLP ingest: how metrics arrive and how series are identified.
- Alert rules: rules that read a metric or a pinned derived metric.
- Dashboards: widgets that read the same specs.
- MCP setup: the metrics tools, their token prices and their scopes.
- Retention: how long raw points and rollups are kept.