Skip to content
bucker

Application performance: services, traces, profiles and release verdicts

How a person or an agent asks "where did the time go, and is it us or them?" from the spans a project already sends. This page says what each read answers, which ones are switched off until the deployment turns a pillar on, and the rules every number follows. It does not repeat the span ingest contract (see OTLP) or how traces are kept (see Sampling).

Everything here is read through the REST API, the dashboard or the MCP apm toolset (the table of every tool, scope and budget is MCP tools). Reads need event:read (telemetry) or project:read (configuration). Changes need apm:write and go through the governed path: a plan, a person's decision, a signed receipt. See Governed actions.

Where the number comes from

Every answer says what it rests on. These rules hold for all the reads below.

  • Evidence and basis. An answer carries the basis it was computed on (exact, rollup, stored_sampled), the rows it scanned, and the watermark of the rollup it read. A figure from a rollup that cannot see below its own top lists says it is a lower bound.
  • Counted before sampling. Requests and errors in a service's RED figures are counted before any sampling rule runs, so sampling does not shrink them. Percentiles come from merged histograms and are published as accurate to about 12 percent, not as exact.
  • Three kinds of "I do not know". not_available means the pillar that computes the figure is off. not_checked means the source has not shipped. not_enough_data means a check had too little on one side. None of them reads as zero, and none reads as passed.
  • Names are untrusted text. Service, endpoint, operation, release and stack-frame names are chosen by your code and by the people who call it. For an agent they are scrubbed, scanned for injection, capped, and listed in the result's trust.untrustedFields. Treat them as data.
  • Handles, not dumps. An agent search returns ids to open, not span bodies. Attribute values are not returned at all.

What is off until a pillar is on

Four capabilities that multiply write or read volume on the shared database are behind pillars. Each is a flag the operator of the deployment sets (pillar.<name>.enabled) after checking the capacity gauges it depends on. All are off by default, and a flag that cannot be read counts as off. A customer cannot switch one.

Pillar Turns on While it is off
apm_red_rollups the per-service hourly request, error and latency rollup RED columns and service_get figures answer not_available
service_edges the call graph between services and per-endpoint counters the service map, dependencies blocks and api_endpoints_list answer not_available
profiles storing and rolling up profiles hydrate_profile and profiles_compare have nothing to read
span_metrics computing span metric rules no rule produces a row; last-hour figures read null, never zero

Services

  • services_list lists the project's services with tier, lifecycle, owners, open issue counts, coverage gaps and the latest version. With telemetry access it adds the RED columns, and include: ["map"] adds the service map. A token without event:read still gets the list, with summary.redWithheld: true. A service whose summary is not computed yet is listed with null figures rather than dropped.
  • service_get is one service in one place: what people declared, what telemetry observed and what is wrong. The base answer carries identity, service-level RED, open issues, latency regressions, rules that fired and failed release verdicts. Opt-in blocks add versions, endpoints, the map, declared facts, dependencies and health, each priced in tokens so an agent can plan its call.
  • services_update declares owners, tier, lifecycle, links, contacts and relations. It is always decided by a person. A service declared by a repository file (bucker.yml, a Backstage catalog) is refused with SOURCE_IS_REPOSITORY: change the file instead.

The dashboard's service map is described under Dashboards.

Traces

  • trace_get returns one trace as a bounded bundle: the critical path (the chain of spans last to finish, with each span's own contribution), the spans that spent the most time themselves, error spans, span links, and how much of the trace the caller could not read. Spans in other projects of your organization that the token cannot read are counted with the reason project_not_readable, never shown. When the trace contains a model call it adds a small genAi block (cost, time in the model and in tools, token counts, no prompt or response text).
  • traces_search finds whole traces by a closed JSON spec: up to three named span conditions, an expression over them (has, and, or, not, and the structural child and descendant), and trace-level filters. "Traces where checkout called payments and payments never called the cache" is one query. There is no SQL, and the spec cannot name another project. It reads what was kept (stored_sampled): a trace holding a span stored under a sampling rate below 1 is marked complete: false. A scan that would exceed its budget is refused before it runs, with the estimate.
  • join_coverage_get answers how well the pillars connect: the share of replay requests that reach a stored span, of errors that reach a stored replay, and of log records that reach a span, and why the rest do not (no_id, sampled, not_found). It also flags a replay kept at a higher rate than the traces it should join.
  • sampling_config_explain answers why a service is kept at the share it is: one ordered chain of the head rate the SDKs report, the upstream decisions, the server rules in order, the quota gauge and the classes the floor never drops, with the kept share before and after sampling. Values an SDK reports about itself are labelled client_reported. To change a rate, use the sampling rules in Sampling.

Endpoints and dependencies

  • api_endpoints_list is the API inventory derived from server spans, one row per METHOD /route, with first and last seen and the release that first showed it. Authentication, public reach and personal-data redaction are hints with their basis stated, and unknown when there is no basis. It needs security:read and the service_edges pillar.
  • dependencies_get answers "is it us or them?": the third-party hosts the project calls, grouped into vendors, with calls, error split, p95 and the issues touched. It adds a verdict upstream suspected only when errors concentrate on one vendor, the other hosts were observed healthy, and the vendor's public status feed was read, was fresh and says degraded. upstream.notSuspectedBecause names what withheld the verdict. It probes nothing. The status text it drafts is for a person: nothing here publishes an incident.

Profiles

hydrate_profile shows where a service spends its time as a bounded flame summary, in two modes. With a blobId it unpacks one stored profile (basis exact) and gives the hottest frames by self and by total time with their callers. With service, type and windowHours it sums the hourly rollup (basis rollup, no call paths, a frame outside the per-hour top lists is a lower bound).

profiles_compare returns the change in each frame's share of its own profile, in points, for two blobs or two windows of the same type, with the sample total of each side so a move on a thin profile is not read like one on a thick profile. Profiles of different types are refused. A side whose object has passed retention is a 409 naming the side.

Span metrics

A span metric turns the spans a filter selects into a one-minute series (a count, a latency distribution, or the sum of a numeric attribute) grouped by up to four attributes. It is counted at the span writer before sampling, so the series stays exact under sampling, and it is a derived rollup, not a custom metric: nothing here is billed.

  • span_metrics_create proposes a rule. It lands in shadow: computed and never served, so a person sees what it would produce first. span_metric_update moves it to active, changes it, or disables it; a person decides every change except turning off a rule nothing reads. Undo is operation_revert.
  • span_metrics_list shows each rule and its state. Last-hour figures exist only for an active rule.
  • Limits are stated in the list answer so an agent plans inside them: 25 rules in force per project, 4 group attributes, 1,000 combinations an hour per rule (the rest fold into an __other__ row that is still counted), and no identifier-class group key such as user.*, session.id or a trace id. A span metric never becomes a per-person store.
  • Read the values through metrics_query with source span_metric (see Metrics explorer).

Release verdicts

release_verdict_get returns the verdict on one release of one service in one environment: version N against N-1 over equal-traffic windows, with each check's result, threshold, sample sizes and watermark. It never computes: it returns the verdict the worker already stored, so a halt or a rollback is proposed from the same answer everyone sees. A release with no stored verdict is "not found", which is not the same as passed.

The four answers are passed, failed, not_enough_data and inconclusive (an effect passed its threshold without significance). judgedBy names what the verdict was judged by: an approved expectation, the project's policy, or the built-in defaults.

release_expectation_propose lets an agent draft what a release is expected to do, as at most ten checks from a closed list (error_rate, p95_latency, crash_free_sessions and others), and a person approves it. An expectation can only add checks: it is judged in addition to the default policy, so a draft cannot loosen a threshold or silence a failed verdict. Loosening the policy is a person's act with no tool. See Releases.

Assertions and probes

Code that calls assertAlways, assertSometimes or assertUnreachable sends one aggregate a minute. assertions_list reads it per assertion and release (evaluations, violations, rate per million), flagging a thin sample below 30 evaluations. The behaviour is in Production assertions.

state_probes_propose is how an agent asks for a short-lived probe at a file:line when an investigation lacks evidence. It never starts one: a person approves it first, a denied file or credential-shaped name is refused, and a probe lives at most 240 minutes.

What turns it on

You want You need
RED, the map, endpoint inventory the operator has enabled apm_red_rollups and service_edges; services sending spans
Profiles the profiles pillar, and an SDK or collector sending profiles
Span metrics the span_metrics pillar; a person to move a rule from shadow to active
Agent writes a token with apm:write (and ci:write for delivery), which a person grants at consent

Every MCP toolset is open on this deployment: the SG-059 holds were lifted by owner decision on 2026-11-16, with no signed-off security review of the authorization server recorded. A write still asks a person in the way its risk class says.