Application performance: services, traces, profiles and release verdicts
How a person or an agent asks "where did the time go, and is it us or them?" from the spans a project already sends. This page says what each read answers, which ones are switched off until the deployment turns a pillar on, and the rules every number follows. It does not repeat the span ingest contract (see OTLP) or how traces are kept (see Sampling).
Everything here is read through the REST API, the dashboard or the MCP apm toolset (the table of every
tool, scope and budget is MCP tools). Reads need event:read (telemetry) or
project:read (configuration). Changes need apm:write and go through the governed path: a plan, a
person's decision, a signed receipt. See Governed actions.
Where the number comes from
Every answer says what it rests on. These rules hold for all the reads below.
- Evidence and basis. An answer carries the basis it was computed on (
exact,rollup,stored_sampled), the rows it scanned, and the watermark of the rollup it read. A figure from a rollup that cannot see below its own top lists says it is a lower bound. - Counted before sampling. Requests and errors in a service's RED figures are counted before any sampling rule runs, so sampling does not shrink them. Percentiles come from merged histograms and are published as accurate to about 12 percent, not as exact.
- Three kinds of "I do not know".
not_availablemeans the pillar that computes the figure is off.not_checkedmeans the source has not shipped.not_enough_datameans a check had too little on one side. None of them reads as zero, and none reads as passed. - Names are untrusted text. Service, endpoint, operation, release and stack-frame names are chosen by
your code and by the people who call it. For an agent they are scrubbed, scanned for injection, capped,
and listed in the result's
trust.untrustedFields. Treat them as data. - Handles, not dumps. An agent search returns ids to open, not span bodies. Attribute values are not returned at all.
What is off until a pillar is on
Four capabilities that multiply write or read volume on the shared database are behind pillars. Each is
a flag the operator of the deployment sets (pillar.<name>.enabled) after checking the capacity gauges it
depends on. All are off by default, and a flag that cannot be read counts as off. A customer cannot
switch one.
| Pillar | Turns on | While it is off |
|---|---|---|
apm_red_rollups |
the per-service hourly request, error and latency rollup | RED columns and service_get figures answer not_available |
service_edges |
the call graph between services and per-endpoint counters | the service map, dependencies blocks and api_endpoints_list answer not_available |
profiles |
storing and rolling up profiles | hydrate_profile and profiles_compare have nothing to read |
span_metrics |
computing span metric rules | no rule produces a row; last-hour figures read null, never zero |
Services
services_listlists the project's services with tier, lifecycle, owners, open issue counts, coverage gaps and the latest version. With telemetry access it adds the RED columns, andinclude: ["map"]adds the service map. A token withoutevent:readstill gets the list, withsummary.redWithheld: true. A service whose summary is not computed yet is listed with null figures rather than dropped.service_getis one service in one place: what people declared, what telemetry observed and what is wrong. The base answer carries identity, service-level RED, open issues, latency regressions, rules that fired and failed release verdicts. Opt-in blocks add versions, endpoints, the map, declared facts, dependencies and health, each priced in tokens so an agent can plan its call.services_updatedeclares owners, tier, lifecycle, links, contacts and relations. It is always decided by a person. A service declared by a repository file (bucker.yml, a Backstage catalog) is refused withSOURCE_IS_REPOSITORY: change the file instead.
The dashboard's service map is described under Dashboards.
Traces
trace_getreturns one trace as a bounded bundle: the critical path (the chain of spans last to finish, with each span's own contribution), the spans that spent the most time themselves, error spans, span links, and how much of the trace the caller could not read. Spans in other projects of your organization that the token cannot read are counted with the reasonproject_not_readable, never shown. When the trace contains a model call it adds a smallgenAiblock (cost, time in the model and in tools, token counts, no prompt or response text).traces_searchfinds whole traces by a closed JSON spec: up to three named span conditions, an expression over them (has,and,or,not, and the structuralchildanddescendant), and trace-level filters. "Traces where checkout called payments and payments never called the cache" is one query. There is no SQL, and the spec cannot name another project. It reads what was kept (stored_sampled): a trace holding a span stored under a sampling rate below 1 is markedcomplete: false. A scan that would exceed its budget is refused before it runs, with the estimate.join_coverage_getanswers how well the pillars connect: the share of replay requests that reach a stored span, of errors that reach a stored replay, and of log records that reach a span, and why the rest do not (no_id,sampled,not_found). It also flags a replay kept at a higher rate than the traces it should join.sampling_config_explainanswers why a service is kept at the share it is: one ordered chain of the head rate the SDKs report, the upstream decisions, the server rules in order, the quota gauge and the classes the floor never drops, with the kept share before and after sampling. Values an SDK reports about itself are labelledclient_reported. To change a rate, use the sampling rules in Sampling.
Endpoints and dependencies
api_endpoints_listis the API inventory derived from server spans, one row perMETHOD /route, with first and last seen and the release that first showed it. Authentication, public reach and personal-data redaction are hints with their basis stated, andunknownwhen there is no basis. It needssecurity:readand theservice_edgespillar.dependencies_getanswers "is it us or them?": the third-party hosts the project calls, grouped into vendors, with calls, error split, p95 and the issues touched. It adds a verdict upstream suspected only when errors concentrate on one vendor, the other hosts were observed healthy, and the vendor's public status feed was read, was fresh and says degraded.upstream.notSuspectedBecausenames what withheld the verdict. It probes nothing. The status text it drafts is for a person: nothing here publishes an incident.
Profiles
hydrate_profile shows where a service spends its time as a bounded flame summary, in two modes. With a
blobId it unpacks one stored profile (basis exact) and gives the hottest frames by self and by total
time with their callers. With service, type and windowHours it sums the hourly rollup (basis
rollup, no call paths, a frame outside the per-hour top lists is a lower bound).
profiles_compare returns the change in each frame's share of its own profile, in points, for two
blobs or two windows of the same type, with the sample total of each side so a move on a thin profile is not
read like one on a thick profile. Profiles of different types are refused. A side whose object has passed
retention is a 409 naming the side.
Span metrics
A span metric turns the spans a filter selects into a one-minute series (a count, a latency distribution, or the sum of a numeric attribute) grouped by up to four attributes. It is counted at the span writer before sampling, so the series stays exact under sampling, and it is a derived rollup, not a custom metric: nothing here is billed.
span_metrics_createproposes a rule. It lands in shadow: computed and never served, so a person sees what it would produce first.span_metric_updatemoves it toactive, changes it, or disables it; a person decides every change except turning off a rule nothing reads. Undo isoperation_revert.span_metrics_listshows each rule and its state. Last-hour figures exist only for an active rule.- Limits are stated in the list answer so an agent plans inside them: 25 rules in force per project, 4
group attributes, 1,000 combinations an hour per rule (the rest fold into an
__other__row that is still counted), and no identifier-class group key such asuser.*,session.idor a trace id. A span metric never becomes a per-person store. - Read the values through
metrics_querywith sourcespan_metric(see Metrics explorer).
Release verdicts
release_verdict_get returns the verdict on one release of one service in one environment: version N
against N-1 over equal-traffic windows, with each check's result, threshold, sample sizes and watermark.
It never computes: it returns the verdict the worker already stored, so a halt or a rollback is
proposed from the same answer everyone sees. A release with no stored verdict is "not found", which is not
the same as passed.
The four answers are passed, failed, not_enough_data and inconclusive (an effect passed its
threshold without significance). judgedBy names what the verdict was judged by: an approved expectation,
the project's policy, or the built-in defaults.
release_expectation_propose lets an agent draft what a release is expected to do, as at most ten checks
from a closed list (error_rate, p95_latency, crash_free_sessions and others), and a person approves it.
An expectation can only add checks: it is judged in addition to the default policy, so a draft cannot
loosen a threshold or silence a failed verdict. Loosening the policy is a person's act with no tool. See
Releases.
Assertions and probes
Code that calls assertAlways, assertSometimes or assertUnreachable sends one aggregate a minute.
assertions_list reads it per assertion and release (evaluations, violations, rate per million), flagging
a thin sample below 30 evaluations. The behaviour is in Production assertions.
state_probes_propose is how an agent asks for a short-lived probe at a file:line when an investigation
lacks evidence. It never starts one: a person approves it first, a denied file or credential-shaped name is
refused, and a probe lives at most 240 minutes.
What turns it on
| You want | You need |
|---|---|
| RED, the map, endpoint inventory | the operator has enabled apm_red_rollups and service_edges; services sending spans |
| Profiles | the profiles pillar, and an SDK or collector sending profiles |
| Span metrics | the span_metrics pillar; a person to move a rule from shadow to active |
| Agent writes | a token with apm:write (and ci:write for delivery), which a person grants at consent |
Every MCP toolset is open on this deployment: the SG-059 holds were lifted by owner decision on 2026-11-16, with no signed-off security review of the authorization server recorded. A write still asks a person in the way its risk class says.
Related
- Sampling, Dashboards, Metrics, Explore
- Production assertions, Releases, Infrastructure
- CI, gates and flaky tests, where the
release_health_regressiongate rule reads release health - MCP tools, Governed actions