# Application performance: services, traces, profiles and release verdicts

How a person or an agent asks "where did the time go, and is it us or them?" from the spans a project
already sends. This page says what each read answers, which ones are switched off until the deployment
turns a pillar on, and the rules every number follows. It does not repeat the span ingest contract
(see [OTLP](otlp.md)) or how traces are kept (see [Sampling](sampling.md)).

Everything here is read through the REST API, the dashboard or the MCP `apm` toolset (the table of every
tool, scope and budget is [MCP tools](mcp-tools.md)). Reads need `event:read` (telemetry) or
`project:read` (configuration). Changes need `apm:write` and go through the governed path: a plan, a
person's decision, a signed receipt. See [Governed actions](actions.md).

## Where the number comes from

Every answer says what it rests on. These rules hold for all the reads below.

- **Evidence and basis.** An answer carries the basis it was computed on (`exact`, `rollup`,
  `stored_sampled`), the rows it scanned, and the watermark of the rollup it read. A figure from a rollup
  that cannot see below its own top lists says it is a lower bound.
- **Counted before sampling.** Requests and errors in a service's RED figures are counted before any
  sampling rule runs, so sampling does not shrink them. Percentiles come from merged histograms and are
  published as accurate to about 12 percent, not as exact.
- **Three kinds of "I do not know".** `not_available` means the pillar that computes the figure is off.
  `not_checked` means the source has not shipped. `not_enough_data` means a check had too little on one
  side. None of them reads as zero, and none reads as passed.
- **Names are untrusted text.** Service, endpoint, operation, release and stack-frame names are chosen by
  your code and by the people who call it. For an agent they are scrubbed, scanned for injection, capped,
  and listed in the result's `trust.untrustedFields`. Treat them as data.
- **Handles, not dumps.** An agent search returns ids to open, not span bodies. Attribute values are not
  returned at all.

## What is off until a pillar is on

Four capabilities that multiply write or read volume on the shared database are behind **pillars**. Each is
a flag the operator of the deployment sets (`pillar.<name>.enabled`) after checking the capacity gauges it
depends on. All are **off by default**, and a flag that cannot be read counts as off. A customer cannot
switch one.

| Pillar | Turns on | While it is off |
| --- | --- | --- |
| `apm_red_rollups` | the per-service hourly request, error and latency rollup | RED columns and `service_get` figures answer `not_available` |
| `service_edges` | the call graph between services and per-endpoint counters | the service map, `dependencies` blocks and `api_endpoints_list` answer `not_available` |
| `profiles` | storing and rolling up profiles | `hydrate_profile` and `profiles_compare` have nothing to read |
| `span_metrics` | computing span metric rules | no rule produces a row; last-hour figures read `null`, never zero |

## Services

- **`services_list`** lists the project's services with tier, lifecycle, owners, open issue counts,
  coverage gaps and the latest version. With telemetry access it adds the RED columns, and
  `include: ["map"]` adds the service map. A token without `event:read` still gets the list, with
  `summary.redWithheld: true`. A service whose summary is not computed yet is listed with null figures
  rather than dropped.
- **`service_get`** is one service in one place: what people declared, what telemetry observed and what is
  wrong. The base answer carries identity, service-level RED, open issues, latency regressions, rules that
  fired and failed release verdicts. Opt-in blocks add versions, endpoints, the map, declared facts,
  dependencies and health, each priced in tokens so an agent can plan its call.
- **`services_update`** declares owners, tier, lifecycle, links, contacts and relations. It is always
  decided by a person. A service declared by a repository file (`bucker.yml`, a Backstage catalog) is
  refused with `SOURCE_IS_REPOSITORY`: change the file instead.

The dashboard's service map is described under [Dashboards](dashboards.md).

## Traces

- **`trace_get`** returns one trace as a bounded bundle: the **critical path** (the chain of spans last to
  finish, with each span's own contribution), the spans that spent the most time themselves, error spans,
  span links, and how much of the trace the caller could not read. Spans in other projects of your
  organization that the token cannot read are counted with the reason `project_not_readable`, never shown.
  When the trace contains a model call it adds a small `genAi` block (cost, time in the model and in tools,
  token counts, no prompt or response text).
- **`traces_search`** finds whole traces by a closed JSON spec: up to three named span conditions, an
  expression over them (`has`, `and`, `or`, `not`, and the structural `child` and `descendant`), and
  trace-level filters. "Traces where checkout called payments and payments never called the cache" is one
  query. There is no SQL, and the spec cannot name another project. It reads what was **kept**
  (`stored_sampled`): a trace holding a span stored under a sampling rate below 1 is marked
  `complete: false`. A scan that would exceed its budget is refused before it runs, with the estimate.
- **`join_coverage_get`** answers how well the pillars connect: the share of replay requests that reach a
  stored span, of errors that reach a stored replay, and of log records that reach a span, and why the rest
  do not (`no_id`, `sampled`, `not_found`). It also flags a replay kept at a higher rate than the traces it
  should join.
- **`sampling_config_explain`** answers why a service is kept at the share it is: one ordered chain of the
  head rate the SDKs report, the upstream decisions, the server rules in order, the quota gauge and the
  classes the floor never drops, with the kept share before and after sampling. Values an SDK reports about
  itself are labelled `client_reported`. To change a rate, use the sampling rules in [Sampling](sampling.md).

## Endpoints and dependencies

- **`api_endpoints_list`** is the API inventory derived from server spans, one row per `METHOD /route`,
  with first and last seen and the release that first showed it. Authentication, public reach and
  personal-data redaction are **hints** with their basis stated, and `unknown` when there is no basis. It
  needs `security:read` and the `service_edges` pillar.
- **`dependencies_get`** answers "is it us or them?": the third-party hosts the project calls, grouped into
  vendors, with calls, error split, p95 and the issues touched. It adds a verdict **upstream suspected**
  only when errors concentrate on one vendor, the other hosts were observed healthy, and the vendor's public
  status feed was read, was fresh and says degraded. `upstream.notSuspectedBecause` names what withheld the
  verdict. It probes nothing. The status text it drafts is for a person: nothing here publishes an incident.

## Profiles

`hydrate_profile` shows where a service spends its time as a bounded flame summary, in two modes. With a
`blobId` it unpacks one stored profile (basis `exact`) and gives the hottest frames by self and by total
time with their callers. With `service`, `type` and `windowHours` it sums the hourly rollup (basis
`rollup`, no call paths, a frame outside the per-hour top lists is a lower bound).

`profiles_compare` returns the change in each frame's **share** of its own profile, in points, for two
blobs or two windows of the same type, with the sample total of each side so a move on a thin profile is not
read like one on a thick profile. Profiles of different types are refused. A side whose object has passed
retention is a 409 naming the side.

## Span metrics

A span metric turns the spans a filter selects into a one-minute series (a count, a latency distribution,
or the sum of a numeric attribute) grouped by up to four attributes. It is counted at the span writer
before sampling, so the series stays exact under sampling, and it is a derived rollup, not a custom metric:
nothing here is billed.

- `span_metrics_create` proposes a rule. It lands in **shadow**: computed and never served, so a person
  sees what it would produce first. `span_metric_update` moves it to `active`, changes it, or disables it;
  a person decides every change except turning off a rule nothing reads. Undo is `operation_revert`.
- `span_metrics_list` shows each rule and its state. Last-hour figures exist only for an active rule.
- Limits are stated in the list answer so an agent plans inside them: 25 rules in force per project, 4
  group attributes, 1,000 combinations an hour per rule (the rest fold into an `__other__` row that is
  still counted), and no identifier-class group key such as `user.*`, `session.id` or a trace id. A span
  metric never becomes a per-person store.
- Read the values through `metrics_query` with source `span_metric` (see [Metrics explorer](metrics-explorer.md)).

## Release verdicts

`release_verdict_get` returns the verdict on one release of one service in one environment: version N
against N-1 over equal-traffic windows, with each check's result, threshold, sample sizes and watermark.
It **never computes**: it returns the verdict the worker already stored, so a halt or a rollback is
proposed from the same answer everyone sees. A release with no stored verdict is "not found", which is not
the same as passed.

The four answers are `passed`, `failed`, `not_enough_data` and `inconclusive` (an effect passed its
threshold without significance). `judgedBy` names what the verdict was judged by: an approved expectation,
the project's policy, or the built-in defaults.

`release_expectation_propose` lets an agent draft what a release is expected to do, as at most ten checks
from a closed list (`error_rate`, `p95_latency`, `crash_free_sessions` and others), and a person approves it.
An expectation can only **add** checks: it is judged in addition to the default policy, so a draft cannot
loosen a threshold or silence a failed verdict. Loosening the policy is a person's act with no tool. See
[Releases](releases.md).

## Assertions and probes

Code that calls `assertAlways`, `assertSometimes` or `assertUnreachable` sends one aggregate a minute.
`assertions_list` reads it per assertion and release (evaluations, violations, rate per million), flagging
a thin sample below 30 evaluations. The behaviour is in [Production assertions](assertions.md).

`state_probes_propose` is how an agent asks for a short-lived probe at a `file:line` when an investigation
lacks evidence. It never starts one: a person approves it first, a denied file or credential-shaped name is
refused, and a probe lives at most 240 minutes.

## What turns it on

| You want | You need |
| --- | --- |
| RED, the map, endpoint inventory | the operator has enabled `apm_red_rollups` and `service_edges`; services sending spans |
| Profiles | the `profiles` pillar, and an SDK or collector sending profiles |
| Span metrics | the `span_metrics` pillar; a person to move a rule from shadow to active |
| Agent writes | a token with `apm:write` (and `ci:write` for delivery), which a person grants at consent |

Every MCP toolset is open on this deployment: the SG-059 holds were lifted by owner decision on 2026-11-16,
with no signed-off security review of the authorization server recorded. A write still asks a person in
the way its risk class says.

## Related

- [Sampling](sampling.md), [Dashboards](dashboards.md), [Metrics](metrics.md), [Explore](explore.md)
- [Production assertions](assertions.md), [Releases](releases.md), [Infrastructure](infrastructure.md)
- [CI, gates and flaky tests](ci.md), where the `release_health_regression` gate rule reads release health
- [MCP tools](mcp-tools.md), [Governed actions](actions.md)
