# Uptime monitoring

An uptime monitor probes a public URL on an interval and opens an issue when it has failed enough
times in a row to rule out a blip. A monitor opens one issue per outage, adds nothing while it stays
down, and resolves the issue when it recovers. A TLS certificate close to expiry is its own issue,
because a site can be up with a certificate nine days from ending the party.

Create one in the console under **Monitors → Uptime**, or with `POST
/orgs/{org}/projects/{project}/uptime-monitors`.

## What counts as a failure

Every probe is judged in this order, and the first thing that fails names the failure:

1. The request itself: a timeout, a refused connection, too many redirects.
2. **Expected status** (default 200). Skipped when you assert on the status code yourself (below).
3. **Expected body contains** (optional substring).
4. **Assertions** (optional): the structured checks below.

`failureThreshold` (default 3) consecutive failures open the issue; `recoveryThreshold` (default 2)
consecutive successes resolve it.

## Assertions

A 200 with `{"status":"degraded"}` is down, and a substring cannot say so reliably. An assertion is
a check on the response, and they combine with **all**, **any** and **not**:

```json
{
  "all": [
    { "source": "json", "path": "$.status", "op": "eq", "value": "ok" },
    { "source": "latency_ms", "op": "lt", "value": 800 },
    { "not": { "source": "header", "name": "x-maintenance", "op": "exists" } }
  ]
}
```

| `source` | Fields | `op` |
| --- | --- | --- |
| `status` | `value` (100-599) | `eq` `neq` `lt` `lte` `gt` `gte` |
| `latency_ms` | `value` (ms) | `lt` `lte` `gt` `gte` |
| `header` | `name`, and `value` unless the op is `exists` or `not_exists` | `exists` `not_exists` `eq` `neq` `contains` |
| `json` | `path`, and `value` unless the op is `exists` or `not_exists` | `exists` `not_exists` `eq` `neq` `lt` `lte` `gt` `gte` `contains` |
| `body` | `value` | `contains` `not_contains` |

- **JSON paths** are `$`, `.key`, `['key']` and `[0]`: no wildcards, filters or recursive descent.
  The body is read up to 64 KiB; a longer body is cut, and a path into the cut part reports that.
- **Combinators** nest up to four deep and 25 assertions in total.
- **No regular expressions.** A pattern you supply would run against a response your server
  controls, which is a denial-of-service waiting for a bad pattern.
- A tree that asserts on `status` **replaces** the single expected status, so "any 2xx" or "200 or
  204" can be said. Otherwise the expected status and body checks still apply.

When one fails, the check's error names the assertion and what was seen: `Assertion failed: json
$.status = "ok", got "degraded"`. It appears in the result list and on the outage issue. What was
seen comes from your server's response, so it is shortened and shown as text.

Set `assertions` to `null` on a `PATCH` to remove them. The console edits flat lists (all of / any
of, each row optionally negated); a deeper tree saved through the API is shown as such and left as it
is when you save the form.

## Probe locations

One probe location cannot tell "your site is down" from "our probe's network blipped". With more
than one, a monitor is **DOWN only when at least two locations have each failed their threshold**; a
single location failing is recorded against that location and pages nobody. With a single location
listed, that one is enough.

- Pick locations with `regions` (create or `PATCH`); `GET .../uptime-monitors/regions` lists what
  your deployment runs. No `regions` means the primary location only, scheduled as it always was.
- Each location probes on the monitor's own interval and keeps its own streaks, shown per monitor.
- Every result carries the `region` it was probed from.
- These are **probe** locations, not data residency. A probe stores nothing regional: every result
  is written to the one database, and the name says where the request came from, not where any data
  lives.

On Bucker's hosted service the primary location, `us`, is the only one today, so
`GET .../uptime-monitors/regions` answers `{"primary":"us","available":["us"]}` and every monitor
is probed from there. Further locations are added when customers need two-location quorum; when
they are, they appear in that list and in the monitor form's location picker, and nothing about
existing monitors changes until you list a location on them.

**Operators.** A location exists when a prober runs in it. Run a worker machine
there with `MONITORS_PROBE_REGION=<name>` (default `us`) and list every location in
`UPTIME_PROBE_REGIONS` (comma-separated) on the api, which refuses a monitor naming any other, since
a location with no prober would be a monitor that is never checked there. Only the `us` machine
probes monitors with no listed locations.

## Trace propagation

By default each probe sends a W3C `traceparent` header, so a Bucker-instrumented target continues
the probe's trace instead of starting an unrelated one. Each result records the `traceId` it sent:
search it in your tracing tools to find the target's own trace for that check. Turn it off with
`propagateTrace: false` if the target rejects unknown headers; a `traceparent` you set in the
monitor's own headers wins.

## Suggested monitors

**Monitors → Uptime → Suggest monitors from my errors** lists the hosts your recent errors were
requesting, most frequent first. It **suggests and never creates**: choosing one opens the ordinary
create form with the URL filled in, and nothing exists until you save it. Hosts that already have a
monitor are left out, and so are internal names and private addresses. (The list comes from error
events, which anyone holding a DSN can write; treat it as a convenience, not as evidence.)

## Limits

- **Targets must be public.** A probe starts from Bucker's network, so a monitor can watch only an
  absolute `http://` or `https://` address that the public internet can reach, with no credentials in
  it. `localhost`, internal-only names and private network addresses are refused when you save, and a
  name that resolves to a private address is refused before each probe, at every redirect too. For a
  service that is not reachable from the internet, a [cron monitor](crons.md) that the service checks
  in to is the alternative.
- **At most 100 uptime monitors per project**, disabled ones included. The next create is refused:
  "This project already has 100 uptime monitors; delete one before creating another".
- **At least 30 seconds between probes.** `intervalSeconds` runs from 30 to 86,400 and defaults to 60;
  `timeoutMs` runs from 500 to 60,000 and defaults to 10,000.

## On a status page

A [status page](status-pages.md) component can be bound to an uptime monitor, and the monitor then
decides the component's status: by default **Major outage** while it is DOWN and **Operational** while
it is UP. With `degradedAboveLatencyMs` on the binding, a monitor that is up but averaged slower than
that over its last five checks shows **Degraded performance**. A monitor that has not run yet, is
disabled or was deleted contributes nothing, so its component shows **No data**, never Operational.

**Monitors → Uptime → Show these on a status page** appears once the project has an uptime monitor. It
opens the status page setup with this project selected, where each web address you add becomes an
uptime monitor in the project, a component on the page, and the binding between them in one request.

## Owner and mute

Uptime monitors take the same `ownerTeamId` / `ownerPrincipalId` and `mutedUntil` as
[cron monitors](crons.md): a `notify_owner` alert action routes the monitor's issues to its owner,
and a mute (up to 30 days) holds the alerts back while the monitor keeps checking. Muted alerts are
recorded as *suppressed* in the alert history, naming the mute.

## Phase timing and failure kinds

Every check records how long each phase of the request took, for each request it made (the first and up
to five redirects): **DNS**, **connect**, **TLS** (https only), **first byte** and **download**, plus the
TLS and HTTP versions. A phase the request never reached is reported as *not reached*, not as 0 ms. The
results panel shows them as a waterfall under **Timing**, with the figures beside the bars, and the
rollup returns p50 and p95 per phase with the number of results they cover.

A failure names the phase it happened in:

| Kind | Meaning |
| --- | --- |
| `TIMEOUT_CONNECT` | The TCP connection was not established in time |
| `TIMEOUT_FIRST_BYTE` | Connected, but no response header arrived in time |
| `TIMEOUT` | The deadline passed while resolving, or while the body was arriving |
| `CONNECTION_REFUSED` / `CONNECTION_RESET` | The target refused, or dropped, the connection |
| `TLS_HANDSHAKE` | The handshake failed or did not finish |
| `TLS_CERT_EXPIRED` / `TLS_CERT_UNTRUSTED` / `TLS_HOSTNAME_MISMATCH` | The certificate is at fault |
| `HTTP_PROTOCOL` | The reply was not valid HTTP |
| `CONNECTION` | Any other connection failure, and a name that did not resolve |
| `CONFIGURATION` | The check itself cannot run (see below) |

A name that does not resolve and a name Bucker refuses to contact (a private address) read exactly the
same, on purpose: telling them apart would let a monitor probe which internal names resolve from
Bucker's network.

**A check that cannot run is not an outage.** A kind this deployment cannot run, a stored spec that no
longer parses, request headers that no longer open, or a URL that is not a URL opens one
`check_maintenance` warning on the project and records no result: the monitor's status, its failure
streak and your uptime figures are left alone. The warning resolves itself the next time the check runs.

Every request carries `x-bucker-check: <check id>` beside the `Bucker-Uptime/1.0` user agent, so you can
recognise (or block) one check at your edge. It cannot be overridden by the monitor's own headers.

**HTTP/2.** Set `httpVersion: "h2"` when creating or updating a monitor to ask for HTTP/2 over https. It
is a preference: a server that will not negotiate it is checked over HTTP/1.1 and the result says `1.1`.
The check's definition lives in its columns and, for this preference, in a versioned `spec`.

## Reading checks from an agent

`checks_list` returns the project's checks with their last result (and the phase it failed in), 24 h
uptime with its sample size and the share of expected checks that ran, owner and projected runs a month.
`check_get` returns one check's definition (every secret as `{{SECRET.NAME}}`), its versions with
digests, bindings and last five failures, and with `include: ["results"]` up to 30 days of runs with a
cursor. Both say what they scanned and whether the answer was cut. Response text from your server
(assertion values, URLs a redirect went to) is returned only to a token that also holds `event:read`.
`GET .../uptime-monitors/{slug}/versions` returns the same change history over REST.

## Agent-authored checks: preview, propose, change

An agent never creates or changes a check on its own. Three tools (and the matching REST routes) are
the whole path, and a person decides every step that matters.

**1. `check_preview_run`** (`POST …/uptime-monitors/preview-run`) runs a full spec once from up to
three deployed locations (today `us`) and returns the result, per-phase timings, warnings and a
projection. It waits 20 seconds for the worker and otherwise returns a task handle; call it again
with `taskHandle` (`GET …/preview-run/:handle`) to read it. The result is kept 24 hours as
`uptime_check_results` rows with trigger `preview`, which no uptime figure counts.

A preview of a spec nobody has approved sends only what cannot change the target: `GET`, `HEAD` and
`OPTIONS`. A `POST`, `PUT`, `PATCH` or `DELETE`, and any step that needs a secret variable, comes back
`not_run: needs approval` and is listed, so the approver knows which steps first run live. An origin
on the organization's **check policy** (`previewSafeOrigins`, edited only by a person) runs in full,
except that a preview of a mutating method never follows a redirect (the probe repeats the method
and body on every hop, so a `307` off the safe origin would replay the `POST` there); it reports
`REDIRECT_NOT_FOLLOWED` instead, and the live check follows it as configured. A preview never resolves a secret variable, and an existing check's own sealed headers are never
sent. The target and every redirect hop go through the same guarded lookup as a scheduled check: a
private, loopback, link-local or cloud-metadata address sends nothing and answers "The target could
not be resolved to a public address". Warnings: the target answers 403 to the Bucker user agent, a
redirect leaves the host, the certificate expires within 14 days.

The projection is `⌈2,592,000 / interval⌉ × locations × (1 + retry rate)` runs a month, with the
retry rate on its own line. An HTTP check is unmetered.

**2. `check_propose`** (`POST …/uptime-monitors/propose`) opens the approval that takes a previewed
spec live (`check.create`). It is ask-floored: no policy row and no review mode applies it, so it only
ever returns an approval to decide. It is refused without a preview of the **exact** spec under 24
hours old, and the check stays an inert `PROPOSED` shell (origin `AGENT`, never scheduled) until a
person approves. The plan shows the check with its secrets as references, the classifier's verdict,
the projection and the steps that were not previewed; the approver's yes is bound to the spec's
fingerprint, so what runs is what was previewed. A go-live's inverse is a **pause**; only a person
deletes a check (`check.delete` is human-only and has no tool).

**3. `check_update`** (and `PATCH` for an agent, which takes `reason` and `approvalRequestId`) states a
desired end, and the engine derives the act from the before-and-after diff:

| The diff | Act | Decision |
| --- | --- | --- |
| a neutral tuning change (an interval or timeout within 10 %, nothing else) | `check.update` | an org ALLOW row applies it only when it changes no assertion or location, raises projected runs by under 10 % and touches no gate, release or status setting; otherwise it asks, whatever ALLOW row exists (the registry's conditional ask floor) |
| a change that makes the check fire more (an added assertion, a lower threshold) | `check.update` | asks |
| a change that makes it fire less (a looser or removed assertion, a higher threshold or `failureThreshold`, a longer interval, fewer locations, a relaxed `executionRule`) | `check.narrow` | ask floor, a TTL of at most 7 days, the earlier values come back at its end, CRITICAL behind a status component |
| `muted: true` / `false` | `check.mute` / `check.unmute` | a mute asks and ends by the clock; an unmute is allowed |
| `paused: true` / `false` | `check.pause` / `check.resume` | a pause asks (ask floor), ends at its TTL, CRITICAL behind a status component |

Prefer a mute to a pause: a mute keeps the evidence (runs, issues and status continue and each held
alert is recorded as suppressed), while a pause turns the check's components and gates to "no data".
An agent cannot change a check's target (URL, method, headers, body): that is a different check.

### The direction classifier

`classifyCheckChange(before, after)` reads the two states field by field and returns `fires_more`,
`neutral` or `fires_less`, with the flags `assertionChanged` and `projectedRunsRaised`. Any comparison
it cannot prove is no looser reads `fires_less`, because a wrong `fires_less` costs one human yes and a
wrong `neutral` is a silenced check nobody was asked about.

### The check policy

`GET`/`PUT /orgs/:org/check-policy` (`check_policy.update`: human-only, `org:admin`, which only an
owner holds). `previewSafeOrigins` (origins where a preview may send any method), `intervalFloorSeconds`
(default 300: no agent-authored check runs more often), `agentChecksPerProject` (default 10). Every
write bumps `version`, and a plan names the policy it was made under. A proposal under the floor, at
an undeployed location, with more than five locations, or an eleventh agent-made check in a project is
refused before an approval exists.

## Probing a public target as evidence: `probe_run`

When an agent is about to say "the site is down" or "the certificate is wrong", it can look from outside first.
`probe_run` (`POST …/projects/:p/probe-run`, then `GET …/probe-run/:handle`) runs **one** DNS, TCP, TLS or HTTP
(`GET` or `HEAD`) probe from the primary location against a public target and returns the phases, the failure
kind, TLS facts (protocol, expiry, subject, issuer), DNS answers from public resolvers, the status, a few
response headers and at most 2 KiB of the body. It waits up to 20 seconds for the worker and otherwise answers
with a handle. The api never dials: the probe is a job on `bucker-workers`.

**What it sends.** One request of the kind named. There is no field for a header, a body, a cookie or a secret,
and a path carries no query string. Every request carries `x-bucker-check` so a target's owner can recognise it.
No ICMP, no traceroute.

**Which hosts.** `project:read` may probe a host the project already names: a check's target, a deploy's URL
host, a host its client spans or its error events carried in the last 14 days. Any other host needs
`synthetics:write`. A host that resolves, directly or through a CNAME or a redirect hop, to a private, loopback,
link-local or cloud-metadata address answers one fixed message ("The target could not be resolved to a public
address") under every scope, and a DNS answer that carries such an address is not shown.

**Limits.** Port 25 is not probed (and a short deny list of file-sharing, RPC, UPnP and memcached ports);
at most 3 distinct ports of a host a day per principal; 20 probes every 10 minutes per principal and 200 a day
per organization, counted on the person an OAuth client acts for and on the client itself; a repeat of the same
target inside 60 seconds is answered from the last result and spends nothing; an agent's probe is charged to its
RUNS budget.

**What comes back is data.** Everything the target sent (error text, certificate names, DNS answers, the final
URL, the host a redirect left for, header values and the body) is scrubbed, injection-scanned, capped and listed
as untrusted. A body the scanner flags is returned as its length and SHA-256 only. The result is kept 24 hours
on the task and, when a check of the project names the host, as a result of that check with trigger `probe`,
which no uptime figure counts. One audit row per call records the host, the kind, the principal, the transport
and the OAuth client id.

## Finding what no check covers: `checks_coverage_get`

`checks_coverage_get` reads what real traffic touches and no check exercises. It creates nothing,
asks nobody and schedules nothing. It reports three kinds of gap: server endpoints by route and method
(the top 90 percent of traffic, ranked by share, with the error rate), external hosts your app calls, and
journeys, which read `not_available` with the reason until journey data exists, never an empty list that
reads as "all covered".

- A gap carries a **draft**: the arguments of a `check_propose` call, to be previewed with
  `check_preview_run` first. It is built only for a `GET` or `HEAD` route without path parameters and a
  known origin, and only when every text in it came through the scrub unchanged; otherwise it is withheld
  with the reason. A proposal made from a gap runs one autonomy rung lower, because its text came from
  traffic.
- With the request-rollup flag on (`pillar.apm_red_rollups`) counts are the pre-sampling rollup
  (`rollup_unsampled`); otherwise they are the newest 100,000 stored server spans of seven days
  (`stored_sampled`, with `truncated` when the cap was hit). Traffic is cached an hour per project and
  environment; checks and suppressions are read fresh.
- `checks_coverage_update` **suppresses** one gap for at most 30 days (7 by default). A suppression hides
  a gap, so it asks a person; the gap stays listed with who suppressed it and until when, and ending it
  early is `operation_revert` on that operation.

## Short-lived probes: `probe_monitors_create`

`probe_run` looks once. `probe_monitors_create` starts a capped **probe monitor** of a host the project
already calls: HTTP `GET` or `HEAD`, TCP, TLS or DNS; an interval of at least 300 seconds; a lifetime of at
most 30 days (7 by default); a public host the project has called on at least three of the last fourteen
days; at most ten live probes per requester. There is no argument for a header, body, secret, location,
alert destination or status binding. A person approves it, unless the agent has earned the
`probe_creation` rung. It lands in **shadow**: nothing pages and nothing feeds a status page until a
person takes it live with a check approval, and it ends by itself at its lifetime. Stop it early with
`operation_revert`.

## Healing a browser check: `check_heal_propose`

When a browser check fails because a button moved, `check_heal_propose` proposes moving the locator of
**one** click or fill step. It always asks a person; the check is not edited until a person approves a plan
bound to its digest, and the previous selector is the stored inverse. It refuses an assertion step or a
navigation, a replacement equal to the current selector, and a step with three heal proposals in a row that
nobody approved (that is reported as a check-maintenance warning instead). The replacement came from page
text an agent read, so the proposal never runs above ASK.

## What a check saw: `hydrate_check_run`

An outage issue that a check opened advertises `hydrate_check_run`: for one run, the failing phase and
step, the probe's trace id, the failing regions, the deploys and change events nearest the window (with
their distance in minutes), one scrubbed response excerpt and the check's most recent on-demand re-runs.
The bundle is built by rules when the outage opens and read back; no model is asked. The excerpt is target
text, so it is served only to a token that also holds `event:read`; any other token gets its length and
SHA-256. The triage label reads `unavailable` until a model labelled it and is never a verdict.

`checks_coverage_get` sits in the `synthetics` toolset, and the tools here that write need `synthetics:write`. The holds on
the wider MCP surface were lifted by owner decision on 2026-11-16 without a signed-off security review of the
OAuth server; see [MCP setup](mcp-setup.md). For the infrastructure side, including restarts and rollbacks,
see [Infrastructure](infrastructure.md#asking-for-a-change-restarts-scaling-rollbacks-and-collector-rollouts).

