Uptime monitoring
An uptime monitor probes a public URL on an interval and opens an issue when it has failed enough times in a row to rule out a blip. A monitor opens one issue per outage, adds nothing while it stays down, and resolves the issue when it recovers. A TLS certificate close to expiry is its own issue, because a site can be up with a certificate nine days from ending the party.
Create one in the console under Monitors → Uptime, or with POST /orgs/{org}/projects/{project}/uptime-monitors.
What counts as a failure
Every probe is judged in this order, and the first thing that fails names the failure:
- The request itself: a timeout, a refused connection, too many redirects.
- Expected status (default 200). Skipped when you assert on the status code yourself (below).
- Expected body contains (optional substring).
- Assertions (optional): the structured checks below.
failureThreshold (default 3) consecutive failures open the issue; recoveryThreshold (default 2)
consecutive successes resolve it.
Assertions
A 200 with {"status":"degraded"} is down, and a substring cannot say so reliably. An assertion is
a check on the response, and they combine with all, any and not:
{
"all": [
{ "source": "json", "path": "$.status", "op": "eq", "value": "ok" },
{ "source": "latency_ms", "op": "lt", "value": 800 },
{ "not": { "source": "header", "name": "x-maintenance", "op": "exists" } }
]
}
source |
Fields | op |
|---|---|---|
status |
value (100-599) |
eq neq lt lte gt gte |
latency_ms |
value (ms) |
lt lte gt gte |
header |
name, and value unless the op is exists or not_exists |
exists not_exists eq neq contains |
json |
path, and value unless the op is exists or not_exists |
exists not_exists eq neq lt lte gt gte contains |
body |
value |
contains not_contains |
- JSON paths are
$,.key,['key']and[0]: no wildcards, filters or recursive descent. The body is read up to 64 KiB; a longer body is cut, and a path into the cut part reports that. - Combinators nest up to four deep and 25 assertions in total.
- No regular expressions. A pattern you supply would run against a response your server controls, which is a denial-of-service waiting for a bad pattern.
- A tree that asserts on
statusreplaces the single expected status, so "any 2xx" or "200 or 204" can be said. Otherwise the expected status and body checks still apply.
When one fails, the check's error names the assertion and what was seen: Assertion failed: json $.status = "ok", got "degraded". It appears in the result list and on the outage issue. What was
seen comes from your server's response, so it is shortened and shown as text.
Set assertions to null on a PATCH to remove them. The console edits flat lists (all of / any
of, each row optionally negated); a deeper tree saved through the API is shown as such and left as it
is when you save the form.
Probe locations
One probe location cannot tell "your site is down" from "our probe's network blipped". With more than one, a monitor is DOWN only when at least two locations have each failed their threshold; a single location failing is recorded against that location and pages nobody. With a single location listed, that one is enough.
- Pick locations with
regions(create orPATCH);GET .../uptime-monitors/regionslists what your deployment runs. Noregionsmeans the primary location only, scheduled as it always was. - Each location probes on the monitor's own interval and keeps its own streaks, shown per monitor.
- Every result carries the
regionit was probed from. - These are probe locations, not data residency. A probe stores nothing regional: every result is written to the one database, and the name says where the request came from, not where any data lives.
On Bucker's hosted service the primary location, us, is the only one today, so
GET .../uptime-monitors/regions answers {"primary":"us","available":["us"]} and every monitor
is probed from there. Further locations are added when customers need two-location quorum; when
they are, they appear in that list and in the monitor form's location picker, and nothing about
existing monitors changes until you list a location on them.
Operators. A location exists when a prober runs in it. Run a worker machine
there with MONITORS_PROBE_REGION=<name> (default us) and list every location in
UPTIME_PROBE_REGIONS (comma-separated) on the api, which refuses a monitor naming any other, since
a location with no prober would be a monitor that is never checked there. Only the us machine
probes monitors with no listed locations.
Trace propagation
By default each probe sends a W3C traceparent header, so a Bucker-instrumented target continues
the probe's trace instead of starting an unrelated one. Each result records the traceId it sent:
search it in your tracing tools to find the target's own trace for that check. Turn it off with
propagateTrace: false if the target rejects unknown headers; a traceparent you set in the
monitor's own headers wins.
Suggested monitors
Monitors → Uptime → Suggest monitors from my errors lists the hosts your recent errors were requesting, most frequent first. It suggests and never creates: choosing one opens the ordinary create form with the URL filled in, and nothing exists until you save it. Hosts that already have a monitor are left out, and so are internal names and private addresses. (The list comes from error events, which anyone holding a DSN can write; treat it as a convenience, not as evidence.)
Limits
- Targets must be public. A probe starts from Bucker's network, so a monitor can watch only an
absolute
http://orhttps://address that the public internet can reach, with no credentials in it.localhost, internal-only names and private network addresses are refused when you save, and a name that resolves to a private address is refused before each probe, at every redirect too. For a service that is not reachable from the internet, a cron monitor that the service checks in to is the alternative. - At most 100 uptime monitors per project, disabled ones included. The next create is refused: "This project already has 100 uptime monitors; delete one before creating another".
- At least 30 seconds between probes.
intervalSecondsruns from 30 to 86,400 and defaults to 60;timeoutMsruns from 500 to 60,000 and defaults to 10,000.
On a status page
A status page component can be bound to an uptime monitor, and the monitor then
decides the component's status: by default Major outage while it is DOWN and Operational while
it is UP. With degradedAboveLatencyMs on the binding, a monitor that is up but averaged slower than
that over its last five checks shows Degraded performance. A monitor that has not run yet, is
disabled or was deleted contributes nothing, so its component shows No data, never Operational.
Monitors → Uptime → Show these on a status page appears once the project has an uptime monitor. It opens the status page setup with this project selected, where each web address you add becomes an uptime monitor in the project, a component on the page, and the binding between them in one request.
Owner and mute
Uptime monitors take the same ownerTeamId / ownerPrincipalId and mutedUntil as
cron monitors: a notify_owner alert action routes the monitor's issues to its owner,
and a mute (up to 30 days) holds the alerts back while the monitor keeps checking. Muted alerts are
recorded as suppressed in the alert history, naming the mute.
Phase timing and failure kinds
Every check records how long each phase of the request took, for each request it made (the first and up to five redirects): DNS, connect, TLS (https only), first byte and download, plus the TLS and HTTP versions. A phase the request never reached is reported as not reached, not as 0 ms. The results panel shows them as a waterfall under Timing, with the figures beside the bars, and the rollup returns p50 and p95 per phase with the number of results they cover.
A failure names the phase it happened in:
| Kind | Meaning |
|---|---|
TIMEOUT_CONNECT |
The TCP connection was not established in time |
TIMEOUT_FIRST_BYTE |
Connected, but no response header arrived in time |
TIMEOUT |
The deadline passed while resolving, or while the body was arriving |
CONNECTION_REFUSED / CONNECTION_RESET |
The target refused, or dropped, the connection |
TLS_HANDSHAKE |
The handshake failed or did not finish |
TLS_CERT_EXPIRED / TLS_CERT_UNTRUSTED / TLS_HOSTNAME_MISMATCH |
The certificate is at fault |
HTTP_PROTOCOL |
The reply was not valid HTTP |
CONNECTION |
Any other connection failure, and a name that did not resolve |
CONFIGURATION |
The check itself cannot run (see below) |
A name that does not resolve and a name Bucker refuses to contact (a private address) read exactly the same, on purpose: telling them apart would let a monitor probe which internal names resolve from Bucker's network.
A check that cannot run is not an outage. A kind this deployment cannot run, a stored spec that no
longer parses, request headers that no longer open, or a URL that is not a URL opens one
check_maintenance warning on the project and records no result: the monitor's status, its failure
streak and your uptime figures are left alone. The warning resolves itself the next time the check runs.
Every request carries x-bucker-check: <check id> beside the Bucker-Uptime/1.0 user agent, so you can
recognise (or block) one check at your edge. It cannot be overridden by the monitor's own headers.
HTTP/2. Set httpVersion: "h2" when creating or updating a monitor to ask for HTTP/2 over https. It
is a preference: a server that will not negotiate it is checked over HTTP/1.1 and the result says 1.1.
The check's definition lives in its columns and, for this preference, in a versioned spec.
Reading checks from an agent
checks_list returns the project's checks with their last result (and the phase it failed in), 24 h
uptime with its sample size and the share of expected checks that ran, owner and projected runs a month.
check_get returns one check's definition (every secret as {{SECRET.NAME}}), its versions with
digests, bindings and last five failures, and with include: ["results"] up to 30 days of runs with a
cursor. Both say what they scanned and whether the answer was cut. Response text from your server
(assertion values, URLs a redirect went to) is returned only to a token that also holds event:read.
GET .../uptime-monitors/{slug}/versions returns the same change history over REST.
Agent-authored checks: preview, propose, change
An agent never creates or changes a check on its own. Three tools (and the matching REST routes) are the whole path, and a person decides every step that matters.
1. check_preview_run (POST …/uptime-monitors/preview-run) runs a full spec once from up to
three deployed locations (today us) and returns the result, per-phase timings, warnings and a
projection. It waits 20 seconds for the worker and otherwise returns a task handle; call it again
with taskHandle (GET …/preview-run/:handle) to read it. The result is kept 24 hours as
uptime_check_results rows with trigger preview, which no uptime figure counts.
A preview of a spec nobody has approved sends only what cannot change the target: GET, HEAD and
OPTIONS. A POST, PUT, PATCH or DELETE, and any step that needs a secret variable, comes back
not_run: needs approval and is listed, so the approver knows which steps first run live. An origin
on the organization's check policy (previewSafeOrigins, edited only by a person) runs in full,
except that a preview of a mutating method never follows a redirect (the probe repeats the method
and body on every hop, so a 307 off the safe origin would replay the POST there); it reports
REDIRECT_NOT_FOLLOWED instead, and the live check follows it as configured. A preview never resolves a secret variable, and an existing check's own sealed headers are never
sent. The target and every redirect hop go through the same guarded lookup as a scheduled check: a
private, loopback, link-local or cloud-metadata address sends nothing and answers "The target could
not be resolved to a public address". Warnings: the target answers 403 to the Bucker user agent, a
redirect leaves the host, the certificate expires within 14 days.
The projection is ⌈2,592,000 / interval⌉ × locations × (1 + retry rate) runs a month, with the
retry rate on its own line. An HTTP check is unmetered.
2. check_propose (POST …/uptime-monitors/propose) opens the approval that takes a previewed
spec live (check.create). It is ask-floored: no policy row and no review mode applies it, so it only
ever returns an approval to decide. It is refused without a preview of the exact spec under 24
hours old, and the check stays an inert PROPOSED shell (origin AGENT, never scheduled) until a
person approves. The plan shows the check with its secrets as references, the classifier's verdict,
the projection and the steps that were not previewed; the approver's yes is bound to the spec's
fingerprint, so what runs is what was previewed. A go-live's inverse is a pause; only a person
deletes a check (check.delete is human-only and has no tool).
3. check_update (and PATCH for an agent, which takes reason and approvalRequestId) states a
desired end, and the engine derives the act from the before-and-after diff:
| The diff | Act | Decision |
|---|---|---|
| a neutral tuning change (an interval or timeout within 10 %, nothing else) | check.update |
an org ALLOW row applies it only when it changes no assertion or location, raises projected runs by under 10 % and touches no gate, release or status setting; otherwise it asks, whatever ALLOW row exists (the registry's conditional ask floor) |
| a change that makes the check fire more (an added assertion, a lower threshold) | check.update |
asks |
a change that makes it fire less (a looser or removed assertion, a higher threshold or failureThreshold, a longer interval, fewer locations, a relaxed executionRule) |
check.narrow |
ask floor, a TTL of at most 7 days, the earlier values come back at its end, CRITICAL behind a status component |
muted: true / false |
check.mute / check.unmute |
a mute asks and ends by the clock; an unmute is allowed |
paused: true / false |
check.pause / check.resume |
a pause asks (ask floor), ends at its TTL, CRITICAL behind a status component |
Prefer a mute to a pause: a mute keeps the evidence (runs, issues and status continue and each held alert is recorded as suppressed), while a pause turns the check's components and gates to "no data". An agent cannot change a check's target (URL, method, headers, body): that is a different check.
The direction classifier
classifyCheckChange(before, after) reads the two states field by field and returns fires_more,
neutral or fires_less, with the flags assertionChanged and projectedRunsRaised. Any comparison
it cannot prove is no looser reads fires_less, because a wrong fires_less costs one human yes and a
wrong neutral is a silenced check nobody was asked about.
The check policy
GET/PUT /orgs/:org/check-policy (check_policy.update: human-only, org:admin, which only an
owner holds). previewSafeOrigins (origins where a preview may send any method), intervalFloorSeconds
(default 300: no agent-authored check runs more often), agentChecksPerProject (default 10). Every
write bumps version, and a plan names the policy it was made under. A proposal under the floor, at
an undeployed location, with more than five locations, or an eleventh agent-made check in a project is
refused before an approval exists.
Probing a public target as evidence: probe_run
When an agent is about to say "the site is down" or "the certificate is wrong", it can look from outside first.
probe_run (POST …/projects/:p/probe-run, then GET …/probe-run/:handle) runs one DNS, TCP, TLS or HTTP
(GET or HEAD) probe from the primary location against a public target and returns the phases, the failure
kind, TLS facts (protocol, expiry, subject, issuer), DNS answers from public resolvers, the status, a few
response headers and at most 2 KiB of the body. It waits up to 20 seconds for the worker and otherwise answers
with a handle. The api never dials: the probe is a job on bucker-workers.
What it sends. One request of the kind named. There is no field for a header, a body, a cookie or a secret,
and a path carries no query string. Every request carries x-bucker-check so a target's owner can recognise it.
No ICMP, no traceroute.
Which hosts. project:read may probe a host the project already names: a check's target, a deploy's URL
host, a host its client spans or its error events carried in the last 14 days. Any other host needs
synthetics:write. A host that resolves, directly or through a CNAME or a redirect hop, to a private, loopback,
link-local or cloud-metadata address answers one fixed message ("The target could not be resolved to a public
address") under every scope, and a DNS answer that carries such an address is not shown.
Limits. Port 25 is not probed (and a short deny list of file-sharing, RPC, UPnP and memcached ports); at most 3 distinct ports of a host a day per principal; 20 probes every 10 minutes per principal and 200 a day per organization, counted on the person an OAuth client acts for and on the client itself; a repeat of the same target inside 60 seconds is answered from the last result and spends nothing; an agent's probe is charged to its RUNS budget.
What comes back is data. Everything the target sent (error text, certificate names, DNS answers, the final
URL, the host a redirect left for, header values and the body) is scrubbed, injection-scanned, capped and listed
as untrusted. A body the scanner flags is returned as its length and SHA-256 only. The result is kept 24 hours
on the task and, when a check of the project names the host, as a result of that check with trigger probe,
which no uptime figure counts. One audit row per call records the host, the kind, the principal, the transport
and the OAuth client id.
Finding what no check covers: checks_coverage_get
checks_coverage_get reads what real traffic touches and no check exercises. It creates nothing,
asks nobody and schedules nothing. It reports three kinds of gap: server endpoints by route and method
(the top 90 percent of traffic, ranked by share, with the error rate), external hosts your app calls, and
journeys, which read not_available with the reason until journey data exists, never an empty list that
reads as "all covered".
- A gap carries a draft: the arguments of a
check_proposecall, to be previewed withcheck_preview_runfirst. It is built only for aGETorHEADroute without path parameters and a known origin, and only when every text in it came through the scrub unchanged; otherwise it is withheld with the reason. A proposal made from a gap runs one autonomy rung lower, because its text came from traffic. - With the request-rollup flag on (
pillar.apm_red_rollups) counts are the pre-sampling rollup (rollup_unsampled); otherwise they are the newest 100,000 stored server spans of seven days (stored_sampled, withtruncatedwhen the cap was hit). Traffic is cached an hour per project and environment; checks and suppressions are read fresh. checks_coverage_updatesuppresses one gap for at most 30 days (7 by default). A suppression hides a gap, so it asks a person; the gap stays listed with who suppressed it and until when, and ending it early isoperation_reverton that operation.
Short-lived probes: probe_monitors_create
probe_run looks once. probe_monitors_create starts a capped probe monitor of a host the project
already calls: HTTP GET or HEAD, TCP, TLS or DNS; an interval of at least 300 seconds; a lifetime of at
most 30 days (7 by default); a public host the project has called on at least three of the last fourteen
days; at most ten live probes per requester. There is no argument for a header, body, secret, location,
alert destination or status binding. A person approves it, unless the agent has earned the
probe_creation rung. It lands in shadow: nothing pages and nothing feeds a status page until a
person takes it live with a check approval, and it ends by itself at its lifetime. Stop it early with
operation_revert.
Healing a browser check: check_heal_propose
When a browser check fails because a button moved, check_heal_propose proposes moving the locator of
one click or fill step. It always asks a person; the check is not edited until a person approves a plan
bound to its digest, and the previous selector is the stored inverse. It refuses an assertion step or a
navigation, a replacement equal to the current selector, and a step with three heal proposals in a row that
nobody approved (that is reported as a check-maintenance warning instead). The replacement came from page
text an agent read, so the proposal never runs above ASK.
What a check saw: hydrate_check_run
An outage issue that a check opened advertises hydrate_check_run: for one run, the failing phase and
step, the probe's trace id, the failing regions, the deploys and change events nearest the window (with
their distance in minutes), one scrubbed response excerpt and the check's most recent on-demand re-runs.
The bundle is built by rules when the outage opens and read back; no model is asked. The excerpt is target
text, so it is served only to a token that also holds event:read; any other token gets its length and
SHA-256. The triage label reads unavailable until a model labelled it and is never a verdict.
checks_coverage_get sits in the synthetics toolset, and the tools here that write need synthetics:write. The holds on
the wider MCP surface were lifted by owner decision on 2026-11-16 without a signed-off security review of the
OAuth server; see MCP setup. For the infrastructure side, including restarts and rollbacks,
see Infrastructure.