Skip to content
bucker

Uptime monitoring

An uptime monitor probes a public URL on an interval and opens an issue when it has failed enough times in a row to rule out a blip. A monitor opens one issue per outage, adds nothing while it stays down, and resolves the issue when it recovers. A TLS certificate close to expiry is its own issue, because a site can be up with a certificate nine days from ending the party.

Create one in the console under Monitors → Uptime, or with POST /orgs/{org}/projects/{project}/uptime-monitors.

What counts as a failure

Every probe is judged in this order, and the first thing that fails names the failure:

  1. The request itself: a timeout, a refused connection, too many redirects.
  2. Expected status (default 200). Skipped when you assert on the status code yourself (below).
  3. Expected body contains (optional substring).
  4. Assertions (optional): the structured checks below.

failureThreshold (default 3) consecutive failures open the issue; recoveryThreshold (default 2) consecutive successes resolve it.

Assertions

A 200 with {"status":"degraded"} is down, and a substring cannot say so reliably. An assertion is a check on the response, and they combine with all, any and not:

{
  "all": [
    { "source": "json", "path": "$.status", "op": "eq", "value": "ok" },
    { "source": "latency_ms", "op": "lt", "value": 800 },
    { "not": { "source": "header", "name": "x-maintenance", "op": "exists" } }
  ]
}
source Fields op
status value (100-599) eq neq lt lte gt gte
latency_ms value (ms) lt lte gt gte
header name, and value unless the op is exists or not_exists exists not_exists eq neq contains
json path, and value unless the op is exists or not_exists exists not_exists eq neq lt lte gt gte contains
body value contains not_contains
  • JSON paths are $, .key, ['key'] and [0]: no wildcards, filters or recursive descent. The body is read up to 64 KiB; a longer body is cut, and a path into the cut part reports that.
  • Combinators nest up to four deep and 25 assertions in total.
  • No regular expressions. A pattern you supply would run against a response your server controls, which is a denial-of-service waiting for a bad pattern.
  • A tree that asserts on status replaces the single expected status, so "any 2xx" or "200 or 204" can be said. Otherwise the expected status and body checks still apply.

When one fails, the check's error names the assertion and what was seen: Assertion failed: json $.status = "ok", got "degraded". It appears in the result list and on the outage issue. What was seen comes from your server's response, so it is shortened and shown as text.

Set assertions to null on a PATCH to remove them. The console edits flat lists (all of / any of, each row optionally negated); a deeper tree saved through the API is shown as such and left as it is when you save the form.

Probe locations

One probe location cannot tell "your site is down" from "our probe's network blipped". With more than one, a monitor is DOWN only when at least two locations have each failed their threshold; a single location failing is recorded against that location and pages nobody. With a single location listed, that one is enough.

  • Pick locations with regions (create or PATCH); GET .../uptime-monitors/regions lists what your deployment runs. No regions means the primary location only, scheduled as it always was.
  • Each location probes on the monitor's own interval and keeps its own streaks, shown per monitor.
  • Every result carries the region it was probed from.
  • These are probe locations, not data residency. A probe stores nothing regional: every result is written to the one database, and the name says where the request came from, not where any data lives.

On Bucker's hosted service the primary location, us, is the only one today, so GET .../uptime-monitors/regions answers {"primary":"us","available":["us"]} and every monitor is probed from there. Further locations are added when customers need two-location quorum; when they are, they appear in that list and in the monitor form's location picker, and nothing about existing monitors changes until you list a location on them.

Operators. A location exists when a prober runs in it. Run a worker machine there with MONITORS_PROBE_REGION=<name> (default us) and list every location in UPTIME_PROBE_REGIONS (comma-separated) on the api, which refuses a monitor naming any other, since a location with no prober would be a monitor that is never checked there. Only the us machine probes monitors with no listed locations.

Trace propagation

By default each probe sends a W3C traceparent header, so a Bucker-instrumented target continues the probe's trace instead of starting an unrelated one. Each result records the traceId it sent: search it in your tracing tools to find the target's own trace for that check. Turn it off with propagateTrace: false if the target rejects unknown headers; a traceparent you set in the monitor's own headers wins.

Suggested monitors

Monitors → Uptime → Suggest monitors from my errors lists the hosts your recent errors were requesting, most frequent first. It suggests and never creates: choosing one opens the ordinary create form with the URL filled in, and nothing exists until you save it. Hosts that already have a monitor are left out, and so are internal names and private addresses. (The list comes from error events, which anyone holding a DSN can write; treat it as a convenience, not as evidence.)

Limits

  • Targets must be public. A probe starts from Bucker's network, so a monitor can watch only an absolute http:// or https:// address that the public internet can reach, with no credentials in it. localhost, internal-only names and private network addresses are refused when you save, and a name that resolves to a private address is refused before each probe, at every redirect too. For a service that is not reachable from the internet, a cron monitor that the service checks in to is the alternative.
  • At most 100 uptime monitors per project, disabled ones included. The next create is refused: "This project already has 100 uptime monitors; delete one before creating another".
  • At least 30 seconds between probes. intervalSeconds runs from 30 to 86,400 and defaults to 60; timeoutMs runs from 500 to 60,000 and defaults to 10,000.

On a status page

A status page component can be bound to an uptime monitor, and the monitor then decides the component's status: by default Major outage while it is DOWN and Operational while it is UP. With degradedAboveLatencyMs on the binding, a monitor that is up but averaged slower than that over its last five checks shows Degraded performance. A monitor that has not run yet, is disabled or was deleted contributes nothing, so its component shows No data, never Operational.

Monitors → Uptime → Show these on a status page appears once the project has an uptime monitor. It opens the status page setup with this project selected, where each web address you add becomes an uptime monitor in the project, a component on the page, and the binding between them in one request.

Owner and mute

Uptime monitors take the same ownerTeamId / ownerPrincipalId and mutedUntil as cron monitors: a notify_owner alert action routes the monitor's issues to its owner, and a mute (up to 30 days) holds the alerts back while the monitor keeps checking. Muted alerts are recorded as suppressed in the alert history, naming the mute.

Phase timing and failure kinds

Every check records how long each phase of the request took, for each request it made (the first and up to five redirects): DNS, connect, TLS (https only), first byte and download, plus the TLS and HTTP versions. A phase the request never reached is reported as not reached, not as 0 ms. The results panel shows them as a waterfall under Timing, with the figures beside the bars, and the rollup returns p50 and p95 per phase with the number of results they cover.

A failure names the phase it happened in:

Kind Meaning
TIMEOUT_CONNECT The TCP connection was not established in time
TIMEOUT_FIRST_BYTE Connected, but no response header arrived in time
TIMEOUT The deadline passed while resolving, or while the body was arriving
CONNECTION_REFUSED / CONNECTION_RESET The target refused, or dropped, the connection
TLS_HANDSHAKE The handshake failed or did not finish
TLS_CERT_EXPIRED / TLS_CERT_UNTRUSTED / TLS_HOSTNAME_MISMATCH The certificate is at fault
HTTP_PROTOCOL The reply was not valid HTTP
CONNECTION Any other connection failure, and a name that did not resolve
CONFIGURATION The check itself cannot run (see below)

A name that does not resolve and a name Bucker refuses to contact (a private address) read exactly the same, on purpose: telling them apart would let a monitor probe which internal names resolve from Bucker's network.

A check that cannot run is not an outage. A kind this deployment cannot run, a stored spec that no longer parses, request headers that no longer open, or a URL that is not a URL opens one check_maintenance warning on the project and records no result: the monitor's status, its failure streak and your uptime figures are left alone. The warning resolves itself the next time the check runs.

Every request carries x-bucker-check: <check id> beside the Bucker-Uptime/1.0 user agent, so you can recognise (or block) one check at your edge. It cannot be overridden by the monitor's own headers.

HTTP/2. Set httpVersion: "h2" when creating or updating a monitor to ask for HTTP/2 over https. It is a preference: a server that will not negotiate it is checked over HTTP/1.1 and the result says 1.1. The check's definition lives in its columns and, for this preference, in a versioned spec.

Reading checks from an agent

checks_list returns the project's checks with their last result (and the phase it failed in), 24 h uptime with its sample size and the share of expected checks that ran, owner and projected runs a month. check_get returns one check's definition (every secret as {{SECRET.NAME}}), its versions with digests, bindings and last five failures, and with include: ["results"] up to 30 days of runs with a cursor. Both say what they scanned and whether the answer was cut. Response text from your server (assertion values, URLs a redirect went to) is returned only to a token that also holds event:read. GET .../uptime-monitors/{slug}/versions returns the same change history over REST.

Agent-authored checks: preview, propose, change

An agent never creates or changes a check on its own. Three tools (and the matching REST routes) are the whole path, and a person decides every step that matters.

1. check_preview_run (POST …/uptime-monitors/preview-run) runs a full spec once from up to three deployed locations (today us) and returns the result, per-phase timings, warnings and a projection. It waits 20 seconds for the worker and otherwise returns a task handle; call it again with taskHandle (GET …/preview-run/:handle) to read it. The result is kept 24 hours as uptime_check_results rows with trigger preview, which no uptime figure counts.

A preview of a spec nobody has approved sends only what cannot change the target: GET, HEAD and OPTIONS. A POST, PUT, PATCH or DELETE, and any step that needs a secret variable, comes back not_run: needs approval and is listed, so the approver knows which steps first run live. An origin on the organization's check policy (previewSafeOrigins, edited only by a person) runs in full, except that a preview of a mutating method never follows a redirect (the probe repeats the method and body on every hop, so a 307 off the safe origin would replay the POST there); it reports REDIRECT_NOT_FOLLOWED instead, and the live check follows it as configured. A preview never resolves a secret variable, and an existing check's own sealed headers are never sent. The target and every redirect hop go through the same guarded lookup as a scheduled check: a private, loopback, link-local or cloud-metadata address sends nothing and answers "The target could not be resolved to a public address". Warnings: the target answers 403 to the Bucker user agent, a redirect leaves the host, the certificate expires within 14 days.

The projection is ⌈2,592,000 / interval⌉ × locations × (1 + retry rate) runs a month, with the retry rate on its own line. An HTTP check is unmetered.

2. check_propose (POST …/uptime-monitors/propose) opens the approval that takes a previewed spec live (check.create). It is ask-floored: no policy row and no review mode applies it, so it only ever returns an approval to decide. It is refused without a preview of the exact spec under 24 hours old, and the check stays an inert PROPOSED shell (origin AGENT, never scheduled) until a person approves. The plan shows the check with its secrets as references, the classifier's verdict, the projection and the steps that were not previewed; the approver's yes is bound to the spec's fingerprint, so what runs is what was previewed. A go-live's inverse is a pause; only a person deletes a check (check.delete is human-only and has no tool).

3. check_update (and PATCH for an agent, which takes reason and approvalRequestId) states a desired end, and the engine derives the act from the before-and-after diff:

The diff Act Decision
a neutral tuning change (an interval or timeout within 10 %, nothing else) check.update an org ALLOW row applies it only when it changes no assertion or location, raises projected runs by under 10 % and touches no gate, release or status setting; otherwise it asks, whatever ALLOW row exists (the registry's conditional ask floor)
a change that makes the check fire more (an added assertion, a lower threshold) check.update asks
a change that makes it fire less (a looser or removed assertion, a higher threshold or failureThreshold, a longer interval, fewer locations, a relaxed executionRule) check.narrow ask floor, a TTL of at most 7 days, the earlier values come back at its end, CRITICAL behind a status component
muted: true / false check.mute / check.unmute a mute asks and ends by the clock; an unmute is allowed
paused: true / false check.pause / check.resume a pause asks (ask floor), ends at its TTL, CRITICAL behind a status component

Prefer a mute to a pause: a mute keeps the evidence (runs, issues and status continue and each held alert is recorded as suppressed), while a pause turns the check's components and gates to "no data". An agent cannot change a check's target (URL, method, headers, body): that is a different check.

The direction classifier

classifyCheckChange(before, after) reads the two states field by field and returns fires_more, neutral or fires_less, with the flags assertionChanged and projectedRunsRaised. Any comparison it cannot prove is no looser reads fires_less, because a wrong fires_less costs one human yes and a wrong neutral is a silenced check nobody was asked about.

The check policy

GET/PUT /orgs/:org/check-policy (check_policy.update: human-only, org:admin, which only an owner holds). previewSafeOrigins (origins where a preview may send any method), intervalFloorSeconds (default 300: no agent-authored check runs more often), agentChecksPerProject (default 10). Every write bumps version, and a plan names the policy it was made under. A proposal under the floor, at an undeployed location, with more than five locations, or an eleventh agent-made check in a project is refused before an approval exists.

Probing a public target as evidence: probe_run

When an agent is about to say "the site is down" or "the certificate is wrong", it can look from outside first. probe_run (POST …/projects/:p/probe-run, then GET …/probe-run/:handle) runs one DNS, TCP, TLS or HTTP (GET or HEAD) probe from the primary location against a public target and returns the phases, the failure kind, TLS facts (protocol, expiry, subject, issuer), DNS answers from public resolvers, the status, a few response headers and at most 2 KiB of the body. It waits up to 20 seconds for the worker and otherwise answers with a handle. The api never dials: the probe is a job on bucker-workers.

What it sends. One request of the kind named. There is no field for a header, a body, a cookie or a secret, and a path carries no query string. Every request carries x-bucker-check so a target's owner can recognise it. No ICMP, no traceroute.

Which hosts. project:read may probe a host the project already names: a check's target, a deploy's URL host, a host its client spans or its error events carried in the last 14 days. Any other host needs synthetics:write. A host that resolves, directly or through a CNAME or a redirect hop, to a private, loopback, link-local or cloud-metadata address answers one fixed message ("The target could not be resolved to a public address") under every scope, and a DNS answer that carries such an address is not shown.

Limits. Port 25 is not probed (and a short deny list of file-sharing, RPC, UPnP and memcached ports); at most 3 distinct ports of a host a day per principal; 20 probes every 10 minutes per principal and 200 a day per organization, counted on the person an OAuth client acts for and on the client itself; a repeat of the same target inside 60 seconds is answered from the last result and spends nothing; an agent's probe is charged to its RUNS budget.

What comes back is data. Everything the target sent (error text, certificate names, DNS answers, the final URL, the host a redirect left for, header values and the body) is scrubbed, injection-scanned, capped and listed as untrusted. A body the scanner flags is returned as its length and SHA-256 only. The result is kept 24 hours on the task and, when a check of the project names the host, as a result of that check with trigger probe, which no uptime figure counts. One audit row per call records the host, the kind, the principal, the transport and the OAuth client id.

Finding what no check covers: checks_coverage_get

checks_coverage_get reads what real traffic touches and no check exercises. It creates nothing, asks nobody and schedules nothing. It reports three kinds of gap: server endpoints by route and method (the top 90 percent of traffic, ranked by share, with the error rate), external hosts your app calls, and journeys, which read not_available with the reason until journey data exists, never an empty list that reads as "all covered".

  • A gap carries a draft: the arguments of a check_propose call, to be previewed with check_preview_run first. It is built only for a GET or HEAD route without path parameters and a known origin, and only when every text in it came through the scrub unchanged; otherwise it is withheld with the reason. A proposal made from a gap runs one autonomy rung lower, because its text came from traffic.
  • With the request-rollup flag on (pillar.apm_red_rollups) counts are the pre-sampling rollup (rollup_unsampled); otherwise they are the newest 100,000 stored server spans of seven days (stored_sampled, with truncated when the cap was hit). Traffic is cached an hour per project and environment; checks and suppressions are read fresh.
  • checks_coverage_update suppresses one gap for at most 30 days (7 by default). A suppression hides a gap, so it asks a person; the gap stays listed with who suppressed it and until when, and ending it early is operation_revert on that operation.

Short-lived probes: probe_monitors_create

probe_run looks once. probe_monitors_create starts a capped probe monitor of a host the project already calls: HTTP GET or HEAD, TCP, TLS or DNS; an interval of at least 300 seconds; a lifetime of at most 30 days (7 by default); a public host the project has called on at least three of the last fourteen days; at most ten live probes per requester. There is no argument for a header, body, secret, location, alert destination or status binding. A person approves it, unless the agent has earned the probe_creation rung. It lands in shadow: nothing pages and nothing feeds a status page until a person takes it live with a check approval, and it ends by itself at its lifetime. Stop it early with operation_revert.

Healing a browser check: check_heal_propose

When a browser check fails because a button moved, check_heal_propose proposes moving the locator of one click or fill step. It always asks a person; the check is not edited until a person approves a plan bound to its digest, and the previous selector is the stored inverse. It refuses an assertion step or a navigation, a replacement equal to the current selector, and a step with three heal proposals in a row that nobody approved (that is reported as a check-maintenance warning instead). The replacement came from page text an agent read, so the proposal never runs above ASK.

What a check saw: hydrate_check_run

An outage issue that a check opened advertises hydrate_check_run: for one run, the failing phase and step, the probe's trace id, the failing regions, the deploys and change events nearest the window (with their distance in minutes), one scrubbed response excerpt and the check's most recent on-demand re-runs. The bundle is built by rules when the outage opens and read back; no model is asked. The excerpt is target text, so it is served only to a token that also holds event:read; any other token gets its length and SHA-256. The triage label reads unavailable until a model labelled it and is never a verdict.

checks_coverage_get sits in the synthetics toolset, and the tools here that write need synthetics:write. The holds on the wider MCP surface were lifted by owner decision on 2026-11-16 without a signed-off security review of the OAuth server; see MCP setup. For the infrastructure side, including restarts and rollbacks, see Infrastructure.