Skip to content
bucker

Alert rules

Who is told about a fire (routing policies), how a burst becomes one notification (correlation groups), SLO budgets and burn alerts, and finding what nothing watches are on Alert routing, correlation groups, SLOs and coverage.

Bucker has one alert router. An issue rule watches issues (a new issue, a regression, a count of events on one issue, a lifecycle change). A metric rule watches an aggregate over a window. Both kinds of rule reach the same channels, use the same debounce ledger, and write the same alert-event history. A rule's type picks what it watches. It does not pick a separate delivery system.

Rules live under a project:

POST   /orgs/{org}/projects/{project}/alert-rules
GET    /orgs/{org}/projects/{project}/alert-rules
PATCH  /orgs/{org}/projects/{project}/alert-rules/{ruleId}
DELETE /orgs/{org}/projects/{project}/alert-rules/{ruleId}

In the dashboard, open Alerts › Rules.

New user feedback

The feedback_received condition (New user feedback in the editor) fires when a user report linked to the issue arrives and is not filed as spam. It is its own signal, like a lifecycle change. It never fires on telemetry, telemetry never fires it, and it cannot be combined with first_seen, regression, event_count, user_count or issue_lifecycle in one rule. The usual debounce applies, so a burst of reports pages once per cooldown. A report that arrives before its error is grouped pages as soon as it is linked to the issue. See User feedback.

{ "type": "issue", "conditions": [{ "id": "feedback_received" }], "actions": [...] }

A new survey answer

The survey_answer_received condition fires once per newly stored answer of one survey on a project issue rule. It takes { survey, question?, score?: { comparator, value }, choices? }: survey is the survey's slug, question narrows to one question, score compares a numeric answer and choices lists choice keys. Pacing is one cooldown per rule, survey and question, so a flood pages once per cooldown. It cannot be combined with other issue conditions or filters, and it is refused on an org-wide rule. The answer text attached to a notification is untrusted: it is sanitized, escaped in Slack and email, replaced by a placeholder when quarantined and withheld in the narrower health data modes. A rule written before its survey is launched is allowed; a slug that matches nothing simply never fires. See Surveys.

{ "type": "issue", "conditions": [{ "id": "survey_answer_received", "survey": "nps-q4", "score": { "comparator": "lte", "value": 6 } }], "actions": [...] }

Org-wide rules

An org-wide rule covers every project in the organization, including projects created after the rule. "Page me on any new FATAL in production, any project" is one rule, not one rule per project.

POST   /orgs/{org}/alert-rules
GET    /orgs/{org}/alert-rules
GET    /orgs/{org}/alert-rules/{ruleId}
PATCH  /orgs/{org}/alert-rules/{ruleId}
DELETE /orgs/{org}/alert-rules/{ruleId}
  • Same DSL. The body is the same as for a project rule. Bucker evaluates the rule separately for each project it covers. Each project gets its own cooldown and its own alert events, so an org-wide metric rule that breaches in one project pages for that project alone.
  • A project filter limits coverage. { "id": "project", "slugs": ["sandbox"], "match": "not_in" } skips a project. "match": "in" covers only the listed projects. A project rule cannot carry this filter.
  • Owners and admins only. Creating, editing or deleting an org-wide rule needs write on the organization. Anyone who can read the organization can list the rules.
  • Restrictions. An org-wide rule can only notify org-wide channels. It cannot use the event anomaly condition, because that condition's baseline is learned per project. Use telemetry_anomaly instead, or a project rule.

In the dashboard, an org-wide rule appears in every project's rule list, marked "all projects". The rule editor offers Applies to › Every project in the organization to owners and admins.

What a metric rule can watch

A metric rule has exactly one watched condition:

Condition Watches Threshold
metric error events: event_count, user_count, issue_count typed
anomaly error events, as an hourly series learned
telemetry spans, logs or custom metrics: any Explore aggregate, or a counter's delta or rate typed
telemetry_anomaly spans, logs or custom metrics, as an hourly series learned

Spans and logs

A telemetry condition is an Explore query that has a threshold. It uses Explore's filter operators, aggregate functions and attribute keys. Bucker compiles the condition with the same builder that draws the Explore chart, so the chart and the alert read the query the same way.

"p95 of checkout spans above 2 s for 10 minutes":

{
  "name": "Slow checkout",
  "type": "metric",
  "conditions": [
    {
      "id": "telemetry",
      "dataset": "spans",
      "query": [{ "key": "span.name", "op": "eq", "value": "checkout" }],
      "aggregate": "p95",
      "field": "span.duration",
      "comparator": "gt",
      "threshold": 2000,
      "windowMinutes": 10
    }
  ],
  "actions": [{ "kind": "notify_channel", "channelId": "…" }]
}

"More than 50 ERROR logs from billing in 5 minutes":

{
  "id": "telemetry",
  "dataset": "logs",
  "query": [
    { "key": "log.severity", "op": "gte", "value": "error" },
    { "key": "service.name", "op": "eq", "value": "billing" }
  ],
  "aggregate": "count",
  "comparator": "gt",
  "threshold": 50,
  "windowMinutes": 5
}
  • Datasets. spans, logs and metrics (see below).
  • Aggregates. count, count_unique, sum, avg, min, max, p50, p75, p95 and p99. Every aggregate except count needs a field.
  • Operators. eq, neq, contains, not_contains, starts_with, gt, gte, lt, lte, exists and not_exists. Filters in query are combined with AND.
  • Span keys. These are the same keys Explore uses: span.name, span.kind, span.status, span.duration (in milliseconds), environment, release, resource.* and any attribute name.
  • Log keys. These are log.severity, log.severity_number, log.body, trace_id, environment, release and any attribute name. log.severity gte error compares OTel severity numbers, so warn counts as less severe than error. Log records have no span.* or resource.* fields. Bucker refuses a rule that names one when you save it, not later when the rule runs.
  • Rule filters. A telemetry rule takes only environment and release filters. A span has no event level, so put severity or status in the query instead.
  • No data is not a breach. An avg or p95 over an empty window has no value. The rule reads "no data" and does not fire, even if its comparator is lt. A count over an empty window is a real zero.
  • Time. The window is measured on the span's start time or the log record's timestamp. Event-count metrics are measured on arrival time instead.

Custom metrics

With "dataset": "metrics", the condition watches metrics your services send over OTLP /v1/metrics. The rule has to name the metric: a metric.name eq filter is required, and Bucker refuses the rule at save time without one. The other keys are Explore's metrics dataset: metric.value (a gauge or sum point), metric.count, metric.sum, metric.min, metric.max, metric.unit, metric.kind, service, environment, release and any attribute.

The aggregate is always explicit. Gauges and sampled values use the ordinary aggregates over the window: "queue depth above 100 at any point in 10 minutes" is aggregate: "max", field: "metric.value", and "average cache hit rate under 80%" is avg. p95 is the 95th percentile of the stored point values, not of a histogram's buckets; for a histogram percentile use histogram_p95 (below).

A counter needs one more word. Summing the stored values of a cumulative counter adds up running totals, which is not a number anyone wants. Set counter to read it properly:

counter Value
delta The increase over the window.
rate The increase per second: the delta divided by the window length.

Per series, a point with delta temporality is its own increase. A cumulative point is its difference from the previous point, and a drop is read as a restart, so the new value is the increase. The increases of all matching series are then summed. The first point of a series that has no earlier point within 15 minutes contributes nothing. counter needs "aggregate": "sum" (no field, or metric.value) and applies only to monotonic sum metrics; up-down counters are read as gauges.

"More than 5 errors per second for 10 minutes":

{
  "id": "telemetry",
  "dataset": "metrics",
  "query": [{ "key": "metric.name", "op": "eq", "value": "http.errors" }],
  "aggregate": "sum",
  "counter": "rate",
  "comparator": "gt",
  "threshold": 5,
  "windowMinutes": 10
}

Counters and histograms by name. The aggregates counter_increase, counter_rate, histogram_p50, histogram_p75, histogram_p90, histogram_p95, histogram_p99, histogram_mean and histogram_rate are the same functions Explore offers on the metrics dataset, and a rule and the chart beside it compute one number one way. counter_rate and counter_increase are "aggregate": "sum" with counter rate and delta under one word; the histogram functions merge the stored buckets, so "aggregate": "histogram_p95" is the 95th percentile of the observations, not of the per-point totals. They need the metrics dataset and a named metric, take no field and no counter, and are refused otherwise. A percentile reads "no data", and does not fire, below 20 observations or when the series behind it use different bucket layouts (the rule's value is then "no data"). A learned threshold (telemetry_anomaly) trains on an hourly series, so it accepts counter_increase but not counter_rate or the histogram functions, and a backtest of a histogram rule is refused.

The rule editor has a Custom metrics option under Watch. It seeds the metric.name filter and offers a Read the metric as choice (stored points, counter increase, counter rate). Grouping, warning thresholds, comparison with an earlier period, recovery messages and the backtest work as they do for spans and logs. A window with no points reads "no data" and does not fire.

telemetry_anomaly takes the same dataset, query, aggregate and field, plus the event anomaly's sensitivity, direction and minValue. Bucker scores the last complete hour against an hour-of-week baseline. It learns that baseline from the last four weeks of the same query. The learned threshold switches on only after four weeks of history. Until then, the rule's reason says when it will switch on.

Warning, recovery, percent change and fan-out

The metric and telemetry conditions accept four optional fields. A rule that leaves them out behaves as it did before they existed.

Field Effect
warningThreshold A second line on the same comparator. It must be crossed before threshold is. Crossing it fires a warning. Crossing threshold fires critical, even inside the cooldown, because the alert has escalated. Going from critical back to warning is recorded but not announced.
notifyOnRecovery When a warning or critical series returns to normal, Bucker sends a resolved message. The message goes out even inside the cooldown. On a PagerDuty channel the incident gets a resolve. telemetry_anomaly accepts this field too.
comparison { "kind": "previous_period", "offsetMinutes": 1440 } compares the window with the same window that many minutes earlier. The value and the thresholds then count as percent change. If the earlier window held nothing, the rule reads "no baseline" and does not fire.
groupBy { "by": "environment" }, { "by": "release" } or { "by": "tag", "key": "region" } fans the rule out over that key. Each value gets its own cooldown, state and recovery. On a span or log rule, tag names an attribute. A rule evaluates at most 50 groups per pass, the busiest first, and its reason says when it cut the rest.

A group that goes quiet still recovers. If an environment stops erroring completely, it drops out of the query. Bucker still evaluates every group that is breaching or warning in its ledger, so that environment's incident gets its resolved message.

{
  "id": "metric",
  "aggregate": "event_count",
  "windowMinutes": 10,
  "comparator": "gt",
  "threshold": 200,
  "warningThreshold": 100,
  "comparison": { "kind": "previous_period", "offsetMinutes": 1440 },
  "groupBy": { "by": "environment" },
  "notifyOnRecovery": true
}

The webhook payload carries severity, recovered, group and comparison in alert.metric. See alert-webhooks.md.

Maintenance windows and mutes

A maintenance window keeps alert notifications quiet during planned work. It covers alert rules, the anomaly detector, and uptime and cron monitors, because monitors raise their alerts through the same router. A window covers one project, or every project in the org, including projects created later.

GET    /orgs/{org}/maintenance-windows
POST   /orgs/{org}/maintenance-windows        { name, startsAt, endsAt, projectSlug?, reason? }
POST   /orgs/{org}/maintenance-windows        { name, statusMaintenanceId }   # follow a status-page maintenance
POST   /orgs/{org}/maintenance-windows/{id}/end
DELETE /orgs/{org}/maintenance-windows/{id}
  • Suppressed, never dropped. Bucker still records every fire a window covers. It is written to the alert history as an alert event with status SUPPRESSED, and its suppression field names the window. Nothing is delivered. The fire still counts for the rule's cooldown, so if a condition is still true when the window ends, the alert goes out when the cooldown next expires.
  • Following a status page. A window bound to a status-page maintenance follows that maintenance's state. It is quiet while the maintenance runs, or while it is scheduled and inside its times. When the maintenance is marked complete, the quiet ends too.
  • Bounds. A window lasts at most 14 days. A window over one project needs write on that project. A window over the whole org needs write on the org.

A rule mute silences one rule until a given time:

POST   /orgs/{org}/projects/{project}/alert-rules/{ruleId}/mute   { minutes | until, scope: "rule" | "me" }
DELETE /orgs/{org}/projects/{project}/alert-rules/{ruleId}/mute?scope=rule|me
  • scope: "rule" mutes the rule for everyone. Its fires are recorded as SUPPRESSED. It needs project write and writes an alert_rule.mute audit row.
  • scope: "me" keeps the rule firing for everyone else. Only your own address is left out of its email deliveries.
  • A mute lasts at most 30 days and ends by itself. The dashboard's rule menu offers 30 m, 1 h, 4 h and 24 h. A Slack alert card offers Mute rule 1h and Mute rule 4h.

Silencing by an agent or a delegated client (the plan shape)

A person's own mute, window and edit stay direct, within the limits above. An agent, a service or a person's OAuth client silences through a governed plan: the same routes answer with a plan, a person decides it, and the change applies once, against the revision the person saw.

POST /orgs/{org}/projects/{project}/alert-rules/{ruleId}/mute      { minutes, reason, approvalRequestId? }
DELETE …/alert-rules/{ruleId}/mute?reason=…&approvalRequestId=…    # an unmute: usually applies at once
POST /orgs/{org}/maintenance-windows    { name, durationMinutes, startInMinutes?, recurrence?, scope?, projectSlug, reason, approvalRequestId? }
POST …/alert-rules/{ruleId}/edit        { patch, reason, minutes?, approvalRequestId? }
POST …/alert-rules/silencing/preview    { ruleId, mute | unmute | disable | enable | narrow }
  • Every silence has an end. A mute, a disable and a narrowing state minutes (at most 7 days for an agent, 30 for a person); a window states durationMinutes (at most 72 hours for an agent, 14 days for a person). The operation expires with the silence and the TTL sweeper reverts it: an unmute, an enable, or, for a narrowing, the stored before values.
  • The act is read from the diff, never from an argument. An edit that makes a rule fire less (a higher threshold on a greater-than rule, a longer window, one more filter, a longer cooldown, one fewer destination, enabled: false) is alert_rule.narrow or .disable and needs minutes; anything else is alert_rule.update. A mixed edit reads as the riskier half. Ending a silence (unmute, enable, ending or cancelling a window) restores alerting and is ELEVATED.
  • The class follows what is covered. A silence over a paging destination (PagerDuty, Opsgenie, incident.io, JSM, ...) for more than four hours, or over an org-wide rule or window, is CRITICAL, which no agent is allowed by default. A mute of a rule that pages is held at ASK by review mode whatever the ceiling.
  • A rule edit carries its backtest. The door replays the rule before and after over the last seven days (the Backtest section below) and binds the counts into the plan an approver reads. A change with no backtest is refused before an approval request exists. The whole result is stored on the request; a backtest more than an hour old at decision time closes the request TARGET_CHANGED and the next call recomputes it. Backtests are limited to ten per principal per ten minutes, and an update costs two.
  • Windows can recur and be scoped. recurrence is an RRULE in UTC (at most twelve occurrences over ninety days, each at most fourteen days). scope narrows a window to rules carrying one of its tags and watching one of its environments; a rule that watches every environment is not covered by a window for one of them. A recurring window's time-box runs to the end of its LAST occurrence, so an agent (72 hours) cannot open a series longer than that. Ending or cancelling a recurring window ends the whole series, and its end cannot be moved (that would resize every occurrence): end it and create another.
  • The plain doors are not a way around the plan. An agent's or a delegated client's PATCH of a rule that makes it fire less is refused (SILENCING_EDIT) with the governed form above; one that makes it fire more is unchanged. DELETE of a rule by an agent or a delegated client is refused (HUMAN_ONLY).
  • Deleting a rule is a seven-day tombstone. The row, its history and its evaluations stay; every list, read and evaluation pass skips it; a person restores it (alert_rule.restore, asked for) inside the week. Deleting stays a person's act.
  • Ending a silence always runs. At its end time the sweeper ends the silence even when a person edited the rule meanwhile (a rename, a cooldown), as long as what the operation set is still in force. A field a person has since rewritten is theirs: the sweeper leaves it and asks.
  • Disabling. A person's enabled: false states its own end (30 days) in disabledUntil; a rule listed with include: ["windows"] over MCP shows mutedUntil, disabledUntil, its lifecycle and the windows covering it.

Backtest: how often would this rule have fired?

Before you save a rule, the editor's Backtest panel replays it over the project's last 7 days. The API is:

POST /orgs/{org}/projects/{project}/alert-rules/backtest   { "rule": { …the rule… }, "days": 7 }

The replay uses the live evaluator. Bucker turns the stored history into the points the live pass would have evaluated: a metric value per group at each step, or an issue signal for each new issue, regression, lifecycle change and event. It runs each point through the same rule code, with the rule's own cooldown kept in memory. Nothing is written and nothing is delivered. The call needs only read on the project.

The response gives the count and its denominators:

Field Meaning
fires Notifications after the cooldown. This is the headline number.
matches Evaluations that met the rule. This is what would have fired with no cooldown.
suppressedByCooldown Matches the cooldown swallowed.
evaluations Points or signals replayed.
recoveries Resolved messages, for a rule with notifyOnRecovery.
delivery.chatMessagesPerDestination Messages each Slack, Discord, Teams or chat-webhook destination would have received after the chat volume ceiling (12 per 5 minutes, then one rollup).
fireTimes, firesByDay When the fires would have happened.
approximations Each way the replay is coarser than the live pass.

The replay is coarser than the live pass in a few ways, and the response lists the ones that apply:

  • A metric rule is evaluated once per window (5 to 60 minutes) rather than every minute, so a breach shorter than that can be missed.
  • An issue's status is taken from the signal (new, regressed, ongoing), not from its stored history, so mutes that have since ended are not replayed.
  • At most 20,000 events are replayed.
  • The event anomaly condition is not backtested, because its baseline lives in the detector's stored profile. telemetry_anomaly is replayed hour by hour.

Is my rule being evaluated?

"No alert fired" can mean nothing breached, or nothing was looked at. Bucker records the difference, for each organization and each evaluator, and shows it next to the rule.

The rules list (GET /orgs/{org}/projects/{project}/alert-rules, and the org-wide list) returns an evaluation block per rule:

Field Meaning
state ok, lagging, broken or unknown.
lastEvaluatedAt When the evaluator last evaluated this rule (refreshed at most every five minutes while healthy). A rule with no record of its own reads your organization's newest pass instead.
lastOutcome, consecutiveFailures How the rule's own last evaluation ended (ok, or the failure class) and how many failures in a row it has had.
lagMs How far behind the oldest due rule was. A rule due every minute and reached every minute has none.
quarantined, brokenSince The rule is set aside, which is what broken means: there is no second threshold. It is not being evaluated, and is never reported as OK. brokenSince is when this episode began.
errorClass Why, in a closed vocabulary: timeout, dependency, invalid_query, bad_config, evaluation_error or dispatch_error. Never the error text, which can carry your query.
quarantinedSince, nextRetryAt When it was set aside and when it is next tried.

lagMs is per organization for the rule's evaluator, not per rule. The quarantine and the rule's own lastEvaluatedAt, lastOutcome and consecutiveFailures are per rule. lagging means the evaluator is more than three passes behind for your organization. For a metric rule it also means the evaluator has written nothing for three passes, because silence is not health. unknown means nothing has reported on the rule: it is disabled, or your organization has had no pass in 30 days. Anomaly rules report too: the anomaly pass uses the same ledger, so a learned-threshold rule that keeps failing is broken like any other.

Per-organization batch. One organization has at most 100 metric rules, 100 anomaly rules, and 100 issue rules per issue evaluated in a pass. The rest wait and the window rotates, so each rule is reached within ceil(n / 100) passes. A big backlog in one organization costs that organization lag and no one else's.

Quarantine, which is BROKEN. A rule that throws on its own account three passes in a row is set aside. It reads broken (not_evaluated), its owner is told through the rule's own actions (an ordinary alert delivery, with its retries and receipts), and it is tried again every 15 minutes. The rule's creator and the organization's owners also get one alert_rule_broken notification per episode, in the inbox and by email unless they turned it off. A rule that is still failing does not repeat it; one that recovers and breaks again does, as a new episode. Neither notice is held back by a mute or a maintenance window: a rule that is not running is not something a silence on its alerts should hide. The delivered notice records the silence it passed in its suppression field (kind: "quarantine_notice", overrode: "rule_muted" | "maintenance" | "monitor_muted"). It comes back on its first successful pass, when you edit it, or when you retry it:

POST /orgs/{org}/projects/{project}/alert-rules/{ruleId}/retry-evaluation

The retry needs write on the project and a person: an agent fixes the rule through its own update instead, which releases the quarantine. It answers 409 for a rule that is not quarantined. If the rule still fails, it is quarantined again after three more passes.

Only a rule's own failure counts. If every rule of your organization fails in a pass, or the database times out, the failure is recorded against the evaluator and no rule is set aside. Otherwise one database incident would quarantine every tenant at once.

A rule that no longer parses is listed with definitionValid: false and empty conditions, filters and actions, and its broken state beside it. One such rule never fails the list.

No lost page. A fire is claimed on the cooldown ledger and then delivered. If delivery throws, Bucker writes the alert deliveries it was about to make as FAILED rows due now, and the ordinary retry pass sends them: the page arrives, once, on the next pass. Only if those rows cannot be written either is the claim released, so the next pass fires the alert afresh. This holds for issue, metric and anomaly rules alike.

What was decided, and when. Each fire and each failure is recorded, plus about one in ten of the other evaluations, in alert_evaluations (value, threshold, outcome, and a minimal reason), kept 14 days. It is the evidence behind "why did this fire" and "why did it not". Every change that moves what a rule would fire or who hears of it is recorded in alert_rule_state_changes, with the values before and after: a mute or unmute (from the API, Slack, or your own "mute for me"), a maintenance window starting, ending or being deleted (one row per rule it covers), an edit that enables or disables a rule or changes its conditions, filters, actions or cooldown (from the dashboard, the API, bucker.yml or an agent), and a rule created by chat setup.

Notices that have no rule. Usage, spend and spike notices write a delivery receipt in the same shape as an alert delivery (status, attempts, lastError, deliveredAt). A send that reached nobody is recorded as FAILED and is not retried, because the fact it reports is on the billing page.

Archive integrity. Each time the retention pass archives a day, it compares every archive with its recorded row count and checksum and with its manifest, before the day's hot partition is dropped, and writes an archive_integrity row.

When a group stops reporting: NO_DATA

A group that is breaching or at warning and then stops reporting is not a group that recovered. For a rule over a value that can be absent (an average, a percentile, a rate, a percent change with no baseline) the evaluator holds such a group as NO_DATA on the rule's ledger and does not resolve it. The ledger's states are one closed set: OK, WARNING, BREACHING, NO_DATA.

  • Held, not resolved. Entering NO_DATA sends nothing and closes nothing downstream. It writes the ledger row once; staying there writes nothing. A rule that asked to be told of recoveries is not told, because nothing recovered.
  • For the rule's retention. noDataGroupRetentionMinutes (1 to 10,080, default 60) is how long the ledger keeps the quiet group. When it runs out the ledger forgets the group, still without a "resolved" message: a person, or the data returning, ends the incident. You can set it on a rule (API, bucker.yml, the rule editor for metric rules) and clear it with null.
  • Data comes back. If it is healthy, a rule with notifyOnRecovery sends a recovery ("recovered from a gap in data"). If it is still breaching, the cooldown decides: inside it nothing is sent (the page nobody resolved is not repeated), past it you get the ordinary reminder.
  • What does not change. A count with no matches is a real zero, so an empty event_count group still recovers. A group that was never open stays OK on no data. A learned (anomaly) threshold has its own "no data" verdict and is unaffected.

The rules list shows each rule's open groups (breaching, warning, no data) and GET .../alert-rules/{ruleId}/state lists them with when a quiet group will be forgotten. GET .../alert-rules?state=NO_DATA narrows the list (OK means a rule with none open).

Shadow mode: agent-made rules deliver nothing until a person says yes

A rule an agent writes over MCP (alert_rules_create) or over REST as an agent, a service or a delegated client starts in SHADOW. A person's own session, the dashboard, still writes a LIVE rule. The lifecycle is stored on the rule (SHADOW, LIVE or EXPIRED) and is on every rule response and in alert_rules_list: it is never a tool argument, and the plain edit routes refuse a lifecycle key.

  • It is evaluated like a live rule and delivers nothing. Every fire writes one SUPPRESSED alert event per action with suppression = {kind: "shadow"}. A mute, a maintenance window and the "rule is not running" notice cannot change that: one gate, in the router, lets only a LIVE rule deliver, so a rule that is somehow EXPIRED and still enabled delivers nothing either. The alert_volume widget counts these rows as their own SHADOW series, not as pages a mute swallowed. Shadow evaluation counts against the same per-organization evaluation caps as a live rule.
  • At most 20 shadow rules per project per agent. The 21st is refused before a person is asked. The rule records who authored it, the delegation chain, the grant the call stood on (authorGrant, a person-issued tool grant naming only alert_rules_create is how an agent is held to drafts) and the plan digest.
  • The record. GET …/alert-rules/:ruleId/shadow and the promote dialog show what the rule did on real traffic: the fires it would have delivered, recoveries, groups, the destinations it would have reached (and which page a person), the fires the cooldown swallowed (an exact floor, plus an estimate from the sampled evaluations), and the MON-01 backtest it was proposed with, beside the actual fires per day.
  • Promotion is alert_rule.promote, derived from a change of lifecycle: alert_rules_update with lifecycle: "LIVE", or the promote button. It needs at least one day of evidence and is refused while a Monitor Quality blocking key (missing_owner, missing_route, rule_broken) is set, read live. For an agent it is asked, with the evidence on the card and its digest on the audit row, and a rule that carries a paging destination is always ask-floored. A person promotes from their own session directly.
  • Demotion puts a live rule back into shadow for a stated time (lifecycle: "SHADOW" with minutes, at most seven days for an agent and thirty for a person): alert_rule.demote, silencing-class and CRITICAL on a rule that pages a person or burns an SLO budget. The rule is promoted again by itself at the end.
  • Editing a shadow rule is ELEVATED, and an edit that makes it fire less needs no end time: nothing is delivered either way. Adding a paging destination still asks.
  • A shadow rule nobody promotes lapses after 14 days: it becomes EXPIRED, disabled and archived under the seven-day tombstone, with a state-change row and an audit row (alert_rule.shadow_expired) as the receipt, and window_ended goes to the sponsor. After three days of evidence the sponsor gets a shadow_report_ready notice, unless a promotion request is already open. A rule that was LIVE and was demoted is never archived.

Monitor Quality: a daily score per rule, and a gate for agent-made rules

Every enabled rule gets a score from 0 to 100 once a day, computed by the alerts scheduler from rows Bucker already stores. There is no model and no estimate: each key is a published rule, and the thresholds are versioned as ruleSetVersion (monitor_quality_v1), so a changed definition is visible on the rows it produced. The rules list shows the score and its keys beside each rule name; GET …/alert-rules/quality and the MCP tool monitor_quality_get return the same rows, worst first.

Key Set when Points off
muted_over_60_days one continuous mute (back-to-back mutes are one) has run for 60 days or more and is still in force 15
stuck_in_alert_over_60_days a metric rule has a BREACHING or WARNING group open whose first unrecovered fire since the last recovery is 60 days old 15
noisy_top_5_percent among the noisiest 5 % of the project's rules by delivered fires over 10 days (at least 10 fires) 10
missing_owner no resolvable owner (below) 20, blocking
missing_route no destination the rule can deliver to 30, blocking
broken_channel a destination is disabled, gone, or has had three dead deliveries and none delivered in 7 days 15
fired_never_resolved the rule asks for recovery notices, fired three times or more in 90 days, and never recovered 10
acted_on_rate_low under 0.2 of at least 30 fires in 90 days were acted on 15
co_firing_80_percent fires in the same 5-minute window as one other rule at least 80 % of the time (at least 10 windows in 30 days) 10
rule_broken BROKEN (the rule cannot be evaluated) 30, blocking
never_fired_180_days a live rule at least 180 days old that has not fired in 180 days 10

A fire is acted on when its issue was assigned, resolved, archived or commented on within 24 hours. The rate carries its 95 % Wilson interval and the number of fires it rests on, because a rate over a handful of fires is a guess. Weeks in which the alerts scheduler was down are left out of the baseline: a gap of more than six hours between its successful runs marks every 7-day bucket it touches, so silence is not learned as noise.

An owner is resolvable when the rule's creator is a person who still belongs to the organization, or the project has an ownership rule that names at least one owner. A rule an agent drafted has no person who made it, so it is an ownership rule that makes it owned. A destination is reachable when it is a channel of the rule's organization that its project can use and that is enabled, or an owner route whose fallback channel is. A channel id of another organization reads as gone.

Three keys are named by the rule set and cannot be computed yet: a composite rule that points at a deleted rule (MON-25), a missing evaluation delay on a late-arriving dataset (MON-07), and a page nobody acknowledged (MON-29). They are listed as notChecked on every read: an absent key is not a clear one for these.

Blocking keys. missing_owner, missing_route and rule_broken stop an agent-authored rule from being promoted from SHADOW to live (MON-03). The promotion gate reads them live, not from the stored day: isPromotionBlocked(ruleId, orgId) in code/domain/src/modules/alerts/quality.ts. It fails closed: a rule that is not the organization's is blocked.

Cleaning up in batches

An agent proposes a batch of at most 25 changes with monitor_cleanup_propose: a mute, a disable or a narrowing edit, each with the minutes it lasts (at most seven days for an agent, the same bound as a mute), or a reroute of one destination to a fallback channel (never onto a paging channel). One rule appears once.

  • It is ask-floored: no policy row, review mode or autonomy rung lets a batch apply on its own, so the first call is always an approval card. One person decides the whole batch, and an approver may narrow it to a subset.
  • Its class is the highest item's: SENSITIVE for a silencing item, CRITICAL when a rule that pages is silenced for more than four hours (which no agent is allowed by default), never below SENSITIVE. A reroute that takes a paging destination away (a pager swapped for a chat channel) is a silence with no end, so it is CRITICAL too; a pager swapped for another pager is not a silence.
  • It is applied item by item in one operation, against the revisions of the project's rules it was asked on: an edit to any rule of the project between the yes and the retry closes it TARGET_CHANGED and nothing is applied. Each item writes its own state-change row and audit row, and the batch writes one more.
  • Every item carries its own inverse (unmute, enable, restore the before values, reroute back), so operation_revert puts the whole batch back.
  • A mute ends by its own clock. A disable and a narrowing are put back by the alerts scheduler's cleanup_lapse pass (every minute) once their minutes have passed, and only while the rule still carries exactly what the batch wrote: a rule someone has edited since is theirs. The restore, its state-change row and its audit row (monitor_cleanup.lapse) land in one transaction.
  • The default window cap is one cleanup per agent per day.

The family is monitor_cleanup (code/domain/src/modules/alerts/cleanup-family.ts); the daily pass and the score are code/domain/src/modules/alerts/quality.ts, registered in the detector registry as monitor_quality (a finding, not an issue: it starts in shadow and opens nothing). The pass runs every six hours, checked every fifteen minutes so a worker restarted more often still scores; it rewrites the day's row, and records a finding once per rule per day, again only when the rule's keys move inside the day.

Why did, or didn't, it fire

GET .../alert-events?explain={eventId} (or the MCP tool alert_events_list with include: ["explain"]) returns one chain for an alert event:

evaluated -> value -> state -> outcome -> suppression -> routing -> delivery

Each link says where it came from: recorded (a row written at the time: the alert event and its rendered payload, a sampled evaluation, the scheduler's runs, the rule's state-change log), replayed (the live evaluator run over stored telemetry and the definition rebuilt for that moment, the same replay the backtest uses, narrowed to one instant) or not_available, with the reason. There is no "unknown".

For a moment with no event (?ruleId={id}&at={time}) the outcome is the evaluator's (fire, no_match, debounced, disabled, no_data, evaluation_failed, ...) or not_evaluated. A rule that was never looked at, because no scheduler pass ran around that moment or the rule was quarantined, is reported as not_evaluated with the gap (the last pass before, the first after, how many were missed). It is never reported as no_match, which would tell you the threshold was fine when nothing was checked.

What can be replayed. The definition at a past moment is rebuilt from the rule's state-change log by undoing the changes logged after it, and the silencing in force from the same log and the maintenance windows. Nothing older than the log (and the 14 days of sampled evaluations, the 30 days of scheduler runs) is rebuilt: the read says not_available. A replay answers whether the reading crossed a line; the debounce ledger is not stored per instant, so a cooldown is named only when a recorded alert inside it proves one. An issue rule fires on a signal rather than a measured value and is not replayed at an instant. Where a recorded value and the replay disagree (telemetry pruned since) the explanation shows both.

Group labels, the evaluator's reason (it embeds the group), window and route names and a delivery's last error did not come from Bucker: they are returned as labelled, bounded, control-free text. The MCP explanation needs event:read.

alert_rule_get returns one rule in full (thresholds as numbers and as labelled one-line summaries, filters, actions, modifiers, tags, lifecycle, mute and disable state, who manages it, evaluation health), and with include its open groups and its state-change history (before, after, actor, mode and approval id).

Where this is defined

  • Shadow mode: code/domain/src/modules/alerts/shadow.ts (the cap, the passes, a person's own promote and demote), shadow-evidence.ts (the record), the delivery gate in suppression.ts, the promote act in rule-family.ts and the demote act in silencing-family.ts, the routes in code/api/src/modules/alerts/routes-shadow.ts.

  • The rule DSL: code/domain/src/modules/alerts/rules.ts.

  • The telemetry builder: code/domain/src/modules/alerts/telemetry-metric.ts, which reuses code/domain/src/modules/explore/span-sql-builder.ts.

  • The learned threshold: code/domain/src/modules/alerts/telemetry-anomaly.ts, which uses code/domain/src/modules/anomaly/seasonality.ts.

  • Groups, comparison and the points a rule is evaluated at: code/domain/src/modules/alerts/metric-measure.ts. Warning, critical and recovery: evaluateMetricRule in code/domain/src/modules/alerts/evaluator.ts.

  • Evaluator liveness and the tenant batch: code/domain/src/modules/alerts/evaluator-liveness.ts. Quarantine: code/domain/src/modules/alerts/rule-quarantine.ts. Rule health and BROKEN: code/domain/src/modules/alerts/health.ts. The retry route and the evaluation block: code/api/src/modules/alerts/routes-evaluation.ts.

  • The claim-and-dispatch step shared by the three evaluators: code/domain/src/modules/alerts/claim-dispatch.ts. The sampled evaluation record: code/domain/src/modules/alerts/evaluations.ts. The state log: code/domain/src/modules/alerts/state-changes.ts.

  • NO_DATA and the retention: evaluateMetricRule in evaluator.ts, the ledger write in evaluateMetricPoint (evaluation.ts), ALERT_GROUP_STATES in rules.ts.

  • The explanation: code/domain/src/modules/alerts/explain.ts, which replays one instant with replayMetricRuleAt in backtest.ts. The reads (alertEventsList, alertRuleGet, the rules list by state): code/domain/src/modules/alerts/reads.ts.

  • Monitor Quality: code/domain/src/modules/alerts/quality.ts (the keys, the score, the daily pass, the gate), the read route in code/api/src/modules/alerts/routes.ts, the scheduler step runQualityPass in code/workers/src/alerts/scheduler.ts, and the batch family in code/domain/src/modules/alerts/cleanup-family.ts.