Alert rules
Who is told about a fire (routing policies), how a burst becomes one notification (correlation groups), SLO budgets and burn alerts, and finding what nothing watches are on Alert routing, correlation groups, SLOs and coverage.
Bucker has one alert router. An issue rule watches issues (a new issue, a
regression, a count of events on one issue, a lifecycle change). A metric rule
watches an aggregate over a window. Both kinds of rule reach the same channels, use the
same debounce ledger, and write the same alert-event history. A rule's type picks what
it watches. It does not pick a separate delivery system.
Rules live under a project:
POST /orgs/{org}/projects/{project}/alert-rules
GET /orgs/{org}/projects/{project}/alert-rules
PATCH /orgs/{org}/projects/{project}/alert-rules/{ruleId}
DELETE /orgs/{org}/projects/{project}/alert-rules/{ruleId}
In the dashboard, open Alerts › Rules.
New user feedback
The feedback_received condition (New user feedback in the editor) fires when a user
report linked to the issue arrives and is not filed as spam. It is its own signal, like a
lifecycle change. It never fires on telemetry, telemetry never fires it, and it cannot be
combined with first_seen, regression, event_count, user_count or
issue_lifecycle in one rule. The usual debounce applies, so a burst of reports pages
once per cooldown. A report that arrives before its error is grouped pages as soon as it
is linked to the issue. See User feedback.
{ "type": "issue", "conditions": [{ "id": "feedback_received" }], "actions": [...] }
A new survey answer
The survey_answer_received condition fires once per newly stored answer of one survey on a project issue
rule. It takes { survey, question?, score?: { comparator, value }, choices? }: survey is the survey's slug,
question narrows to one question, score compares a numeric answer and choices lists choice keys. Pacing is
one cooldown per rule, survey and question, so a flood pages once per cooldown. It cannot be combined with other
issue conditions or filters, and it is refused on an org-wide rule. The answer text attached to a notification
is untrusted: it is sanitized, escaped in Slack and email, replaced by a placeholder when quarantined and
withheld in the narrower health data modes. A rule written before its survey is launched is allowed; a slug
that matches nothing simply never fires. See Surveys.
{ "type": "issue", "conditions": [{ "id": "survey_answer_received", "survey": "nps-q4", "score": { "comparator": "lte", "value": 6 } }], "actions": [...] }
Org-wide rules
An org-wide rule covers every project in the organization, including projects created after the rule. "Page me on any new FATAL in production, any project" is one rule, not one rule per project.
POST /orgs/{org}/alert-rules
GET /orgs/{org}/alert-rules
GET /orgs/{org}/alert-rules/{ruleId}
PATCH /orgs/{org}/alert-rules/{ruleId}
DELETE /orgs/{org}/alert-rules/{ruleId}
- Same DSL. The body is the same as for a project rule. Bucker evaluates the rule separately for each project it covers. Each project gets its own cooldown and its own alert events, so an org-wide metric rule that breaches in one project pages for that project alone.
- A
projectfilter limits coverage.{ "id": "project", "slugs": ["sandbox"], "match": "not_in" }skips a project."match": "in"covers only the listed projects. A project rule cannot carry this filter. - Owners and admins only. Creating, editing or deleting an org-wide rule needs
writeon the organization. Anyone who can read the organization can list the rules. - Restrictions. An org-wide rule can only notify org-wide channels. It cannot use
the event
anomalycondition, because that condition's baseline is learned per project. Usetelemetry_anomalyinstead, or a project rule.
In the dashboard, an org-wide rule appears in every project's rule list, marked "all projects". The rule editor offers Applies to › Every project in the organization to owners and admins.
What a metric rule can watch
A metric rule has exactly one watched condition:
| Condition | Watches | Threshold |
|---|---|---|
metric |
error events: event_count, user_count, issue_count |
typed |
anomaly |
error events, as an hourly series | learned |
telemetry |
spans, logs or custom metrics: any Explore aggregate, or a counter's delta or rate | typed |
telemetry_anomaly |
spans, logs or custom metrics, as an hourly series | learned |
Spans and logs
A telemetry condition is an Explore query that has a threshold. It uses Explore's
filter operators, aggregate functions and attribute keys. Bucker compiles the condition
with the same builder that draws the Explore chart, so the chart and the alert read the
query the same way.
"p95 of checkout spans above 2 s for 10 minutes":
{
"name": "Slow checkout",
"type": "metric",
"conditions": [
{
"id": "telemetry",
"dataset": "spans",
"query": [{ "key": "span.name", "op": "eq", "value": "checkout" }],
"aggregate": "p95",
"field": "span.duration",
"comparator": "gt",
"threshold": 2000,
"windowMinutes": 10
}
],
"actions": [{ "kind": "notify_channel", "channelId": "…" }]
}
"More than 50 ERROR logs from billing in 5 minutes":
{
"id": "telemetry",
"dataset": "logs",
"query": [
{ "key": "log.severity", "op": "gte", "value": "error" },
{ "key": "service.name", "op": "eq", "value": "billing" }
],
"aggregate": "count",
"comparator": "gt",
"threshold": 50,
"windowMinutes": 5
}
- Datasets.
spans,logsandmetrics(see below). - Aggregates.
count,count_unique,sum,avg,min,max,p50,p75,p95andp99. Every aggregate exceptcountneeds afield. - Operators.
eq,neq,contains,not_contains,starts_with,gt,gte,lt,lte,existsandnot_exists. Filters inqueryare combined with AND. - Span keys. These are the same keys Explore uses:
span.name,span.kind,span.status,span.duration(in milliseconds),environment,release,resource.*and any attribute name. - Log keys. These are
log.severity,log.severity_number,log.body,trace_id,environment,releaseand any attribute name.log.severity gte errorcompares OTel severity numbers, sowarncounts as less severe thanerror. Log records have nospan.*orresource.*fields. Bucker refuses a rule that names one when you save it, not later when the rule runs. - Rule filters. A telemetry rule takes only
environmentandreleasefilters. A span has no event level, so put severity or status in the query instead. - No data is not a breach. An
avgorp95over an empty window has no value. The rule reads "no data" and does not fire, even if its comparator islt. A count over an empty window is a real zero. - Time. The window is measured on the span's start time or the log record's timestamp. Event-count metrics are measured on arrival time instead.
Custom metrics
With "dataset": "metrics", the condition watches metrics your services send over OTLP
/v1/metrics. The rule has to name the metric: a metric.name eq filter is required, and
Bucker refuses the rule at save time without one. The other keys are Explore's metrics
dataset: metric.value (a gauge or sum point), metric.count, metric.sum, metric.min,
metric.max, metric.unit, metric.kind, service, environment, release and any
attribute.
The aggregate is always explicit. Gauges and sampled values use the ordinary aggregates over
the window: "queue depth above 100 at any point in 10 minutes" is
aggregate: "max", field: "metric.value", and "average cache hit rate under 80%" is avg.
p95 is the 95th percentile of the stored point values, not of a histogram's buckets; for
a histogram percentile use histogram_p95 (below).
A counter needs one more word. Summing the stored values of a cumulative counter adds up
running totals, which is not a number anyone wants. Set counter to read it properly:
counter |
Value |
|---|---|
delta |
The increase over the window. |
rate |
The increase per second: the delta divided by the window length. |
Per series, a point with delta temporality is its own increase. A cumulative point is its
difference from the previous point, and a drop is read as a restart, so the new value is the
increase. The increases of all matching series are then summed. The first point of a series
that has no earlier point within 15 minutes contributes nothing. counter needs
"aggregate": "sum" (no field, or metric.value) and applies only to monotonic sum
metrics; up-down counters are read as gauges.
"More than 5 errors per second for 10 minutes":
{
"id": "telemetry",
"dataset": "metrics",
"query": [{ "key": "metric.name", "op": "eq", "value": "http.errors" }],
"aggregate": "sum",
"counter": "rate",
"comparator": "gt",
"threshold": 5,
"windowMinutes": 10
}
Counters and histograms by name. The aggregates counter_increase, counter_rate,
histogram_p50, histogram_p75, histogram_p90, histogram_p95, histogram_p99,
histogram_mean and histogram_rate are the same functions Explore offers on the metrics
dataset, and a rule and the chart beside it compute one number one way. counter_rate
and counter_increase are "aggregate": "sum" with counter rate and delta under one
word; the histogram functions merge the stored buckets, so
"aggregate": "histogram_p95" is the 95th percentile of the observations, not of the
per-point totals. They need the metrics dataset and a named metric, take no field and no
counter, and are refused otherwise. A percentile reads "no data", and does not fire, below
20 observations or when the series behind it use different bucket layouts (the rule's value
is then "no data"). A learned threshold (telemetry_anomaly)
trains on an hourly series, so it accepts counter_increase but not counter_rate or the
histogram functions, and a backtest of a histogram rule is refused.
The rule editor has a Custom metrics option under Watch. It seeds the metric.name
filter and offers a Read the metric as choice (stored points, counter increase, counter
rate). Grouping, warning thresholds, comparison with an earlier period, recovery messages
and the backtest work as they do for spans and logs. A window with no points reads "no
data" and does not fire.
telemetry_anomaly takes the same dataset, query, aggregate and field, plus
the event anomaly's sensitivity, direction and minValue. Bucker scores the last
complete hour against an hour-of-week baseline. It learns that baseline from the last
four weeks of the same query. The learned threshold switches on only after four weeks
of history. Until then, the rule's reason says when it will switch on.
Warning, recovery, percent change and fan-out
The metric and telemetry conditions accept four optional fields. A rule that leaves
them out behaves as it did before they existed.
| Field | Effect |
|---|---|
warningThreshold |
A second line on the same comparator. It must be crossed before threshold is. Crossing it fires a warning. Crossing threshold fires critical, even inside the cooldown, because the alert has escalated. Going from critical back to warning is recorded but not announced. |
notifyOnRecovery |
When a warning or critical series returns to normal, Bucker sends a resolved message. The message goes out even inside the cooldown. On a PagerDuty channel the incident gets a resolve. telemetry_anomaly accepts this field too. |
comparison |
{ "kind": "previous_period", "offsetMinutes": 1440 } compares the window with the same window that many minutes earlier. The value and the thresholds then count as percent change. If the earlier window held nothing, the rule reads "no baseline" and does not fire. |
groupBy |
{ "by": "environment" }, { "by": "release" } or { "by": "tag", "key": "region" } fans the rule out over that key. Each value gets its own cooldown, state and recovery. On a span or log rule, tag names an attribute. A rule evaluates at most 50 groups per pass, the busiest first, and its reason says when it cut the rest. |
A group that goes quiet still recovers. If an environment stops erroring completely, it drops out of the query. Bucker still evaluates every group that is breaching or warning in its ledger, so that environment's incident gets its resolved message.
{
"id": "metric",
"aggregate": "event_count",
"windowMinutes": 10,
"comparator": "gt",
"threshold": 200,
"warningThreshold": 100,
"comparison": { "kind": "previous_period", "offsetMinutes": 1440 },
"groupBy": { "by": "environment" },
"notifyOnRecovery": true
}
The webhook payload carries severity, recovered, group and comparison in
alert.metric. See alert-webhooks.md.
Maintenance windows and mutes
A maintenance window keeps alert notifications quiet during planned work. It covers alert rules, the anomaly detector, and uptime and cron monitors, because monitors raise their alerts through the same router. A window covers one project, or every project in the org, including projects created later.
GET /orgs/{org}/maintenance-windows
POST /orgs/{org}/maintenance-windows { name, startsAt, endsAt, projectSlug?, reason? }
POST /orgs/{org}/maintenance-windows { name, statusMaintenanceId } # follow a status-page maintenance
POST /orgs/{org}/maintenance-windows/{id}/end
DELETE /orgs/{org}/maintenance-windows/{id}
- Suppressed, never dropped. Bucker still records every fire a window covers. It is
written to the alert history as an alert event with status
SUPPRESSED, and itssuppressionfield names the window. Nothing is delivered. The fire still counts for the rule's cooldown, so if a condition is still true when the window ends, the alert goes out when the cooldown next expires. - Following a status page. A window bound to a status-page maintenance follows that maintenance's state. It is quiet while the maintenance runs, or while it is scheduled and inside its times. When the maintenance is marked complete, the quiet ends too.
- Bounds. A window lasts at most 14 days. A window over one project needs
writeon that project. A window over the whole org needswriteon the org.
A rule mute silences one rule until a given time:
POST /orgs/{org}/projects/{project}/alert-rules/{ruleId}/mute { minutes | until, scope: "rule" | "me" }
DELETE /orgs/{org}/projects/{project}/alert-rules/{ruleId}/mute?scope=rule|me
scope: "rule"mutes the rule for everyone. Its fires are recorded asSUPPRESSED. It needs projectwriteand writes analert_rule.muteaudit row.scope: "me"keeps the rule firing for everyone else. Only your own address is left out of its email deliveries.- A mute lasts at most 30 days and ends by itself. The dashboard's rule menu offers 30 m, 1 h, 4 h and 24 h. A Slack alert card offers Mute rule 1h and Mute rule 4h.
Silencing by an agent or a delegated client (the plan shape)
A person's own mute, window and edit stay direct, within the limits above. An agent, a service or a person's OAuth client silences through a governed plan: the same routes answer with a plan, a person decides it, and the change applies once, against the revision the person saw.
POST /orgs/{org}/projects/{project}/alert-rules/{ruleId}/mute { minutes, reason, approvalRequestId? }
DELETE …/alert-rules/{ruleId}/mute?reason=…&approvalRequestId=… # an unmute: usually applies at once
POST /orgs/{org}/maintenance-windows { name, durationMinutes, startInMinutes?, recurrence?, scope?, projectSlug, reason, approvalRequestId? }
POST …/alert-rules/{ruleId}/edit { patch, reason, minutes?, approvalRequestId? }
POST …/alert-rules/silencing/preview { ruleId, mute | unmute | disable | enable | narrow }
- Every silence has an end. A mute, a disable and a narrowing state
minutes(at most 7 days for an agent, 30 for a person); a window statesdurationMinutes(at most 72 hours for an agent, 14 days for a person). The operation expires with the silence and the TTL sweeper reverts it: an unmute, an enable, or, for a narrowing, the stored before values. - The act is read from the diff, never from an argument. An edit that makes a rule fire less (a higher
threshold on a greater-than rule, a longer window, one more filter, a longer cooldown, one fewer destination,
enabled: false) isalert_rule.narrowor.disableand needsminutes; anything else isalert_rule.update. A mixed edit reads as the riskier half. Ending a silence (unmute,enable, ending or cancelling a window) restores alerting and is ELEVATED. - The class follows what is covered. A silence over a paging destination (PagerDuty, Opsgenie, incident.io, JSM, ...) for more than four hours, or over an org-wide rule or window, is CRITICAL, which no agent is allowed by default. A mute of a rule that pages is held at ASK by review mode whatever the ceiling.
- A rule edit carries its backtest. The door replays the rule before and after over the last seven days
(the Backtest section below) and binds the counts into the plan an approver reads. A change with no backtest
is refused before an approval request exists. The whole result is stored on the request; a backtest more than
an hour old at decision time closes the request
TARGET_CHANGEDand the next call recomputes it. Backtests are limited to ten per principal per ten minutes, and an update costs two. - Windows can recur and be scoped.
recurrenceis an RRULE in UTC (at most twelve occurrences over ninety days, each at most fourteen days).scopenarrows a window to rules carrying one of itstagsand watching one of itsenvironments; a rule that watches every environment is not covered by a window for one of them. A recurring window's time-box runs to the end of its LAST occurrence, so an agent (72 hours) cannot open a series longer than that. Ending or cancelling a recurring window ends the whole series, and its end cannot be moved (that would resize every occurrence): end it and create another. - The plain doors are not a way around the plan. An agent's or a delegated client's
PATCHof a rule that makes it fire less is refused (SILENCING_EDIT) with the governed form above; one that makes it fire more is unchanged.DELETEof a rule by an agent or a delegated client is refused (HUMAN_ONLY). - Deleting a rule is a seven-day tombstone. The row, its history and its evaluations stay; every list, read
and evaluation pass skips it; a person restores it (
alert_rule.restore, asked for) inside the week. Deleting stays a person's act. - Ending a silence always runs. At its end time the sweeper ends the silence even when a person edited the rule meanwhile (a rename, a cooldown), as long as what the operation set is still in force. A field a person has since rewritten is theirs: the sweeper leaves it and asks.
- Disabling. A person's
enabled: falsestates its own end (30 days) indisabledUntil; a rule listed withinclude: ["windows"]over MCP showsmutedUntil,disabledUntil, its lifecycle and the windows covering it.
Backtest: how often would this rule have fired?
Before you save a rule, the editor's Backtest panel replays it over the project's last 7 days. The API is:
POST /orgs/{org}/projects/{project}/alert-rules/backtest { "rule": { …the rule… }, "days": 7 }
The replay uses the live evaluator. Bucker turns the stored history into the points the
live pass would have evaluated: a metric value per group at each step, or an issue
signal for each new issue, regression, lifecycle change and event. It runs each point
through the same rule code, with the rule's own cooldown kept in memory. Nothing is
written and nothing is delivered. The call needs only read on the project.
The response gives the count and its denominators:
| Field | Meaning |
|---|---|
fires |
Notifications after the cooldown. This is the headline number. |
matches |
Evaluations that met the rule. This is what would have fired with no cooldown. |
suppressedByCooldown |
Matches the cooldown swallowed. |
evaluations |
Points or signals replayed. |
recoveries |
Resolved messages, for a rule with notifyOnRecovery. |
delivery.chatMessagesPerDestination |
Messages each Slack, Discord, Teams or chat-webhook destination would have received after the chat volume ceiling (12 per 5 minutes, then one rollup). |
fireTimes, firesByDay |
When the fires would have happened. |
approximations |
Each way the replay is coarser than the live pass. |
The replay is coarser than the live pass in a few ways, and the response lists the ones that apply:
- A metric rule is evaluated once per window (5 to 60 minutes) rather than every minute, so a breach shorter than that can be missed.
- An issue's status is taken from the signal (new, regressed, ongoing), not from its stored history, so mutes that have since ended are not replayed.
- At most 20,000 events are replayed.
- The event
anomalycondition is not backtested, because its baseline lives in the detector's stored profile.telemetry_anomalyis replayed hour by hour.
Is my rule being evaluated?
"No alert fired" can mean nothing breached, or nothing was looked at. Bucker records the difference, for each organization and each evaluator, and shows it next to the rule.
The rules list (GET /orgs/{org}/projects/{project}/alert-rules, and the org-wide list)
returns an evaluation block per rule:
| Field | Meaning |
|---|---|
state |
ok, lagging, broken or unknown. |
lastEvaluatedAt |
When the evaluator last evaluated this rule (refreshed at most every five minutes while healthy). A rule with no record of its own reads your organization's newest pass instead. |
lastOutcome, consecutiveFailures |
How the rule's own last evaluation ended (ok, or the failure class) and how many failures in a row it has had. |
lagMs |
How far behind the oldest due rule was. A rule due every minute and reached every minute has none. |
quarantined, brokenSince |
The rule is set aside, which is what broken means: there is no second threshold. It is not being evaluated, and is never reported as OK. brokenSince is when this episode began. |
errorClass |
Why, in a closed vocabulary: timeout, dependency, invalid_query, bad_config, evaluation_error or dispatch_error. Never the error text, which can carry your query. |
quarantinedSince, nextRetryAt |
When it was set aside and when it is next tried. |
lagMs is per organization for the rule's evaluator, not per rule. The quarantine and the
rule's own lastEvaluatedAt, lastOutcome and consecutiveFailures are per rule. lagging means the evaluator is more than three passes behind for
your organization. For a metric rule it also means the evaluator has written nothing for
three passes, because silence is not health. unknown means nothing has reported on the
rule: it is disabled, or your organization has had no pass in 30 days. Anomaly rules report
too: the anomaly pass uses the same ledger, so a learned-threshold rule that keeps failing is
broken like any other.
Per-organization batch. One organization has at most 100 metric rules, 100 anomaly rules, and
100 issue rules per issue evaluated in a pass. The rest wait and the window rotates, so each rule is reached within ceil(n / 100)
passes. A big backlog in one organization costs that organization lag and no one else's.
Quarantine, which is BROKEN. A rule that throws on its own account three passes in a row is set aside.
It reads broken (not_evaluated), its owner is told through the rule's own actions (an ordinary
alert delivery, with its retries and receipts), and it is tried again every 15 minutes.
The rule's creator and the organization's owners also get one alert_rule_broken notification
per episode, in the inbox and by email unless they turned it off. A rule that is still failing
does not repeat it; one that recovers and breaks again does, as a new episode. Neither notice
is held back by a mute or a maintenance window: a rule that is not running is not something a
silence on its alerts should hide. The delivered notice records the silence it passed in its
suppression field (kind: "quarantine_notice", overrode: "rule_muted" | "maintenance" | "monitor_muted"). It
comes back on its first successful pass, when you edit it, or when you retry it:
POST /orgs/{org}/projects/{project}/alert-rules/{ruleId}/retry-evaluation
The retry needs write on the project and a person: an agent fixes the rule through its own
update instead, which releases the quarantine. It answers 409 for a rule that is not
quarantined. If the rule still fails, it is quarantined again after three more passes.
Only a rule's own failure counts. If every rule of your organization fails in a pass, or the database times out, the failure is recorded against the evaluator and no rule is set aside. Otherwise one database incident would quarantine every tenant at once.
A rule that no longer parses is listed with definitionValid: false and empty conditions,
filters and actions, and its broken state beside it. One such rule never fails the list.
No lost page. A fire is claimed on the cooldown ledger and then delivered. If delivery
throws, Bucker writes the alert deliveries it was about to make as FAILED rows due now, and
the ordinary retry pass sends them: the page arrives, once, on the next pass. Only if those rows
cannot be written either is the claim released, so the next pass fires the alert afresh. This
holds for issue, metric and anomaly rules alike.
What was decided, and when. Each fire and each failure is recorded, plus about one in ten of
the other evaluations, in alert_evaluations (value, threshold, outcome, and a minimal reason),
kept 14 days. It is the evidence behind "why did this fire" and "why did it not". Every change
that moves what a rule would fire or who hears of it is recorded in alert_rule_state_changes,
with the values before and after: a mute or unmute (from the API, Slack, or your own "mute for
me"), a maintenance window starting, ending or being deleted (one row per rule it covers), an
edit that enables or disables a rule or changes its conditions, filters, actions or cooldown
(from the dashboard, the API, bucker.yml or an agent), and a rule created by chat setup.
Notices that have no rule. Usage, spend and spike notices write a delivery receipt in
the same shape as an alert delivery (status, attempts, lastError, deliveredAt). A
send that reached nobody is recorded as FAILED and is not retried, because the fact it
reports is on the billing page.
Archive integrity. Each time the retention pass archives a day, it compares every
archive with its recorded row count and checksum and with its manifest, before the day's hot
partition is dropped, and writes an archive_integrity row.
When a group stops reporting: NO_DATA
A group that is breaching or at warning and then stops reporting is not a group that
recovered. For a rule over a value that can be absent (an average, a percentile, a rate, a
percent change with no baseline) the evaluator holds such a group as NO_DATA on the
rule's ledger and does not resolve it. The ledger's states are one closed set:
OK, WARNING, BREACHING, NO_DATA.
- Held, not resolved. Entering
NO_DATAsends nothing and closes nothing downstream. It writes the ledger row once; staying there writes nothing. A rule that asked to be told of recoveries is not told, because nothing recovered. - For the rule's retention.
noDataGroupRetentionMinutes(1 to 10,080, default 60) is how long the ledger keeps the quiet group. When it runs out the ledger forgets the group, still without a "resolved" message: a person, or the data returning, ends the incident. You can set it on a rule (API,bucker.yml, the rule editor for metric rules) and clear it withnull. - Data comes back. If it is healthy, a rule with
notifyOnRecoverysends a recovery ("recovered from a gap in data"). If it is still breaching, the cooldown decides: inside it nothing is sent (the page nobody resolved is not repeated), past it you get the ordinary reminder. - What does not change. A count with no matches is a real zero, so an empty
event_countgroup still recovers. A group that was never open staysOKon no data. A learned (anomaly) threshold has its own "no data" verdict and is unaffected.
The rules list shows each rule's open groups (breaching, warning, no data) and
GET .../alert-rules/{ruleId}/state lists them with when a quiet group will be forgotten.
GET .../alert-rules?state=NO_DATA narrows the list (OK means a rule with none open).
Shadow mode: agent-made rules deliver nothing until a person says yes
A rule an agent writes over MCP (alert_rules_create) or over REST as an agent, a service or a delegated client
starts in SHADOW. A person's own session, the dashboard, still writes a LIVE rule. The lifecycle is stored on
the rule (SHADOW, LIVE or EXPIRED) and is on every rule response and in alert_rules_list: it is never a tool
argument, and the plain edit routes refuse a lifecycle key.
- It is evaluated like a live rule and delivers nothing. Every fire writes one
SUPPRESSEDalert event per action withsuppression = {kind: "shadow"}. A mute, a maintenance window and the "rule is not running" notice cannot change that: one gate, in the router, lets only a LIVE rule deliver, so a rule that is somehow EXPIRED and still enabled delivers nothing either. Thealert_volumewidget counts these rows as their ownSHADOWseries, not as pages a mute swallowed. Shadow evaluation counts against the same per-organization evaluation caps as a live rule. - At most 20 shadow rules per project per agent. The 21st is refused before a person is asked. The rule
records who authored it, the delegation chain, the grant the call stood on (
authorGrant, a person-issued tool grant naming onlyalert_rules_createis how an agent is held to drafts) and the plan digest. - The record.
GET …/alert-rules/:ruleId/shadowand the promote dialog show what the rule did on real traffic: the fires it would have delivered, recoveries, groups, the destinations it would have reached (and which page a person), the fires the cooldown swallowed (an exact floor, plus an estimate from the sampled evaluations), and the MON-01 backtest it was proposed with, beside the actual fires per day. - Promotion is
alert_rule.promote, derived from a change of lifecycle:alert_rules_updatewithlifecycle: "LIVE", or the promote button. It needs at least one day of evidence and is refused while a Monitor Quality blocking key (missing_owner,missing_route,rule_broken) is set, read live. For an agent it is asked, with the evidence on the card and its digest on the audit row, and a rule that carries a paging destination is always ask-floored. A person promotes from their own session directly. - Demotion puts a live rule back into shadow for a stated time (
lifecycle: "SHADOW"withminutes, at most seven days for an agent and thirty for a person):alert_rule.demote, silencing-class and CRITICAL on a rule that pages a person or burns an SLO budget. The rule is promoted again by itself at the end. - Editing a shadow rule is ELEVATED, and an edit that makes it fire less needs no end time: nothing is delivered either way. Adding a paging destination still asks.
- A shadow rule nobody promotes lapses after 14 days: it becomes EXPIRED, disabled and archived under the
seven-day tombstone, with a state-change row and an audit row (
alert_rule.shadow_expired) as the receipt, andwindow_endedgoes to the sponsor. After three days of evidence the sponsor gets ashadow_report_readynotice, unless a promotion request is already open. A rule that was LIVE and was demoted is never archived.
Monitor Quality: a daily score per rule, and a gate for agent-made rules
Every enabled rule gets a score from 0 to 100 once a day, computed by the alerts scheduler from rows Bucker
already stores. There is no model and no estimate: each key is a published rule, and the thresholds are
versioned as ruleSetVersion (monitor_quality_v1), so a changed definition is visible on the rows it
produced. The rules list shows the score and its keys beside each rule name; GET …/alert-rules/quality and
the MCP tool monitor_quality_get return the same rows, worst first.
| Key | Set when | Points off |
|---|---|---|
muted_over_60_days |
one continuous mute (back-to-back mutes are one) has run for 60 days or more and is still in force | 15 |
stuck_in_alert_over_60_days |
a metric rule has a BREACHING or WARNING group open whose first unrecovered fire since the last recovery is 60 days old | 15 |
noisy_top_5_percent |
among the noisiest 5 % of the project's rules by delivered fires over 10 days (at least 10 fires) | 10 |
missing_owner |
no resolvable owner (below) | 20, blocking |
missing_route |
no destination the rule can deliver to | 30, blocking |
broken_channel |
a destination is disabled, gone, or has had three dead deliveries and none delivered in 7 days | 15 |
fired_never_resolved |
the rule asks for recovery notices, fired three times or more in 90 days, and never recovered | 10 |
acted_on_rate_low |
under 0.2 of at least 30 fires in 90 days were acted on | 15 |
co_firing_80_percent |
fires in the same 5-minute window as one other rule at least 80 % of the time (at least 10 windows in 30 days) | 10 |
rule_broken |
BROKEN (the rule cannot be evaluated) | 30, blocking |
never_fired_180_days |
a live rule at least 180 days old that has not fired in 180 days | 10 |
A fire is acted on when its issue was assigned, resolved, archived or commented on within 24 hours. The rate carries its 95 % Wilson interval and the number of fires it rests on, because a rate over a handful of fires is a guess. Weeks in which the alerts scheduler was down are left out of the baseline: a gap of more than six hours between its successful runs marks every 7-day bucket it touches, so silence is not learned as noise.
An owner is resolvable when the rule's creator is a person who still belongs to the organization, or the project has an ownership rule that names at least one owner. A rule an agent drafted has no person who made it, so it is an ownership rule that makes it owned. A destination is reachable when it is a channel of the rule's organization that its project can use and that is enabled, or an owner route whose fallback channel is. A channel id of another organization reads as gone.
Three keys are named by the rule set and cannot be computed yet: a composite rule that points at a deleted
rule (MON-25), a missing evaluation delay on a late-arriving dataset (MON-07), and a page nobody acknowledged
(MON-29). They are listed as notChecked on every read: an absent key is not a clear one for these.
Blocking keys. missing_owner, missing_route and rule_broken stop an agent-authored rule from being
promoted from SHADOW to live (MON-03). The promotion gate reads them live, not from the stored day:
isPromotionBlocked(ruleId, orgId) in code/domain/src/modules/alerts/quality.ts. It fails closed: a rule
that is not the organization's is blocked.
Cleaning up in batches
An agent proposes a batch of at most 25 changes with monitor_cleanup_propose: a mute, a disable or a
narrowing edit, each with the minutes it lasts (at most seven days for an agent, the same bound as a mute),
or a reroute of one destination to a fallback channel (never onto a paging channel). One rule appears once.
- It is ask-floored: no policy row, review mode or autonomy rung lets a batch apply on its own, so the first call is always an approval card. One person decides the whole batch, and an approver may narrow it to a subset.
- Its class is the highest item's: SENSITIVE for a silencing item, CRITICAL when a rule that pages is silenced for more than four hours (which no agent is allowed by default), never below SENSITIVE. A reroute that takes a paging destination away (a pager swapped for a chat channel) is a silence with no end, so it is CRITICAL too; a pager swapped for another pager is not a silence.
- It is applied item by item in one operation, against the revisions of the project's rules it was asked
on: an edit to any rule of the project between the yes and the retry closes it
TARGET_CHANGEDand nothing is applied. Each item writes its own state-change row and audit row, and the batch writes one more. - Every item carries its own inverse (unmute, enable, restore the before values, reroute back), so
operation_revertputs the whole batch back. - A mute ends by its own clock. A disable and a narrowing are put back by the alerts scheduler's
cleanup_lapsepass (every minute) once their minutes have passed, and only while the rule still carries exactly what the batch wrote: a rule someone has edited since is theirs. The restore, its state-change row and its audit row (monitor_cleanup.lapse) land in one transaction. - The default window cap is one cleanup per agent per day.
The family is monitor_cleanup (code/domain/src/modules/alerts/cleanup-family.ts); the daily pass and the
score are code/domain/src/modules/alerts/quality.ts, registered in the detector registry as monitor_quality
(a finding, not an issue: it starts in shadow and opens nothing). The pass runs every six hours, checked every
fifteen minutes so a worker restarted more often still scores; it rewrites the day's row, and records a finding
once per rule per day, again only when the rule's keys move inside the day.
Why did, or didn't, it fire
GET .../alert-events?explain={eventId} (or the MCP tool alert_events_list with
include: ["explain"]) returns one chain for an alert event:
evaluated -> value -> state -> outcome -> suppression -> routing -> delivery
Each link says where it came from: recorded (a row written at the time: the alert event
and its rendered payload, a sampled evaluation, the scheduler's runs, the rule's state-change
log), replayed (the live evaluator run over stored telemetry and the definition rebuilt
for that moment, the same replay the backtest uses, narrowed to one instant) or
not_available, with the reason. There is no "unknown".
For a moment with no event (?ruleId={id}&at={time}) the outcome is the evaluator's
(fire, no_match, debounced, disabled, no_data, evaluation_failed, ...) or
not_evaluated. A rule that was never looked at, because no scheduler pass ran around that
moment or the rule was quarantined, is reported as not_evaluated with the gap (the last pass
before, the first after, how many were missed). It is never reported as no_match, which
would tell you the threshold was fine when nothing was checked.
What can be replayed. The definition at a past moment is rebuilt from the rule's state-change
log by undoing the changes logged after it, and the silencing in force from the same log and
the maintenance windows. Nothing older than the log (and the 14 days of sampled evaluations,
the 30 days of scheduler runs) is rebuilt: the read says not_available. A replay answers
whether the reading crossed a line; the debounce ledger is not stored per instant, so a
cooldown is named only when a recorded alert inside it proves one. An issue rule fires on a
signal rather than a measured value and is not replayed at an instant. Where a recorded value
and the replay disagree (telemetry pruned since) the explanation shows both.
Group labels, the evaluator's reason (it embeds the group), window and route names and a
delivery's last error did not come from Bucker: they are returned as labelled, bounded,
control-free text. The MCP explanation needs event:read.
alert_rule_get returns one rule in full (thresholds as numbers and as labelled one-line
summaries, filters, actions, modifiers, tags, lifecycle, mute and disable state, who manages
it, evaluation health), and with include its open groups and its state-change history
(before, after, actor, mode and approval id).
Where this is defined
Shadow mode:
code/domain/src/modules/alerts/shadow.ts(the cap, the passes, a person's own promote and demote),shadow-evidence.ts(the record), the delivery gate insuppression.ts, the promote act inrule-family.tsand the demote act insilencing-family.ts, the routes incode/api/src/modules/alerts/routes-shadow.ts.The rule DSL:
code/domain/src/modules/alerts/rules.ts.The telemetry builder:
code/domain/src/modules/alerts/telemetry-metric.ts, which reusescode/domain/src/modules/explore/span-sql-builder.ts.The learned threshold:
code/domain/src/modules/alerts/telemetry-anomaly.ts, which usescode/domain/src/modules/anomaly/seasonality.ts.Groups, comparison and the points a rule is evaluated at:
code/domain/src/modules/alerts/metric-measure.ts. Warning, critical and recovery:evaluateMetricRuleincode/domain/src/modules/alerts/evaluator.ts.Evaluator liveness and the tenant batch:
code/domain/src/modules/alerts/evaluator-liveness.ts. Quarantine:code/domain/src/modules/alerts/rule-quarantine.ts. Rule health and BROKEN:code/domain/src/modules/alerts/health.ts. The retry route and theevaluationblock:code/api/src/modules/alerts/routes-evaluation.ts.The claim-and-dispatch step shared by the three evaluators:
code/domain/src/modules/alerts/claim-dispatch.ts. The sampled evaluation record:code/domain/src/modules/alerts/evaluations.ts. The state log:code/domain/src/modules/alerts/state-changes.ts.NO_DATA and the retention:
evaluateMetricRuleinevaluator.ts, the ledger write inevaluateMetricPoint(evaluation.ts),ALERT_GROUP_STATESinrules.ts.The explanation:
code/domain/src/modules/alerts/explain.ts, which replays one instant withreplayMetricRuleAtinbacktest.ts. The reads (alertEventsList,alertRuleGet, the rules list by state):code/domain/src/modules/alerts/reads.ts.Monitor Quality:
code/domain/src/modules/alerts/quality.ts(the keys, the score, the daily pass, the gate), the read route incode/api/src/modules/alerts/routes.ts, the scheduler steprunQualityPassincode/workers/src/alerts/scheduler.ts, and the batch family incode/domain/src/modules/alerts/cleanup-family.ts.