# Alert rules

Who is told about a fire (routing policies), how a burst becomes one notification (correlation groups),
SLO budgets and burn alerts, and finding what nothing watches are on [Alert routing, correlation groups,
SLOs and coverage](alert-routing.md).

Bucker has one alert router. An **issue rule** watches issues (a new issue, a
regression, a count of events on one issue, a lifecycle change). A **metric rule**
watches an aggregate over a window. Both kinds of rule reach the same channels, use the
same debounce ledger, and write the same alert-event history. A rule's `type` picks what
it watches. It does not pick a separate delivery system.

Rules live under a project:

```
POST   /orgs/{org}/projects/{project}/alert-rules
GET    /orgs/{org}/projects/{project}/alert-rules
PATCH  /orgs/{org}/projects/{project}/alert-rules/{ruleId}
DELETE /orgs/{org}/projects/{project}/alert-rules/{ruleId}
```

In the dashboard, open **Alerts › Rules**.

### New user feedback

The `feedback_received` condition (**New user feedback** in the editor) fires when a user
report linked to the issue arrives and is not filed as spam. It is its own signal, like a
lifecycle change. It never fires on telemetry, telemetry never fires it, and it cannot be
combined with `first_seen`, `regression`, `event_count`, `user_count` or
`issue_lifecycle` in one rule. The usual debounce applies, so a burst of reports pages
once per cooldown. A report that arrives before its error is grouped pages as soon as it
is linked to the issue. See [User feedback](user-feedback.md).

```json
{ "type": "issue", "conditions": [{ "id": "feedback_received" }], "actions": [...] }
```

### A new survey answer

The `survey_answer_received` condition fires once per newly stored answer of one survey on a project issue
rule. It takes `{ survey, question?, score?: { comparator, value }, choices? }`: `survey` is the survey's slug,
`question` narrows to one question, `score` compares a numeric answer and `choices` lists choice keys. Pacing is
one cooldown per rule, survey and question, so a flood pages once per cooldown. It cannot be combined with other
issue conditions or filters, and it is refused on an org-wide rule. The answer text attached to a notification
is untrusted: it is sanitized, escaped in Slack and email, replaced by a placeholder when quarantined and
withheld in the narrower health data modes. A rule written before its survey is launched is allowed; a slug
that matches nothing simply never fires. See [Surveys](surveys.md).

```json
{ "type": "issue", "conditions": [{ "id": "survey_answer_received", "survey": "nps-q4", "score": { "comparator": "lte", "value": 6 } }], "actions": [...] }
```

### Org-wide rules

An **org-wide rule** covers every project in the organization, including projects
created after the rule. "Page me on any new FATAL in production, any project" is one
rule, not one rule per project.

```
POST   /orgs/{org}/alert-rules
GET    /orgs/{org}/alert-rules
GET    /orgs/{org}/alert-rules/{ruleId}
PATCH  /orgs/{org}/alert-rules/{ruleId}
DELETE /orgs/{org}/alert-rules/{ruleId}
```

- **Same DSL.** The body is the same as for a project rule. Bucker evaluates the rule
  separately for each project it covers. Each project gets its own cooldown and its own
  alert events, so an org-wide metric rule that breaches in one project pages for that
  project alone.
- **A `project` filter limits coverage.** `{ "id": "project", "slugs": ["sandbox"],
  "match": "not_in" }` skips a project. `"match": "in"` covers only the listed projects.
  A project rule cannot carry this filter.
- **Owners and admins only.** Creating, editing or deleting an org-wide rule needs
  `write` on the organization. Anyone who can read the organization can list the rules.
- **Restrictions.** An org-wide rule can only notify org-wide channels. It cannot use
  the event `anomaly` condition, because that condition's baseline is learned per
  project. Use `telemetry_anomaly` instead, or a project rule.

In the dashboard, an org-wide rule appears in every project's rule list, marked "all
projects". The rule editor offers **Applies to › Every project in the organization**
to owners and admins.

## What a metric rule can watch

A metric rule has exactly one *watched* condition:

| Condition | Watches | Threshold |
| --- | --- | --- |
| `metric` | error events: `event_count`, `user_count`, `issue_count` | typed |
| `anomaly` | error events, as an hourly series | learned |
| `telemetry` | spans, logs or custom metrics: any Explore aggregate, or a counter's delta or rate | typed |
| `telemetry_anomaly` | spans, logs or custom metrics, as an hourly series | learned |

### Spans and logs

A `telemetry` condition is an Explore query that has a threshold. It uses Explore's
filter operators, aggregate functions and attribute keys. Bucker compiles the condition
with the same builder that draws the Explore chart, so the chart and the alert read the
query the same way.

"p95 of checkout spans above 2 s for 10 minutes":

```json
{
  "name": "Slow checkout",
  "type": "metric",
  "conditions": [
    {
      "id": "telemetry",
      "dataset": "spans",
      "query": [{ "key": "span.name", "op": "eq", "value": "checkout" }],
      "aggregate": "p95",
      "field": "span.duration",
      "comparator": "gt",
      "threshold": 2000,
      "windowMinutes": 10
    }
  ],
  "actions": [{ "kind": "notify_channel", "channelId": "…" }]
}
```

"More than 50 ERROR logs from billing in 5 minutes":

```json
{
  "id": "telemetry",
  "dataset": "logs",
  "query": [
    { "key": "log.severity", "op": "gte", "value": "error" },
    { "key": "service.name", "op": "eq", "value": "billing" }
  ],
  "aggregate": "count",
  "comparator": "gt",
  "threshold": 50,
  "windowMinutes": 5
}
```

- **Datasets.** `spans`, `logs` and `metrics` (see below).
- **Aggregates.** `count`, `count_unique`, `sum`, `avg`, `min`, `max`, `p50`, `p75`,
  `p95` and `p99`. Every aggregate except `count` needs a `field`.
- **Operators.** `eq`, `neq`, `contains`, `not_contains`, `starts_with`, `gt`, `gte`,
  `lt`, `lte`, `exists` and `not_exists`. Filters in `query` are combined with AND.
- **Span keys.** These are the same keys Explore uses: `span.name`, `span.kind`,
  `span.status`, `span.duration` (in milliseconds), `environment`, `release`,
  `resource.*` and any attribute name.
- **Log keys.** These are `log.severity`, `log.severity_number`, `log.body`,
  `trace_id`, `environment`, `release` and any attribute name. `log.severity gte
  error` compares OTel severity numbers, so `warn` counts as less severe than
  `error`. Log records have no `span.*` or `resource.*` fields. Bucker refuses a rule
  that names one when you save it, not later when the rule runs.
- **Rule filters.** A telemetry rule takes only `environment` and `release` filters.
  A span has no event level, so put severity or status in the query instead.
- **No data is not a breach.** An `avg` or `p95` over an empty window has no value.
  The rule reads "no data" and does not fire, even if its comparator is `lt`. A count
  over an empty window is a real zero.
- **Time.** The window is measured on the span's start time or the log record's
  timestamp. Event-count metrics are measured on arrival time instead.

### Custom metrics

With `"dataset": "metrics"`, the condition watches metrics your services send over OTLP
`/v1/metrics`. The rule has to name the metric: a `metric.name` `eq` filter is required, and
Bucker refuses the rule at save time without one. The other keys are Explore's `metrics`
dataset: `metric.value` (a gauge or sum point), `metric.count`, `metric.sum`, `metric.min`,
`metric.max`, `metric.unit`, `metric.kind`, `service`, `environment`, `release` and any
attribute.

The aggregate is always explicit. Gauges and sampled values use the ordinary aggregates over
the window: "queue depth above 100 at any point in 10 minutes" is
`aggregate: "max", field: "metric.value"`, and "average cache hit rate under 80%" is `avg`.
`p95` is the 95th percentile of the stored point values, not of a histogram's buckets; for
a histogram percentile use `histogram_p95` (below).

A counter needs one more word. Summing the stored values of a cumulative counter adds up
running totals, which is not a number anyone wants. Set `counter` to read it properly:

| `counter` | Value |
| --- | --- |
| `delta` | The increase over the window. |
| `rate` | The increase per second: the delta divided by the window length. |

Per series, a point with delta temporality is its own increase. A cumulative point is its
difference from the previous point, and a drop is read as a restart, so the new value is the
increase. The increases of all matching series are then summed. The first point of a series
that has no earlier point within 15 minutes contributes nothing. `counter` needs
`"aggregate": "sum"` (no `field`, or `metric.value`) and applies only to monotonic `sum`
metrics; up-down counters are read as gauges.

"More than 5 errors per second for 10 minutes":

```json
{
  "id": "telemetry",
  "dataset": "metrics",
  "query": [{ "key": "metric.name", "op": "eq", "value": "http.errors" }],
  "aggregate": "sum",
  "counter": "rate",
  "comparator": "gt",
  "threshold": 5,
  "windowMinutes": 10
}
```

**Counters and histograms by name.** The aggregates `counter_increase`, `counter_rate`,
`histogram_p50`, `histogram_p75`, `histogram_p90`, `histogram_p95`, `histogram_p99`,
`histogram_mean` and `histogram_rate` are the same functions Explore offers on the metrics
dataset, and a rule and the chart beside it compute **one number one way**. `counter_rate`
and `counter_increase` are `"aggregate": "sum"` with `counter` `rate` and `delta` under one
word; the histogram functions merge the stored buckets, so
`"aggregate": "histogram_p95"` is the 95th percentile of the observations, not of the
per-point totals. They need the metrics dataset and a named metric, take no `field` and no
`counter`, and are refused otherwise. A percentile reads "no data", and does not fire, below
20 observations or when the series behind it use different bucket layouts (the rule's value
is then "no data"). A learned threshold (`telemetry_anomaly`)
trains on an hourly series, so it accepts `counter_increase` but not `counter_rate` or the
histogram functions, and a backtest of a histogram rule is refused.

The rule editor has a **Custom metrics** option under *Watch*. It seeds the `metric.name`
filter and offers a *Read the metric as* choice (stored points, counter increase, counter
rate). Grouping, warning thresholds, comparison with an earlier period, recovery messages
and the backtest work as they do for spans and logs. A window with no points reads "no
data" and does not fire.

`telemetry_anomaly` takes the same `dataset`, `query`, `aggregate` and `field`, plus
the event anomaly's `sensitivity`, `direction` and `minValue`. Bucker scores the last
complete hour against an hour-of-week baseline. It learns that baseline from the last
four weeks of the same query. The learned threshold switches on only after four weeks
of history. Until then, the rule's reason says when it will switch on.

## Warning, recovery, percent change and fan-out

The `metric` and `telemetry` conditions accept four optional fields. A rule that leaves
them out behaves as it did before they existed.

| Field | Effect |
| --- | --- |
| `warningThreshold` | A second line on the same comparator. It must be crossed before `threshold` is. Crossing it fires a **warning**. Crossing `threshold` fires **critical**, even inside the cooldown, because the alert has escalated. Going from critical back to warning is recorded but not announced. |
| `notifyOnRecovery` | When a warning or critical series returns to normal, Bucker sends a **resolved** message. The message goes out even inside the cooldown. On a PagerDuty channel the incident gets a `resolve`. `telemetry_anomaly` accepts this field too. |
| `comparison` | `{ "kind": "previous_period", "offsetMinutes": 1440 }` compares the window with the same window that many minutes earlier. The value and the thresholds then count as **percent change**. If the earlier window held nothing, the rule reads "no baseline" and does not fire. |
| `groupBy` | `{ "by": "environment" }`, `{ "by": "release" }` or `{ "by": "tag", "key": "region" }` fans the rule out over that key. Each value gets its own cooldown, state and recovery. On a span or log rule, `tag` names an attribute. A rule evaluates at most 50 groups per pass, the busiest first, and its reason says when it cut the rest. |

A group that goes quiet still recovers. If an environment stops erroring completely, it
drops out of the query. Bucker still evaluates every group that is breaching or warning
in its ledger, so that environment's incident gets its resolved message.

```json
{
  "id": "metric",
  "aggregate": "event_count",
  "windowMinutes": 10,
  "comparator": "gt",
  "threshold": 200,
  "warningThreshold": 100,
  "comparison": { "kind": "previous_period", "offsetMinutes": 1440 },
  "groupBy": { "by": "environment" },
  "notifyOnRecovery": true
}
```

The webhook payload carries `severity`, `recovered`, `group` and `comparison` in
`alert.metric`. See [alert-webhooks.md](alert-webhooks.md).

## Maintenance windows and mutes

A **maintenance window** keeps alert notifications quiet during planned work. It covers
alert rules, the anomaly detector, and uptime and cron monitors, because monitors raise
their alerts through the same router. A window covers one project, or every project in
the org, including projects created later.

```
GET    /orgs/{org}/maintenance-windows
POST   /orgs/{org}/maintenance-windows        { name, startsAt, endsAt, projectSlug?, reason? }
POST   /orgs/{org}/maintenance-windows        { name, statusMaintenanceId }   # follow a status-page maintenance
POST   /orgs/{org}/maintenance-windows/{id}/end
DELETE /orgs/{org}/maintenance-windows/{id}
```

- **Suppressed, never dropped.** Bucker still records every fire a window covers. It is
  written to the alert history as an alert event with status `SUPPRESSED`, and its
  `suppression` field names the window. Nothing is delivered. The fire still counts for
  the rule's cooldown, so if a condition is still true when the window ends, the alert
  goes out when the cooldown next expires.
- **Following a status page.** A window bound to a status-page maintenance follows that
  maintenance's state. It is quiet while the maintenance runs, or while it is scheduled
  and inside its times. When the maintenance is marked complete, the quiet ends too.
- **Bounds.** A window lasts at most 14 days. A window over one project needs `write` on
  that project. A window over the whole org needs `write` on the org.

A **rule mute** silences one rule until a given time:

```
POST   /orgs/{org}/projects/{project}/alert-rules/{ruleId}/mute   { minutes | until, scope: "rule" | "me" }
DELETE /orgs/{org}/projects/{project}/alert-rules/{ruleId}/mute?scope=rule|me
```

- `scope: "rule"` mutes the rule for everyone. Its fires are recorded as `SUPPRESSED`.
  It needs project `write` and writes an `alert_rule.mute` audit row.
- `scope: "me"` keeps the rule firing for everyone else. Only your own address is left
  out of its email deliveries.
- A mute lasts at most 30 days and ends by itself. The dashboard's rule menu offers 30 m,
  1 h, 4 h and 24 h. A Slack alert card offers **Mute rule 1h** and **Mute rule 4h**.

### Silencing by an agent or a delegated client (the plan shape)

A person's own mute, window and edit stay direct, within the limits above. An **agent, a service or a person's
OAuth client** silences through a governed plan: the same routes answer with a plan, a person decides it, and the
change applies once, against the revision the person saw.

```
POST /orgs/{org}/projects/{project}/alert-rules/{ruleId}/mute      { minutes, reason, approvalRequestId? }
DELETE …/alert-rules/{ruleId}/mute?reason=…&approvalRequestId=…    # an unmute: usually applies at once
POST /orgs/{org}/maintenance-windows    { name, durationMinutes, startInMinutes?, recurrence?, scope?, projectSlug, reason, approvalRequestId? }
POST …/alert-rules/{ruleId}/edit        { patch, reason, minutes?, approvalRequestId? }
POST …/alert-rules/silencing/preview    { ruleId, mute | unmute | disable | enable | narrow }
```

- **Every silence has an end.** A mute, a disable and a narrowing state `minutes` (at most 7 days for an agent, 30
  for a person); a window states `durationMinutes` (at most 72 hours for an agent, 14 days for a person). The
  operation expires with the silence and the TTL sweeper reverts it: an unmute, an enable, or, for a narrowing,
  the stored before values.
- **The act is read from the diff, never from an argument.** An edit that makes a rule fire less (a higher
  threshold on a greater-than rule, a longer window, one more filter, a longer cooldown, one fewer destination,
  `enabled: false`) is `alert_rule.narrow` or `.disable` and needs `minutes`; anything else is
  `alert_rule.update`. A mixed edit reads as the riskier half. Ending a silence (`unmute`, `enable`, ending or
  cancelling a window) restores alerting and is ELEVATED.
- **The class follows what is covered.** A silence over a paging destination (PagerDuty, Opsgenie, incident.io,
  JSM, ...) for more than four hours, or over an org-wide rule or window, is CRITICAL, which no agent is
  allowed by default. A mute of a rule that pages is held at ASK by review mode whatever the ceiling.
- **A rule edit carries its backtest.** The door replays the rule before and after over the last seven days
  (the Backtest section below) and binds the counts into the plan an approver reads. A change with no backtest
  is refused before an approval request exists. The whole result is stored on the request; a backtest more than
  an hour old at decision time closes the request `TARGET_CHANGED` and the next call recomputes it. Backtests are
  limited to ten per principal per ten minutes, and an update costs two.
- **Windows can recur and be scoped.** `recurrence` is an RRULE in UTC (at most twelve occurrences over ninety
  days, each at most fourteen days). `scope` narrows a window to rules carrying one of its `tags` and watching
  one of its `environments`; a rule that watches every environment is not covered by a window for one of them.
  A recurring window's time-box runs to the end of its LAST occurrence, so an agent (72 hours) cannot open a
  series longer than that. Ending or cancelling a recurring window ends the whole series, and its end cannot be
  moved (that would resize every occurrence): end it and create another.
- **The plain doors are not a way around the plan.** An agent's or a delegated client's `PATCH` of a rule that
  makes it fire less is refused (`SILENCING_EDIT`) with the governed form above; one that makes it fire more is
  unchanged. `DELETE` of a rule by an agent or a delegated client is refused (`HUMAN_ONLY`).
- **Deleting a rule is a seven-day tombstone.** The row, its history and its evaluations stay; every list, read
  and evaluation pass skips it; a person restores it (`alert_rule.restore`, asked for) inside the week. Deleting
  stays a person's act.
- **Ending a silence always runs.** At its end time the sweeper ends the silence even when a person edited the
  rule meanwhile (a rename, a cooldown), as long as what the operation set is still in force. A field a person
  has since rewritten is theirs: the sweeper leaves it and asks.
- **Disabling.** A person's `enabled: false` states its own end (30 days) in `disabledUntil`; a rule listed with
  `include: ["windows"]` over MCP shows `mutedUntil`, `disabledUntil`, its lifecycle and the windows covering it.

## Backtest: how often would this rule have fired?

Before you save a rule, the editor's **Backtest** panel replays it over the project's
last 7 days. The API is:

```
POST /orgs/{org}/projects/{project}/alert-rules/backtest   { "rule": { …the rule… }, "days": 7 }
```

The replay uses the live evaluator. Bucker turns the stored history into the points the
live pass would have evaluated: a metric value per group at each step, or an issue
signal for each new issue, regression, lifecycle change and event. It runs each point
through the same rule code, with the rule's own cooldown kept in memory. Nothing is
written and nothing is delivered. The call needs only `read` on the project.

The response gives the count and its denominators:

| Field | Meaning |
| --- | --- |
| `fires` | Notifications after the cooldown. This is the headline number. |
| `matches` | Evaluations that met the rule. This is what would have fired with no cooldown. |
| `suppressedByCooldown` | Matches the cooldown swallowed. |
| `evaluations` | Points or signals replayed. |
| `recoveries` | Resolved messages, for a rule with `notifyOnRecovery`. |
| `delivery.chatMessagesPerDestination` | Messages each Slack, Discord, Teams or chat-webhook destination would have received after the chat volume ceiling (12 per 5 minutes, then one rollup). |
| `fireTimes`, `firesByDay` | When the fires would have happened. |
| `approximations` | Each way the replay is coarser than the live pass. |

The replay is coarser than the live pass in a few ways, and the response lists the ones
that apply:

- A metric rule is evaluated once per window (5 to 60 minutes) rather than every minute,
  so a breach shorter than that can be missed.
- An issue's status is taken from the signal (new, regressed, ongoing), not from its
  stored history, so mutes that have since ended are not replayed.
- At most 20,000 events are replayed.
- The event `anomaly` condition is not backtested, because its baseline lives in the
  detector's stored profile. `telemetry_anomaly` is replayed hour by hour.

## Is my rule being evaluated?

"No alert fired" can mean nothing breached, or nothing was looked at. Bucker records the
difference, for each organization and each evaluator, and shows it next to the rule.

The rules list (`GET /orgs/{org}/projects/{project}/alert-rules`, and the org-wide list)
returns an `evaluation` block per rule:

| Field | Meaning |
| --- | --- |
| `state` | `ok`, `lagging`, `broken` or `unknown`. |
| `lastEvaluatedAt` | When the evaluator last evaluated this rule (refreshed at most every five minutes while healthy). A rule with no record of its own reads your organization's newest pass instead. |
| `lastOutcome`, `consecutiveFailures` | How the rule's own last evaluation ended (`ok`, or the failure class) and how many failures in a row it has had. |
| `lagMs` | How far behind the oldest due rule was. A rule due every minute and reached every minute has none. |
| `quarantined`, `brokenSince` | The rule is set aside, which is what **broken** means: there is no second threshold. It is **not being evaluated**, and is never reported as OK. `brokenSince` is when this episode began. |
| `errorClass` | Why, in a closed vocabulary: `timeout`, `dependency`, `invalid_query`, `bad_config`, `evaluation_error` or `dispatch_error`. Never the error text, which can carry your query. |
| `quarantinedSince`, `nextRetryAt` | When it was set aside and when it is next tried. |

`lagMs` is per organization for the rule's evaluator, not per rule. The quarantine and the
rule's own `lastEvaluatedAt`, `lastOutcome` and `consecutiveFailures` are per rule. `lagging` means the evaluator is more than three passes behind for
your organization. For a metric rule it also means the evaluator has written nothing for
three passes, because silence is not health. `unknown` means nothing has reported on the
rule: it is disabled, or your organization has had no pass in 30 days. Anomaly rules report
too: the anomaly pass uses the same ledger, so a learned-threshold rule that keeps failing is
broken like any other.

**Per-organization batch.** One organization has at most 100 metric rules, 100 anomaly rules, and
100 issue rules per issue evaluated in a pass. The rest wait and the window rotates, so each rule is reached within `ceil(n / 100)`
passes. A big backlog in one organization costs that organization lag and no one else's.

**Quarantine, which is BROKEN.** A rule that throws on its own account three passes in a row is set aside.
It reads `broken` (`not_evaluated`), its owner is told through the rule's own actions (an ordinary
alert delivery, with its retries and receipts), and it is tried again every 15 minutes.
The rule's creator and the organization's owners also get one `alert_rule_broken` notification
per episode, in the inbox and by email unless they turned it off. A rule that is still failing
does not repeat it; one that recovers and breaks again does, as a new episode. Neither notice
is held back by a mute or a maintenance window: a rule that is not running is not something a
silence on its alerts should hide. The delivered notice records the silence it passed in its
`suppression` field (`kind: "quarantine_notice"`, `overrode: "rule_muted" | "maintenance" |
"monitor_muted"`). It
comes back on its first successful pass, when you edit it, or when you retry it:

```
POST /orgs/{org}/projects/{project}/alert-rules/{ruleId}/retry-evaluation
```

The retry needs `write` on the project and a person: an agent fixes the rule through its own
update instead, which releases the quarantine. It answers `409` for a rule that is not
quarantined. If the rule still fails, it is quarantined again after three more passes.

Only a rule's own failure counts. If every rule of your organization fails in a pass, or the
database times out, the failure is recorded against the evaluator and no rule is set aside.
Otherwise one database incident would quarantine every tenant at once.

**A rule that no longer parses** is listed with `definitionValid: false` and empty conditions,
filters and actions, and its `broken` state beside it. One such rule never fails the list.

**No lost page.** A fire is claimed on the cooldown ledger and then delivered. If delivery
throws, Bucker writes the alert deliveries it was about to make as `FAILED` rows due now, and
the ordinary retry pass sends them: the page arrives, once, on the next pass. Only if those rows
cannot be written either is the claim released, so the next pass fires the alert afresh. This
holds for issue, metric and anomaly rules alike.

**What was decided, and when.** Each fire and each failure is recorded, plus about one in ten of
the other evaluations, in `alert_evaluations` (value, threshold, outcome, and a minimal reason),
kept 14 days. It is the evidence behind "why did this fire" and "why did it not". Every change
that moves what a rule would fire or who hears of it is recorded in `alert_rule_state_changes`,
with the values before and after: a mute or unmute (from the API, Slack, or your own "mute for
me"), a maintenance window starting, ending or being deleted (one row per rule it covers), an
edit that enables or disables a rule or changes its conditions, filters, actions or cooldown
(from the dashboard, the API, `bucker.yml` or an agent), and a rule created by chat setup.

**Notices that have no rule.** Usage, spend and spike notices write a delivery receipt in
the same shape as an alert delivery (`status`, `attempts`, `lastError`, `deliveredAt`). A
send that reached nobody is recorded as `FAILED` and is not retried, because the fact it
reports is on the billing page.

**Archive integrity.** Each time the retention pass archives a day, it compares every
archive with its recorded row count and checksum and with its manifest, before the day's hot
partition is dropped, and writes an `archive_integrity` row.

## When a group stops reporting: NO_DATA

A group that is breaching or at warning and then **stops reporting** is not a group that
recovered. For a rule over a value that can be absent (an average, a percentile, a rate, a
percent change with no baseline) the evaluator holds such a group as **`NO_DATA`** on the
rule's ledger and does not resolve it. The ledger's states are one closed set:
`OK`, `WARNING`, `BREACHING`, `NO_DATA`.

- **Held, not resolved.** Entering `NO_DATA` sends nothing and closes nothing downstream. It
  writes the ledger row once; staying there writes nothing. A rule that asked to be told of
  recoveries is not told, because nothing recovered.
- **For the rule's retention.** `noDataGroupRetentionMinutes` (1 to 10,080, default 60) is
  how long the ledger keeps the quiet group. When it runs out the ledger **forgets** the
  group, still without a "resolved" message: a person, or the data returning, ends the
  incident. You can set it on a rule (API, `bucker.yml`, the rule editor for metric rules) and
  clear it with `null`.
- **Data comes back.** If it is healthy, a rule with `notifyOnRecovery` sends a recovery
  ("recovered from a gap in data"). If it is still breaching, the cooldown decides: inside it
  nothing is sent (the page nobody resolved is not repeated), past it you get the ordinary
  reminder.
- **What does not change.** A count with no matches is a real zero, so an empty
  `event_count` group still recovers. A group that was never open stays `OK` on no data. A
  learned (anomaly) threshold has its own "no data" verdict and is unaffected.

The rules list shows each rule's open groups (breaching, warning, no data) and
`GET .../alert-rules/{ruleId}/state` lists them with when a quiet group will be forgotten.
`GET .../alert-rules?state=NO_DATA` narrows the list (`OK` means a rule with none open).

## Shadow mode: agent-made rules deliver nothing until a person says yes

A rule an agent writes over MCP (`alert_rules_create`) or over REST as an agent, a service or a delegated client
**starts in SHADOW**. A person's own session, the dashboard, still writes a LIVE rule. The `lifecycle` is stored on
the rule (`SHADOW`, `LIVE` or `EXPIRED`) and is on every rule response and in `alert_rules_list`: it is never a tool
argument, and the plain edit routes refuse a lifecycle key.

- **It is evaluated like a live rule and delivers nothing.** Every fire writes one `SUPPRESSED` alert event per
  action with `suppression = {kind: "shadow"}`. A mute, a maintenance window and the "rule is not running" notice
  cannot change that: one gate, in the router, lets only a LIVE rule deliver, so a rule that is somehow EXPIRED
  and still enabled delivers nothing either. The `alert_volume` widget counts these rows as their own `SHADOW`
  series, not as pages a mute swallowed. Shadow evaluation counts against the same per-organization evaluation
  caps as a live rule.
- **At most 20 shadow rules per project per agent.** The 21st is refused before a person is asked. The rule
  records who authored it, the delegation chain, the grant the call stood on (`authorGrant`, a person-issued tool
  grant naming only `alert_rules_create` is how an agent is held to drafts) and the plan digest.
- **The record.** `GET …/alert-rules/:ruleId/shadow` and the promote dialog show what the rule did on real traffic:
  the fires it would have delivered, recoveries, groups, the destinations it would have reached (and which page a
  person), the fires the cooldown swallowed (an exact floor, plus an estimate from the sampled evaluations), and the
  MON-01 backtest it was proposed with, beside the actual fires per day.
- **Promotion** is `alert_rule.promote`, derived from a change of lifecycle: `alert_rules_update` with
  `lifecycle: "LIVE"`, or the promote button. It needs at least one day of evidence and is refused while a Monitor
  Quality blocking key (`missing_owner`, `missing_route`, `rule_broken`) is set, read live. For an agent it is asked,
  with the evidence on the card and its digest on the audit row, and a rule that carries a paging destination is
  always ask-floored. A person promotes from their own session directly.
- **Demotion** puts a live rule back into shadow for a stated time (`lifecycle: "SHADOW"` with `minutes`, at most
  seven days for an agent and thirty for a person): `alert_rule.demote`, silencing-class and CRITICAL on a rule that
  pages a person or burns an SLO budget. The rule is promoted again by itself at the end.
- **Editing a shadow rule** is ELEVATED, and an edit that makes it fire less needs no end time: nothing is
  delivered either way. Adding a paging destination still asks.
- **A shadow rule nobody promotes lapses after 14 days**: it becomes EXPIRED, disabled and archived under the
  seven-day tombstone, with a state-change row and an audit row (`alert_rule.shadow_expired`) as the receipt, and
  `window_ended` goes to the sponsor. After three days of evidence the sponsor gets a `shadow_report_ready` notice,
  unless a promotion request is already open. A rule that was LIVE and was demoted is never archived.

## Monitor Quality: a daily score per rule, and a gate for agent-made rules

Every enabled rule gets a score from 0 to 100 once a day, computed by the alerts scheduler from rows Bucker
already stores. There is no model and no estimate: each **key** is a published rule, and the thresholds are
versioned as `ruleSetVersion` (`monitor_quality_v1`), so a changed definition is visible on the rows it
produced. The rules list shows the score and its keys beside each rule name; `GET …/alert-rules/quality` and
the MCP tool `monitor_quality_get` return the same rows, worst first.

| Key | Set when | Points off |
| --- | --- | --- |
| `muted_over_60_days` | one continuous mute (back-to-back mutes are one) has run for 60 days or more and is still in force | 15 |
| `stuck_in_alert_over_60_days` | a metric rule has a BREACHING or WARNING group open whose first unrecovered fire since the last recovery is 60 days old | 15 |
| `noisy_top_5_percent` | among the noisiest 5 % of the project's rules by delivered fires over 10 days (at least 10 fires) | 10 |
| `missing_owner` | no resolvable owner (below) | 20, **blocking** |
| `missing_route` | no destination the rule can deliver to | 30, **blocking** |
| `broken_channel` | a destination is disabled, gone, or has had three dead deliveries and none delivered in 7 days | 15 |
| `fired_never_resolved` | the rule asks for recovery notices, fired three times or more in 90 days, and never recovered | 10 |
| `acted_on_rate_low` | under 0.2 of at least 30 fires in 90 days were acted on | 15 |
| `co_firing_80_percent` | fires in the same 5-minute window as one other rule at least 80 % of the time (at least 10 windows in 30 days) | 10 |
| `rule_broken` | BROKEN (the rule cannot be evaluated) | 30, **blocking** |
| `never_fired_180_days` | a live rule at least 180 days old that has not fired in 180 days | 10 |

A **fire is acted on** when its issue was assigned, resolved, archived or commented on within 24 hours. The
rate carries its 95 % Wilson interval and the number of fires it rests on, because a rate over a handful of fires
is a guess. **Weeks in which the alerts scheduler was down are left out of the baseline**: a gap of more than
six hours between its successful runs marks every 7-day bucket it touches, so silence is not learned as noise.

An **owner** is resolvable when the rule's creator is a person who still belongs to the organization, or the
project has an ownership rule that names at least one owner. A rule an agent drafted has no person who made
it, so it is an ownership rule that makes it owned. A **destination** is reachable when it is a channel of the
rule's organization that its project can use and that is enabled, or an owner route whose fallback channel is.
A channel id of another organization reads as gone.

Three keys are named by the rule set and **cannot be computed yet**: a composite rule that points at a deleted
rule (MON-25), a missing evaluation delay on a late-arriving dataset (MON-07), and a page nobody acknowledged
(MON-29). They are listed as `notChecked` on every read: an absent key is not a clear one for these.

**Blocking keys.** `missing_owner`, `missing_route` and `rule_broken` stop an agent-authored rule from being
promoted from SHADOW to live (MON-03). The promotion gate reads them live, not from the stored day:
`isPromotionBlocked(ruleId, orgId)` in `code/domain/src/modules/alerts/quality.ts`. It fails closed: a rule
that is not the organization's is blocked.

### Cleaning up in batches

An agent proposes a batch of at most 25 changes with `monitor_cleanup_propose`: a mute, a disable or a
narrowing edit, each with the minutes it lasts (at most seven days for an agent, the same bound as a mute),
or a reroute of one destination to a fallback channel (never onto a paging channel). One rule appears once.

- It is **ask-floored**: no policy row, review mode or autonomy rung lets a batch apply on its own, so the first
  call is always an approval card. One person decides the whole batch, and an approver may narrow it to a subset.
- Its **class is the highest item's**: SENSITIVE for a silencing item, CRITICAL when a rule that pages is
  silenced for more than four hours (which no agent is allowed by default), never below SENSITIVE. A reroute
  that takes a paging destination away (a pager swapped for a chat channel) is a silence with no end, so it is
  CRITICAL too; a pager swapped for another pager is not a silence.
- It is **applied item by item in one operation**, against the revisions of the project's rules it was asked
  on: an edit to any rule of the project between the yes and the retry closes it `TARGET_CHANGED` and nothing is
  applied. Each item writes its own state-change row and audit row, and the batch writes one more.
- Every item carries **its own inverse** (unmute, enable, restore the before values, reroute back), so
  `operation_revert` puts the whole batch back.
- A mute ends by its own clock. A disable and a narrowing are put back by the alerts scheduler's `cleanup_lapse`
  pass (every minute) once their minutes have passed, and only while the rule still carries exactly what the batch
  wrote: a rule someone has edited since is theirs. The restore, its state-change row and its audit row
  (`monitor_cleanup.lapse`) land in one transaction.
- The default window cap is one cleanup per agent per day.

The family is `monitor_cleanup` (`code/domain/src/modules/alerts/cleanup-family.ts`); the daily pass and the
score are `code/domain/src/modules/alerts/quality.ts`, registered in the detector registry as `monitor_quality`
(a finding, not an issue: it starts in shadow and opens nothing). The pass runs every six hours, checked every
fifteen minutes so a worker restarted more often still scores; it rewrites the day's row, and records a finding
once per rule per day, again only when the rule's keys move inside the day.

## Why did, or didn't, it fire

`GET .../alert-events?explain={eventId}` (or the MCP tool `alert_events_list` with
`include: ["explain"]`) returns one chain for an alert event:

`evaluated` -> `value` -> `state` -> `outcome` -> `suppression` -> `routing` -> `delivery`

Each link says where it came from: **`recorded`** (a row written at the time: the alert event
and its rendered payload, a sampled evaluation, the scheduler's runs, the rule's state-change
log), **`replayed`** (the live evaluator run over stored telemetry and the definition rebuilt
for that moment, the same replay the backtest uses, narrowed to one instant) or
**`not_available`**, with the reason. There is no "unknown".

For a moment with no event (`?ruleId={id}&at={time}`) the outcome is the evaluator's
(`fire`, `no_match`, `debounced`, `disabled`, `no_data`, `evaluation_failed`, ...) **or
`not_evaluated`**. A rule that was never looked at, because no scheduler pass ran around that
moment or the rule was quarantined, is reported as `not_evaluated` with the gap (the last pass
before, the first after, how many were missed). It is never reported as `no_match`, which
would tell you the threshold was fine when nothing was checked.

What can be replayed. The definition at a past moment is rebuilt from the rule's state-change
log by undoing the changes logged after it, and the silencing in force from the same log and
the maintenance windows. Nothing older than the log (and the 14 days of sampled evaluations,
the 30 days of scheduler runs) is rebuilt: the read says `not_available`. A replay answers
whether the **reading** crossed a line; the debounce ledger is not stored per instant, so a
cooldown is named only when a recorded alert inside it proves one. An issue rule fires on a
signal rather than a measured value and is not replayed at an instant. Where a recorded value
and the replay disagree (telemetry pruned since) the explanation shows both.

Group labels, the evaluator's reason (it embeds the group), window and route names and a
delivery's last error did not come from Bucker: they are returned as labelled, bounded,
control-free text. The MCP explanation needs `event:read`.

`alert_rule_get` returns one rule in full (thresholds as numbers and as labelled one-line
summaries, filters, actions, modifiers, tags, lifecycle, mute and disable state, who manages
it, evaluation health), and with `include` its open groups and its state-change history
(before, after, actor, mode and approval id).

## Where this is defined

- Shadow mode: `code/domain/src/modules/alerts/shadow.ts` (the cap, the passes, a person's own promote and demote),
  `shadow-evidence.ts` (the record), the delivery gate in `suppression.ts`, the promote act in `rule-family.ts` and the
  demote act in `silencing-family.ts`, the routes in `code/api/src/modules/alerts/routes-shadow.ts`.

- The rule DSL: `code/domain/src/modules/alerts/rules.ts`.
- The telemetry builder: `code/domain/src/modules/alerts/telemetry-metric.ts`, which
  reuses `code/domain/src/modules/explore/span-sql-builder.ts`.
- The learned threshold: `code/domain/src/modules/alerts/telemetry-anomaly.ts`, which
  uses `code/domain/src/modules/anomaly/seasonality.ts`.
- Groups, comparison and the points a rule is evaluated at:
  `code/domain/src/modules/alerts/metric-measure.ts`. Warning, critical and recovery:
  `evaluateMetricRule` in `code/domain/src/modules/alerts/evaluator.ts`.
- Evaluator liveness and the tenant batch: `code/domain/src/modules/alerts/evaluator-liveness.ts`.
  Quarantine: `code/domain/src/modules/alerts/rule-quarantine.ts`. Rule health and BROKEN:
  `code/domain/src/modules/alerts/health.ts`. The retry route and the `evaluation` block:
  `code/api/src/modules/alerts/routes-evaluation.ts`.
- The claim-and-dispatch step shared by the three evaluators:
  `code/domain/src/modules/alerts/claim-dispatch.ts`. The sampled evaluation record:
  `code/domain/src/modules/alerts/evaluations.ts`. The state log:
  `code/domain/src/modules/alerts/state-changes.ts`.
- NO_DATA and the retention: `evaluateMetricRule` in `evaluator.ts`, the ledger write in
  `evaluateMetricPoint` (`evaluation.ts`), `ALERT_GROUP_STATES` in `rules.ts`.
- The explanation: `code/domain/src/modules/alerts/explain.ts`, which replays one instant with
  `replayMetricRuleAt` in `backtest.ts`. The reads (`alertEventsList`, `alertRuleGet`, the
  rules list by state): `code/domain/src/modules/alerts/reads.ts`.
- Monitor Quality: `code/domain/src/modules/alerts/quality.ts` (the keys, the score, the daily pass, the gate),
  the read route in `code/api/src/modules/alerts/routes.ts`, the scheduler step `runQualityPass` in
  `code/workers/src/alerts/scheduler.ts`, and the batch family in
  `code/domain/src/modules/alerts/cleanup-family.ts`.
