# Logs beyond the hot store: archive, rehydration, facets, parse rules, patterns and findings

[Logs](logs.md) describes what a stored record holds and how the hot store is searched. This page
covers what sits around it: the archive of older days and how to search or bring one back,
the facets that name the keys you filter on, the parse rules that turn a line into fields, the
patterns the miner finds, the findings that say what is wrong with how a project sends logs, and the
log-based metric that counts lines before anything drops them. Quotas and purging are on the
[Logs](logs.md#quotas) page; the doors that deliver logs are in [Log drains](log-drains.md) and
[OTLP ingest](otlp.md), and the keep, drop, sample and transform rules are in
[Log pipeline rules](log-pipeline.md).

## What is off by default

Every capability on this page that adds write or read volume on the shared database sits behind a
**pillar**: an operator flag, `pillar.<name>.enabled`, that ships **off**. A flag that cannot be
read counts as off. An operator turns one on in the admin console, and turning it on is refused
while a capacity gauge it depends on is in breach unless the operator gives a reason. Turning one
off is never refused. A person using the product does not set a pillar; what they see while one is
off is below.

| Pillar | Gives | While it is off |
| --- | --- | --- |
| `cold_tier` and `cold_logs` (both) | Archiving log days to encrypted Parquet; `logs_cold_search`; rehydration | Nothing is archived. `logs_cold_search_preview` answers `enabled: false, reason: pillar_off` with zeros; `logs_cold_search` is refused with a 403 that names the flag. Days past the hot window are gone, not archived. |
| `log_templates` | Log patterns (`logs_patterns_get`), the volume advisor | No template is mined or counted. (`pattern` on `log_record_get` reads `not_available` either way today.) |
| `log_stages` | Parse, rename, remove and severity rules at ingest | The batch passes through unchanged. Rules can still be stored. |
| `log_metrics` | The log-based metric stage | No point is written and no rule is read. |
| `log_index` | The trigram index behind body search | A body search reads a bounded slice and says so ([Logs](logs.md#searching-the-body)). |
| `log_classes` | A retention class per record | Writers use the old table, as before. |
| `log_units` | Log-unit metering of ingest, drains and archive scans | Scans are priced and shown on the preview, never recorded. |

If a feature on this page returns nothing and you expected something, ask whoever operates your
deployment whether its pillar is on. The reason is stated in the answer where the door can say it.

## Searching the archive

The archive holds a project's log days after they leave the hot window, one encrypted Parquet object
per project and day. A search over it is priced before it runs and bounded.

1. **Price it.** `logs_cold_search_preview` takes a window (`from`, `to`, or `days`, default 30, at
   most 90) and reads the catalogue only: archived objects, rows and stored bytes, the estimated
   scan units, whether the scan guard allows it, the caller's remaining query-scan budget, the
   project's archived days and its running rehydrations with their expiry. No log text is returned.
2. **Search it.** `logs_cold_search` takes the same window and filters: `service`,
   `minSeverityNumber`, `traceId`, `templateHash`, `sessionId`, `issueId`, `release` and
   `environment`, each exact (a list matches any), plus `text`, a case-insensitive substring of the
   body. Rows come back newest first, 50 by default and at most 200, with a cursor that is bound to
   the request it came from.

```json
{ "org": "acme", "project": "checkout", "days": 30 }
```

What to know before relying on it:

- A window is at most **90 days**. The guard refuses a query that would scan more than **50 GiB**,
  and a project that has scanned more than **200 GiB** that day. A refusal is data on the preview
  (`guard.allowed: false` and which limit), not an error.
- Structured filters are pushed down into the objects. `text` is applied to the rows the engine
  returns (up to 10,000), and the answer says how many it examined and whether that was complete.
  There is **no aggregation** over the archive: for numbers use `logs_query` on the hot store.
- A search over your query-scan budget does not fail: it returns `status: "budget_paused"` and
  waits for a person to raise the budget. Ask again afterwards; a page is a fresh read, priced again.
- Each row says `tier: cold`. Archived text was scrubbed when it was written, and it passes the same
  agent-tier scrub and injection scanner as a hot line on the way out.
- **Price.** Bytes scanned are priced in log units and shown on the preview and on every answer. While
  the log-unit price is undecided the figure is shown and **not charged** (`chargedUnits: 0`).
- It needs `event:read` on the project; the `logs` toolset carries both tools.

A project can also have its archive copied, in an open format, to a bucket you own: Bucker writes
Parquet files and a manifest per day to a destination a person configured, and does not read that
bucket back. That is an export, not a search; see [Exports](exports.md).

## Bringing one archived day back

`log_rehydration_propose` copies one archived day (`day: "YYYY-MM-DD"`, with a `reason` of at most
500 characters) back into the hot store, so the hot search, `logs_query` and alerts can read it.

- It is always a request, never an immediate action. The first call returns an `input_required` ask
  that shows the day, the archived rows and bytes the plan reads from the catalogue, and the expiry.
  Only the retry that names a **person's approval** applies, once. No policy, review mode or autonomy
  rung turns it automatic.
- Saying yes records the rehydration; a worker then copies the rows in under retention class `r7` and
  **re-applies the current scrub rules** on the way, then removes them **seven days** later. At most
  **10 million records**. A rehydrated row is never archived a second time.
- If the day was rewritten between the ask and the yes (an erasure, for example) the apply is refused
  as a moved target and nothing starts.
- End one early with `operation_revert` on that operation.
- It needs `logs:write`, a scope the MCP server now lists (the SG-059 holds were lifted by owner
  decision on 2026-11-16, with no signed-off security review of the OAuth server; see
  [MCP setup](mcp-setup.md)).

## Facets

A facet is a name for a key you filter on. `log_facets_create` declares one in three shapes: a
**key**, an **alias** standing for up to five sender spellings of the same thing (`userId` and
`usr.id` for `user.id`), or an **extraction**: a pattern with a named group over the body or one
attribute. A type (string, number, duration, bytes), a unit and a display name can be set. Up to
**100** facets a project, **10** of them extractions.

- Nothing stored is rewritten. An alias is resolved at query time, so it covers records already stored.
- A plain facet or an extraction applies straight away. An **alias asks a person** when an alert
  rule, saved view, widget, log metric or quota already reads one of the keys it covers, because the
  alias changes what that reader matches.
- A name the log search already means something by, a pattern that can backtrack, one with a
  backreference or no named group, and a name already taken are refused before anyone is asked.
- `logs_facets_get` is the cheap map of a project: each facet with its coverage, the top 20 values
  and, for numbers, min, max, p50 and p95, computed over at most 10,000 matching records and marked
  `~` when that is a sample. `log_facets_update` changes or retires one.

## Parse rules

A parse rule extracts fields from the lines of **one log template**, so a line like
`paid order=8841 ms=212` gets `order` and `ms` as attributes. An agent writes one from real lines:
`logs_patterns_get` with `include: ["samples"]` and a `templateHash` returns up to 50 lines of that
template, then `log_parse_rules_create` proposes a `parse_regex` or `parse_kv` rule.

Bucker does not trust the author. The rule is replayed on a different half of that template's lines,
ones the agent was never shown, against published bounds: at least 50 unseen lines, 80 percent
matched, every declared field present on 80 percent of them and 95 percent of its values of the
declared type, and a cost within 200 microseconds a record. It lands in **shadow** (it counts and
extracts nothing visible); a rule that fails is **quarantined** and cannot be enabled. Going live is
`log_parse_rules_update` with `status: "active"`, which **asks a person**, who sees the replay. A
project holds at most 5 shadow rules and 50 transform rules. At ingest a parse rule runs only while
`log_stages` is on ([Log pipeline rules](log-pipeline.md#transform-rules-processing-at-ingest)).

## Patterns

The miner reduces log lines to templates (the line with its variables taken out) and counts them.
`logs_patterns_get` returns each with its count, share, worst severity, services, first and last
seen and up to three record ids to open with `log_record_get`. Asked for a comparison it says what
is new, what went away and what is spiking against the same hour a week earlier, which facet values
are over-represented among errors, and which change events landed in between. No model clusters
anything. The text of a template is untrusted: one that looks like an instruction comes back as its
count alone.

`log_patterns_update` mutes a template for up to 30 days (`mutedUntil`) or unmutes it (`null`). A
mute **asks a person** and is refused while an issue for that template is open at high priority;
unmuting only makes the product say more and applies without asking. A mute that runs out reopens
the template with nothing to sweep.

## Findings

A daily pass writes findings about how a project sends logs. `log_findings_list` returns them by
`family`:

- `health`: a service that sends spans and no logs, severity that lives in the body, logs with no
  trace id, records that hit the intake caps, busy templates with no attributes, the daily template
  cap reached, SDKs far behind;
- `volume`: patterns that cost the most and are read the least, each with a ready proposal for a
  shadow drop or sample rule. Nothing is applied; the rule it would create lands in shadow and going
  live is a person's act. A pattern that an alert, saved view, widget or saved query reads is never
  advised for dropping. This needs `log_templates`;
- `hygiene`: a weekly judgement of which rules, facets, aliases, log metrics and quotas have stopped
  earning their keep (a rule nothing matched in 30 days, one in shadow for 14 days or more, a facet
  nothing reads, a metric with no point in 30 days, a quota whose busiest day never passed half its
  limit). Nothing is deleted or changed by the check; a cleanup is an act a person approves.

A finding clears itself the first pass its check passes. `log_findings_update` suppresses one for up
to 30 days (7 by default), which asks a person and needs a reason, or reopens it. A caller holding
only `project:read` sees a template's hash and no template text, because raw log text needs
`event:read`.

## Counting lines before they drop: log-based metrics

A log metric counts the records a filter selects, or aggregates a numeric facet of them (sum, min,
max, a distribution with p50 to p99), into one-minute buckets grouped by up to four declared facets.
It runs **after** parse rules and **before** every drop, so a line a sampling rule or quota later
removes still counted. That is what lets you turn a noisy line into a count and drop the line.

- Points are stored apart from custom metrics, are not billed as custom metrics, and do not count
  against the 1,000-series cap. A project may have 25 metrics in force.
- A group facet must be a declared, active attribute facet. A facet that names a person (user or
  subject id, email, session id, client address) is refused as a group: no point is keyed by a person.
- Read them in Explore through the `log_metric` dataset. They are written only while `log_metrics` is on.
- **Convert then drop** is one approved plan with two parts: the metric, active at once, and a drop or
  sample rule over the same filter, born in shadow. The drop goes live only after the metric matched
  the raw count within 1 percent over 24 hours. The `log_pipeline_preview` tool shows which readers
  (alerts, widgets) could move onto the metric and which read the lines themselves.

**Not yet reachable.** The governed acts that create, change or retire a log metric, a quota or a
conversion (`log_metric.*`, `log_quota.*`, `log_conversion.create`) are defined and registered, and
listed as planned in [Actions](actions.md), but no route or MCP tool submits them today. What a
person or agent can do now is read the state, as in [Quotas](logs.md#quotas).

## The HTTP log door

Request logs from a host that cannot run a script go to `…/http-log/`; the format, the limits and the
rejection reasons are under [The http-log door](log-drains.md#the-http-log-door), and what the
classifier does with the lines is in [Web agents](web-agents.md).

## See also

[Logs](logs.md) · [Log pipeline rules](log-pipeline.md) · [Log drains](log-drains.md) ·
[Scrubbing and privacy](scrubbing-and-privacy.md) · [Agent governance](agent-governance.md) ·
[Actions](actions.md) · [MCP setup](mcp-setup.md)
