Skip to content
bucker

Logs beyond the hot store: archive, rehydration, facets, parse rules, patterns and findings

Logs describes what a stored record holds and how the hot store is searched. This page covers what sits around it: the archive of older days and how to search or bring one back, the facets that name the keys you filter on, the parse rules that turn a line into fields, the patterns the miner finds, the findings that say what is wrong with how a project sends logs, and the log-based metric that counts lines before anything drops them. Quotas and purging are on the Logs page; the doors that deliver logs are in Log drains and OTLP ingest, and the keep, drop, sample and transform rules are in Log pipeline rules.

What is off by default

Every capability on this page that adds write or read volume on the shared database sits behind a pillar: an operator flag, pillar.<name>.enabled, that ships off. A flag that cannot be read counts as off. An operator turns one on in the admin console, and turning it on is refused while a capacity gauge it depends on is in breach unless the operator gives a reason. Turning one off is never refused. A person using the product does not set a pillar; what they see while one is off is below.

Pillar Gives While it is off
cold_tier and cold_logs (both) Archiving log days to encrypted Parquet; logs_cold_search; rehydration Nothing is archived. logs_cold_search_preview answers enabled: false, reason: pillar_off with zeros; logs_cold_search is refused with a 403 that names the flag. Days past the hot window are gone, not archived.
log_templates Log patterns (logs_patterns_get), the volume advisor No template is mined or counted. (pattern on log_record_get reads not_available either way today.)
log_stages Parse, rename, remove and severity rules at ingest The batch passes through unchanged. Rules can still be stored.
log_metrics The log-based metric stage No point is written and no rule is read.
log_index The trigram index behind body search A body search reads a bounded slice and says so (Logs).
log_classes A retention class per record Writers use the old table, as before.
log_units Log-unit metering of ingest, drains and archive scans Scans are priced and shown on the preview, never recorded.

If a feature on this page returns nothing and you expected something, ask whoever operates your deployment whether its pillar is on. The reason is stated in the answer where the door can say it.

Searching the archive

The archive holds a project's log days after they leave the hot window, one encrypted Parquet object per project and day. A search over it is priced before it runs and bounded.

  1. Price it. logs_cold_search_preview takes a window (from, to, or days, default 30, at most 90) and reads the catalogue only: archived objects, rows and stored bytes, the estimated scan units, whether the scan guard allows it, the caller's remaining query-scan budget, the project's archived days and its running rehydrations with their expiry. No log text is returned.
  2. Search it. logs_cold_search takes the same window and filters: service, minSeverityNumber, traceId, templateHash, sessionId, issueId, release and environment, each exact (a list matches any), plus text, a case-insensitive substring of the body. Rows come back newest first, 50 by default and at most 200, with a cursor that is bound to the request it came from.
{ "org": "acme", "project": "checkout", "days": 30 }

What to know before relying on it:

  • A window is at most 90 days. The guard refuses a query that would scan more than 50 GiB, and a project that has scanned more than 200 GiB that day. A refusal is data on the preview (guard.allowed: false and which limit), not an error.
  • Structured filters are pushed down into the objects. text is applied to the rows the engine returns (up to 10,000), and the answer says how many it examined and whether that was complete. There is no aggregation over the archive: for numbers use logs_query on the hot store.
  • A search over your query-scan budget does not fail: it returns status: "budget_paused" and waits for a person to raise the budget. Ask again afterwards; a page is a fresh read, priced again.
  • Each row says tier: cold. Archived text was scrubbed when it was written, and it passes the same agent-tier scrub and injection scanner as a hot line on the way out.
  • Price. Bytes scanned are priced in log units and shown on the preview and on every answer. While the log-unit price is undecided the figure is shown and not charged (chargedUnits: 0).
  • It needs event:read on the project; the logs toolset carries both tools.

A project can also have its archive copied, in an open format, to a bucket you own: Bucker writes Parquet files and a manifest per day to a destination a person configured, and does not read that bucket back. That is an export, not a search; see Exports.

Bringing one archived day back

log_rehydration_propose copies one archived day (day: "YYYY-MM-DD", with a reason of at most 500 characters) back into the hot store, so the hot search, logs_query and alerts can read it.

  • It is always a request, never an immediate action. The first call returns an input_required ask that shows the day, the archived rows and bytes the plan reads from the catalogue, and the expiry. Only the retry that names a person's approval applies, once. No policy, review mode or autonomy rung turns it automatic.
  • Saying yes records the rehydration; a worker then copies the rows in under retention class r7 and re-applies the current scrub rules on the way, then removes them seven days later. At most 10 million records. A rehydrated row is never archived a second time.
  • If the day was rewritten between the ask and the yes (an erasure, for example) the apply is refused as a moved target and nothing starts.
  • End one early with operation_revert on that operation.
  • It needs logs:write, a scope the MCP server now lists (the SG-059 holds were lifted by owner decision on 2026-11-16, with no signed-off security review of the OAuth server; see MCP setup).

Facets

A facet is a name for a key you filter on. log_facets_create declares one in three shapes: a key, an alias standing for up to five sender spellings of the same thing (userId and usr.id for user.id), or an extraction: a pattern with a named group over the body or one attribute. A type (string, number, duration, bytes), a unit and a display name can be set. Up to 100 facets a project, 10 of them extractions.

  • Nothing stored is rewritten. An alias is resolved at query time, so it covers records already stored.
  • A plain facet or an extraction applies straight away. An alias asks a person when an alert rule, saved view, widget, log metric or quota already reads one of the keys it covers, because the alias changes what that reader matches.
  • A name the log search already means something by, a pattern that can backtrack, one with a backreference or no named group, and a name already taken are refused before anyone is asked.
  • logs_facets_get is the cheap map of a project: each facet with its coverage, the top 20 values and, for numbers, min, max, p50 and p95, computed over at most 10,000 matching records and marked ~ when that is a sample. log_facets_update changes or retires one.

Parse rules

A parse rule extracts fields from the lines of one log template, so a line like paid order=8841 ms=212 gets order and ms as attributes. An agent writes one from real lines: logs_patterns_get with include: ["samples"] and a templateHash returns up to 50 lines of that template, then log_parse_rules_create proposes a parse_regex or parse_kv rule.

Bucker does not trust the author. The rule is replayed on a different half of that template's lines, ones the agent was never shown, against published bounds: at least 50 unseen lines, 80 percent matched, every declared field present on 80 percent of them and 95 percent of its values of the declared type, and a cost within 200 microseconds a record. It lands in shadow (it counts and extracts nothing visible); a rule that fails is quarantined and cannot be enabled. Going live is log_parse_rules_update with status: "active", which asks a person, who sees the replay. A project holds at most 5 shadow rules and 50 transform rules. At ingest a parse rule runs only while log_stages is on (Log pipeline rules).

Patterns

The miner reduces log lines to templates (the line with its variables taken out) and counts them. logs_patterns_get returns each with its count, share, worst severity, services, first and last seen and up to three record ids to open with log_record_get. Asked for a comparison it says what is new, what went away and what is spiking against the same hour a week earlier, which facet values are over-represented among errors, and which change events landed in between. No model clusters anything. The text of a template is untrusted: one that looks like an instruction comes back as its count alone.

log_patterns_update mutes a template for up to 30 days (mutedUntil) or unmutes it (null). A mute asks a person and is refused while an issue for that template is open at high priority; unmuting only makes the product say more and applies without asking. A mute that runs out reopens the template with nothing to sweep.

Findings

A daily pass writes findings about how a project sends logs. log_findings_list returns them by family:

  • health: a service that sends spans and no logs, severity that lives in the body, logs with no trace id, records that hit the intake caps, busy templates with no attributes, the daily template cap reached, SDKs far behind;
  • volume: patterns that cost the most and are read the least, each with a ready proposal for a shadow drop or sample rule. Nothing is applied; the rule it would create lands in shadow and going live is a person's act. A pattern that an alert, saved view, widget or saved query reads is never advised for dropping. This needs log_templates;
  • hygiene: a weekly judgement of which rules, facets, aliases, log metrics and quotas have stopped earning their keep (a rule nothing matched in 30 days, one in shadow for 14 days or more, a facet nothing reads, a metric with no point in 30 days, a quota whose busiest day never passed half its limit). Nothing is deleted or changed by the check; a cleanup is an act a person approves.

A finding clears itself the first pass its check passes. log_findings_update suppresses one for up to 30 days (7 by default), which asks a person and needs a reason, or reopens it. A caller holding only project:read sees a template's hash and no template text, because raw log text needs event:read.

Counting lines before they drop: log-based metrics

A log metric counts the records a filter selects, or aggregates a numeric facet of them (sum, min, max, a distribution with p50 to p99), into one-minute buckets grouped by up to four declared facets. It runs after parse rules and before every drop, so a line a sampling rule or quota later removes still counted. That is what lets you turn a noisy line into a count and drop the line.

  • Points are stored apart from custom metrics, are not billed as custom metrics, and do not count against the 1,000-series cap. A project may have 25 metrics in force.
  • A group facet must be a declared, active attribute facet. A facet that names a person (user or subject id, email, session id, client address) is refused as a group: no point is keyed by a person.
  • Read them in Explore through the log_metric dataset. They are written only while log_metrics is on.
  • Convert then drop is one approved plan with two parts: the metric, active at once, and a drop or sample rule over the same filter, born in shadow. The drop goes live only after the metric matched the raw count within 1 percent over 24 hours. The log_pipeline_preview tool shows which readers (alerts, widgets) could move onto the metric and which read the lines themselves.

Not yet reachable. The governed acts that create, change or retire a log metric, a quota or a conversion (log_metric.*, log_quota.*, log_conversion.create) are defined and registered, and listed as planned in Actions, but no route or MCP tool submits them today. What a person or agent can do now is read the state, as in Quotas.

The HTTP log door

Request logs from a host that cannot run a script go to …/http-log/; the format, the limits and the rejection reasons are under The http-log door, and what the classifier does with the lines is in Web agents.

See also

Logs · Log pipeline rules · Log drains · Scrubbing and privacy · Agent governance · Actions · MCP setup