Logs beyond the hot store: archive, rehydration, facets, parse rules, patterns and findings
Logs describes what a stored record holds and how the hot store is searched. This page covers what sits around it: the archive of older days and how to search or bring one back, the facets that name the keys you filter on, the parse rules that turn a line into fields, the patterns the miner finds, the findings that say what is wrong with how a project sends logs, and the log-based metric that counts lines before anything drops them. Quotas and purging are on the Logs page; the doors that deliver logs are in Log drains and OTLP ingest, and the keep, drop, sample and transform rules are in Log pipeline rules.
What is off by default
Every capability on this page that adds write or read volume on the shared database sits behind a
pillar: an operator flag, pillar.<name>.enabled, that ships off. A flag that cannot be
read counts as off. An operator turns one on in the admin console, and turning it on is refused
while a capacity gauge it depends on is in breach unless the operator gives a reason. Turning one
off is never refused. A person using the product does not set a pillar; what they see while one is
off is below.
| Pillar | Gives | While it is off |
|---|---|---|
cold_tier and cold_logs (both) |
Archiving log days to encrypted Parquet; logs_cold_search; rehydration |
Nothing is archived. logs_cold_search_preview answers enabled: false, reason: pillar_off with zeros; logs_cold_search is refused with a 403 that names the flag. Days past the hot window are gone, not archived. |
log_templates |
Log patterns (logs_patterns_get), the volume advisor |
No template is mined or counted. (pattern on log_record_get reads not_available either way today.) |
log_stages |
Parse, rename, remove and severity rules at ingest | The batch passes through unchanged. Rules can still be stored. |
log_metrics |
The log-based metric stage | No point is written and no rule is read. |
log_index |
The trigram index behind body search | A body search reads a bounded slice and says so (Logs). |
log_classes |
A retention class per record | Writers use the old table, as before. |
log_units |
Log-unit metering of ingest, drains and archive scans | Scans are priced and shown on the preview, never recorded. |
If a feature on this page returns nothing and you expected something, ask whoever operates your deployment whether its pillar is on. The reason is stated in the answer where the door can say it.
Searching the archive
The archive holds a project's log days after they leave the hot window, one encrypted Parquet object per project and day. A search over it is priced before it runs and bounded.
- Price it.
logs_cold_search_previewtakes a window (from,to, ordays, default 30, at most 90) and reads the catalogue only: archived objects, rows and stored bytes, the estimated scan units, whether the scan guard allows it, the caller's remaining query-scan budget, the project's archived days and its running rehydrations with their expiry. No log text is returned. - Search it.
logs_cold_searchtakes the same window and filters:service,minSeverityNumber,traceId,templateHash,sessionId,issueId,releaseandenvironment, each exact (a list matches any), plustext, a case-insensitive substring of the body. Rows come back newest first, 50 by default and at most 200, with a cursor that is bound to the request it came from.
{ "org": "acme", "project": "checkout", "days": 30 }
What to know before relying on it:
- A window is at most 90 days. The guard refuses a query that would scan more than 50 GiB,
and a project that has scanned more than 200 GiB that day. A refusal is data on the preview
(
guard.allowed: falseand which limit), not an error. - Structured filters are pushed down into the objects.
textis applied to the rows the engine returns (up to 10,000), and the answer says how many it examined and whether that was complete. There is no aggregation over the archive: for numbers uselogs_queryon the hot store. - A search over your query-scan budget does not fail: it returns
status: "budget_paused"and waits for a person to raise the budget. Ask again afterwards; a page is a fresh read, priced again. - Each row says
tier: cold. Archived text was scrubbed when it was written, and it passes the same agent-tier scrub and injection scanner as a hot line on the way out. - Price. Bytes scanned are priced in log units and shown on the preview and on every answer. While
the log-unit price is undecided the figure is shown and not charged (
chargedUnits: 0). - It needs
event:readon the project; thelogstoolset carries both tools.
A project can also have its archive copied, in an open format, to a bucket you own: Bucker writes Parquet files and a manifest per day to a destination a person configured, and does not read that bucket back. That is an export, not a search; see Exports.
Bringing one archived day back
log_rehydration_propose copies one archived day (day: "YYYY-MM-DD", with a reason of at most
500 characters) back into the hot store, so the hot search, logs_query and alerts can read it.
- It is always a request, never an immediate action. The first call returns an
input_requiredask that shows the day, the archived rows and bytes the plan reads from the catalogue, and the expiry. Only the retry that names a person's approval applies, once. No policy, review mode or autonomy rung turns it automatic. - Saying yes records the rehydration; a worker then copies the rows in under retention class
r7and re-applies the current scrub rules on the way, then removes them seven days later. At most 10 million records. A rehydrated row is never archived a second time. - If the day was rewritten between the ask and the yes (an erasure, for example) the apply is refused as a moved target and nothing starts.
- End one early with
operation_reverton that operation. - It needs
logs:write, a scope the MCP server now lists (the SG-059 holds were lifted by owner decision on 2026-11-16, with no signed-off security review of the OAuth server; see MCP setup).
Facets
A facet is a name for a key you filter on. log_facets_create declares one in three shapes: a
key, an alias standing for up to five sender spellings of the same thing (userId and
usr.id for user.id), or an extraction: a pattern with a named group over the body or one
attribute. A type (string, number, duration, bytes), a unit and a display name can be set. Up to
100 facets a project, 10 of them extractions.
- Nothing stored is rewritten. An alias is resolved at query time, so it covers records already stored.
- A plain facet or an extraction applies straight away. An alias asks a person when an alert rule, saved view, widget, log metric or quota already reads one of the keys it covers, because the alias changes what that reader matches.
- A name the log search already means something by, a pattern that can backtrack, one with a backreference or no named group, and a name already taken are refused before anyone is asked.
logs_facets_getis the cheap map of a project: each facet with its coverage, the top 20 values and, for numbers, min, max, p50 and p95, computed over at most 10,000 matching records and marked~when that is a sample.log_facets_updatechanges or retires one.
Parse rules
A parse rule extracts fields from the lines of one log template, so a line like
paid order=8841 ms=212 gets order and ms as attributes. An agent writes one from real lines:
logs_patterns_get with include: ["samples"] and a templateHash returns up to 50 lines of that
template, then log_parse_rules_create proposes a parse_regex or parse_kv rule.
Bucker does not trust the author. The rule is replayed on a different half of that template's lines,
ones the agent was never shown, against published bounds: at least 50 unseen lines, 80 percent
matched, every declared field present on 80 percent of them and 95 percent of its values of the
declared type, and a cost within 200 microseconds a record. It lands in shadow (it counts and
extracts nothing visible); a rule that fails is quarantined and cannot be enabled. Going live is
log_parse_rules_update with status: "active", which asks a person, who sees the replay. A
project holds at most 5 shadow rules and 50 transform rules. At ingest a parse rule runs only while
log_stages is on (Log pipeline rules).
Patterns
The miner reduces log lines to templates (the line with its variables taken out) and counts them.
logs_patterns_get returns each with its count, share, worst severity, services, first and last
seen and up to three record ids to open with log_record_get. Asked for a comparison it says what
is new, what went away and what is spiking against the same hour a week earlier, which facet values
are over-represented among errors, and which change events landed in between. No model clusters
anything. The text of a template is untrusted: one that looks like an instruction comes back as its
count alone.
log_patterns_update mutes a template for up to 30 days (mutedUntil) or unmutes it (null). A
mute asks a person and is refused while an issue for that template is open at high priority;
unmuting only makes the product say more and applies without asking. A mute that runs out reopens
the template with nothing to sweep.
Findings
A daily pass writes findings about how a project sends logs. log_findings_list returns them by
family:
health: a service that sends spans and no logs, severity that lives in the body, logs with no trace id, records that hit the intake caps, busy templates with no attributes, the daily template cap reached, SDKs far behind;volume: patterns that cost the most and are read the least, each with a ready proposal for a shadow drop or sample rule. Nothing is applied; the rule it would create lands in shadow and going live is a person's act. A pattern that an alert, saved view, widget or saved query reads is never advised for dropping. This needslog_templates;hygiene: a weekly judgement of which rules, facets, aliases, log metrics and quotas have stopped earning their keep (a rule nothing matched in 30 days, one in shadow for 14 days or more, a facet nothing reads, a metric with no point in 30 days, a quota whose busiest day never passed half its limit). Nothing is deleted or changed by the check; a cleanup is an act a person approves.
A finding clears itself the first pass its check passes. log_findings_update suppresses one for up
to 30 days (7 by default), which asks a person and needs a reason, or reopens it. A caller holding
only project:read sees a template's hash and no template text, because raw log text needs
event:read.
Counting lines before they drop: log-based metrics
A log metric counts the records a filter selects, or aggregates a numeric facet of them (sum, min, max, a distribution with p50 to p99), into one-minute buckets grouped by up to four declared facets. It runs after parse rules and before every drop, so a line a sampling rule or quota later removes still counted. That is what lets you turn a noisy line into a count and drop the line.
- Points are stored apart from custom metrics, are not billed as custom metrics, and do not count against the 1,000-series cap. A project may have 25 metrics in force.
- A group facet must be a declared, active attribute facet. A facet that names a person (user or subject id, email, session id, client address) is refused as a group: no point is keyed by a person.
- Read them in Explore through the
log_metricdataset. They are written only whilelog_metricsis on. - Convert then drop is one approved plan with two parts: the metric, active at once, and a drop or
sample rule over the same filter, born in shadow. The drop goes live only after the metric matched
the raw count within 1 percent over 24 hours. The
log_pipeline_previewtool shows which readers (alerts, widgets) could move onto the metric and which read the lines themselves.
Not yet reachable. The governed acts that create, change or retire a log metric, a quota or a
conversion (log_metric.*, log_quota.*, log_conversion.create) are defined and registered, and
listed as planned in Actions, but no route or MCP tool submits them today. What a
person or agent can do now is read the state, as in Quotas.
The HTTP log door
Request logs from a host that cannot run a script go to …/http-log/; the format, the limits and the
rejection reasons are under The http-log door, and what the
classifier does with the lines is in Web agents.
See also
Logs · Log pipeline rules · Log drains · Scrubbing and privacy · Agent governance · Actions · MCP setup