# Logs: keys on every record, and how agents and people read them

A log record carries the keys it is joined and filtered on, and every door that reads logs
speaks the same search grammar. This page is the reference for what a stored record holds, how
`where` and `fields` work, what a search says about itself, and the three MCP tools that read
logs. The search box and its grammar are described under
[Searching logs](explore.md#searching-logs); how records arrive is in [OTLP ingest](otlp.md) and
[Log drains](log-drains.md). The archive, rehydration, facets, parse rules, patterns, findings and
log-based metrics are in [Logs beyond the hot store](log-tiers.md).

## What a record carries

| Field | What it is |
| --- | --- |
| `service` | The emitting service: OTLP `service.name`, or the drain's service (one per Vercel project, Worker, Heroku app or Supabase service). Null when the sender named none, and on every record written before the column existed. |
| `resource` | At most 16 allow-listed resource facts, strings only, each cut to 200 characters: `service.version`, `host.name`, `k8s.pod.name`, `k8s.namespace.name`, `cloud.region`, `faas.name`, `vercel.deployment.id`, `vercel.request.id`. A Vercel drain carries the last two on each line, so a record's own value wins over the batch's. Anything else a sender puts in its resource is not stored on the row. |
| `entityId` | The infrastructure entity the record's resource resolved to, when entity resolution has run. |
| `subjectKey` | Reserved for the keyed pseudonym of the person a record is about. **Empty until F-04 ships**: nothing writes it today, so it is not offered as a join key (`session_id` is in the same position). Erasure already matches it, so rows written after F-04 are reached without a change. |
| `traceId`, `spanId`, `issueId`, `release`, `environment`, `sampleRate` | As before. |

`service` and `resource` go through the same stored-tier scrub as the body and the attributes
before they are written: a credential in a host name is redacted, and what is not sensitive is
kept. A drain still drops the client address, the referer and the query string; nothing here
changes that.

In the Explore logs dataset the keys are `service`, `entity_id` and the path `resource.<key>`:
`service:checkout`, `resource.host.name:ip-10-0-0-7`. They work in the log search, in Explore
aggregates, in saved queries and in the MCP tools alike.

## Searching: `where`, `fields` and the evidence

`GET /orgs/:org/projects/:project/logs` takes the grammar in `query` (the name the log view has
always used) and, under the name the MCP tool uses, in `where`; if both are sent they are ANDed.
It also takes `fields`, up to ten attribute keys (or `resource.<key>`) separated by commas, and
returns each row's present keys as `projected`, so a reader gets `http.route` without the whole
attribute bag. More than ten fields is a 400.

Every page now says what it rests on, as `evidence`:

| Field | Meaning |
| --- | --- |
| `compiled` | The parse of `where`: `filters`, `text` and, for an `OR` or a negated group, `expression`. Null when no `where` was sent. |
| `window` | `from`, `to` and `defaulted`. A search the caller bounded by neither time, ids, a trace, a span nor an issue is bounded to the last 24 hours by the search itself (`defaulted: true`); the REST list keeps its older behaviour of reading newest-first from the whole table when it is sent no window. |
| `scanned`, `truncated`, `scannedThrough` | With a scan cap (the MCP tool sets one): how many matching rows were counted, up to the newest 100,000, whether more matched, and how far back the scan reached (the window's start when it was covered whole). A page never reaches past a cut scan. Null and false when the search was not capped, which is the dashboard's case. |
| `tier`, `index` | The tier that served it (`hot`) and the index behind it. A search with a body term says `trigram` (the body index exists and every partition has it) or `none`; any other search names the access path its filters let the planner use: `primary_key`, `project_trace_time`, `issue_time`, `project_service_time` or `project_time`. The planner chooses; this is the shape's. |
| `rowsExamined` | The rows the read looked at. Without the index a body search reads the newest 100,000 rows of its window and says how many it read; with it, the matching rows counted (`scanned`). Null when the search was not capped. |
| `nextArrivalCursor` | Where "what arrived since" resumes; see [Reading what arrived](#reading-what-arrived-since-a-cursor-not-a-stream). |
| `floorNote` | Records at error severity or worse, records with an exception, records linked to an issue and records on an error trace are never sampled away. Anything else may have been, and says so in its `sampleRate`. |
| `sampleRates` | The distinct sample rates below 1 on this page. A count over a sampled record is an estimate of `1 / rate` per row. |

`GET /orgs/:org/projects/:project/logs/:recordId` returns one record with its joins (below). A
record of another project, or of another organization, is a 404 identical to an unknown id.

## Searching the body

Three body terms, all case-insensitive, all combinable with the grammar and with each other:

- **contains**: `q` (and a free-text term in `query` or `where`), `body ILIKE '%term%'`;
- **prefix**: `bodyPrefix`, `body ILIKE 'term%'`;
- **regex**: `bodyRegex`, a POSIX regular expression (`body ~* pattern`; note `\y` is the word boundary and `\b` is a backspace). At most **100 characters**, **no backreferences** (`\1`..`\9`), and cancelled after **10 seconds** with a 422 that says so. A pattern longer than that, with a backreference or that does not compile is a 400. `*:term` over every attribute value is not offered: a blanket index over attribute values is a write cost this deployment does not take on.

`LIKE` characters in a term (`%`, `_`) are literals.

**With the index.** `log_records_body_trgm_idx` is a `pg_trgm` GIN index on `body` (`fastupdate`
on) on every `log_records` partition, behind the `log_index` pillar. It is built by code and by an
operator script, never by a migration: a migration would build it on every partition inside one
transaction on the busiest insert table, whether or not the operator chose its write cost.

- New partitions are indexed as they are created once the pillar is on (`pillar.log_index.enabled`,
  switched by the operator after the insert-cost harness, H-30).
- Existing partitions are indexed by `pnpm -C code --filter @bucker/api backfill-log-trgm` (dry run
  by default, `-- --apply` to build, `-- --days N` to take the newest N days first): one partition
  at a time, `CREATE INDEX CONCURRENTLY`, then attached to the parent index, resumable, and refused
  while the pillar is off. The DEFAULT partition is indexed in a full run.
- The response says `index: trigram` only when the parent index exists and every partition, the
  DEFAULT included, carries a valid trigram index of its own. A half-finished backfill reads
  `none`. (This is read from the partitions, not from the parent's own valid flag: Postgres sets
  that flag only on an attach, so a parent left invalid by a day that retention has since dropped
  would otherwise read `none` for good. A backfill stopped between building a day's index and
  attaching it attaches it on the next run rather than skipping the day.)
- Turning the pillar off stops this code indexing new partitions; it does not drop the index, and
  once the parent index exists Postgres indexes every new partition itself. Removing it is an
  operator's `DROP INDEX log_records_body_trgm_idx`.

**Without it** (`index: none`). A body search does not run into the statement timeout. It reads the
newest 100,000 rows of its window that pass the other filters, newest first, applies the body term
to them, and says so: `rowsExamined`, `truncated: true` and `scannedThrough`, the timestamp of the
oldest row it read. When the page is not full and the window is not covered, `hasMore` is true and
`nextCursor` continues from the last row read, so the next call reads the next slice. A search
with no window is bounded to 24 hours, never more than the deployment's hot window.

## Reading what arrived since: a cursor, not a stream

A person's live tail is a server-sent stream. An agent gets no stream: `search_logs` takes
`afterArrivalCursor`, the `nextArrivalCursor` of an earlier answer, and returns only the records
**received** after it, oldest first, so "what came in since my last read" is one budgeted call.

- Every `search_logs` answer carries `search.nextArrivalCursor`. Start from the one on any ordinary
  read, or from a change event's time, and pass it back. An ordinary read that found nothing hands
  out its own instant, so a quiet project can be watched from "now".
- An arrival read looks at the last **5 minutes of record time**: `receivedAt` is not indexed, and
  this window keeps the read a range scan on the timestamp index. A record stamped by a clock more
  than five minutes behind its arrival is in a search, not in an arrival read.
- Nothing new returns an empty page and the same cursor. When a page was cut for the token budget
  the cursor resumes after the last record kept, so a record dropped for size is read next time.
- **At most 6 arrival reads a minute per principal per project**; the seventh is a 429 carrying the
  seconds to wait. An ordinary read is not counted. The limit is per api machine.

### The dashboard tail's bounds

`GET …/logs/stream` is bounded so one tenant cannot starve ingest of the primary's 8 connections:

| Bound | Value |
| --- | --- |
| Open tails | 3 per project and 10 per organization, per api machine; the next is a 429 |
| Poll interval | 2 seconds, 10 seconds after 30 seconds with nothing new, back to 2 on a record |
| One tick's query | a 2 second statement timeout; a tick that cannot finish is retried from the same cursor |
| A bad `bodyRegex` | a 400 before the stream opens (too long, a backreference, or a pattern Postgres cannot compile), taking no slot |
| High volume | above 500 matching records a second a tick keeps a trace-hash sample (a trace is kept or dropped whole) and sends `{ kind: 'sampled', rate, note: 'sampled at 0.333' }` |
| A capped tick | sends `{ kind: 'capped', shown, skipped }`: how many matching records it did not deliver |

## The MCP tools

| Tool | What it does | Needs |
| --- | --- | --- |
| `search_logs` | Rows. `where`, `fields`, a `cursor` and the evidence above; every hit carries its `id`, `service`, `sampleRate` and the short id of its issue. An empty answer says why (`not_ingested`, `no_data_in_window`, `filtered_out`). | `project:read` |
| `logs_query` | Numbers: counts and measures per bucket and group, through the query printer. | `event:read` |
| `log_record_get` | One record with its joins in 1,200 tokens. | `event:read` |

Everything these tools return that came from outside Bucker (bodies, attribute keys and values,
service names, group labels, titles) passes the agent-tier scrub and the injection scanner and is
named in `trust.untrustedFields`. Attributes, projected `fields` included, are scrubbed as a bag,
so the rules that match on a key apply: `user.id`, `user.email` and `user.ip_address` come back
hashed, and a credential-named key comes back filtered.

### `logs_query`

"ERROR lines by service every 5 minutes for 6 hours" is one call:

```json
{ "org": "acme", "project": "checkout", "where": "level:>=error",
  "groupBy": ["service"], "intervalMinutes": 5, "windowHours": 6 }
```

- `where` is the log search grammar, **AND only**: an `OR` or a negated group is refused with the
  parse echoed, rather than narrowed to a different question. A simple negation (`!env:staging`)
  is fine.
- `measures` (up to 4): `count` (the default), `count_unique`, `sum`, `avg`, `min`, `max`, `p50`,
  `p75`, `p95`, `p99`. All but `count` take a `field`.
- `groupBy` takes up to three keys. **Each key is its own breakdown** over the same filters and
  window: three marginal series, not the cross product. The printer groups by one key per run.
- A series is a dense array per measure: `v.<measure>[i]` is the bucket at `from + i × step`.
- Each breakdown says `countBasis`: `exact`, or `sampled` when any scanned row was stored
  sampled, in which case every count is a lower bound and the `sampling` block gives the number
  of sampled rows, the lowest rate and the corrected estimate.
- The scan is the newest 100,000 matching rows inside the window (at most 30 days) and says so
  when it was cut. It is charged to the QUERY_SCAN budget before it runs and logged on the `mcp`
  surface.
- With more than one key, a later page carries only the breakdowns that still have groups to
  read; each names its `groupBy`.
- When the answer would exceed its budget the smallest groups are dropped, `omitted` says how
  many, and no cursor is offered; ask again with a smaller `limit`.

### `log_record_get`

The record, plus:

| Join | Source | When it is empty |
| --- | --- | --- |
| `issue` | the issue the record became: short id, status, level, title | `no_key`, or `not_found` |
| `trace` | a summary of the record's trace: span count, duration, services, error spans, the root. At most 200 spans are read; `truncated` says when more exist. | `no_key`, or `not_kept` (logs and traces are sampled independently, so a log can name a trace that was not stored) |
| `release` | the release and its deploy change event: when, the summary, minutes between | `no_key`; the deploy is null when none was recorded |
| `flags` | the variants the request's root span carried (`flagsBasis: trace_root`), else the variants the issue's events carried over a span of time that covers the record (`issue_events`) | `no_key`, or `not_found` |
| `pattern` | a placeholder until log patterns ship | always `not_available` |
| `neighbours` | the five nearest records in time, within 15 minutes, on the trace, else on the service, else in the project | `not_found` |

The bundle is shrunk in a fixed order to fit: attribute values, then neighbour bodies, then
neighbours from the farthest, then the attribute bag. The head of the body, the severity, the
service and every join's key are never cut, and `omitted` counts what was.

## Quotas

A log quota caps one **scope**, a service or a filter in the log grammar, at a number of records or
bytes per UTC day. Past the cap a record that is not protected is dropped, or sampled at a stated rate
(`overflow: drop | sample`), and counted under the drop reason `log_quota`. Records at error severity
or worse, records with an exception, records linked to an issue and records on an error trace
**always pass**, whatever the quota says; they are counted as `floorKept`.

A quota is `shadow` (counts what it would have dropped and drops nothing), `active` or `disabled`.
Each has a `warnAtPercent` threshold and a `resetHourUtc` for where its day starts.

`GET /orgs/:org/projects/:project/log-quotas` (project `read`) returns every quota with today's state:

| Field | Meaning |
| --- | --- |
| `used`, `percentUsed` | What the quota let through today, in its unit. |
| `dropped` | What it removed; for a shadow quota, what it would have removed. |
| `floorKept` | Protected records that arrived after the cap and were stored anyway. |
| `warnedAt`, `cappedAt` | When the warning threshold and the cap were first passed today. |
| `overshoot` | The per-machine note below. |

A quota may overshoot its limit. Counters live in each api machine and are merged every ten seconds, so
with N machines it can overshoot by up to N times ten seconds of traffic, and by one record (or one
record's bytes) per batch in flight. A record that matches several quotas is dropped by the first capped
one and charges no other for it. If the quota store cannot be read the batch is stored whole: a quota
fails open.

**Authoring.** This door only reads. The governed acts that create or change a quota (`log_quota.create`,
`log_quota.update`, always asked of a person, never automatic) are defined but no route or MCP tool
submits them yet; see [Actions](actions.md). To shape volume today, use [Log pipeline rules](log-pipeline.md).

## Drains

`GET /orgs/:org/projects/:project/log-drains` (project `read`) says how logs reach the project: each
drain's coverage state, last delivery, records in the last hour against its 14-day normal, rejected and
lost counts in 24 hours, and losses by a closed reason, broken drains first. The states are `live`,
`lossy`, `silent` and `unknown`. Because the counters are hourly, a drain that delivers every five
minutes is declared silent after about three hours here and after fifteen minutes in the finding that
reads the same drain, and the answer says so in `resolution`. The answer carries no credential and no
platform-chosen text, and a platform price only where Bucker holds one (otherwise
`not_checked`: it bills on your own account). Setting drains up is in [Log drains](log-drains.md).

## Purging records

A person who finds a secret or personal data in stored logs can delete the affected records, or have
them rewritten under the current scrub rules. Both are a **preview, then a scheduled job**, and nothing is
deleted by an agent.

Who: a person on their own session holding org `admin` (an organization owner). An agent, a service
principal, an OAuth-client token and a derived credential are refused by name before any row is read.
There is no MCP tool for it. The jobs list and a job read are org `admin` too, because a selection can
hold a literal about a person.

1. **Preview.** `POST /orgs/:org/projects/:project/log-purges/preview` with `kind` (`purge` or `rescrub`
   as the API names them) and a `selection`: a closed UTC window (`from`, `to`, not in the future, at
   most 400 days) and an optional `where` in the log grammar. The job is stored as `previewed` with the
   count it matches per day. Nothing is deleted. The selection is fenced at the instant of the preview:
   records received afterwards are never in it.
2. **Challenge.** `POST …/log-purges/:jobId/schedule/challenge` with a `reason` (10 to 500 characters)
   and `acknowledgeMatched`, the count the preview showed. It returns a step-up challenge bound to this
   job, that reason and that count. It is refused with a 409 unless the job is `previewed`, and with a
   `STEP_UP_REQUIRED` error when the person has no second factor enrolled.
3. **Schedule.** `POST …/log-purges/:jobId/schedule` with the same two fields and `stepUp`, the answer
   to the challenge. The count is re-checked: if the selection now matches a different number, the call
   is refused and you preview again. A project has one open job at a time. It answers 202 with
   `executeAfter`: the job starts **72 hours** after the server clock says it was scheduled, and until
   then it can be cancelled (`POST …/cancel`).
4. **Run.** After the window a worker deletes (or rewrites) the records per UTC day in batches of
   10,000, and a crash resumes where the last batch ended. On completion a signed receipt is added to
   the organization's receipt log with the matched, removed and rewritten counts.

The step-up is enforced when the deployment runs in production, and a proof is single use: it is spent
by the schedule call, a spent code is refused on the next job, and a refused or missing proof leaves the
job `previewed`. The audit row for the schedule carries `metadata.stepUp`, the method used.

What it does **not** touch: rollups and derived series keep their counts (they hold no record content),
and the receipt names the stores that still count the removed records instead of pretending otherwise.
A rewrite re-runs `body`, `attributes`, `resource`, `service`, `environment` and `release` through the
stored-tier scrub with your rules as they are now. For a finding, `GET …/findings/:findingId/leak-response`
lays out the three responses (fix at source, add a scrub rule, purge or rewrite stored logs) and how many
stored records the selection would match; the order it recommends is redact first, then rewrite.

## When a capability is off

Several log features sit behind operator flags that ship off ([the table](log-tiers.md#what-is-off-by-default)).
What you see while one is off: a body search without the trigram index runs, bounded, and says
`index: none` with the rows it examined; `logs_patterns_get` has no templates to return; archive search says `reason: pillar_off` and is refused; parse and rename rules are
stored and do nothing. No answer pretends a feature ran when it did not.

## What this page does not cover

- **Exception records inside a logs export are not split out.** A log record that carries
  `exception.*` becomes an issue at ingest, and the export of a project's logs still lists it as
  one row among the others. Separating the exception records in an export is open.
- **`subjectKey` and `session_id`** are not populated; see above.
- **The `pattern` join** on `log_record_get` is still `not_available` on every record. Templates are read with `logs_patterns_get` instead, see [Logs beyond the hot store](log-tiers.md#patterns).
- **Insert cost of the index** is measured by the capacity harness before an operator switches the pillar on; this page states the mechanism, not the measurement.
