# Explore: spans, error events, logs and metrics

Explore answers questions over a project's telemetry with one query engine. This page says
what it does today and where its limits are.

## Datasets

The **Dataset** switcher at the start of the query row picks one of four tables. The
aggregate, field, group-by and filter controls work the same way on each.

| Dataset | Rows | Time column | Promoted keys |
| --- | --- | --- | --- |
| **Spans** | every span from OTLP, the SDK's tracing, and Sentry SDK transactions | span start | `span.name`, `span.kind`, `span.status`, `span.duration` (ms), `environment`, `release`, `trace_id`, `resource.<key>` |
| **Errors** | every error event that reached an issue | the time Bucker **received** it | `level`, `message`, `error.type`, `error.value`, `environment`, `release`, `platform`, `transaction`, `server_name`, `trace_id`, `span_id`, `issue`, `user` (hashed) |
| **Logs** | every OTLP or SDK log record | the record's timestamp | `severity`, `severity_number`, `message`, `environment`, `release`, `trace_id`, `span_id`, `issue` |
| **Metrics** | every data point received over OTLP `/v1/metrics` ([OTLP](otlp.md#metrics)) | the point's timestamp | `metric.name`, `metric.unit`, `metric.kind`, `metric.temporality`, `service`, `environment`, `release`, `trace_id` (the first exemplar's trace), `series`; numeric: `metric.value`, `metric.count`, `metric.sum`, `metric.min`, `metric.max` |

Any other key is looked up among the row's attributes: span attributes, log attributes, a
metric point's attributes, or an event's `tags`. An event whose tags are not a key/value map counts as having no tags.

The Errors dataset filters on the time the event was received, not the time the SDK
stamped it. Only the received time is indexed per project. For a live SDK the two are
seconds apart.

Querying **Errors** needs the `issue:read` scope on the token, because those rows are
issue data. A token limited to an incident cannot query it.

### Metrics

A gauge or sum point has `metric.value`; a histogram, exponential histogram or summary point has
`metric.count` (observations) and `metric.sum`, so `avg` of `metric.value` grouped by
`metric.name` charts a gauge, and `sum` of `metric.count` counts observations. Percentiles over
a histogram METRIC are percentiles of the per-point values you pick, not of the underlying
observations: the bucket layout is stored but not aggregated in SQL.

A point that carried an **exemplar** has a `trace_id`, so a row in **Show matching rows** links
to the trace the value was observed in, and **Same trace** joins metrics to spans, logs and errors
in either direction. A point with no exemplar has no trace and never matches a join.

If a metric's attributes include something unbounded (a user or request id), Bucker keeps the
first 1,000 distinct series per metric per day and reports the rest on the drops panel as
`metric_series_cap`; see [OTLP](otlp.md#metrics).

## Aggregates

Count, unique values, sum, average, min, max and p50/p75/p95/p99, optionally grouped by one
key and bucketed over time. Every query has three limits, and the answer always says which
of them applied:

- at most 30 days;
- at most 100,000 rows read (the newest rows win, and the result says so);
- at most 50 series, and grouping is refused past 1,000 distinct values.

**Show the compiled query**, under every result, prints the exact statement that produced
the numbers, with its values in order (`$1`, `$2`, …).

### Counters and histograms (metrics dataset)

`p95` on a metric is the 95th percentile of the stored point *values*, which for a histogram
are per-point totals: the p95 of how many requests each scrape counted, not of how long they
took. Seven more functions read a metric by its meaning, and they exist only on the
**Metrics** dataset. They take no field, and the filters must name the metric
(`metric.name` equals):

| Function | Answers |
| --- | --- |
| `counter_increase` | How much a counter grew in the window. |
| `counter_rate` | The same, per second of the window (or of the bucket, in a series). |
| `histogram_p50`, `_p75`, `_p90`, `_p95`, `_p99` | The percentile of the *observations*, from merged buckets. |
| `histogram_mean` | The sum of the observations over their count. |
| `histogram_rate` | Observations per second. |

**Counters.** Per series, a delta point is its own increase and a cumulative point is its
difference from the previous point of the same series; a fall is read as a restart, and the
new value is the increase. Two replicas of one service are two series (they differ by
instance), so they are never differenced against each other. A series' first point, with no
earlier point within 15 minutes, contributes nothing.

**Histograms.** Per series, cumulative points are differenced bucket by bucket in time order
(delta points are used as sent). A restart is a falling count, a falling bucket, a changed
`startTime` or a changed bucket layout, and the point is then taken whole. Increments are
summed across series that share a bucket layout (`layoutDigest`); exponential histograms at
different scales are downscaled to the smallest, which is exact. The percentile is
interpolated inside the bucket that holds the rank, the way Prometheus `histogram_quantile`
does (linearly for explicit buckets, geometrically for exponential ones), and **the bounds of
that bucket are returned as the answer's `interval`**: the true percentile is in it.

Every answer carries `groups[].detail`:

| Field | Meaning |
| --- | --- |
| `status` | `ok`, `no_data`, `insufficient_observations`, `layouts_differ` or `unreadable_buckets` |
| `basis` | `buckets` |
| `observations` | Observations merged into the answer |
| `interval` | `[lower, upper]` of the bucket holding the rank; `upper` is `null` for the open top bucket |
| `layouts` | The bucket layouts merged (a hash of the bounds) |
| `pointsMerged`, `resetsDetected` | Points differenced, and restarts read through |

Below **20 observations** a percentile is `insufficient_observations` with `value: null`,
never a number. Series whose explicit layouts differ are never merged: the answer is a
**422 `layouts_differ`**, or one answer per layout (`layout` on each group) when the request
sets `"onLayouts": "split"`. A mean and a rate merge no buckets, so they do not need layouts
to agree. Summary quantiles cannot be merged and are skipped, with a warning.

The response also carries `metric: { seriesBasis, resetsDetected, instancesMerged }`:
`instance_key` when every series the read touched was told apart by its instance id,
`no_instance_key` when it could not tell two replicas apart (a counter can then over-count),
and how many replicas were merged. `sum` over a **cumulative monotonic counter** adds up
running totals; the answer says so in `warnings` (`cumulative_sum_adds_running_totals`) with a
`suggestedSpec` for the `counter_rate` question you probably meant. An alert condition with
the same function and filters computes the same number: see
[Alert rules](alert-rules.md#custom-metrics).

## From a chart to Compare

On a span chart, drag across the region that looks wrong, or pick its first and last bucket
under the chart and press **Compare this window**. The **Spike vs baseline** tab opens with
that region as the spike window and the same length of time just before it as the baseline,
and ranks the attributes that differ. It is offered for spans, the dataset Compare ranks.

## Same-trace join

**Same trace** keeps only rows whose trace also has, or has no, matching rows in another
dataset. Some questions it answers:

- spans of traces that logged an `error` (dataset Spans, join Logs, `severity is error`);
- log records of traces that raised an error event (dataset Logs, join Errors);
- checkout spans whose trace has no error (join mode "Trace has no").

A row with no trace id never matches a join, in either mode. The joined rows are looked up
in the same time window.

## Matching rows and columns

**Show matching rows** under a result lists the rows behind it, 20 at a time, newest
first. **Columns** picks what the table shows, from the dataset's promoted keys and the
attribute keys found in recent rows, or any key you type. The picked columns stay in the
URL with the rest of the query.

## Reading one trace

The **Trace** tab draws a trace as a table, one row per span, nested under its parent.

- **Collapse** a span with the arrow beside its name. The row then says how many spans it
  hides and the time they cover, for example "+37 spans, 212 ms".
- A row says how many log records were emitted in the span and how many error events it
  raised ("2 logs · 1 error"). Error counts need the `issue:read` scope.
- **Open** a span by its name, or with `j`/`k` and Enter. The pane beside the table shows
  its status, timing, attributes, span events and resource, the log records emitted in it,
  and its error events with a link to the issue each one belongs to. The span is in the URL
  as `?span=`, so **Copy link** points at that one span.
- **Zoom** by dragging across the ruler above the timeline, or with **Zoom timeline to this
  span** in the pane. **Reset zoom** shows the whole trace again. A span outside the zoomed
  window keeps its row and loses its bar.

Attributes, events and log bodies are shown exactly as the instrumented service sent them,
labelled as untrusted text.

## Searching logs

The **Logs** search box speaks the same search grammar as the Issues list and the events
table (`@bucker/shared/search` is the one parser behind all three): `key:value` terms,
`OR`, parentheses, `!` negation, `[a,b]` lists and `*` wildcards, as well as text. Terms
next to each other must all match; `OR` (capitals, as in Issues) lets either side match,
and `AND` binds tighter than `OR`. Unlike Issues, a key the registry does not know is not
an error: log attributes are free-form, so any key is read as an attribute path.

| You type | It means |
| --- | --- |
| `user.id:42` | the attribute equals `42`, whether it was sent as a number or as text |
| `!environment:staging` (or `-environment:staging`) | not equal |
| `duration_ms:>500`, `>=`, `<`, `<=` | a numeric comparison |
| `http.url:*stripe*` / `http.route:/api/*` | contains / starts with |
| `severity:[warn,error]` (or `severity:warn,error`) | one of several values |
| `user.id:7 OR severity:fatal` | either side matches; `AND` binds tighter than `OR` |
| `(user.id:7 OR plan:pro) !severity:error` | parentheses group; `!` also negates a group |
| `severity:>=warn` | that severity or worse, using OTel severity numbers |
| `has:tenant` / `!has:tenant` (or `-has:tenant`) | the attribute is present / absent |
| `"card declined"`, `retry` | text in the message; every term must match |

`level:` means `severity:`; `trace:`, `span:` and `env:` mean `trace_id:`, `span_id:` and
`environment:`. `service:` is the emitting service of a record and `resource.<key>:` addresses
the resource facts a record keeps (`resource.host.name:ip-10-0-0-7`); both are described, with the
other keys on a log record, in [Logs](logs.md). A term that does not parse (an unclosed parenthesis, a dangling `OR`, `key:`) is searched
for as text, never refused. `-key:value` is the older spelling of `!key:value` and keeps
working. A `*suffix` or `a*b` pattern matches every part of it somewhere in the value, not
strictly at the end. The
line under the box shows what the search was read as.

- **Show ±60s around this record**, in an open record, shows every record from the 60
  seconds either side of it, in the same environment, with the search cleared.
- **Pin to top** keeps a record above the stream. Pins are in the URL and in a saved view.
- **Follow** adds new records to the top of the stream as they arrive, with the same
  search applied. It shows at most 100 records per two seconds and says so when more
  arrived. It works on the first page of a relative time range. If the live connection
  fails it refreshes every few seconds. It follows by the time Bucker received a record,
  over records stamped within the last five minutes.
- **Chart** draws the same search as counts, or unique values of one attribute, over
  time, optionally grouped by one key. It runs through Explore's logs dataset with the same
  limits.
- The bookmark button saves the search, severity, environment, trace, time range and pins
  as a named log view.

## Saved queries

The bookmark button beside **Run query** saves the question under a name, privately. Its
menu pins it, shares it with the organization (admins and owners can share), or deletes
it. Opening a saved query fills in the query row; it does not run it until you press
**Run query**. Saved queries are the same saved views the issue stream uses, with
`surface: explore`, so they follow the same visibility rule. A name is unique per person
per organization across issue views, saved queries and saved log views.

## Dashboard widgets from Explore

**Save as widget** adds the question to a dashboard as an **Explore query** widget, drawn
as a line chart, bars, a big number or a table. The widget uses the dashboard's time range
in place of the query's, and keeps every Explore limit. Dashboards cover at most 30 days for
these widgets. The widget says how many rows it read, and says so when the row limit cut
the answer short.

Dashboards never read raw error events, so a query over the **Errors** dataset, or one
joined to it, cannot become a widget. Chart errors with the issue widgets.

**Add widget** on a dashboard also offers **Note**, a text tile. It renders headings, lists,
bold, emphasis, code and `http(s)` links. It never renders HTML, so a tag you type shows as
text.

## Sampling rules

**Settings › Projects › Sampling rules** decides what share of a project's spans and logs
Bucker stores, without changing any SDK. A rule has conditions in the Explore key
vocabulary and a share to keep: `span.name is GET /health` at 0% drops health checks;
`http.route starts with /search` at 10% keeps a tenth of search traffic. Rules apply in
order and the first match decides. A record no rule matches is stored.

Four things always hold:

- **Errors are kept.** A span with an error status keeps every span of its trace that
  arrived with it. A log at ERROR or worse, or one carrying an exception, is always kept.
- **A trace is kept or dropped whole.** The decision is a stable hash of the trace id, so
  the spans of one trace agree across batches and across OTLP and SDK ingest. A log on a
  trace follows its trace.
- **The rate is stored.** Every kept span or log record records the share it was kept at.
  Explore counts what is stored, says how many of the rows it read were sampled and at what
  lowest rate, and gives an estimate of the count before sampling only when you ask for it.
- **The rule says what it did.** Each rule shows how many records it kept and dropped since
  it was created.

Records a rule drops on OTLP ingest are not billed. They appear on the drops panel under
the sampling-rule reason. SDK spans and logs dropped by a rule were already counted when the
envelope arrived, so they show on the rule's own counts and not as a second drop. A changed
rule applies within five seconds.

## API

| Route | Purpose |
| --- | --- |
| `POST /orgs/:org/projects/:project/explore/aggregate` | aggregate over `dataset: spans \| events \| logs \| metrics`, with `filters`, an optional `sameTrace: { dataset, filters, negate }` and, for `histogram_*`, `onLayouts: refuse \| split` |
| `POST /orgs/:org/projects/:project/explore/rows` | raw rows with `columns` (at most 20), keyset-paginated by `cursor` |
| `GET /orgs/:org/projects/:project/explore/attributes?dataset=` | the key menu for one dataset |
| `GET /orgs/:org/projects/:project/explore/traces/:traceId/waterfall` | the trace's rows, each with `logCount` and `errorCount` (null without `issue:read`) |
| `GET /orgs/:org/projects/:project/explore/traces/:traceId/spans/:spanId` | one span with up to 50 of its logs and 20 of its error events; `errorsHidden: true` without `issue:read` |
| `POST /orgs/:org/projects/:project/explore/spans/aggregate` | the original span-only aggregate, unchanged |
| `GET /orgs/:org/projects/:project/logs?query=` | log search with the grammar above (`where=` is the same grammar under the MCP tool's name); `ids=` fetches pinned records; `fields=` projects up to ten attribute keys per row; the page's `interpreted` says what ran and its `evidence` says what it rests on ([Logs](logs.md)) |
| `GET /orgs/:org/projects/:project/logs/:recordId` | one record with its issue, trace summary, release and deploy, flags and neighbours ([Logs](logs.md)) |
| `GET /orgs/:org/projects/:project/logs/stream` | the live tail, as server-sent events; each event's `id` resumes it |
| `GET`/`POST /orgs/:org/projects/:project/sampling-rules`, `PATCH`/`DELETE …/:ruleId` | manage sampling rules: `{ name, signal: spans \| logs, filters, rate: 0..1, enabled }`; reads need project read, writes project write, and every write is audited |
| `POST /orgs/:org/views` with `surface: explore` and `query: { kind: 'explore', query }` | save a query; `GET /orgs/:org/views?surface=explore` lists them |
| `PATCH /orgs/:org/dashboards/:id` with a widget `{ type: 'explore_query', config: { query, projectSlug?, display } }` | add a widget; `display` is `line`, `bar`, `big_number` or `table` |

## Ask in plain words

Press **Ctrl/⌘ + /** anywhere, or the **Ask** button beside the issue, Explore and log search
boxes. Type a question such as `p95 checkout latency by release last 24h`; Bucker compiles it
with a fixed phrase table into the query that surface already runs — an issue search, an Explore
aggregate, or a log search — and shows that query, editable, with every phrase it recognised,
before anything runs. Nothing is sent to a model and nothing is changed: running it is the same
read as typing the query yourself (`POST /orgs/:org/ask` returns the compiled query only).

A *why* question ("why did WEB-42 start failing?") is not answered by the compiler. It offers to
open root-cause analysis on the named issue, where hypotheses carry their own confidence and
evidence, and any action you take from there goes through the usual autonomy and approval rules.

## The query log

Every read Explore, log search, trace detail and the exports compile is recorded, so "which
agent read this last week, and on whose behalf" has an answer. A run is one row: who ran it (the
principal and its kind, the person sponsoring an agent, the OAuth client when there was one),
from where (`dashboard`, `api`, `cli`, `mcp`, `export`), what it was (a digest of the query's
shape, and a digest of the exact statement it compiled to) and what it cost (rows scanned and
returned, duration, whether the scan was truncated). A refused run is recorded too, with the
reason code.

Filter values are never stored: a value can be a customer identifier, so the log keeps only the
digests. Two runs of the same question over different time ranges share a `specDigest`.
A dashboard widget or an alert rule does not write a row per refresh; it adds to one hourly
summary row, which is kept 400 days. Raw rows are kept 30 days.

| Request | Returns |
| --- | --- |
| `GET /orgs/:org/projects/:project/query-runs` | the log, newest first, with the usual `cursor`; filters `surface`, `principalKind`, `since` |
| `GET /orgs/:org/projects/:project/query-runs/summary` | the last seven days per surface and principal kind, widget and alert summaries included |

With `org:read` you see every principal's runs in the project. Without it (a token scoped to
the project, an agent token) you see only your own, and the response says `scope: "own"`.
The dashboard and the CLI label their own requests with an `x-bucker-surface` header; anything
else is `api`. The label is attribution only, nothing is decided on it.
