Explore: spans, error events, logs and metrics
Explore answers questions over a project's telemetry with one query engine. This page says what it does today and where its limits are.
Datasets
The Dataset switcher at the start of the query row picks one of four tables. The aggregate, field, group-by and filter controls work the same way on each.
| Dataset | Rows | Time column | Promoted keys |
|---|---|---|---|
| Spans | every span from OTLP, the SDK's tracing, and Sentry SDK transactions | span start | span.name, span.kind, span.status, span.duration (ms), environment, release, trace_id, resource.<key> |
| Errors | every error event that reached an issue | the time Bucker received it | level, message, error.type, error.value, environment, release, platform, transaction, server_name, trace_id, span_id, issue, user (hashed) |
| Logs | every OTLP or SDK log record | the record's timestamp | severity, severity_number, message, environment, release, trace_id, span_id, issue |
| Metrics | every data point received over OTLP /v1/metrics (OTLP) |
the point's timestamp | metric.name, metric.unit, metric.kind, metric.temporality, service, environment, release, trace_id (the first exemplar's trace), series; numeric: metric.value, metric.count, metric.sum, metric.min, metric.max |
Any other key is looked up among the row's attributes: span attributes, log attributes, a
metric point's attributes, or an event's tags. An event whose tags are not a key/value map counts as having no tags.
The Errors dataset filters on the time the event was received, not the time the SDK stamped it. Only the received time is indexed per project. For a live SDK the two are seconds apart.
Querying Errors needs the issue:read scope on the token, because those rows are
issue data. A token limited to an incident cannot query it.
Metrics
A gauge or sum point has metric.value; a histogram, exponential histogram or summary point has
metric.count (observations) and metric.sum, so avg of metric.value grouped by
metric.name charts a gauge, and sum of metric.count counts observations. Percentiles over
a histogram METRIC are percentiles of the per-point values you pick, not of the underlying
observations: the bucket layout is stored but not aggregated in SQL.
A point that carried an exemplar has a trace_id, so a row in Show matching rows links
to the trace the value was observed in, and Same trace joins metrics to spans, logs and errors
in either direction. A point with no exemplar has no trace and never matches a join.
If a metric's attributes include something unbounded (a user or request id), Bucker keeps the
first 1,000 distinct series per metric per day and reports the rest on the drops panel as
metric_series_cap; see OTLP.
Aggregates
Count, unique values, sum, average, min, max and p50/p75/p95/p99, optionally grouped by one key and bucketed over time. Every query has three limits, and the answer always says which of them applied:
- at most 30 days;
- at most 100,000 rows read (the newest rows win, and the result says so);
- at most 50 series, and grouping is refused past 1,000 distinct values.
Show the compiled query, under every result, prints the exact statement that produced
the numbers, with its values in order ($1, $2, …).
Counters and histograms (metrics dataset)
p95 on a metric is the 95th percentile of the stored point values, which for a histogram
are per-point totals: the p95 of how many requests each scrape counted, not of how long they
took. Seven more functions read a metric by its meaning, and they exist only on the
Metrics dataset. They take no field, and the filters must name the metric
(metric.name equals):
| Function | Answers |
|---|---|
counter_increase |
How much a counter grew in the window. |
counter_rate |
The same, per second of the window (or of the bucket, in a series). |
histogram_p50, _p75, _p90, _p95, _p99 |
The percentile of the observations, from merged buckets. |
histogram_mean |
The sum of the observations over their count. |
histogram_rate |
Observations per second. |
Counters. Per series, a delta point is its own increase and a cumulative point is its difference from the previous point of the same series; a fall is read as a restart, and the new value is the increase. Two replicas of one service are two series (they differ by instance), so they are never differenced against each other. A series' first point, with no earlier point within 15 minutes, contributes nothing.
Histograms. Per series, cumulative points are differenced bucket by bucket in time order
(delta points are used as sent). A restart is a falling count, a falling bucket, a changed
startTime or a changed bucket layout, and the point is then taken whole. Increments are
summed across series that share a bucket layout (layoutDigest); exponential histograms at
different scales are downscaled to the smallest, which is exact. The percentile is
interpolated inside the bucket that holds the rank, the way Prometheus histogram_quantile
does (linearly for explicit buckets, geometrically for exponential ones), and the bounds of
that bucket are returned as the answer's interval: the true percentile is in it.
Every answer carries groups[].detail:
| Field | Meaning |
|---|---|
status |
ok, no_data, insufficient_observations, layouts_differ or unreadable_buckets |
basis |
buckets |
observations |
Observations merged into the answer |
interval |
[lower, upper] of the bucket holding the rank; upper is null for the open top bucket |
layouts |
The bucket layouts merged (a hash of the bounds) |
pointsMerged, resetsDetected |
Points differenced, and restarts read through |
Below 20 observations a percentile is insufficient_observations with value: null,
never a number. Series whose explicit layouts differ are never merged: the answer is a
422 layouts_differ, or one answer per layout (layout on each group) when the request
sets "onLayouts": "split". A mean and a rate merge no buckets, so they do not need layouts
to agree. Summary quantiles cannot be merged and are skipped, with a warning.
The response also carries metric: { seriesBasis, resetsDetected, instancesMerged }:
instance_key when every series the read touched was told apart by its instance id,
no_instance_key when it could not tell two replicas apart (a counter can then over-count),
and how many replicas were merged. sum over a cumulative monotonic counter adds up
running totals; the answer says so in warnings (cumulative_sum_adds_running_totals) with a
suggestedSpec for the counter_rate question you probably meant. An alert condition with
the same function and filters computes the same number: see
Alert rules.
From a chart to Compare
On a span chart, drag across the region that looks wrong, or pick its first and last bucket under the chart and press Compare this window. The Spike vs baseline tab opens with that region as the spike window and the same length of time just before it as the baseline, and ranks the attributes that differ. It is offered for spans, the dataset Compare ranks.
Same-trace join
Same trace keeps only rows whose trace also has, or has no, matching rows in another dataset. Some questions it answers:
- spans of traces that logged an
error(dataset Spans, join Logs,severity is error); - log records of traces that raised an error event (dataset Logs, join Errors);
- checkout spans whose trace has no error (join mode "Trace has no").
A row with no trace id never matches a join, in either mode. The joined rows are looked up in the same time window.
Matching rows and columns
Show matching rows under a result lists the rows behind it, 20 at a time, newest first. Columns picks what the table shows, from the dataset's promoted keys and the attribute keys found in recent rows, or any key you type. The picked columns stay in the URL with the rest of the query.
Reading one trace
The Trace tab draws a trace as a table, one row per span, nested under its parent.
- Collapse a span with the arrow beside its name. The row then says how many spans it hides and the time they cover, for example "+37 spans, 212 ms".
- A row says how many log records were emitted in the span and how many error events it
raised ("2 logs · 1 error"). Error counts need the
issue:readscope. - Open a span by its name, or with
j/kand Enter. The pane beside the table shows its status, timing, attributes, span events and resource, the log records emitted in it, and its error events with a link to the issue each one belongs to. The span is in the URL as?span=, so Copy link points at that one span. - Zoom by dragging across the ruler above the timeline, or with Zoom timeline to this span in the pane. Reset zoom shows the whole trace again. A span outside the zoomed window keeps its row and loses its bar.
Attributes, events and log bodies are shown exactly as the instrumented service sent them, labelled as untrusted text.
Searching logs
The Logs search box speaks the same search grammar as the Issues list and the events
table (@bucker/shared/search is the one parser behind all three): key:value terms,
OR, parentheses, ! negation, [a,b] lists and * wildcards, as well as text. Terms
next to each other must all match; OR (capitals, as in Issues) lets either side match,
and AND binds tighter than OR. Unlike Issues, a key the registry does not know is not
an error: log attributes are free-form, so any key is read as an attribute path.
| You type | It means |
|---|---|
user.id:42 |
the attribute equals 42, whether it was sent as a number or as text |
!environment:staging (or -environment:staging) |
not equal |
duration_ms:>500, >=, <, <= |
a numeric comparison |
http.url:*stripe* / http.route:/api/* |
contains / starts with |
severity:[warn,error] (or severity:warn,error) |
one of several values |
user.id:7 OR severity:fatal |
either side matches; AND binds tighter than OR |
(user.id:7 OR plan:pro) !severity:error |
parentheses group; ! also negates a group |
severity:>=warn |
that severity or worse, using OTel severity numbers |
has:tenant / !has:tenant (or -has:tenant) |
the attribute is present / absent |
"card declined", retry |
text in the message; every term must match |
level: means severity:; trace:, span: and env: mean trace_id:, span_id: and
environment:. service: is the emitting service of a record and resource.<key>: addresses
the resource facts a record keeps (resource.host.name:ip-10-0-0-7); both are described, with the
other keys on a log record, in Logs. A term that does not parse (an unclosed parenthesis, a dangling OR, key:) is searched
for as text, never refused. -key:value is the older spelling of !key:value and keeps
working. A *suffix or a*b pattern matches every part of it somewhere in the value, not
strictly at the end. The
line under the box shows what the search was read as.
- Show ±60s around this record, in an open record, shows every record from the 60 seconds either side of it, in the same environment, with the search cleared.
- Pin to top keeps a record above the stream. Pins are in the URL and in a saved view.
- Follow adds new records to the top of the stream as they arrive, with the same search applied. It shows at most 100 records per two seconds and says so when more arrived. It works on the first page of a relative time range. If the live connection fails it refreshes every few seconds. It follows by the time Bucker received a record, over records stamped within the last five minutes.
- Chart draws the same search as counts, or unique values of one attribute, over time, optionally grouped by one key. It runs through Explore's logs dataset with the same limits.
- The bookmark button saves the search, severity, environment, trace, time range and pins as a named log view.
Saved queries
The bookmark button beside Run query saves the question under a name, privately. Its
menu pins it, shares it with the organization (admins and owners can share), or deletes
it. Opening a saved query fills in the query row; it does not run it until you press
Run query. Saved queries are the same saved views the issue stream uses, with
surface: explore, so they follow the same visibility rule. A name is unique per person
per organization across issue views, saved queries and saved log views.
Dashboard widgets from Explore
Save as widget adds the question to a dashboard as an Explore query widget, drawn as a line chart, bars, a big number or a table. The widget uses the dashboard's time range in place of the query's, and keeps every Explore limit. Dashboards cover at most 30 days for these widgets. The widget says how many rows it read, and says so when the row limit cut the answer short.
Dashboards never read raw error events, so a query over the Errors dataset, or one joined to it, cannot become a widget. Chart errors with the issue widgets.
Add widget on a dashboard also offers Note, a text tile. It renders headings, lists,
bold, emphasis, code and http(s) links. It never renders HTML, so a tag you type shows as
text.
Sampling rules
Settings › Projects › Sampling rules decides what share of a project's spans and logs
Bucker stores, without changing any SDK. A rule has conditions in the Explore key
vocabulary and a share to keep: span.name is GET /health at 0% drops health checks;
http.route starts with /search at 10% keeps a tenth of search traffic. Rules apply in
order and the first match decides. A record no rule matches is stored.
Four things always hold:
- Errors are kept. A span with an error status keeps every span of its trace that arrived with it. A log at ERROR or worse, or one carrying an exception, is always kept.
- A trace is kept or dropped whole. The decision is a stable hash of the trace id, so the spans of one trace agree across batches and across OTLP and SDK ingest. A log on a trace follows its trace.
- The rate is stored. Every kept span or log record records the share it was kept at. Explore counts what is stored, says how many of the rows it read were sampled and at what lowest rate, and gives an estimate of the count before sampling only when you ask for it.
- The rule says what it did. Each rule shows how many records it kept and dropped since it was created.
Records a rule drops on OTLP ingest are not billed. They appear on the drops panel under the sampling-rule reason. SDK spans and logs dropped by a rule were already counted when the envelope arrived, so they show on the rule's own counts and not as a second drop. A changed rule applies within five seconds.
API
| Route | Purpose |
|---|---|
POST /orgs/:org/projects/:project/explore/aggregate |
aggregate over dataset: spans | events | logs | metrics, with filters, an optional sameTrace: { dataset, filters, negate } and, for histogram_*, onLayouts: refuse | split |
POST /orgs/:org/projects/:project/explore/rows |
raw rows with columns (at most 20), keyset-paginated by cursor |
GET /orgs/:org/projects/:project/explore/attributes?dataset= |
the key menu for one dataset |
GET /orgs/:org/projects/:project/explore/traces/:traceId/waterfall |
the trace's rows, each with logCount and errorCount (null without issue:read) |
GET /orgs/:org/projects/:project/explore/traces/:traceId/spans/:spanId |
one span with up to 50 of its logs and 20 of its error events; errorsHidden: true without issue:read |
POST /orgs/:org/projects/:project/explore/spans/aggregate |
the original span-only aggregate, unchanged |
GET /orgs/:org/projects/:project/logs?query= |
log search with the grammar above (where= is the same grammar under the MCP tool's name); ids= fetches pinned records; fields= projects up to ten attribute keys per row; the page's interpreted says what ran and its evidence says what it rests on (Logs) |
GET /orgs/:org/projects/:project/logs/:recordId |
one record with its issue, trace summary, release and deploy, flags and neighbours (Logs) |
GET /orgs/:org/projects/:project/logs/stream |
the live tail, as server-sent events; each event's id resumes it |
GET/POST /orgs/:org/projects/:project/sampling-rules, PATCH/DELETE …/:ruleId |
manage sampling rules: { name, signal: spans | logs, filters, rate: 0..1, enabled }; reads need project read, writes project write, and every write is audited |
POST /orgs/:org/views with surface: explore and query: { kind: 'explore', query } |
save a query; GET /orgs/:org/views?surface=explore lists them |
PATCH /orgs/:org/dashboards/:id with a widget { type: 'explore_query', config: { query, projectSlug?, display } } |
add a widget; display is line, bar, big_number or table |
Ask in plain words
Press Ctrl/⌘ + / anywhere, or the Ask button beside the issue, Explore and log search
boxes. Type a question such as p95 checkout latency by release last 24h; Bucker compiles it
with a fixed phrase table into the query that surface already runs — an issue search, an Explore
aggregate, or a log search — and shows that query, editable, with every phrase it recognised,
before anything runs. Nothing is sent to a model and nothing is changed: running it is the same
read as typing the query yourself (POST /orgs/:org/ask returns the compiled query only).
A why question ("why did WEB-42 start failing?") is not answered by the compiler. It offers to open root-cause analysis on the named issue, where hypotheses carry their own confidence and evidence, and any action you take from there goes through the usual autonomy and approval rules.
The query log
Every read Explore, log search, trace detail and the exports compile is recorded, so "which
agent read this last week, and on whose behalf" has an answer. A run is one row: who ran it (the
principal and its kind, the person sponsoring an agent, the OAuth client when there was one),
from where (dashboard, api, cli, mcp, export), what it was (a digest of the query's
shape, and a digest of the exact statement it compiled to) and what it cost (rows scanned and
returned, duration, whether the scan was truncated). A refused run is recorded too, with the
reason code.
Filter values are never stored: a value can be a customer identifier, so the log keeps only the
digests. Two runs of the same question over different time ranges share a specDigest.
A dashboard widget or an alert rule does not write a row per refresh; it adds to one hourly
summary row, which is kept 400 days. Raw rows are kept 30 days.
| Request | Returns |
|---|---|
GET /orgs/:org/projects/:project/query-runs |
the log, newest first, with the usual cursor; filters surface, principalKind, since |
GET /orgs/:org/projects/:project/query-runs/summary |
the last seven days per surface and principal kind, widget and alert summaries included |
With org:read you see every principal's runs in the project. Without it (a token scoped to
the project, an agent token) you see only your own, and the response says scope: "own".
The dashboard and the CLI label their own requests with an x-bucker-surface header; anything
else is api. The label is attribution only, nothing is decided on it.