Skip to content
bucker

Explore: spans, error events, logs and metrics

Explore answers questions over a project's telemetry with one query engine. This page says what it does today and where its limits are.

Datasets

The Dataset switcher at the start of the query row picks one of four tables. The aggregate, field, group-by and filter controls work the same way on each.

Dataset Rows Time column Promoted keys
Spans every span from OTLP, the SDK's tracing, and Sentry SDK transactions span start span.name, span.kind, span.status, span.duration (ms), environment, release, trace_id, resource.<key>
Errors every error event that reached an issue the time Bucker received it level, message, error.type, error.value, environment, release, platform, transaction, server_name, trace_id, span_id, issue, user (hashed)
Logs every OTLP or SDK log record the record's timestamp severity, severity_number, message, environment, release, trace_id, span_id, issue
Metrics every data point received over OTLP /v1/metrics (OTLP) the point's timestamp metric.name, metric.unit, metric.kind, metric.temporality, service, environment, release, trace_id (the first exemplar's trace), series; numeric: metric.value, metric.count, metric.sum, metric.min, metric.max

Any other key is looked up among the row's attributes: span attributes, log attributes, a metric point's attributes, or an event's tags. An event whose tags are not a key/value map counts as having no tags.

The Errors dataset filters on the time the event was received, not the time the SDK stamped it. Only the received time is indexed per project. For a live SDK the two are seconds apart.

Querying Errors needs the issue:read scope on the token, because those rows are issue data. A token limited to an incident cannot query it.

Metrics

A gauge or sum point has metric.value; a histogram, exponential histogram or summary point has metric.count (observations) and metric.sum, so avg of metric.value grouped by metric.name charts a gauge, and sum of metric.count counts observations. Percentiles over a histogram METRIC are percentiles of the per-point values you pick, not of the underlying observations: the bucket layout is stored but not aggregated in SQL.

A point that carried an exemplar has a trace_id, so a row in Show matching rows links to the trace the value was observed in, and Same trace joins metrics to spans, logs and errors in either direction. A point with no exemplar has no trace and never matches a join.

If a metric's attributes include something unbounded (a user or request id), Bucker keeps the first 1,000 distinct series per metric per day and reports the rest on the drops panel as metric_series_cap; see OTLP.

Aggregates

Count, unique values, sum, average, min, max and p50/p75/p95/p99, optionally grouped by one key and bucketed over time. Every query has three limits, and the answer always says which of them applied:

  • at most 30 days;
  • at most 100,000 rows read (the newest rows win, and the result says so);
  • at most 50 series, and grouping is refused past 1,000 distinct values.

Show the compiled query, under every result, prints the exact statement that produced the numbers, with its values in order ($1, $2, …).

Counters and histograms (metrics dataset)

p95 on a metric is the 95th percentile of the stored point values, which for a histogram are per-point totals: the p95 of how many requests each scrape counted, not of how long they took. Seven more functions read a metric by its meaning, and they exist only on the Metrics dataset. They take no field, and the filters must name the metric (metric.name equals):

Function Answers
counter_increase How much a counter grew in the window.
counter_rate The same, per second of the window (or of the bucket, in a series).
histogram_p50, _p75, _p90, _p95, _p99 The percentile of the observations, from merged buckets.
histogram_mean The sum of the observations over their count.
histogram_rate Observations per second.

Counters. Per series, a delta point is its own increase and a cumulative point is its difference from the previous point of the same series; a fall is read as a restart, and the new value is the increase. Two replicas of one service are two series (they differ by instance), so they are never differenced against each other. A series' first point, with no earlier point within 15 minutes, contributes nothing.

Histograms. Per series, cumulative points are differenced bucket by bucket in time order (delta points are used as sent). A restart is a falling count, a falling bucket, a changed startTime or a changed bucket layout, and the point is then taken whole. Increments are summed across series that share a bucket layout (layoutDigest); exponential histograms at different scales are downscaled to the smallest, which is exact. The percentile is interpolated inside the bucket that holds the rank, the way Prometheus histogram_quantile does (linearly for explicit buckets, geometrically for exponential ones), and the bounds of that bucket are returned as the answer's interval: the true percentile is in it.

Every answer carries groups[].detail:

Field Meaning
status ok, no_data, insufficient_observations, layouts_differ or unreadable_buckets
basis buckets
observations Observations merged into the answer
interval [lower, upper] of the bucket holding the rank; upper is null for the open top bucket
layouts The bucket layouts merged (a hash of the bounds)
pointsMerged, resetsDetected Points differenced, and restarts read through

Below 20 observations a percentile is insufficient_observations with value: null, never a number. Series whose explicit layouts differ are never merged: the answer is a 422 layouts_differ, or one answer per layout (layout on each group) when the request sets "onLayouts": "split". A mean and a rate merge no buckets, so they do not need layouts to agree. Summary quantiles cannot be merged and are skipped, with a warning.

The response also carries metric: { seriesBasis, resetsDetected, instancesMerged }: instance_key when every series the read touched was told apart by its instance id, no_instance_key when it could not tell two replicas apart (a counter can then over-count), and how many replicas were merged. sum over a cumulative monotonic counter adds up running totals; the answer says so in warnings (cumulative_sum_adds_running_totals) with a suggestedSpec for the counter_rate question you probably meant. An alert condition with the same function and filters computes the same number: see Alert rules.

From a chart to Compare

On a span chart, drag across the region that looks wrong, or pick its first and last bucket under the chart and press Compare this window. The Spike vs baseline tab opens with that region as the spike window and the same length of time just before it as the baseline, and ranks the attributes that differ. It is offered for spans, the dataset Compare ranks.

Same-trace join

Same trace keeps only rows whose trace also has, or has no, matching rows in another dataset. Some questions it answers:

  • spans of traces that logged an error (dataset Spans, join Logs, severity is error);
  • log records of traces that raised an error event (dataset Logs, join Errors);
  • checkout spans whose trace has no error (join mode "Trace has no").

A row with no trace id never matches a join, in either mode. The joined rows are looked up in the same time window.

Matching rows and columns

Show matching rows under a result lists the rows behind it, 20 at a time, newest first. Columns picks what the table shows, from the dataset's promoted keys and the attribute keys found in recent rows, or any key you type. The picked columns stay in the URL with the rest of the query.

Reading one trace

The Trace tab draws a trace as a table, one row per span, nested under its parent.

  • Collapse a span with the arrow beside its name. The row then says how many spans it hides and the time they cover, for example "+37 spans, 212 ms".
  • A row says how many log records were emitted in the span and how many error events it raised ("2 logs · 1 error"). Error counts need the issue:read scope.
  • Open a span by its name, or with j/k and Enter. The pane beside the table shows its status, timing, attributes, span events and resource, the log records emitted in it, and its error events with a link to the issue each one belongs to. The span is in the URL as ?span=, so Copy link points at that one span.
  • Zoom by dragging across the ruler above the timeline, or with Zoom timeline to this span in the pane. Reset zoom shows the whole trace again. A span outside the zoomed window keeps its row and loses its bar.

Attributes, events and log bodies are shown exactly as the instrumented service sent them, labelled as untrusted text.

Searching logs

The Logs search box speaks the same search grammar as the Issues list and the events table (@bucker/shared/search is the one parser behind all three): key:value terms, OR, parentheses, ! negation, [a,b] lists and * wildcards, as well as text. Terms next to each other must all match; OR (capitals, as in Issues) lets either side match, and AND binds tighter than OR. Unlike Issues, a key the registry does not know is not an error: log attributes are free-form, so any key is read as an attribute path.

You type It means
user.id:42 the attribute equals 42, whether it was sent as a number or as text
!environment:staging (or -environment:staging) not equal
duration_ms:>500, >=, <, <= a numeric comparison
http.url:*stripe* / http.route:/api/* contains / starts with
severity:[warn,error] (or severity:warn,error) one of several values
user.id:7 OR severity:fatal either side matches; AND binds tighter than OR
(user.id:7 OR plan:pro) !severity:error parentheses group; ! also negates a group
severity:>=warn that severity or worse, using OTel severity numbers
has:tenant / !has:tenant (or -has:tenant) the attribute is present / absent
"card declined", retry text in the message; every term must match

level: means severity:; trace:, span: and env: mean trace_id:, span_id: and environment:. service: is the emitting service of a record and resource.<key>: addresses the resource facts a record keeps (resource.host.name:ip-10-0-0-7); both are described, with the other keys on a log record, in Logs. A term that does not parse (an unclosed parenthesis, a dangling OR, key:) is searched for as text, never refused. -key:value is the older spelling of !key:value and keeps working. A *suffix or a*b pattern matches every part of it somewhere in the value, not strictly at the end. The line under the box shows what the search was read as.

  • Show ±60s around this record, in an open record, shows every record from the 60 seconds either side of it, in the same environment, with the search cleared.
  • Pin to top keeps a record above the stream. Pins are in the URL and in a saved view.
  • Follow adds new records to the top of the stream as they arrive, with the same search applied. It shows at most 100 records per two seconds and says so when more arrived. It works on the first page of a relative time range. If the live connection fails it refreshes every few seconds. It follows by the time Bucker received a record, over records stamped within the last five minutes.
  • Chart draws the same search as counts, or unique values of one attribute, over time, optionally grouped by one key. It runs through Explore's logs dataset with the same limits.
  • The bookmark button saves the search, severity, environment, trace, time range and pins as a named log view.

Saved queries

The bookmark button beside Run query saves the question under a name, privately. Its menu pins it, shares it with the organization (admins and owners can share), or deletes it. Opening a saved query fills in the query row; it does not run it until you press Run query. Saved queries are the same saved views the issue stream uses, with surface: explore, so they follow the same visibility rule. A name is unique per person per organization across issue views, saved queries and saved log views.

Dashboard widgets from Explore

Save as widget adds the question to a dashboard as an Explore query widget, drawn as a line chart, bars, a big number or a table. The widget uses the dashboard's time range in place of the query's, and keeps every Explore limit. Dashboards cover at most 30 days for these widgets. The widget says how many rows it read, and says so when the row limit cut the answer short.

Dashboards never read raw error events, so a query over the Errors dataset, or one joined to it, cannot become a widget. Chart errors with the issue widgets.

Add widget on a dashboard also offers Note, a text tile. It renders headings, lists, bold, emphasis, code and http(s) links. It never renders HTML, so a tag you type shows as text.

Sampling rules

Settings › Projects › Sampling rules decides what share of a project's spans and logs Bucker stores, without changing any SDK. A rule has conditions in the Explore key vocabulary and a share to keep: span.name is GET /health at 0% drops health checks; http.route starts with /search at 10% keeps a tenth of search traffic. Rules apply in order and the first match decides. A record no rule matches is stored.

Four things always hold:

  • Errors are kept. A span with an error status keeps every span of its trace that arrived with it. A log at ERROR or worse, or one carrying an exception, is always kept.
  • A trace is kept or dropped whole. The decision is a stable hash of the trace id, so the spans of one trace agree across batches and across OTLP and SDK ingest. A log on a trace follows its trace.
  • The rate is stored. Every kept span or log record records the share it was kept at. Explore counts what is stored, says how many of the rows it read were sampled and at what lowest rate, and gives an estimate of the count before sampling only when you ask for it.
  • The rule says what it did. Each rule shows how many records it kept and dropped since it was created.

Records a rule drops on OTLP ingest are not billed. They appear on the drops panel under the sampling-rule reason. SDK spans and logs dropped by a rule were already counted when the envelope arrived, so they show on the rule's own counts and not as a second drop. A changed rule applies within five seconds.

API

Route Purpose
POST /orgs/:org/projects/:project/explore/aggregate aggregate over dataset: spans | events | logs | metrics, with filters, an optional sameTrace: { dataset, filters, negate } and, for histogram_*, onLayouts: refuse | split
POST /orgs/:org/projects/:project/explore/rows raw rows with columns (at most 20), keyset-paginated by cursor
GET /orgs/:org/projects/:project/explore/attributes?dataset= the key menu for one dataset
GET /orgs/:org/projects/:project/explore/traces/:traceId/waterfall the trace's rows, each with logCount and errorCount (null without issue:read)
GET /orgs/:org/projects/:project/explore/traces/:traceId/spans/:spanId one span with up to 50 of its logs and 20 of its error events; errorsHidden: true without issue:read
POST /orgs/:org/projects/:project/explore/spans/aggregate the original span-only aggregate, unchanged
GET /orgs/:org/projects/:project/logs?query= log search with the grammar above (where= is the same grammar under the MCP tool's name); ids= fetches pinned records; fields= projects up to ten attribute keys per row; the page's interpreted says what ran and its evidence says what it rests on (Logs)
GET /orgs/:org/projects/:project/logs/:recordId one record with its issue, trace summary, release and deploy, flags and neighbours (Logs)
GET /orgs/:org/projects/:project/logs/stream the live tail, as server-sent events; each event's id resumes it
GET/POST /orgs/:org/projects/:project/sampling-rules, PATCH/DELETE …/:ruleId manage sampling rules: { name, signal: spans | logs, filters, rate: 0..1, enabled }; reads need project read, writes project write, and every write is audited
POST /orgs/:org/views with surface: explore and query: { kind: 'explore', query } save a query; GET /orgs/:org/views?surface=explore lists them
PATCH /orgs/:org/dashboards/:id with a widget { type: 'explore_query', config: { query, projectSlug?, display } } add a widget; display is line, bar, big_number or table

Ask in plain words

Press Ctrl/⌘ + / anywhere, or the Ask button beside the issue, Explore and log search boxes. Type a question such as p95 checkout latency by release last 24h; Bucker compiles it with a fixed phrase table into the query that surface already runs — an issue search, an Explore aggregate, or a log search — and shows that query, editable, with every phrase it recognised, before anything runs. Nothing is sent to a model and nothing is changed: running it is the same read as typing the query yourself (POST /orgs/:org/ask returns the compiled query only).

A why question ("why did WEB-42 start failing?") is not answered by the compiler. It offers to open root-cause analysis on the named issue, where hypotheses carry their own confidence and evidence, and any action you take from there goes through the usual autonomy and approval rules.

The query log

Every read Explore, log search, trace detail and the exports compile is recorded, so "which agent read this last week, and on whose behalf" has an answer. A run is one row: who ran it (the principal and its kind, the person sponsoring an agent, the OAuth client when there was one), from where (dashboard, api, cli, mcp, export), what it was (a digest of the query's shape, and a digest of the exact statement it compiled to) and what it cost (rows scanned and returned, duration, whether the scan was truncated). A refused run is recorded too, with the reason code.

Filter values are never stored: a value can be a customer identifier, so the log keeps only the digests. Two runs of the same question over different time ranges share a specDigest. A dashboard widget or an alert rule does not write a row per refresh; it adds to one hourly summary row, which is kept 400 days. Raw rows are kept 30 days.

Request Returns
GET /orgs/:org/projects/:project/query-runs the log, newest first, with the usual cursor; filters surface, principalKind, since
GET /orgs/:org/projects/:project/query-runs/summary the last seven days per surface and principal kind, widget and alert summaries included

With org:read you see every principal's runs in the project. Without it (a token scoped to the project, an agent token) you see only your own, and the response says scope: "own". The dashboard and the CLI label their own requests with an x-bucker-surface header; anything else is api. The label is attribution only, nothing is decided on it.