# Experiments

An experiment compares the variants of a feature flag and says what the numbers do and do not show. This page covers the lifecycle (draft, launch, result, decision), what an agent can do and what only a person can, and what is off until an operator or your own agreement turns it on. Experiments are separate from the flag mirror: [Feature flags](feature-flags.md) covers the list of flags your vendors hold, and an experiment is a design that names one of them.

## What Bucker does, and does not do

- **Bucker does not serve flags.** The flag stays at your vendor (PostHog, LaunchDarkly and the others in [Feature flags](feature-flags.md)). An experiment names a flag as the mirror holds it, with its variants, and Bucker measures, checks and records. Bucker serves traffic splits for one thing only: prompt and model arms of an agent, under their own rollout guardian.
- **A draft is inert.** It exposes nobody, drives no flag and is read by nothing that changes anything.
- **A result is evidence, not a verdict.** There is no `significant` field. A result carries estimates with intervals, the number of units in both arms, the engine that computed them, and the checks that ran and the checks that could not.

## The lifecycle

| State | Meaning |
| --- | --- |
| `draft` | A design on file. Editable. |
| `running` | Launched. Measuring. |
| `paused` | Returned to control for everyone. Never resumes by itself. |
| `stopped`, `shipped` | Ended by a person. `shipped` names the variant that was chosen. |
| `archived` | Set aside. Not deleted. |
| `imported` | Copied in from another vendor, with its result as exported. Drives nothing. |

### 1. Draft

A design is the flag and its variants as the mirror names them, the primary metric and its goal (`increase` or `decrease`), the guardrail metrics and which direction is harm, the smallest relative effect worth detecting (`mde`), the share of traffic entering the experiment, the unit of randomisation (`subject` or `account`) and the longest it may run. An unequal split must say why. A variant the flag does not declare, a design with no single control, or weights that do not add up are refused before anything is written, and the refusal names the code.

- MCP: `experiments_create` (a draft), `experiments_update` (edit a draft's design, pause a running experiment, archive one; one change per call). Scope `experiments:write`.
- REST: `POST /orgs/:org/projects/:project/experiments`, `PUT .../experiments/:experimentId/design`.

### 2. Power, before launch

`experiment_power_get` answers how long an experiment must run, or how small an effect it can see, before anyone launches it. Give it a metric (kind, baseline, and for a mean its standard deviation), the number of variants and the traffic share. It returns the smallest detectable effect at 1, 2, 4 and 8 weeks and, when you pass `mde`, the units per arm and the weeks that effect needs. The method is a fixed-horizon two-sample test, named in the answer along with the engine.

The baseline is yours to supply and the answer says so. Traffic is read from the project's own session counts over 28 days unless you pass `dailyUnits`; that read needs `event:read` and is recorded in the query log. A launch re-runs the check: a design that cannot see its own effect inside its longest duration, or has no traffic to size against, is refused. Neither counts as a pass.

### 3. Protocols: the decision rule, written before launch

A protocol is the experiment's pre-registered rule: the primary metric and direction, the minimum effect worth acting on, the guardrails with their margins, alpha, an optional planned sample, and the minimum and maximum run in days. `experiment_protocols_list` shows the revisions an experiment has had; `experiment_protocols_update` adds one to a **draft**.

- A first rule, or a tightening, asks a person.
- A **loosening** (alpha raised, the minimum run shortened, a guardrail dropped or its margin widened, the primary metric changed) is Critical and denied by default until an organization relaxes it to two approvers. The engine works out the direction from the difference between the rule on file and the new one, never from an argument.
- The rule is **frozen at launch**. After that the governed write is refused (`RULE_IS_LOCKED`) and an amendment is a person's act with no tool. Every decision carries the digest of the rule in force.

### 4. Launch

`experiment_propose` proposes a launch (or a resume of a paused one) for a person to approve. It never launches by itself: both carry an ask that no policy, earned rung or review mode removes ([Approvals](approvals.md)). Readiness runs first and refuses before anyone is asked: a flag under a live narrowing or disable, another experiment holding the flag, a named protocol that is not locked, or a design whose power check fails. A yes is bound to the digest of this exact design and is stamped with the person who approved, never the agent that asked.

The same acts are REST routes (`POST .../experiments/:experimentId/launch`, `/pause`, `/resume`, `/archive`, each with a `reason`), under the same governance.

### 5. Result

`experiment_results_get` returns the result. For an experiment run in Bucker it returns, per arm, the exposures, and per metric and treatment arm the relative effect with its interval, the p-value and its Holm-adjusted value, alpha, the n of both arms and the engine with its version, or `not_available` with the reason. Beside them:

- **Validity.** Seventeen checks run before any result is shown (sample-ratio mismatch, no exposures, exposure stalls, a split, bucketing or targeting changed mid-run, a forced variant, a flag state that disagrees with the mirror, a low telemetry join, overlap with another experiment, and others). Each answers `found`, `not_found` or `not_checked` with its reason, and `not_checked` is never a pass. A critical finding **withholds every estimate** and names its code until it clears on fresh data or a person overrides it with a reason; a result under an override is labelled `diagnostic overridden: <code>`.
- **Guardrails.** Four are judged without any configuration: the share of sessions that errored, the share of subjects who raised an error, p95 latency of the routes touched, and, where the project has AI cost, AI cost per exposed subject. Each reads `guardrail_breach`, `guardrail_not_proven`, `within_margin` or `not_checked`. A breach proposes a pause for a person to decide. It never pauses by itself.
- **The rule's recommendation**, labelled as a recommendation only.
- **The trust statement.** Every result lists the checks that ran and the checks that could not, so a reader never meets a bare "significant".

`experiment_get` reads the design, the revision and digest it is bound to, and who launched and who shipped it. `experiments_list` filters by `state`, `source` (`bucker` or `imported`) and `flag`. All three need `project:read` (or `issue:read`).

### 6. Decision, ship and stop: a person's acts

Ending an experiment is human-only. `experiments:ship` is held by organization owners, admins and members, on a person's own session. No agent, service principal, OAuth client or delegated token holds it, and no MCP tool has a verb for it.

| Act | Route |
| --- | --- |
| Ship a variant | `POST /orgs/:org/projects/:project/experiments/:experimentId/ship` (`variantKey`, `reason`) |
| Stop | `POST .../:experimentId/stop` (`reason`) |
| Delete | `POST .../:experimentId/delete` (`reason`) |
| Decide an imported result | `POST .../:experimentId/decision` |

A decision records ship a variant, ship none, or invalid, beside what the pre-registered rule recommended. The person signs the digest of the result they actually saw: if the numbers moved after they read them, the decision is refused (`RESULT_CHANGED`). A decision that differs from the recommendation, or that has no locked rule behind it, requires a reason. The decision is signed so it can be verified offline, and it changes nothing else: shipping the variant and changing the flag at the vendor are separate acts. For an experiment run in Bucker, the decision is built in the domain layer, but there is no REST or MCP door for it yet, so it cannot be recorded today; the decision route above is for imported results only.

## Experiments imported from another vendor

`experiment_import_run` copies one experiment's definition and result from a connected, read-only vendor (PostHog, GrowthBook, Statsig, Datadog) into Bucker's records, through a connection you added in [Connections](connections.md). It never writes to the vendor, never launches or ships, and an imported experiment exposes nobody. The result carries a **fidelity** block naming everything the import lacks: the vendor's own figures beside Bucker's recomputation where Bucker can compute them, Bucker's own checks (sample-ratio mismatch, exposure agreement with flag telemetry, a new-issues guardrail) and `not_checked` where one could not run. Re-running an unchanged export writes nothing. A spent rate budget answers `rate_limited` with `retryAfterSeconds`.

A [`bucker.yml`](config-as-code.md#flags-and-experiments) file can declare a draft or a launch request; it cannot ship, stop, delete or archive, and an imported experiment is refused.

## The statistics engine

Every figure goes through one engine, named in every result as `bucker-stats/<version> + gbstats.<release>`. It is a TypeScript port of GrowthBook's MIT-licensed `gbstats`, held against a corpus generated by the upstream Python, so a number can be traced to the exact method and replayed. If the engine cannot be built, the answer says so: no figure is produced by a different method, and a check that needed it reads `not_checked`.

## Prompt and model arms

An experiment whose arms are an agent's prompt versions or a model route, rather than flag variants, is compared on counts and published tests: the share of sessions with an agent issue, with a retry loop, that never reached a terminal state, and cost per completed session. A judge score is shown as advisory and never halts or decides an arm. The result has the same contract: intervals and the n of both sides, no `significant`, a recommendation only. Launch and each widening stay the prompt rollout's governed acts.

## What is off, and what turns it on

| Thing | State | What turns it on |
| --- | --- | --- |
| The flag mirror that experiments read | Off | The operator switch `pillar.flag_mirror.enabled`, and a flag connection that names a project ([Feature flags](feature-flags.md)) |
| Exposure recording | Off | The operator switch `pillar.flag_exposures.enabled` |
| Per-variant statistics (`experiment_stats`) | Off | The operator switch `pillar.experiment_stats.enabled`; validity checks that read it say `not_checked` until then |
| Any result that reads end-user data | **Off for every project** | The project's product-analytics gate, which an organization owner opens after accepting the amended data processing terms; those terms have not been published yet, so today the gate cannot be opened for any project ([Product analytics gate](product-analytics-gate.md)). While it is shut, a Bucker-run result is withheld and its checks read `not_checked`. |
| Importing from a vendor | Off | A read-only connection to that vendor |

A pillar switch is set through the operator console, which refuses to enable one while a database gauge it depends on is in breach unless the operator gives a reason. If the switch cannot be read, the pillar is off.

## Experiment debt

A running experiment is a thing someone owns. The `experiment-lifecycle/1` rules open a due-dated debt for an experiment that has run 60 days or more with fewer than 100 newly exposed subjects in the last 14, one stopped for 7 days with no recorded decision, one past the longest run its locked rule allows, and one shipped for 7 days while a mirrored environment still serves another variant. A rule that could not look reads `not_checked`, never clear.

## Scopes

| Scope | Lets you | Held by |
| --- | --- | --- |
| `project:read` | Read experiments, results, protocols and power | Anyone who can read the project |
| `experiments:write` | Draft, edit a draft, pause, archive, propose a launch, write a protocol, import | Owners, admins and members, and an MCP token that asks for it. It is one of the 41 scopes the MCP server can ask for; the SG-059 holds on the MCP tool registry were lifted by owner decision on 2026-11-16 ([MCP setup](mcp-setup.md)). |
| `experiments:ship` | Ship, stop, delete, decide, amend a locked rule, override a validity finding | A person on their own session only |

The tools are in the `flags` and `experiments` toolsets ([MCP tools](mcp-tools.md), [Agent governance](agent-governance.md)).
