# CI setup runbook

This page is for a person or a coding agent connecting a repository's CI to Bucker. It is written to be read by a program: every table cell is filled, and a support matrix says `yes`, `no` or `n/a`, never nothing. A docs test fails on an empty cell in this file.

## What you may change

| Thing | An agent may change it | How |
|---|---|---|
| Files in the repository (workflow YAML, test-runner config) | yes | a pull request that a person merges |
| Merging the pull request | no | a person |
| Branch protection and required checks | no | a repository admin |
| Bucker project settings, including turning CI ingest on | no | an organization admin, in the dashboard |
| The GitHub App's permissions (consent) | no | an organization admin |
| Tokens and the DSN | no | a person creates them and stores them as CI secrets or variables |

Doctor reads; it never writes. It makes GET requests only and prints a token's kind and scopes, never the token.

## The loop

1. Run `bucker ci doctor --json` in the repository root, after the tests have written their reports.
2. Read the `checks` array. Each finding has `id`, `status`, `evidence` and `fix`.
3. For each `fail` or `warn` whose fix is a file in the repository, propose the change as a pull request.
4. For each whose fix names a person (consent, ingest switch, branch protection), say so in the pull request and stop.
5. After a person merges, run doctor again.

## Exit codes

| Exit code | Meaning | What to do |
|---|---|---|
| 0 | every check passed or warned | warnings are advice, not blockers |
| 1 | at least one check failed | fix the failed checks; a failure beats a could-not-check |
| 2 | nothing failed but a check that matters could not be made | retry; the server may be older than the CLI, or the token may not reach the project |

A `not_checked` finding carries `blocking`. Only a blocking one moves the exit code to 2. Branch protection that needs admin consent to read is `not_checked` with `blocking: false`.

## The checks

| Check id | Fails when | Warns when | Needs the server |
|---|---|---|---|
| `action_version` | no workflow runs a bucker action or `@bucker/cli` | a reference is a branch, a major tag or an unversioned CLI | no |
| `token` | no token, a 401, or `release:write` missing | the token is personal or an agent's, not an org CI token | yes |
| `dsn` | no DSN, or one that does not parse | never | no |
| `connectivity` | the API or the ingest host is unreachable or answers 5xx | never | no |
| `github_consent` | no connection, or `checks:write` or `actions:read` not granted | never | yes |
| `webhook_deliveries` | no delivery at all in 24 hours | one of `workflow_job`, `check_run`, `merge_group`, `deployment_status` is missing | yes |
| `junit_files` | no JUnit file, or none usable | some unusable, or older than 71 hours | no |
| `coverage_files` | never | no coverage file found | no |
| `identity_stability` | never | more than 20% of test identities appear in only one of the last 30 runs | yes |
| `traceparent` | never | `TRACEPARENT` is absent or malformed | no |
| `merge_queue` | never | `merge_group` events arrive and no workflow triggers on them, or neither exists | yes |
| `gate_setup` | never | the gate check is not required, or production's only protection is Bucker's | yes |

## Uploading results

```bash
bucker ci upload "reports/*.xml" --dsn "$BUCKER_DSN"
```

The command reads the commit, branch, run id, attempt, workflow, job and pull request from the CI system's own variables, and `TRACEPARENT` and `BAGGAGE` (`cicd.pipeline.run.id`) when the pipeline exports them. A flag wins over the environment. It sends each file to `POST /api/:projectId/ci/reports` on the DSN's ingest host with the DSN public key, and exits 0 only when every file was accepted.

| Flag | Meaning | Default |
|---|---|---|
| `--format` | `junit` or `coverage` | `junit` |
| `--sha` | full commit id | the CI variable, then `git rev-parse HEAD` |
| `--branch` | branch name | the CI variable |
| `--run-id` | the CI system's run id | the CI variable, then `BAGGAGE` |
| `--attempt` | attempt number, 1 to 100 | the CI variable, else 1 |
| `--workflow` | workflow name | the CI variable |
| `--job` | job name | the CI variable |
| `--config` | a suffix for test identity: operating system, runtime, matrix key | none |
| `--pr` | comma-separated pull request numbers | the CI variable |
| `--dry-run` | find the files and print what would be sent | off |

### CI systems read

| CI system | Commit | Run id | Job | Pull request |
|---|---|---|---|---|
| GitHub Actions | yes | yes | yes | yes |
| GitLab CI | yes | yes | yes | yes |
| CircleCI | yes | yes | yes | yes |
| Buildkite | yes | yes | yes | yes |
| Jenkins | yes | yes | yes | no |
| Anything else | no | no | no | no |

On a system marked `no`, pass the flag. The commit always falls back to the git checkout.

### Formats the door accepts

| Format | Accepted today | Notes |
|---|---|---|
| JUnit XML | yes | surefire reruns and flaky markers are understood; a DOCTYPE is refused |
| Coverage | no | the door answers with a reason until coverage ingest ships |
| TAP | no | convert to JUnit |
| TRX | no | convert to JUnit |

Limits: 8 MiB per file. A refused file prints the door's reason (`ci_ingest_disabled`, `report_too_large`, `report_format_unsupported`, and the parser's own). A 429 or 5xx is retried three times; a refusal is never retried.

## The GitHub Actions step

```yaml
- uses: bucker/release-action/ci-upload@<full commit sha>
  if: always()
  with:
    dsn: ${{ vars.BUCKER_DSN }}
    files: reports/*.xml
```

Pin the action to a full commit SHA. The `cli-version` input is an exact version.

## What doctor reads

Doctor reads two routes through the control plane with the caller's token. A server that does not serve them yields `not_checked` findings and exit 2, not a failure.

| Route | Used by |
|---|---|
| `GET /orgs/:org/projects/:project/ci/setup` | `token` scopes, `github_consent`, `webhook_deliveries`, `merge_queue`, `gate_setup` |
| `GET /orgs/:org/projects/:project/ci/runs?limit=30` | `identity_stability` |

## DORA metrics

`dora_get` (MCP, `delivery` toolset) and `GET /orgs/:org/projects/:project/ci/dora` return DORA's five metrics for one project and environment: deployment frequency, change lead time, change failure rate, failed deployment recovery time and deployment rework rate. The window defaults to 30 days and is at most 90, and the environment defaults to `production`.

| Rule | What it means |
|---|---|
| Honest denominators | every rate prints its numerator, its denominator and the deploys it could not measure, for example 48 deploys, 41 measured, 7 without a final status |
| Minimum sample | a rate over fewer than 5 measured deploys is `null` with a reason |
| Publishable sample | a rate over fewer than 30 measured deploys exists but is marked `publishable: false`, and a shared report leaves it out |
| No deploys | a project with none reads `not_enough_data` as a whole |
| Evidence | every failed or rework deploy lists its evidence (a rollback, a guardian halt, a failed release verdict, a status incident, a manual mark) and whether the basis is `observed`, `heuristic`, `manual` or `rework` |
| Heuristics | revert and hotfix wording in commit messages counts only in its own `heuristic` column, never in the rate |
| Rule set | the rule set (`dora-rules@1`) is printed with every answer |

A person can mark a deploy's outcome (`POST .../ci/dora/outcome/preview`, then `.../ci/dora/outcome`). The mark is stored beside the observed evidence and labelled `manual`; it never overwrites a deploy's own status. Cohorts by authorship are not built, and the answer says `not_available` rather than returning an empty list.

## Scorecards

A scorecard is a set of rules over data Bucker already holds, evaluated daily for every project you can read. Each rule is a closed predicate over named facts (an owner is defined, an alert rule routes, release health is reported, no overdue reversal debt, an instrumentation score, a gate is enforced). It has a level ladder from 1 to 3. A scorecard **describes and never enforces**: nothing it reports blocks anything.

| Question | Answer |
|---|---|
| Who may read | anyone who can read the projects; only projects the caller can read contribute a row, a count or a level |
| A fact that could not be read | counted as `skipped`, never as a pass and never as a fail |
| Who proposes a rule | an agent or a person, through `scorecards_update` or `POST .../scorecards/rules`; a proposal is always a **draft** |
| What a draft shows | organization admins see how many projects it would fail (shadow); nobody else learns a draft exists |
| Who publishes | **a person with `org:admin`**, on `POST /orgs/:org/scorecards/rules/:ruleSlug/publish` or in the dashboard; there is no tool for publishing, unpublishing or archiving a published rule |
| Limits | at most 100 rules per organization and 5 leaves in one predicate |

Read them with `scorecards_list` or `GET /orgs/:org/scorecards`. With `scorecard: <slug>` and `include: ["outcomes"]` each project's outcome shows the value read, the rows read behind it and the fix each failing leaf points at. Campaigns are not built, and asking for them answers `not_available`.

## Gates

A gate is a named set of rules checked against a pull request, a commit or a deploy. The rule types are `tests_failed`, `new_flaky_test`, `patch_coverage_below`, `live_issues_in_touched_files`, `release_health_regression`, `overdue_reversal_debt` and `synthetic_journeys_failing`. A project has at most 20 gates and a gate at most 12 rules. A rule that cannot be decided reads `not_enough_data`, never `passed`.

A gate moves through three stages, and **every move is a person's act**.

| Stage | What it does | Who moves it up |
|---|---|---|
| `shadow` | evaluates and records; posts to no pull request and blocks nothing | a person with `org:admin` publishes it (`POST .../gates/:gateId/publish`) |
| `warn` | posts a check with a neutral conclusion: it reports and does not block | a person enforces it (`POST .../gates/:gateId/enforce`) |
| `enforced` | the check can block a merge | the step above |

- **Enforcement is earned.** Enforcing needs at least 14 days in shadow and at least 30 recorded check evaluations, and a person confirms with a security key (the step-up route is `.../enforce/challenge`), bound to the gate, its revision, its fail mode and the reason.
- **Agents propose, people publish.** `gates_update` creates a gate in `shadow`, revises a shadow gate, or archives one. A new gate carries a backtest over the last 200 pull requests once applied. A published or enforced gate is refused (`not_shadow`), and `publish`, `enforce` and `disable` are not inputs of the tool.
- **Reading.** `gates_list` shows each gate's stage, fail mode (`closed` or `open`), revision and shadow record. With `include: ["evaluations"]` it shows the stored evaluations: the verdict that was posted, each rule's outcome with its evidence ids and watermark, and the signed entry id. `gate_evaluate` asks what a gate would say about a commit, pull request or environment **now**. It is stored as `adhoc`, never posts a check, never blocks and is never signed.
- **Overrides are a person's.** A person other than the pull request's author, holding `fix:approve`, may override an enforced gate that is failing the commit, with a reason, an expiry and a step-up bound to the evaluation. There is no tool for it. Revoking an override restores the failing verdict, so it is allowed to `ci:write`.
- **Signed.** An enforced verdict is signed, so a result can be checked later.
- **Undecided checks.** A non-shadow check that cannot be decided within 10 minutes is settled by the gate's fail mode.

## Flaky tests

`flaky_tests_list` shows the tests whose outcome depends on something other than the code: the verdict, the executions behind it, the same-commit flips, an age-weighted flake rate, the CODEOWNERS owner, the issue the detector opened, and any quarantine. Minutes lost to flakes are not shown, and the field is absent rather than zero.

`flaky_test_quarantine_create` keeps a flaky test running and reporting while its failures stop failing the gate, for at most 14 days (the default is 7). It is a governed write that asks a person, and the approval card carries the evidence: the verdict, executions, same-commit flips, the flake rate and how much of the project's cap is in use. It is refused before anyone is asked when:

| Refusal | Cause |
|---|---|
| `not_flaky` | the test is not judged flaky, or was never seen to pass and fail on one commit |
| `too_few_samples` | fewer than 10 executions |
| `protected_path` | the test is under a protected path |
| `blamed_production_issue` | the test file changed in a commit blamed for an unresolved production issue |
| `cap_reached` | the project already has 25 active quarantines |

A test cannot be quarantined twice. The default protected paths are `**/auth/**`, `**/authz/**`, `**/migrations/**` and `**/security/**`. Restoring is `operation_revert` on the quarantine's operation, and the sweeper restores it at the end of its lifetime. A renewal is a new call.

## Reruns and cancels

`ci_runs_update` reruns a failed job, reruns the failed set of a run, or cancels a run. The approver sees the jobs that would run, an estimate of minutes and dollars with the price's source and date, and the reruns GitHub still allows. It is refused before anyone is asked if the run or the restart set holds a deploy job (`holds_deploy_job`), the run is still executing, a cancel targets a finished run, nothing failed, or GitHub's 30-day or 50-rerun limits are passed. It also pauses when the caller's CI-minutes budget is spent. GitHub's Actions write permission is an opt-in on the GitHub App that an organization admin grants; without it, nothing can rerun.

## What turns it on

| Capability | Needs |
|---|---|
| Any CI data to read at all | the `ci_ingest` pillar on the deployment (off by default; a refused upload says `ci_ingest_disabled`) and CI ingest turned on for the project by an organization admin |
| Test, coverage and run data | the upload step above, and the GitHub App consent that `bucker ci doctor` checks |
| Agent writes (`gates_update`, `scorecards_update`, `flaky_test_quarantine_create`, `ci_runs_update`, `ci_ingest_filter_update`) | a token with `ci:write`, granted at consent |
| Blocking a merge | a gate a person has published and enforced after the shadow period |

The SG-059 holds on these tools were lifted by owner decision on 2026-11-16, with no signed-off security review of the authorization server recorded. Related: [Releases](releases.md), [Release attestations](release-attestations.md), [Staged rollouts](staged-rollouts.md), [Production assertions](assertions.md), [Application performance](apm.md).
