CI setup runbook
This page is for a person or a coding agent connecting a repository's CI to Bucker. It is written to be read by a program: every table cell is filled, and a support matrix says yes, no or n/a, never nothing. A docs test fails on an empty cell in this file.
What you may change
| Thing | An agent may change it | How |
|---|---|---|
| Files in the repository (workflow YAML, test-runner config) | yes | a pull request that a person merges |
| Merging the pull request | no | a person |
| Branch protection and required checks | no | a repository admin |
| Bucker project settings, including turning CI ingest on | no | an organization admin, in the dashboard |
| The GitHub App's permissions (consent) | no | an organization admin |
| Tokens and the DSN | no | a person creates them and stores them as CI secrets or variables |
Doctor reads; it never writes. It makes GET requests only and prints a token's kind and scopes, never the token.
The loop
- Run
bucker ci doctor --jsonin the repository root, after the tests have written their reports. - Read the
checksarray. Each finding hasid,status,evidenceandfix. - For each
failorwarnwhose fix is a file in the repository, propose the change as a pull request. - For each whose fix names a person (consent, ingest switch, branch protection), say so in the pull request and stop.
- After a person merges, run doctor again.
Exit codes
| Exit code | Meaning | What to do |
|---|---|---|
| 0 | every check passed or warned | warnings are advice, not blockers |
| 1 | at least one check failed | fix the failed checks; a failure beats a could-not-check |
| 2 | nothing failed but a check that matters could not be made | retry; the server may be older than the CLI, or the token may not reach the project |
A not_checked finding carries blocking. Only a blocking one moves the exit code to 2. Branch protection that needs admin consent to read is not_checked with blocking: false.
The checks
| Check id | Fails when | Warns when | Needs the server |
|---|---|---|---|
action_version |
no workflow runs a bucker action or @bucker/cli |
a reference is a branch, a major tag or an unversioned CLI | no |
token |
no token, a 401, or release:write missing |
the token is personal or an agent's, not an org CI token | yes |
dsn |
no DSN, or one that does not parse | never | no |
connectivity |
the API or the ingest host is unreachable or answers 5xx | never | no |
github_consent |
no connection, or checks:write or actions:read not granted |
never | yes |
webhook_deliveries |
no delivery at all in 24 hours | one of workflow_job, check_run, merge_group, deployment_status is missing |
yes |
junit_files |
no JUnit file, or none usable | some unusable, or older than 71 hours | no |
coverage_files |
never | no coverage file found | no |
identity_stability |
never | more than 20% of test identities appear in only one of the last 30 runs | yes |
traceparent |
never | TRACEPARENT is absent or malformed |
no |
merge_queue |
never | merge_group events arrive and no workflow triggers on them, or neither exists |
yes |
gate_setup |
never | the gate check is not required, or production's only protection is Bucker's | yes |
Uploading results
bucker ci upload "reports/*.xml" --dsn "$BUCKER_DSN"
The command reads the commit, branch, run id, attempt, workflow, job and pull request from the CI system's own variables, and TRACEPARENT and BAGGAGE (cicd.pipeline.run.id) when the pipeline exports them. A flag wins over the environment. It sends each file to POST /api/:projectId/ci/reports on the DSN's ingest host with the DSN public key, and exits 0 only when every file was accepted.
| Flag | Meaning | Default |
|---|---|---|
--format |
junit or coverage |
junit |
--sha |
full commit id | the CI variable, then git rev-parse HEAD |
--branch |
branch name | the CI variable |
--run-id |
the CI system's run id | the CI variable, then BAGGAGE |
--attempt |
attempt number, 1 to 100 | the CI variable, else 1 |
--workflow |
workflow name | the CI variable |
--job |
job name | the CI variable |
--config |
a suffix for test identity: operating system, runtime, matrix key | none |
--pr |
comma-separated pull request numbers | the CI variable |
--dry-run |
find the files and print what would be sent | off |
CI systems read
| CI system | Commit | Run id | Job | Pull request |
|---|---|---|---|---|
| GitHub Actions | yes | yes | yes | yes |
| GitLab CI | yes | yes | yes | yes |
| CircleCI | yes | yes | yes | yes |
| Buildkite | yes | yes | yes | yes |
| Jenkins | yes | yes | yes | no |
| Anything else | no | no | no | no |
On a system marked no, pass the flag. The commit always falls back to the git checkout.
Formats the door accepts
| Format | Accepted today | Notes |
|---|---|---|
| JUnit XML | yes | surefire reruns and flaky markers are understood; a DOCTYPE is refused |
| Coverage | no | the door answers with a reason until coverage ingest ships |
| TAP | no | convert to JUnit |
| TRX | no | convert to JUnit |
Limits: 8 MiB per file. A refused file prints the door's reason (ci_ingest_disabled, report_too_large, report_format_unsupported, and the parser's own). A 429 or 5xx is retried three times; a refusal is never retried.
The GitHub Actions step
- uses: bucker/release-action/ci-upload@<full commit sha>
if: always()
with:
dsn: ${{ vars.BUCKER_DSN }}
files: reports/*.xml
Pin the action to a full commit SHA. The cli-version input is an exact version.
What doctor reads
Doctor reads two routes through the control plane with the caller's token. A server that does not serve them yields not_checked findings and exit 2, not a failure.
| Route | Used by |
|---|---|
GET /orgs/:org/projects/:project/ci/setup |
token scopes, github_consent, webhook_deliveries, merge_queue, gate_setup |
GET /orgs/:org/projects/:project/ci/runs?limit=30 |
identity_stability |
DORA metrics
dora_get (MCP, delivery toolset) and GET /orgs/:org/projects/:project/ci/dora return DORA's five metrics for one project and environment: deployment frequency, change lead time, change failure rate, failed deployment recovery time and deployment rework rate. The window defaults to 30 days and is at most 90, and the environment defaults to production.
| Rule | What it means |
|---|---|
| Honest denominators | every rate prints its numerator, its denominator and the deploys it could not measure, for example 48 deploys, 41 measured, 7 without a final status |
| Minimum sample | a rate over fewer than 5 measured deploys is null with a reason |
| Publishable sample | a rate over fewer than 30 measured deploys exists but is marked publishable: false, and a shared report leaves it out |
| No deploys | a project with none reads not_enough_data as a whole |
| Evidence | every failed or rework deploy lists its evidence (a rollback, a guardian halt, a failed release verdict, a status incident, a manual mark) and whether the basis is observed, heuristic, manual or rework |
| Heuristics | revert and hotfix wording in commit messages counts only in its own heuristic column, never in the rate |
| Rule set | the rule set (dora-rules@1) is printed with every answer |
A person can mark a deploy's outcome (POST .../ci/dora/outcome/preview, then .../ci/dora/outcome). The mark is stored beside the observed evidence and labelled manual; it never overwrites a deploy's own status. Cohorts by authorship are not built, and the answer says not_available rather than returning an empty list.
Scorecards
A scorecard is a set of rules over data Bucker already holds, evaluated daily for every project you can read. Each rule is a closed predicate over named facts (an owner is defined, an alert rule routes, release health is reported, no overdue reversal debt, an instrumentation score, a gate is enforced). It has a level ladder from 1 to 3. A scorecard describes and never enforces: nothing it reports blocks anything.
| Question | Answer |
|---|---|
| Who may read | anyone who can read the projects; only projects the caller can read contribute a row, a count or a level |
| A fact that could not be read | counted as skipped, never as a pass and never as a fail |
| Who proposes a rule | an agent or a person, through scorecards_update or POST .../scorecards/rules; a proposal is always a draft |
| What a draft shows | organization admins see how many projects it would fail (shadow); nobody else learns a draft exists |
| Who publishes | a person with org:admin, on POST /orgs/:org/scorecards/rules/:ruleSlug/publish or in the dashboard; there is no tool for publishing, unpublishing or archiving a published rule |
| Limits | at most 100 rules per organization and 5 leaves in one predicate |
Read them with scorecards_list or GET /orgs/:org/scorecards. With scorecard: <slug> and include: ["outcomes"] each project's outcome shows the value read, the rows read behind it and the fix each failing leaf points at. Campaigns are not built, and asking for them answers not_available.
Gates
A gate is a named set of rules checked against a pull request, a commit or a deploy. The rule types are tests_failed, new_flaky_test, patch_coverage_below, live_issues_in_touched_files, release_health_regression, overdue_reversal_debt and synthetic_journeys_failing. A project has at most 20 gates and a gate at most 12 rules. A rule that cannot be decided reads not_enough_data, never passed.
A gate moves through three stages, and every move is a person's act.
| Stage | What it does | Who moves it up |
|---|---|---|
shadow |
evaluates and records; posts to no pull request and blocks nothing | a person with org:admin publishes it (POST .../gates/:gateId/publish) |
warn |
posts a check with a neutral conclusion: it reports and does not block | a person enforces it (POST .../gates/:gateId/enforce) |
enforced |
the check can block a merge | the step above |
- Enforcement is earned. Enforcing needs at least 14 days in shadow and at least 30 recorded check evaluations, and a person confirms with a security key (the step-up route is
.../enforce/challenge), bound to the gate, its revision, its fail mode and the reason. - Agents propose, people publish.
gates_updatecreates a gate inshadow, revises a shadow gate, or archives one. A new gate carries a backtest over the last 200 pull requests once applied. A published or enforced gate is refused (not_shadow), andpublish,enforceanddisableare not inputs of the tool. - Reading.
gates_listshows each gate's stage, fail mode (closedoropen), revision and shadow record. Withinclude: ["evaluations"]it shows the stored evaluations: the verdict that was posted, each rule's outcome with its evidence ids and watermark, and the signed entry id.gate_evaluateasks what a gate would say about a commit, pull request or environment now. It is stored asadhoc, never posts a check, never blocks and is never signed. - Overrides are a person's. A person other than the pull request's author, holding
fix:approve, may override an enforced gate that is failing the commit, with a reason, an expiry and a step-up bound to the evaluation. There is no tool for it. Revoking an override restores the failing verdict, so it is allowed toci:write. - Signed. An enforced verdict is signed, so a result can be checked later.
- Undecided checks. A non-shadow check that cannot be decided within 10 minutes is settled by the gate's fail mode.
Flaky tests
flaky_tests_list shows the tests whose outcome depends on something other than the code: the verdict, the executions behind it, the same-commit flips, an age-weighted flake rate, the CODEOWNERS owner, the issue the detector opened, and any quarantine. Minutes lost to flakes are not shown, and the field is absent rather than zero.
flaky_test_quarantine_create keeps a flaky test running and reporting while its failures stop failing the gate, for at most 14 days (the default is 7). It is a governed write that asks a person, and the approval card carries the evidence: the verdict, executions, same-commit flips, the flake rate and how much of the project's cap is in use. It is refused before anyone is asked when:
| Refusal | Cause |
|---|---|
not_flaky |
the test is not judged flaky, or was never seen to pass and fail on one commit |
too_few_samples |
fewer than 10 executions |
protected_path |
the test is under a protected path |
blamed_production_issue |
the test file changed in a commit blamed for an unresolved production issue |
cap_reached |
the project already has 25 active quarantines |
A test cannot be quarantined twice. The default protected paths are **/auth/**, **/authz/**, **/migrations/** and **/security/**. Restoring is operation_revert on the quarantine's operation, and the sweeper restores it at the end of its lifetime. A renewal is a new call.
Reruns and cancels
ci_runs_update reruns a failed job, reruns the failed set of a run, or cancels a run. The approver sees the jobs that would run, an estimate of minutes and dollars with the price's source and date, and the reruns GitHub still allows. It is refused before anyone is asked if the run or the restart set holds a deploy job (holds_deploy_job), the run is still executing, a cancel targets a finished run, nothing failed, or GitHub's 30-day or 50-rerun limits are passed. It also pauses when the caller's CI-minutes budget is spent. GitHub's Actions write permission is an opt-in on the GitHub App that an organization admin grants; without it, nothing can rerun.
What turns it on
| Capability | Needs |
|---|---|
| Any CI data to read at all | the ci_ingest pillar on the deployment (off by default; a refused upload says ci_ingest_disabled) and CI ingest turned on for the project by an organization admin |
| Test, coverage and run data | the upload step above, and the GitHub App consent that bucker ci doctor checks |
Agent writes (gates_update, scorecards_update, flaky_test_quarantine_create, ci_runs_update, ci_ingest_filter_update) |
a token with ci:write, granted at consent |
| Blocking a merge | a gate a person has published and enforced after the shadow period |
The SG-059 holds on these tools were lifted by owner decision on 2026-11-16, with no signed-off security review of the authorization server recorded. Related: Releases, Release attestations, Staged rollouts, Production assertions, Application performance.