Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

CI/CD

Thirteen workflows guard this repository. This page explains what each one protects, the decisions behind how they are wired, and the failure modes that shaped them.

Pipeline at a glance

WorkflowTriggerGuards against
ci.ymlPR, mainUnformatted code, clippy warnings, failing tests (including the shell integration in real bash, zsh, fish and pwsh), MSRV drift, broken shell scripts, stale reference docs, malformed .po, files missing from the published crate
integration-test.ymlPR (steps skipped without code changes), mainMigration breaking for pyenv / conda / virtualenvwrapper users
coverage.ymlPR, mainUntested code paths going unnoticed
bench.ymlPR (steps skipped without code changes), mainPerformance regressions in parsing and validation
mutants.ymlPR (diff), weekly (full)Tests that execute code without asserting on it
security.ymlPR, main, weeklyVulnerable dependencies, disallowed licences, untrusted sources
msrv-check.ymlCargo.toml/Cargo.lock changesA declared MSRV that no longer compiles
docker-build.ymldocker/** changes, weeklyBroken published images, vulnerable image contents
fuzz.ymlWeeklyParser crashes on hostile input
docs-check.ymlPR touching docs/src/**, docs/po/**, docs/book.toml, docs/theme/**A docs edit that breaks the mdBook build or leaves ko.po stale reaching a release tag
docs.ymlv* tagsBroken documentation site
release-plz.ymlmainManual release mistakes
cache-cleanup.ymlPR closed, weeklyClosed PRs’ cache copies and superseded main rust caches eating the 10 GB allowance and forcing evictions mid-export

Cross-cutting decisions

Cancel PR runs, never cancel main

Every workflow that runs on both uses the same concurrency block:

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}

Pushing a fixup to a PR should abandon the previous run — nobody needs results for a commit that no longer exists. A run on main is different: it produces state that later runs depend on. bench.yml writes the benchmark baseline to gh-pages, and a cancelled main run leaves the next PR comparing against a stale baseline.

release-plz.yml uses the same block with cancel-in-progress: false: a release must never be cancelled, but two releases must never run at once either, and a group delivers both.

The MSRV is verified twice, deliberately

ci.yml has an msrv job that builds and tests on 1.89. msrv-check.yml separately runs cargo msrv verify. These answer different questions:

  • ci.yml — does the code work on the version we claim?
  • msrv-check.yml — is the version we claim still the lowest one that works?

The second catches the case where a dependency bump silently raises the real floor while Cargo.toml still advertises the old one. It only runs when Cargo.toml or Cargo.lock changes, because that is the only way the answer can change.

cargo-msrv itself is pinned to ^0.18: 0.19 requires rustc 1.91, which this project’s own toolchain pin cannot satisfy.

Gate what is stable, track what is noisy

Not every measurement can be a gate. bench.yml splits its benchmarks by how reproducible they are on shared runners:

GroupBenchesObserved spreadBehaviour
CPUparsing, validation1.71x-2.37xFails the build past 250%
Filesystempath_lookup~3.4xRecorded, never fails

find_executable_in calls stat. Across 38 runs of unchanged code on main it produced anywhere from 574 to 1931 ns depending on which runner the job landed on. No threshold both tolerates that and catches a real regression, so the group is tracked on gh-pages for trend visibility and fail-on-alert: false keeps it out of the gate.

This matters because a noisy gate is worse than no gate: it blocks unrelated work and trains reviewers to ignore red.

The CPU row was written as “~1.4x, fails past 150%”, and the numbers did not hold. Measured across the 60 runs on gh-pages, the group swings 1.71x to 2.37x — its noise floor was above its own threshold, so it could not tell a regression from a runner. The way it failed is worth remembering, because a re-run does not clear it: the action compares against the single previous data point, and a run that lands on a fast runner measures low across every bench at once. Commit 9a9241da came in ~1.7x fast, and every PR afterwards compared against that and failed until the next commit reached main and replaced the baseline. 250% clears the measured ceiling. It is a coarse gate on purpose — the regressions worth catching here are algorithmic, and those land far beyond 2.5x.

Mutation testing runs at two scopes

cargo test proves a line executed. It does not prove a test would notice if that line were wrong. cargo-mutants injects deliberate defects — flipping > to >=, replacing a return value — and reports any that the suite fails to catch.

  • On PRs (--in-diff): only the lines this PR changed. Fast enough to gate on.
  • Weekly (full): every candidate in scope. A missed mutant fails the job, as a broken baseline or a timeout does. It tolerated exit 2 while the backlog of earlier escapes was triaged; see Recently closed.

.cargo/mutants.toml carries the exclusions, each with a written rationale. Most exclude code whose mutations can only be killed by spawning a real uv or python — a gap we accept rather than a test hole. Add exclusions there with the reason, not silently.

exclude_re matches the whole mutant description, not the function name. A bare "foo" therefore drops every mutant in foo, including the ones existing tests already kill — the exclusion outlives the reason given for it, and --in-diff never gates that code again. Exclude the specific description instead ("delete match arm \[major\] in foo"), and confirm what actually left the gate by listing both configs and diffing them:

cargo mutants --config <before>.toml --list --file '<glob>' | sort > before.txt
cargo mutants --list --file '<glob>' | sort > after.txt
comm -23 before.txt after.txt

Some mutants cannot be separated that way. Two delete ! in one function produce a byte-identical description, so a whole-function exclusion is the only option available — say so in the comment when that is the reason, instead of letting it read as an equivalence claim.

New Check trait implementations and thin wrappers need direct dispatch tests, or the PR gate reports them as MISSED.

Both scopes run each mutant’s tests under nextest (test_tool = "nextest"), with nextest’s default profile so a caught mutant stops at the first failing test. nextest does not run doctests, so a mutant that only a doctest kills counts as MISSED — cover that behaviour with a unit test. The switch turned up three such mutants in src/core/migrate/common.rs (dir_size, check_name_conflict); unit tests were added for them.

Skip what a PR cannot affect

Docker Integration is the slowest required check, and Benchmarks the slowest optional one. A PR that edits only docs, CLAUDE.md or another workflow cannot change what either of them builds or tests: 6 of the 20 PRs merged between 2026-10-15 and 2026-11-07 were like that, and each ran both in full. Both workflows now start with a scope step that diffs the PR against its merge-base and lets the later steps run only when a file they read changed. Each workflow lists its own inputs in CODE_PATHS. Both watch src/, benches/, locales/, .cargo/, .github/actions/, Cargo.toml, Cargo.lock, build.rs, the toolchain file, root .gitattributes and their own workflow file; integration also watches tests/, .config/ (nextest), docker/ and .dockerignore.

The check stays required and still reports: the job runs, the step decides, and the later steps carry if: steps.scope.outputs.code != 'false'. Two simpler mechanisms do not work here:

  • on.paths skips the whole workflow, and a required check of a skipped workflow stays Pending, which blocks the merge.
  • A job-level if is evaluated before the matrix expands, so a skipped matrix job posts one check with the expression unexpanded (Mutants (full ${{ matrix.shard }}/4) is that), and the three required Docker Integration (<source>) names would never appear.

The step is built to fail toward running everything:

  • It diffs the event’s own base.sha...head.sha, so a re-run or a later push cannot classify a different revision; only PR runs fetch full history (fetch-depth: 0) for the merge-base.
  • --no-renames lists both sides of a rename, so moving a file out of docker/ still counts as touching docker/.
  • -z with grep -z keeps paths unquoted; without it, git prints a non-ASCII or newline-containing name in quotes and a pattern anchored with ^ misses it.
  • Only grep’s “no match” (exit 1) skips. A failed diff (an unknown SHA, unrelated histories), a grep error and any non-PR event run everything. A failed write of the step output fails the job instead of leaving it green with nothing run.

A skipped run is green but did not test anything; the job summary says no code paths changed: tests skipped, not run. The cases above were checked with the step’s script under the runner’s bash --noprofile --norc -eo pipefail and GNU grep before it landed.

One cache per toolchain, one writer per cache

Swatinem/rust-cache keys on shared-key, so jobs sharing a key share a cache:

shared-keyJobs
stable-cici.yml lint / test / uv-floor / shellcheck
coveragecoverage.yml
mutantsmutants.yml
msrvci.yml msrv
msrv-verifymsrv-check.yml
benchbench.yml

MSRV artifacts are kept separate on purpose: a different compiler produces incompatible output, and sharing would mean both jobs permanently invalidating each other.

Within a shared key there is at most one writer. ci.yml marks lint and shellcheck save-if: false so only the test job — which produces the richest artifacts, including test binaries — populates the cache; in mutants.yml neither job writes today: the incremental job saves only on main, but it runs only on pull requests, and the weekly full job has save-if: false. A writer would gain little: cargo-mutants first copies the tree without the checkout’s target/ and builds the baseline there (later mutants reuse that copy’s build), so a restored target/ goes unused and only the registry download would be saved. The baseline build takes about 65s (full run, 2026-11-02), while PR #237’s 55 mutants took 14 minutes.

Note that rust-cache folds workflow-level env into its key hash. Two jobs can declare the same shared-key and still land on different caches if their workflows set different environment variables.

Failure modes worth remembering

These cost real debugging time. They are recorded so the next person recognises them faster.

An unmarked test picks up another test’s fake uv

The v0.17.1 release PR failed Docker Integration in two tests that had passed on the same source a minute earlier, on main. One #[serial] test put a fake uv in front of PATH; an unmarked test resolved uv at the same moment, got the fake one, and both assertions broke. #[serial] only keeps marked tests apart. Every job that runs the suite now uses cargo-nextest, which gives each test its own process: the test, MSRV and Docker integration jobs run cargo nextest run --profile ci (.config/nextest.toml; no retries, so a race stays visible), the test and MSRV jobs add cargo test --doc for the doctests nextest skips, coverage runs cargo llvm-cov nextest --profile ci, and mutants sets test_tool = "nextest". The tests that reach uv are also marked #[serial] or run under the env lock, because a local cargo test still runs them in one process.

Criterion errors corrupt the benchmark parser output

rust-cache restores target/ with the criterion tree present but its sample.json baselines pruned out. Criterion then fails to load a baseline it can see should exist, and writes the error to stdout mid-line:

test clap_parse_create ... Criterion.rs ERROR: error: Failed to access file
".../base/sample.json": No such file or directory
bench:      41,347 ns/iter (+/- 2,364)

benchmark-action’s parser needs test NAME ... bench: N ns/iter on one line. Split in two, it reports “no benchmark result” even though every benchmark ran. A cold cache passes precisely because there is no tree to half-load.

Each bench step therefore starts with rm -rf target/criterion. The gate compares against gh-pages history and never against criterion’s local baseline, so removing it costs nothing.

Two benchmark-action invocations collide on gh-pages

The first invocation fetches gh-pages into a local branch and commits its entry onto it — even on PRs, where it simply never pushes. A second invocation’s identical fetch is then a non-fast-forward update and git rejects it, failing the step before any comparison happens. skip-fetch-gh-pages: true on the second step reuses what the first fetched.

cargo bench | tee swallows failures

Without set -o pipefail the step exits with tee’s status, so a genuinely failing cargo bench reports success and resurfaces one step later as a confusing parse error.

Docs guards run on PRs only for docs paths

docs.yml — the mdBook build, the ko.po staleness round-trip and the Pages deploy — triggers only on v* tags. docs-check.yml runs the same build and the same round-trip on pull requests, but only when the PR touches docs/src/**, docs/po/**, docs/book.toml or docs/theme/**. It is a separate workflow rather than a trigger on docs.yml so a PR run needs neither pages: write nor id-token: write and never enters the pages concurrency group. Two checks in the ci.yml Lint job cover the facts that live outside those paths:

  • scripts/check-doc-references.py compares facts copied into README.md, CONTRIBUTING.md, llms.txt, llms-full.txt and the docs against the code that owns them — version and MSRV from Cargo.toml, reserved names from src/validate.rs, key count from locales/app.yml. Every check is doc-against-code; comparing the three llms files to each other would pass with all three wrong.
  • msgfmt --check on docs/po/*.po catches structural damage such as a msgstr whose trailing newline no longer matches its msgid.

Editing any page under docs/src/ still requires regenerating ko.po in the same commit. See Docs Translation.

Secrets

SecretUsed byPurpose
RELEASE_PLZ_TOKENrelease-plz.ymlPush release PRs and tags
CARGO_REGISTRY_TOKENrelease-plz.ymlPublish to crates.io
CODECOV_TOKENcoverage.ymlUpload coverage reports
GITHUB_TOKENbench, dockerPush gh-pages, publish to ghcr.io

A workflow triggered by Dependabot reads the Dependabot secret store, not the Actions one. A secret both need has to be registered twice — gh secret set CODECOV_TOKEN --app dependabot — or the job fails on every Dependabot PR while passing everywhere else.

Default workflow permissions are read. Workflows that need more declare it explicitly — bench.yml needs contents: write for gh-pages, docker-build.yml needs packages: write for the registry.

Releases

release-plz reads Conventional Commits on main and prepares the release; release-plz.toml holds the policy:

  • release_commits = "^(feat|fix|perf|refactor|revert)" — documentation and chore commits do not trigger a release on their own.
  • features_always_increment_minor = true — a feat bumps the minor version even in 0.x, where cargo’s default would only bump the patch.
  • Changelog generation goes through git-cliff (cliff.toml).

release-plz only rewrites Cargo.toml, Cargo.lock and CHANGELOG.md. The scuv <version> samples in README.md, CLAUDE.md, installation.md and api.md are not its business, so every release PR used to fail the Lint job on check-doc-references.py until someone hand-synced those four files. There is no hook that runs while release-plz builds the PR — the hooks it does have live on the publish path, which is too late — so release-plz.yml carries a step that runs check-doc-references.py --fix and commits the result onto the release branch. A release PR therefore arrives with an extra docs: sync version samples to <version> commit; that is the automation working, not drift. The step pushes with RELEASE_PLZ_TOKEN because the default GITHUB_TOKEN does not re-trigger workflows, and re-runs the full guard afterwards so a reference --fix does not cover still fails the job.

Merging the release PR is what publishes: it creates the tag, the GitHub release, and the crates.io upload — and the v* tag is also what deploys the documentation site.

Known gaps

Accurate as of the last revision of this page. Verify before relying on any of these being fixed.

  • codecov/project has never posted. codecov.yml states a target for it, and the config is live — setting comment.require_changes changed comment behaviour on the next PR. The status itself has never appeared, though, on any PR back through #165, which predates that config by months. The cause is on the Codecov side, in account or organisation settings this repository cannot read, so the relative check (“did this PR drop overall coverage”) is not being enforced. An absolute floor stands in for it in coverage.yml (cargo llvm-cov report --fail-under-lines 80), which catches a collapse but not a slow slide. Requiring the status before it is known to post would leave every PR waiting on a check that never arrives.
  • The coverage floor is absolute, not relative. --fail-under-lines 80 is measured against llvm-cov, which reads 80.97% where Codecov reads 78.5%; the two count different things, so a number taken from the Codecov dashboard would be wrong in the workflow. Coverage can still drift from 81% to 80.1% without tripping it — closing that needs codecov/project, see the gap above.
  • The cache hit its limit. 10.49 GB against the 10 GB allowance on 2026-09-26: 139 caches for main (about 2 GB of them v0-rust-* keyed to the Cargo.lock hash from before the 0.16.0 cargo update) and 118 copies left behind by merged PRs. LRU eviction then ran while a BuildKit export was writing layers and failed a green Docker Integration build with error writing layer blob: not_found (#198). Three changes: every saving step (Swatinem/rust-cache save-if, BuildKit cache-to) now writes only from main — PR runs restore main’s entries and leave no copy; BuildKit exports are ignore-error=true, so a failed cache write is a warning rather than a failed build; and cache-cleanup.yml deletes whatever a PR still leaves when it closes. The one-time deletion of the PR copies and the stale-lockfile rust caches brought usage to 6.4 GB. main still kept every lockfile generation of each rust cache (up to four per job, about 4.4 GB, by 2026-10-04); a weekly job in cache-cleanup.yml now keeps only the newest entry under each restore key (the cache key minus its lockfile hash), the one a later run with that toolchain and environment falls back to.

Recently closed

Left here because the reasoning is worth keeping, not because anything is outstanding.

  • The weekly full mutation run tolerated missed mutants (exit 2) while a backlog was triaged: 55 escapes at first, 46 of them under src/core/migrate/**. The 2026-10-08 run left three, and each was closed: Config::load and UvClient::list_pythons got tests (the latter against a fake uv), and virtualenv_activate’s miss was its cfg(windows) twin, which never compiles on the Linux runner, so it joined the other Windows path helpers in .cargo/mutants.toml. A missed mutant now fails the weekly job.

  • Docs were only verified on release tags: the MSRV 1.89 bump edited pages under docs/src/, its PR went green, and the v0.15.4 tag then failed at the ko.po round-trip. The crate published and the tag was fine — only the Pages deploy stopped, which is the quiet half of the failure and the reason it went unnoticed until someone opened the site. Closed by docs-check.yml, which runs the build and the round-trip on PRs that touch the docs paths.

  • Coverage uploads were rejected for months with Token required because branch is protected while the job reported success, because fail_ci_if_error: false hid it. The org allows tokenless uploads, but that path only covers fork PRs. Fixed by wiring CODECOV_TOKEN and letting a rejected upload fail the job. That fix was half of it: Dependabot-triggered runs read a different secret store, so every Dependabot PR kept failing the same way until the token was registered there too.

  • Every release PR failed the Lint job because release-plz bumps Cargo.toml without touching the version samples in prose. Hand-fixed at v0.15.3, hit again at v0.15.4, and now handled by the sync step described under Releases — the direction the v0.15.3 fix asked for: teach the release to update them rather than loosen the check.

  • Dependabot does not read rust-version, so it raised rust-i18n past the MSRV and every job died at dependency resolution before a line compiled. The group could not land without the MSRV moving to 1.89, which it did. .github/dependabot.yml now carries an ignore for the next known case (serial_test 4.x needs rustc 1.93.1), marked as debt to drop when the MSRV catches up.

  • The weekly full mutation run had never once completed — 60 minutes killed it every time, and GitHub reports a timed-out job as cancelled, which reads as benign. Measured at 385/446 mutants in 60 minutes, so the timeout is now 90.

  • At 90 minutes the weekly full run still timed out as the suite grew (the 2026-10-19 and 2026-11-02 runs). It was split into two round-robin shards, each with its own 90-minute limit, which took 42 and 58 minutes on 2026-11-07; it now runs as four, and each mutant builds with the debug-less mutants profile (.cargo/mutants.toml).

  • aquasecurity/trivy-action@master and astral-sh/setup-uv@v7 are now pinned to release tags. Dependabot could not follow either: it cannot bump a branch ref, and setup-uv stopped publishing moving major tags at v8.

  • Coverage and mutants ran tests as threads of one process after the rest of CI had moved to nextest, so a missing #[serial] could still make them flaky. Both use nextest now. The coverage total did not move — doctests were never in it — and on three sampled files mutants took about half the time it took under cargo test.

  • stable-ci had four writers and, because rust-cache hashes workflow-level env, was never actually shared with coverage.yml or mutants.yml at all. Each now names the key it really uses.

  • release-plz.yml had no concurrency group. It has one now, with cancel-in-progress: false — which delivers the “must always complete” intent that omitting the block only half-achieved.

  • Mutants (diff) had a 20-minute limit, enough for PRs of a few dozen mutants. #279 changed enough code for 177; the limit stopped the run at 147 (about 8 s each, none missed), and GitHub reported it as cancelled. The limit is 45 minutes now.