# How it works

https://shipcheck.run/docs/how-it-works/

> What shipcheck runs, where, and when it trusts a stored result.

## A tool reports the result, never the agent

The agent can start a run. Only shipcheck turns a run into a check, from facts it reads itself: the exit codes of the processes it started, the git tree of the commit, and the config on `main`. An agent that says the tests passed changes nothing.

## What runs

`runner` in `shipcheck.toml` picks the mode of every job, and `[jobs]` moves one job to the other mode in the same run ([modes](https://shipcheck.run/docs/config/#modes)). `shipcheck init` writes host mode when it finds checks in the workflows or a project file it knows, with a `[jobs]` act entry for each job no host check runs as GitHub does. It writes act mode for the whole repository when a job runs a Linux matrix, every job runs on Linux and Docker is running, and when it finds nothing ([which jobs run where](#which-jobs-run-where)).

- Host mode, `runner = "host"`: the commands the owner lists under `[host] checks`, such as `cargo test` or `npm test`, run natively on this machine in an OS sandbox, each in its own clone. A pass means the code passed on this machine's toolchain, operating system and system libraries ([modes](https://shipcheck.run/docs/config/#modes)).
- act mode, `runner = "act"`: the GitHub workflows run in Linux containers with act. This is the mode closest to GitHub's CI: a pass means the workflows as written passed in a container like GitHub's runner. It needs Docker.

shipcheck reads `shipcheck.toml` and `.github/workflows/` from the base branch, found in one order for every command: the base `install` stored for this clone, then the default branch the remote reports (`git ls-remote --symref origin HEAD`), then `origin/main` with a warning. The hooks never ask the remote, so they skip the second step. `init` refuses when the remote does not answer, and nothing posts from the `origin/main` fallback. These docs call that branch `main`. shipcheck never reads them from the branch or from your working copy, so a PR that edits a workflow to `exit 0` or empties `workflows` changes nothing until it merges.

When `paths` is set and a change touches none of those files since it left `main`, nothing runs and every job check is skipped with that reason. shipcheck works this out again on each run, because it depends on where `main` is.

In act mode, before a run, shipcheck edits its own copy of each workflow it runs. The repository's files stay as they are.

- It adds a `workflow_dispatch` trigger, so act can run a workflow that on GitHub runs only on pushes to `main`. A job guarded by `if: github.ref == 'refs/heads/main'`, such as a deploy, is skipped, because the ref is the branch.
- It adds a first and a last step to every job. They print markers with a code that is new for each run, which tell shipcheck that each job started and how it ended.
- It splits each matrix into one job per combination. act does not always pick the same platform for matrix jobs, and separate jobs run the same way every time.

A workflow passes when every job that started reached its end with status `success`, and no job was skipped for an unsupported platform. As on GitHub, a failed job with `continue-on-error: true` does not fail the workflow. act runs a PR's checks as `workflow_dispatch`, so when it skips a job by its `if:`, that skip does not tell whether GitHub would run the job on the PR. So a job act skips by its `if:` fails, with ``failure, act skipped it by its `if:`, which may hold on a PR, and only GitHub can tell``, unless that `if:` is false on every PR:

- `false`
- a push event, such as `github.event_name == 'push'`
- one branch, such as `github.ref == 'refs/heads/main'`
- a tag, such as `startsWith(github.ref, 'refs/tags/')`

A condition counts when it ands one of these terms with the rest. An `||`, or a `!` other than `!=`, could make it hold on a PR, so a condition with one never counts. A job whose `if:` reads the PR, such as `github.event.pull_request.draft == false`, or reads `vars`, fails too. A job that needs a skipped job fails as well when its `if:` uses `always()`, `failure()` or `cancelled()`, since GitHub would run it anyway.

Steps get the same check. A step whose `if:` reads something `workflow_dispatch` and a PR answer differently, the event, `github.ref`, `head_ref`, `base_ref`, the actor or `vars`, gets a `shipcheck if` step before it in shipcheck's copy of the workflow. That also counts when the `if:` reads an `env.` value whose step, job or workflow `env` reads one of them. When act skips the step, the guard fails the job with `act skipped the step '<name>' by its if:, which may hold on a PR, and only GitHub can tell`. A step whose `if:` is false on every PR, or that runs only on `failure()` or `cancelled()`, gets no guard. The guard is part of the workflow as it runs, so the cache key holds it.

The step check has two gaps. It does not see steps inside a composite action. It does not catch a step that act runs but GitHub would skip on the PR.

To fix such a job or step, keep the job on GitHub by leaving its workflow out of `workflows`, or make the `if:` false on every PR with one of the terms above.

## Which jobs run where

`shipcheck init` places each job of a workflow GitHub runs for feature branches:

| Job | For a PR it runs |
|---|---|
| A host check runs it as GitHub does | Here, as a host check |
| No host check runs it as GitHub does, such as a job with a heredoc or `\` step, `sudo` or `apt`, an action no command stands for, a `container:`, a JDK this machine lacks, or a Linux matrix, Docker running | Here, with act, as a `[jobs."<file>/<job>"]` entry with `runner = "act"`. init also writes `network = "internet"`, since such jobs download what their steps set up |
| The same, no Docker | On GitHub Actions |
| A Linux matrix, Docker running, every job on Linux and none running Docker | Here, with act. The whole repository uses act mode, with `network = "internet"` and `shard_tests = true` |
| A matrix with macOS or Windows entries | On GitHub Actions |
| On macOS | Here as a host check on a Mac, on GitHub Actions elsewhere |
| On Windows | On GitHub Actions |
| It reads the PR: through its `if:`, a step's `if:`, an `env`, secrets, `vars` or ref value, a script, Makefile, justfile or `package.json` its command reads, or `HEAD` in a PR workflow | On GitHub Actions |
| It uses a PR-aware action, such as one that lints only the changed lines, or an action that reports to GitHub or publishes | On GitHub Actions |
| Its `runs-on` label has no act image, or it uses a local action | On GitHub Actions |
| It runs Docker or changes the kernel, such as `mount`, `sysctl`, `bwrap` or `modprobe` | On GitHub Actions. colima could run it, and you can move it to a colima VM on init's Checks tab. init never picks that by default, since the VM reaches this machine's loopback ([engines](https://shipcheck.run/docs/config/#engines)) |
| Its host check fails when init tries it | On GitHub Actions, and init prints the end of its log and the fix |

A job goes to act only when every job it needs can run there too. A job a job on GitHub needs stays on GitHub with it. A Rust toolchain or Ruby that this machine lacks does not keep a job off: its check runs on this machine's version, and init lists it under "run on this machine's version, not the workflow's".

A matrix of one combination is a plain job. A matrix exists to test several toolchains or systems, and host mode tests only the one on this machine, so a matrix job runs with act or stays on GitHub. That holds for a matrix such as express's, whose `run` steps never name the matrix, since its setup actions do. When the whole repository uses act mode, a workflow with a job act cannot run as GitHub does stays on GitHub as it is, since act runs a listed workflow whole.

shipcheck does not support Windows yet, so Windows jobs stay on GitHub Actions ([limits](https://shipcheck.run/docs/limits/#windows)).

`init` prints each job that stays on GitHub and why. A workflow with such a job is kept whole: its triggers stay, and each job shipcheck runs gets an `if:` that runs it on GitHub only on the default branch, on the other branches the workflow lists for pushes, on PRs into other branches, for Dependabot and for a fork's PR, and-ed with an `if:` it already has. On a PR into the default branch, GitHub then runs only the jobs that stay, and on the default branch it runs them all. A job that a job on GitHub needs, directly or through others, stays on GitHub too, since GitHub skips a job whose needs it skipped. If you decline the edit, the workflow keeps running fully on GitHub for PRs, beside shipcheck's checks.

The ruleset `init` offers requires `shipcheck/<job>` from the shipcheck App and, from GitHub Actions, the checks of the jobs that stay on GitHub. When their names depend on a matrix `include` or an expression, `init` says so and you add them once a PR shows them. Jobs that stay on GitHub still use Actions minutes.

To move a repository to act anyway, set `runner = "act"` and list the workflows in `workflows` in `shipcheck.toml` ([modes](https://shipcheck.run/docs/config/#modes)). To give a job back to GitHub, remove its check from `[host.checks]` or its `[jobs]` entry, and its `if:` from the workflow.

## When it runs

Runs start from the hooks, as `on` in `shipcheck.toml` says, or from `shipcheck run`.

`shipcheck run` tests the working copy, so the test run an agent does before it commits becomes the check. shipcheck builds a tree from the working copy in a copy of the index: tracked files with their edits and untracked files that are not gitignored. Your branch, index and stash stay as they are.

`--staged` builds it from the index instead. When you commit that same tree, with `shipcheck commit` or with git, the commit hook and `shipcheck post` find the result by tree and nothing runs again. A commit with any other tree, such as one with a file left out, runs again.

- A run from a hook goes to the background. The commit or push carries on at once, and the run's output goes to `logs/background.log` in shipcheck's home.
- There is one background run per repository and branch. A newer commit stops the older run, its containers included, before its own starts, like `cancel-in-progress` in a workflow. A request for the commit already running leaves that run alone, so a push right after a commit does not restart it.
- On a machine signed in with `shipcheck login`, or with `SHIPCHECK_TOKEN` set, a push's run also posts the checks. Once GitHub has the commit, every check shows in progress, and when the run ends it turns green or red. A tree that already passed posts at once. A newer push stops the older run, which posts nothing. A machine that is not signed in writes one line to `logs/background.log` and posts nothing.
- `shipcheck run` runs in the foreground and waits, for when you or an agent want the verdict before going on.

## When a PR changes on GitHub

A rebase or an update on GitHub, such as the Update branch button, makes a new head commit with no local push, so no hook runs. The daemon covers that case. `shipcheck init` and `shipcheck install` start it as a background service.

- It reads the open PRs of each repository it was added to, into the base `install` stored for it: the default branch the remote reported then. With the App installed, shipcheck's service gets GitHub's pull request webhook for each push, rebase, Update branch and merge, and tells the daemon over a socket the daemon keeps open, so the daemon reads the PRs within seconds of the change. While that socket is down, or a head is still waiting, such as for who pushed it, it reads them every 10 seconds by default. Otherwise it reads them every 5 minutes anyway, in case it missed an event.
- When a head has no completed `shipcheck` check and its tree has a stored result, it posts that result at once. A rebase that changes no file has the same tree, so it needs no run.
- When the tree has no result, it runs the checks and posts them, the same way a push does. A newer head stops the older run, which posts nothing.
- It runs a head only when you, or a login `daemon_authors` in `shipcheck.toml` on `main` lists, both opened the PR and pushed that head. It never runs a fork's PR.
- "Update branch" and a rebase on GitHub are pushes by whoever clicked the button. So when you click it on your own PR, your daemon runs the new head. When a teammate clicks it, your daemon refuses the head unless `daemon_authors` lists them. To check it yourself, pull the branch, then run `shipcheck run --sha <head>` and `shipcheck post <head>`.
- The daemon learns who pushed a head from shipcheck's service first, which gets GitHub's pull request webhook for each push to a PR within seconds. Otherwise it reads the PR's timeline, which shows a rebase "Update branch" or a force-push at once. For a merge "Update branch", it reads the merge commit GitHub made, which names the person who clicked. For a PR merged into the branch, such as a stacked PR, it reads who merged it. Last, it reads the repository's events, which GitHub can list hours late. Until it knows, the checks stay queued and it asks again on every poll.
- A head that someone not allowed pushed stays refused for 30 days, and so does every later head that contains it, such as a merge "Update branch" made on top of it. A head whose tree already has a result is never refused: the daemon posts that result.
- With several machines running it, the first to mark a head in progress runs it, and the others wait. Two that start within a few seconds of each other can both run it, and both post the same result.
- When a PR merges into the branch, it checks the branch's new tip too, also when the service cannot say whether a closed PR merged, so `main` has checks of its own, the ones a status badge reads. A squash of a PR whose tree passed posts at once from that result. A commit whose message has `[skip ci]`, such as a release bump, gets none.
- Every 5 minutes it also looks in the folders that hold the repositories it watches, such as `~/Projects`. A clone of another repository is added once its default branch has a `shipcheck.toml` and the App can read the repository, so installing the App on a repository you cloned is enough.

`daemon = false` in `shipcheck.toml` keeps it off a repository, and `shipcheck daemon --uninstall` removes it from the machine.

The daemon's settings live in `~/.local/state/shipcheck/daemon.toml`, next to the repositories it watches. `fetch = "https"` sends shipcheck's own fetches of a GitHub remote over HTTPS with git's credential helper, such as `gh auth git-credential`: the daemon's, a posting run's and the one that finds the default branch. An SSH key that asks for a touch then never signs them. Your remotes, your own git and every push stay as they are.

## Where it runs

In act mode each job gets a new checkout of the tested tree, cloned with copy-on-write from its own warm template, so it starts with the build output of its earlier runs and costs no disk. A working copy is checked out from a commit no ref names, with `HEAD` as parent, as `git stash create` makes. A run never touches your own checkout. Jobs never share a checkout, as on GitHub. A job starts once the jobs in its `needs` pass, and gets their outputs. Gitignored files are not in it.

The checkout sits at a fixed path per repository, slot and job, because builds remember absolute paths. Toolchains and package caches live in Docker volumes per repository, removed after 14 days without a run that uses them. apt's package lists get a volume per slot and job, and `.venv` a new volume each run, off the macOS bind mount. setup-uv steps get `prune-cache: false` in the job's copy, so uv's cache keeps its wheels, and `pnpm/action-setup`'s `run_install` becomes plain `pnpm install` steps after it, since the action prunes the store after an install. Both are fed only by shipcheck's own runs, never by builds on your machine, which would not run in a Linux container.

act runs each job in Docker, never on the machine itself. A job gets its checkout, its cache volumes, no Docker socket, no host token or environment, and by default no network.

A job with `engine = "colima"` runs on a Colima VM that shipcheck starts for that job and deletes after it. The VM shares no folder with this machine. It gets the checkout's tracked files and a `.git` of its own over `colima ssh`, and act runs inside it with `-self-hosted`, so the steps run on the VM, with its own Docker and kernel. The VM reaches this machine's `127.0.0.1` and the internet whatever `network` says, so it is for code you trust ([engines](https://shipcheck.run/docs/config/#engines)).

In host mode each check gets the same kind of clone, but its template is warmed from your checkout at most once an hour, so gitignored `node_modules` and `target/` are there. It runs in `sandbox-exec` on macOS or bubblewrap on Linux: it writes only its clone, a new home and the repository's tool caches, reads the system and your toolchains, and has the network ([config](https://shipcheck.run/docs/config/#modes)).

## When a stored result counts

shipcheck stores each result under a key made of:

- the git tree that ran
- `shipcheck.toml` on `main`
- each workflow as shipcheck runs it, or in host mode each check's command and the output of `versions`
- the act and shipcheck versions
- the pinned images

Two commits with the same tree share a result, so a rebase or a squash that changes nothing needs no new run. A change to the config or the workflows on `main` changes the key, so old results stop counting.

Only a pass is reused. A failure runs again next time, so a flaky failure cannot stick.

Two runs of the same tree at once share one run. The second waits, then reads the result.

## The checks on the PR

shipcheck posts one check per job, `shipcheck/test` for job `test`, as GitHub Actions does, and branch protection requires them ([require the checks](https://shipcheck.run/docs/branch-protection.md)). Each is queued until a machine runs the commit, in progress while it runs, then green or red. A `paths` skip marks every job skipped. A `shipcheck` check beside them sums up the run, with each failed workflow's log tail.

Each job check shows how long the job took and, as `logs` in `shipcheck.toml` says, the end of its log ([config](https://shipcheck.run/docs/config.md)):

- The last 100 lines. A longer log starts with `[...]`, and a host check's log keeps its command line above it, so you still see what ran.
- A failed job lists its failed tests first, then each test's own output, such as its panic, assertion or traceback, even when it is far above the last lines. This works for cargo test, nextest, pytest, jest, vitest and go test.
- When the failures do not fit in GitHub's 65,535 characters, the check keeps the first ones whole and says how many it left out. `shipcheck logs` has the rest.
- Values from the secrets file are replaced with `***`. A log can still hold paths, names or a secret printed in another form.
- A fork's PR never gets a log.

## The report comment

Write a report to `$GITHUB_STEP_SUMMARY` and it appears in one shipcheck comment on the PR. This replaces a step that posts a coverage report with `gh pr comment`, which cannot run under shipcheck: a job has no GitHub token and, by default, no network.

- Each job check gets its own section, failed jobs first. Every step that writes to the file adds to its job's section in step order, as on GitHub.
- The App writes the comment once and then edits it. A later post replaces the sections of the jobs it ran and keeps the others, so a rerun of one job changes only that job's section. A job that ran again and wrote nothing loses its section.
- If someone edited the comment and its hidden markers no longer match, the next post writes the whole comment again from its own result.
- A run where no job wrote a summary posts nothing and deletes nothing.
- Each job keeps at most 20 KB of its summary. The comment stays under GitHub's 65,536 characters, and a section that does not fit says it was left out.
- Values from the secrets file are replaced with `***`.
- A fork's PR never gets the comment, since the fork's own code wrote the Markdown.

It works in both modes. In act mode, act gives each step the file. In host mode, each check gets `GITHUB_STEP_SUMMARY` set to a file in its sandbox. The App needs Pull requests: write for the comment. Until it has it, the checks still post, and `shipcheck post` prints one line saying there is no comment.