How it works

A tool reports the result, never the agent

The agent can start a run. Only shipcheck turns a run into a check, from facts it reads itself: the exit codes of the processes it started, the git tree of the commit, and the config on main. An agent that says the tests passed changes nothing.

What runs

runner in shipcheck.toml picks the mode of every job, and [jobs] moves one job to the other mode in the same run (modes). shipcheck init writes host mode when it finds checks in the workflows or a project file it knows, with a [jobs] act entry for each job no host check runs as GitHub does. It writes act mode for the whole repository when a job runs a Linux matrix, every job runs on Linux and Docker is running, and when it finds nothing (which jobs run where).

• Host mode, runner = "host": the commands the owner lists under [host] checks, such as cargo test or npm test, run natively on this machine in an OS sandbox, each in its own clone. A pass means the code passed on this machine's toolchain, operating system and system libraries (modes).
• act mode, runner = "act": the GitHub workflows run in Linux containers with act. This is the mode closest to GitHub's CI: a pass means the workflows as written passed in a container like GitHub's runner. It needs Docker.

shipcheck reads shipcheck.toml and .github/workflows/ from the base branch, found in one order for every command: the base install stored for this clone, then the default branch the remote reports (git ls-remote --symref origin HEAD), then origin/main with a warning. The hooks never ask the remote, so they skip the second step. init refuses when the remote does not answer, and nothing posts from the origin/main fallback. These docs call that branch main. shipcheck never reads them from the branch or from your working copy, so a PR that edits a workflow to exit 0 or empties workflows changes nothing until it merges.

When paths is set and a change touches none of those files since it left main, nothing runs and every job check is skipped with that reason. shipcheck works this out again on each run, because it depends on where main is.

In act mode, before a run, shipcheck edits its own copy of each workflow it runs. The repository's files stay as they are.

• It adds a workflow_dispatch trigger, so act can run a workflow that on GitHub runs only on pushes to main. A job guarded by if: github.ref == 'refs/heads/main', such as a deploy, is skipped, because the ref is the branch.
• It adds a first and a last step to every job. They print markers with a code that is new for each run, which tell shipcheck that each job started and how it ended.
• It splits each matrix into one job per combination. act does not always pick the same platform for matrix jobs, and separate jobs run the same way every time.

A workflow passes when every job that started reached its end with status success, and no job was skipped for an unsupported platform. As on GitHub, a failed job with continue-on-error: true does not fail the workflow. act runs a PR's checks as workflow_dispatch, so when it skips a job by its if:, that skip does not tell whether GitHub would run the job on the PR. So a job act skips by its if: fails, with failure, act skipped it by its `if:`, which may hold on a PR, and only GitHub can tell, unless that if: is false on every PR:

• false
• a push event, such as github.event_name == 'push'
• one branch, such as github.ref == 'refs/heads/main'
• a tag, such as startsWith(github.ref, 'refs/tags/')

A condition counts when it ands one of these terms with the rest. An ||, or a ! other than !=, could make it hold on a PR, so a condition with one never counts. A job whose if: reads the PR, such as github.event.pull_request.draft == false, or reads vars, fails too. A job that needs a skipped job fails as well when its if: uses always(), failure() or cancelled(), since GitHub would run it anyway.

Steps get the same check. A step whose if: reads something workflow_dispatch and a PR answer differently, the event, github.ref, head_ref, base_ref, the actor or vars, gets a shipcheck if step before it in shipcheck's copy of the workflow. That also counts when the if: reads an env. value whose step, job or workflow env reads one of them. When act skips the step, the guard fails the job with act skipped the step '<name>' by its if:, which may hold on a PR, and only GitHub can tell. A step whose if: is false on every PR, or that runs only on failure() or cancelled(), gets no guard. The guard is part of the workflow as it runs, so the cache key holds it.

The step check has two gaps. It does not see steps inside a composite action. It does not catch a step that act runs but GitHub would skip on the PR.

To fix such a job or step, keep the job on GitHub by leaving its workflow out of workflows, or make the if: false on every PR with one of the terms above.

Which jobs run where

shipcheck init places each job of a workflow GitHub runs for feature branches:

A job goes to act only when every job it needs can run there too. A job a job on GitHub needs stays on GitHub with it. A Rust toolchain or Ruby that this machine lacks does not keep a job off: its check runs on this machine's version, and init lists it under "run on this machine's version, not the workflow's".

A matrix of one combination is a plain job. A matrix exists to test several toolchains or systems, and host mode tests only the one on this machine, so a matrix job runs with act or stays on GitHub. That holds for a matrix such as express's, whose run steps never name the matrix, since its setup actions do. When the whole repository uses act mode, a workflow with a job act cannot run as GitHub does stays on GitHub as it is, since act runs a listed workflow whole.

shipcheck does not support Windows yet, so Windows jobs stay on GitHub Actions (limits).

init prints each job that stays on GitHub and why. A workflow with such a job is kept whole: its triggers stay, and each job shipcheck runs gets an if: that runs it on GitHub only on the default branch, on the other branches the workflow lists for pushes, on PRs into other branches, for Dependabot and for a fork's PR, and-ed with an if: it already has. On a PR into the default branch, GitHub then runs only the jobs that stay, and on the default branch it runs them all. A job that a job on GitHub needs, directly or through others, stays on GitHub too, since GitHub skips a job whose needs it skipped. If you decline the edit, the workflow keeps running fully on GitHub for PRs, beside shipcheck's checks.

The ruleset init offers requires shipcheck/<job> from the shipcheck App and, from GitHub Actions, the checks of the jobs that stay on GitHub. When their names depend on a matrix include or an expression, init says so and you add them once a PR shows them. Jobs that stay on GitHub still use Actions minutes.

To move a repository to act anyway, set runner = "act" and list the workflows in workflows in shipcheck.toml (modes). To give a job back to GitHub, remove its check from [host.checks] or its [jobs] entry, and its if: from the workflow.

When it runs

Runs start from the hooks, as on in shipcheck.toml says, or from shipcheck run.

shipcheck run tests the working copy, so the test run an agent does before it commits becomes the check. shipcheck builds a tree from the working copy in a copy of the index: tracked files with their edits and untracked files that are not gitignored. Your branch, index and stash stay as they are.

--staged builds it from the index instead. When you commit that same tree, with shipcheck commit or with git, the commit hook and shipcheck post find the result by tree and nothing runs again. A commit with any other tree, such as one with a file left out, runs again.

• A run from a hook goes to the background. The commit or push carries on at once, and the run's output goes to logs/background.log in shipcheck's home.
• There is one background run per repository and branch. A newer commit stops the older run, its containers included, before its own starts, like cancel-in-progress in a workflow. A request for the commit already running leaves that run alone, so a push right after a commit does not restart it.
• On a machine signed in with shipcheck login, or with SHIPCHECK_TOKEN set, a push's run also posts the checks. Once GitHub has the commit, every check shows in progress, and when the run ends it turns green or red. A tree that already passed posts at once. A newer push stops the older run, which posts nothing. A machine that is not signed in writes one line to logs/background.log and posts nothing.
• shipcheck run runs in the foreground and waits, for when you or an agent want the verdict before going on.

When a PR changes on GitHub

A rebase or an update on GitHub, such as the Update branch button, makes a new head commit with no local push, so no hook runs. The daemon covers that case. shipcheck init and shipcheck install start it as a background service.

• It reads the open PRs of each repository it was added to, into the base install stored for it: the default branch the remote reported then. With the App installed, shipcheck's service gets GitHub's pull request webhook for each push, rebase, Update branch and merge, and tells the daemon over a socket the daemon keeps open, so the daemon reads the PRs within seconds of the change. While that socket is down, or a head is still waiting, such as for who pushed it, it reads them every 10 seconds by default. Otherwise it reads them every 5 minutes anyway, in case it missed an event.
• When a head has no completed shipcheck check and its tree has a stored result, it posts that result at once. A rebase that changes no file has the same tree, so it needs no run.
• When the tree has no result, it runs the checks and posts them, the same way a push does. A newer head stops the older run, which posts nothing.
• It runs a head only when you, or a login daemon_authors in shipcheck.toml on main lists, both opened the PR and pushed that head. It never runs a fork's PR.
• "Update branch" and a rebase on GitHub are pushes by whoever clicked the button. So when you click it on your own PR, your daemon runs the new head. When a teammate clicks it, your daemon refuses the head unless daemon_authors lists them. To check it yourself, pull the branch, then run shipcheck run --sha <head> and shipcheck post <head>.
• The daemon learns who pushed a head from shipcheck's service first, which gets GitHub's pull request webhook for each push to a PR within seconds. Otherwise it reads the PR's timeline, which shows a rebase "Update branch" or a force-push at once. For a merge "Update branch", it reads the merge commit GitHub made, which names the person who clicked. For a PR merged into the branch, such as a stacked PR, it reads who merged it. Last, it reads the repository's events, which GitHub can list hours late. Until it knows, the checks stay queued and it asks again on every poll.
• A head that someone not allowed pushed stays refused for 30 days, and so does every later head that contains it, such as a merge "Update branch" made on top of it. A head whose tree already has a result is never refused: the daemon posts that result.
• With several machines running it, the first to mark a head in progress runs it, and the others wait. Two that start within a few seconds of each other can both run it, and both post the same result.
• When a PR merges into the branch, it checks the branch's new tip too, also when the service cannot say whether a closed PR merged, so main has checks of its own, the ones a status badge reads. A squash of a PR whose tree passed posts at once from that result. A commit whose message has [skip ci], such as a release bump, gets none.
• Every 5 minutes it also looks in the folders that hold the repositories it watches, such as ~/Projects. A clone of another repository is added once its default branch has a shipcheck.toml and the App can read the repository, so installing the App on a repository you cloned is enough.

daemon = false in shipcheck.toml keeps it off a repository, and shipcheck daemon --uninstall removes it from the machine.

The daemon's settings live in ~/.local/state/shipcheck/daemon.toml, next to the repositories it watches. fetch = "https" sends shipcheck's own fetches of a GitHub remote over HTTPS with git's credential helper, such as gh auth git-credential: the daemon's, a posting run's and the one that finds the default branch. An SSH key that asks for a touch then never signs them. Your remotes, your own git and every push stay as they are.

Where it runs

In act mode each job gets a new checkout of the tested tree, cloned with copy-on-write from its own warm template, so it starts with the build output of its earlier runs and costs no disk. A working copy is checked out from a commit no ref names, with HEAD as parent, as git stash create makes. A run never touches your own checkout. Jobs never share a checkout, as on GitHub. A job starts once the jobs in its needs pass, and gets their outputs. Gitignored files are not in it.

The checkout sits at a fixed path per repository, slot and job, because builds remember absolute paths. Toolchains and package caches live in Docker volumes per repository, removed after 14 days without a run that uses them. apt's package lists get a volume per slot and job, and .venv a new volume each run, off the macOS bind mount. setup-uv steps get prune-cache: false in the job's copy, so uv's cache keeps its wheels, and pnpm/action-setup's run_install becomes plain pnpm install steps after it, since the action prunes the store after an install. Both are fed only by shipcheck's own runs, never by builds on your machine, which would not run in a Linux container.

act runs each job in Docker, never on the machine itself. A job gets its checkout, its cache volumes, no Docker socket, no host token or environment, and by default no network.

A job with engine = "colima" runs on a Colima VM that shipcheck starts for that job and deletes after it. The VM shares no folder with this machine. It gets the checkout's tracked files and a .git of its own over colima ssh, and act runs inside it with -self-hosted, so the steps run on the VM, with its own Docker and kernel. The VM reaches this machine's 127.0.0.1 and the internet whatever network says, so it is for code you trust (engines).

In host mode each check gets the same kind of clone, but its template is warmed from your checkout at most once an hour, so gitignored node_modules and target/ are there. It runs in sandbox-exec on macOS or bubblewrap on Linux: it writes only its clone, a new home and the repository's tool caches, reads the system and your toolchains, and has the network (config).

When a stored result counts

shipcheck stores each result under a key made of:

• the git tree that ran
• shipcheck.toml on main
• each workflow as shipcheck runs it, or in host mode each check's command and the output of versions
• the act and shipcheck versions
• the pinned images

Two commits with the same tree share a result, so a rebase or a squash that changes nothing needs no new run. A change to the config or the workflows on main changes the key, so old results stop counting.

Only a pass is reused. A failure runs again next time, so a flaky failure cannot stick.

Two runs of the same tree at once share one run. The second waits, then reads the result.

The checks on the PR

shipcheck posts one check per job, shipcheck/test for job test, as GitHub Actions does, and branch protection requires them (require the checks). Each is queued until a machine runs the commit, in progress while it runs, then green or red. A paths skip marks every job skipped. A shipcheck check beside them sums up the run, with each failed workflow's log tail.

Each job check shows how long the job took and, as logs in shipcheck.toml says, the end of its log (config):

• The last 100 lines. A longer log starts with [...], and a host check's log keeps its command line above it, so you still see what ran.
• A failed job lists its failed tests first, then each test's own output, such as its panic, assertion or traceback, even when it is far above the last lines. This works for cargo test, nextest, pytest, jest, vitest and go test.
• When the failures do not fit in GitHub's 65,535 characters, the check keeps the first ones whole and says how many it left out. shipcheck logs has the rest.
• Values from the secrets file are replaced with ***. A log can still hold paths, names or a secret printed in another form.
• A fork's PR never gets a log.

The report comment

Write a report to $GITHUB_STEP_SUMMARY and it appears in one shipcheck comment on the PR. This replaces a step that posts a coverage report with gh pr comment, which cannot run under shipcheck: a job has no GitHub token and, by default, no network.

• Each job check gets its own section, failed jobs first. Every step that writes to the file adds to its job's section in step order, as on GitHub.
• The App writes the comment once and then edits it. A later post replaces the sections of the jobs it ran and keeps the others, so a rerun of one job changes only that job's section. A job that ran again and wrote nothing loses its section.
• If someone edited the comment and its hidden markers no longer match, the next post writes the whole comment again from its own result.
• A run where no job wrote a summary posts nothing and deletes nothing.
• Each job keeps at most 20 KB of its summary. The comment stays under GitHub's 65,536 characters, and a section that does not fit says it was left out.
• Values from the secrets file are replaced with ***.
• A fork's PR never gets the comment, since the fork's own code wrote the Markdown.

It works in both modes. In act mode, act gives each step the file. In host mode, each check gets GITHUB_STEP_SUMMARY set to a file in its sandbox. The App needs Pull requests: write for the comment. Until it has it, the checks still post, and shipcheck post prints one line saying there is no comment.

Previous: Commands Next: What a green check proves