Rust builds across worktrees, from 40 seconds to 2
I keep about twenty tasks a day in flight, each on its own branch and worktree, and each one needs to compile the same Rust runtime. For the runtime alone the naive cost is 3 GB of build output and roughly 40 seconds per task. The disk fills before the day ends.
What follows is one machine (32 cores, warm Nix store, btrfs), the ideas that failed, and the combination that worked: worktrunk for worktrees, devenv for the per-task environment, and mr boxington (mbx) for the compiler cache.
All test cases, profiles, and the scripts that produced these numbers are in rust-build-bench. The measured values are in results.md.
Start with Cargo’s timing report
For a 3-crate workspace with one binary, a cold build in a fresh worktree (scripts/cargo-timings.sh):
| wall time | 36 s |
| units | 575 |
| dependency units | 570, about 336 s of CPU |
| workspace crates | 5 units, 14.7 s |
| output | target/, 3.0 GB |
On 32 cores that is about 10x parallelism, so 36 seconds of wall time from roughly 350 seconds of CPU. Nearly all of it is the dependency graph. Your own crates are the last 15 seconds and the link.
The dependencies are the build. Everything else here follows from that.
Worktrees are cheap. Copying them is a trap.
A worktree costs almost nothing. worktrunk created the branch and checkout across three repos in 252 ms. Plain git worktree add takes 82 ms. Orchestration was never the bottleneck.
The next idea is always to copy target/ from a warm worktree. Worktrunk can do it, with reflinks where the filesystem supports them (scripts/copy-ignored.sh):
$ wt -C repos/rig step copy-ignored --from stack/cache --to stack/feature
✓ Copied 21274 files · 10.8 GiB (reflinked, no extra disk)
That is 461 ms and no extra disk. For node_modules, .env, and generated assets, it is a good trick.
For Rust it does nothing. Copy the target directory, then build:
$ cargo build --manifest-path runtime-rs/Cargo.toml -p rig-runtime
Finished `dev` profile in 36 s
Cargo recompiled everything. It rewrote 60,922 files and 2 GB before I stopped waiting. The reason sits in the package ID. For a path package it contains the checkout path, so the same crate under threads/a/rig and threads/b/rig is a different package, with a different -C metadata hash and different output filenames. A copied target/ holds artifacts for the wrong package. Dependencies survive because their package IDs are path-independent. Your crates never do.
Use worktrunk for the worktree and for the ignored files that are not path-keyed. Leave target/ out of the copy.
devenv: one project per task, and the eval cost
Each task gets its own devenv project directory so ports, the process manager, and service state do not collide. That is what lets two tasks run at once, and it means every task pays an evaluation.
Fresh project directory, warm store (scripts/eval.sh):
first print-dev-env |
4.85 to 5.85 s |
| same directory, second call | 207 to 258 ms |
The trace splits the first call:
validating lock: 1.25 s
configuring shell: 3.89 s
evaluating shell: 3.71 s
realizing shell: 1.85 s
selecting drvPath: 0.82 s
extracting shell env: 0.68 s
Two details matter. The shell is not rebuilt: DEVENV_PROFILE is the same store path for every project of the same profile, so only the evaluation repeats. And part of the cost is cache lookups. devenv asks the binary caches about per-project artifacts (tasks.json, devenv-shell-env) that can never be there, because they embed the project path. Turning off devenv’s cachix integration on a warm store took a fresh project from 4.85 s to 3.66 s.
The obvious shortcuts do not work:
| attempt | fresh project |
|---|---|
copy a warm .devenv into the new project |
4.60 s |
| copy with identical content and mtimes, different path | 4.65 s |
identical /nix/store input path, different project root |
4.58 s |
share DEVENV_ROOT |
1.37 s |
Only the last row reuses the evaluation, and it breaks isolation. With a shared root both projects report the same runtime directory, bind the same port, and down in one stops the other. DEVENV_STATE and DEVENV_RUNTIME are derived from the root and are not read from the environment, so you cannot split the eval root from the runtime state. That is an upstream limitation, not a configuration mistake.
Four seconds per task is the price of running tasks in parallel. A resumed task costs 225 ms. I paid it.
mr boxington: cache compiler outputs, not directories
mbx works at the compiler boundary. It sets itself as RUSTC_WRAPPER, derives a key for each compiler invocation, and restores matching outputs from a content-addressed store. What makes it useful here is that it replaces checkout paths with placeholders before they enter the key, so two worktrees with the same content produce the same key.
Measured on the same workspace (scripts/ab.sh):
| cargo | mbx | |
|---|---|---|
| first build, empty store | 36 s | 78 s |
| new checkout, same content | 36 s | 20.6 s, 431 hits and 7 misses |
| one-line edit | 11.4 s | 5.2 s |
| per-worktree disk | 3.0 GB | 12 MB of source, target in the store |
The first build is slower because mbx hashes inputs and learns the dependency predictions. That cost is once per machine. After it, 431 of 438 compiler invocations came from the cache.
The seven misses are the part to understand: reqwest, a git dependency, two of the three workspace crates, and the link. That is the expensive end of the graph, and mbx cannot share it across checkouts on this workspace. A new checkout is still about 1.8x faster, and an edit about 2x.
These are build times inside a devenv shell. A full new task adds the 4.5 seconds of evaluation and process start, which is where the end-to-end figures below come from.
Two settings matter in practice. mbx sets CARGO_INCREMENTAL=0 by default and keeps private incremental state for edited workspace crates. Leave that alone. If you hand incremental back to Cargo, its per-checkout state keeps the dependents out of the shared cache. And plan for cleanup. After nine test checkouts, mbx gc reclaimed 21.2 GiB. A worktree manager that deletes worktrees without calling mbx gc moves the trash rather than removing it.
The alternatives, measured
crane
Split the build into buildDepsOnly and buildPackage (expr/crane.nix). The dependency derivation is keyed on Cargo.toml and Cargo.lock only, so on a warm store a new worktree is a pure hit.
| new worktree | one-line edit | |
|---|---|---|
| crane | 5.4 s | 36 to 41 s |
| mbx | 20 to 31 s | 2 to 5 s |
crane wins the cold start because the whole app derivation is content-keyed. An unmodified checkout is a store hit, your crates included. It loses the edit loop because buildDepsOnly caches against a dummy workspace source, so every edit recompiles the workspace and relinks in the sandbox. Switching crane to the dev profile changed the cold build from 4 minutes 02 seconds to 95 seconds and the binary from 26 MB to 106 MB, but the edit stayed at 36 seconds. The profile is not the lever.
crate2nix
One derivation per crate (scripts/crate2nix.sh). It shares the useful property, since the same content at a different path gives the same store path and a full hit took 1.1 seconds, but the cold build was 6 minutes 08 seconds and a one-line edit 49 seconds. buildRustCrate compiles the changed crate at opt-level=3 with no incremental state. With three crates and one dominant binary, crate granularity buys nothing.
A mount namespace and a reflinked target
The fastest thing I found (scripts/namespace.sh). Give every worktree the same absolute build path inside a private namespace, so Cargo’s path-keyed cache behaves like a content-addressed one, and reflink a warm target in.
| new worktree | edit | |
|---|---|---|
| namespace and reflink | 2.7 s | 1.8 s |
It works. It is also Linux-only, needs unprivileged user namespaces, and puts a virtual path between the editor and the build. Build times again; with devenv’s evaluation a new task lands around 7 seconds. I did not ship it.
The setup that shipped
- worktrunk creates the worktrees and copies ignored files that are not path-keyed:
node_modules,.env, generated assets.target/is excluded. - devenv gives each task its own project with separate ports, process manager, and state. Accept the 4 seconds as the cost of running two tasks at once.
- mbx caches compiler outputs across worktrees, so a new checkout skips 431 invocations and an edit recompiles the changed crates.
A small CLI, pot, wires the three together: it creates the branch and worktree in every repo, prepares the per-task project, and hands the terminal to devenv with the right input paths. The profile is the only place the build changes. In the devenv profile, the runtime process is this (profiles/hybrid):
processes.runtime.exec = ''
cd ${toString inputs.rig}
if [ ! -x runtime-rs/target/debug/rig-runtime ]; then
mbx build --manifest-path runtime-rs/Cargo.toml -p rig-runtime
fi
exec runtime-rs/target/debug/rig-runtime
'';
The version the runtime reports comes from the build environment, not from a file a sandbox cannot see:
# .cargo/config.toml
[env]
RIG_RUNTIME_VERSION = "0.4.187"
// build.rs
let version = std::env::var("RIG_RUNTIME_VERSION")
.unwrap_or_else(|_| "unknown".to_string());
If you also want the 5-second cold start, keep a Nix build of the runtime and choose at run time. Evaluate the derivation’s outPath (about 1 second, builds nothing), exec it when the path exists, otherwise fall back to mbx. End to end that measured 6.5 seconds for a new task and about 2 seconds per later edit. The first edit in a worktree pays mbx’s cold build there, 27 seconds, because its target starts empty. Every edit after that is warm.
What to expect
| new task | later edits | disk per task | |
|---|---|---|---|
| cargo alone | 40 s | 11 s | 3.0 GB |
| worktrunk, devenv, mbx | 31 s | 1.9 s | 12 MB |
| plus crane for cold tasks | 6.5 s | about 2 s | 12 MB plus store |
Numbers from one host and one workspace (results.md). The ratios travel: dependencies are about 90% of the build, path-keyed caching is why copying does not work, and content-keyed caching is why the store does.
Loose ends
- mbx is young, 1.30.0 and MIT. Pin the release binary; it is not in nixpkgs yet. Its first build is slower, and its store needs to be part of your cleanup.
- devenv’s eval cache is keyed on the project root. Sharing it needs
DEVENV_STATEandDEVENV_RUNTIMEoverrides from upstream. Until then, 4 seconds per task. - A pinned Rust toolchain still matters. If two paths build the same binary with different compilers, they are not the same binary. This repo pins one stable version and a drift check enforces it across Nix, CI, and the release scripts.
target/stays out ofcopy-ignoredin every configuration. It is useless for Cargo and pointless for Nix.