Workers, worktrees and fan-out
The orchestrator does the work of a run by starting workers. This guide covers the commands it uses, and how to write a workflow that uses them. You don’t usually run these yourself. You describe what you want in the workflow body, and the orchestrator knows the commands. The examples on this page are files under docs/examples/ that CI validates.
Workers need an orchestrator, so a workflow that uses them sets mode: orchestrated in its frontmatter. In the default mode: single, the run’s one agent does the work itself, and tome worker, tome group and tome worktree create are refused. See Workflows, runs and steps.
Workers
# an agent worker: gets a prompt and reports back
tome worker spawn --name reviewer --prompt "Review src/lib.rs and report what you find"
# a command worker: everything after `--` is the command
tome worker spawn --name tests -- cargo test
An agent worker starts through a harness (--harness picks one, otherwise the workflow’s defaults.harness, otherwise the one its orchestrator runs). It runs in its own session, and it finishes by calling tome worker done --summary "..." or tome worker fail --summary "...". If it exits without doing either, it is marked failed. A command worker is done when it exits 0, and its summary is the exit code and the last lines of output.
Only the orchestrator starts workers. Workers can’t start workers, so if a task needs splitting, a worker says so in its summary and the orchestrator decides.
Then:
tome worker status [name] # status, summary, branch and worktree
tome worker wait <name> # block until it finishes, then print its status
tome worker kill <name> # stop it; it is marked cancelled
Sessions are closed when a worker finishes. --keep-open keeps one open so you can read the output.
Workers get these environment variables: TOME_RUN_ID, TOME_WORKER_ID, TOME_OUTPUT=json and, with a worktree, TOME_WORKTREE. That is why the commands above need no ids.
Here is a workflow that runs the tests as a command worker and starts an agent only if they fail:
---
name: test-then-fix
description: Run the tests as a command worker, and start an agent to fix any failures.
mode: orchestrated
---
Run the test suite and fix what breaks.
## Test
Spawn a command worker named `tests` that runs `cargo test`, and wait for it.
If it succeeded, finish the run successfully.
## Fix
Spawn an agent worker named `fixer` on its own worktree, told to make the
failing tests pass and to commit its work. Give it the tail of the test
output from the `tests` worker's summary. Wait for it. If it failed, fail the
run.
## Verify
Merge the `fixer` branch, run the tests again as a command worker, and finish
the run with the result.
Worktrees
A worker started with --worktree gets its own git worktree of the project, on a new branch named tome/<run>/<name>. It starts from --base (a branch or commit), or from the current HEAD commit. Uncommitted changes in your checkout are not included.
tome worker spawn --name task-1 --worktree --base main --prompt "Add a --verbose flag"
--branch names the branch instead, to follow your team’s convention or to open a pull request from it. The branch must not exist yet, so put something unique in the name, such as a task id:
tome worker spawn --name t004-renderer --worktree --base main --branch task/004-renderer --prompt "..."
Say how to name branches in the workflow body, and the orchestrator passes --branch for you.
The worker’s working directory is the worktree, and it commits its changes to that branch. tome doesn’t merge anything for you. The orchestrator merges the branches with git, which is why prompts should tell workers to commit before they report done.
tome worktree create <name> [--base <ref>] [--branch <name>] makes a worktree that isn’t tied to a worker.
Worktrees stay after a run ends, so you can look at the branches. A run resumed with tome run resume adopts the old run’s worktrees, so tome worktree create <name>, or a worker spawned with --worktree under its old name, gets the old worktree back with its work. tome gc removes them with old runs, and deletes a branch only if it was merged into its base.
Fan-out and fan-in
Fan-out is starting many workers at once. Fan-in is waiting for them and combining what they did. A group ties the two together:
tome group create reviews --fail-fast # optional; a group is also created on first use
tome worker spawn --group reviews --name review-1 --prompt "..."
tome worker spawn --group reviews --name review-2 --prompt "..."
tome group wait reviews # closes the group and blocks until every member is finished
tome group status reviews # every member's result
--fail-fast makes the first failure cancel the other members. tome group close stops a group taking new members.
Example: one reviewer per changed file, then one report.
---
name: fan-out
description: Review every file changed on a branch in parallel, then combine the reviews.
mode: orchestrated
params:
base: {type: string, default: main, description: "Compare the current branch to this one"}
concurrency: 1
---
Review the files changed since {{params.base}}, one worker per file.
## Split
List the files that changed between {{params.base}} and HEAD. If there are
none, finish the run successfully and say so.
## Review
Create a group named `reviews`. For each changed file, spawn an agent worker
in that group, named `review-<n>`, told to review only that file and to
report its findings as its summary. Then wait for the group.
## Combine
Read every member's summary from the group's status. Write one report to
`review.md`, most serious findings first. If any reviewer failed, list it in
the report and carry on. Finish the run successfully with the path to the
report.
When workers change code, give each its own worktree, then merge in a final step:
---
name: fan-in-worktrees
description: Do independent tasks on separate branches, then merge the branches.
mode: orchestrated
params:
tasks: {type: string, description: "One task per line"}
base: {type: string, default: main}
---
Do each of these tasks on its own branch, then merge the results.
{{params.tasks}}
## Implement
Create a group `tasks`. For each task, spawn an agent worker in it with its
own worktree, branching from {{params.base}}:
tome worker spawn --name task-<n> --group tasks --worktree --base {{params.base}} --prompt "<the task>"
Tell each worker to commit its work to its branch before it reports done.
Wait for the group.
## Merge
Merge the branch of every worker that succeeded into the current branch, one
at a time, and resolve conflicts as they come. Run the project's tests after
the last merge. Fail the run if a worker failed, or if the tests fail, and
say which.
Queues
Use a queue when workers and the orchestrator need to pass work around while they run, instead of at the end. A queue belongs to the project, so it also works from your shell, and a run can leave messages for the next one.
tome queue push tasks "refactor the parser" # add a message (`-` reads stdin)
tome queue pull tasks --wait 30s # claim one; exit 3 if there is none
tome queue peek tasks # look at what is waiting, claiming nothing
tome queue ack <id> # remove it once handled
tome queue close tasks # no more messages; pulls report `closed` when drained
tome queue ls # the project's queues and their counts
Outside a run, tome queue uses the project of the directory you are in. Inside a run it uses the run’s project.
A message is text up to about 1 MiB. Put bigger data in a file and send its path. A claimed message that is never acked goes back on the queue when the worker that claimed it finishes, so another worker can pick it up.
A worker reports once, and its session closes when it does, so a worker only works through a queue if its prompt tells it to loop. Say it in the prompt: pull, do the item, ack, and pull again until pull reports closed, then report once. pull exits 3 in two cases, and the JSON status tells them apart: empty means nothing is there yet, and closed means nothing more will come. That is why the orchestrator has to close the queue after its last push. A closed queue stays closed for good, so a workflow that pushes more work in a later round needs a new queue name for that round. Queues outlive runs, so pick names that won’t collide with another run’s, such as tasks-<round>. Workflows that don’t need to stop workers can skip close and have them stop on empty instead.
If every item gets its own worker, you don’t need a queue. Put the item in the worker’s prompt.
Limits and tips
- A worker’s summary is the main channel back to the orchestrator. Ask for a useful one.
- There is no limit on how many workers run at once. Each agent worker is a full harness session, so say in the workflow how many your machine can take, for example “never have more than {{params.workers}} workers running”, with a param that defaults to 1 or 2.
- Worker names must be unique in a run. Without
--name, they arew1,w2and so on. A name shows in the worker’s tab title, intome runs showand in its branch, so say in the workflow how to name workers, for example “name each workert<id>-<short title>”.