jobd

jobd

jobd is a job broker with an MCP server that lets AI agents submit, route and babysit jobs on a few machines of your own. Like task-spooler or pueue, but across all your machines on Tailscale or a LAN, routing each job to a worker with the VRAM free, with no cloud, Kubernetes or Slurm.

# from any machine on your tailnet:
job submit --project myproj --gpu --vram-required 16 --wait -- python train.py
# → routed to whichever worker has ≥16 GB VRAM free, streamed back to your terminal

VRAM routing tracks one GPU per host: the worker reports free memory on GPU index 0 only.

Why it exists

• VRAM-fit routing. Each job goes to a worker whose live free VRAM, RAM, CPUs, capability tags and arch/OS fit it, instead of you guessing which box is free.
• Preemption with a checkpoint window. A higher-priority job can preempt a running one: SIGTERM, a grace window for the workload to save into its JOBD_CHECKPOINT_DIR, then SIGKILL. A preempted job is not re-run; to resume, submit a new job pointed at the checkpoint (docs/preemption.md).
• Survives sessions. Jobs live in the broker, not your shell. Submit, close the laptop, check back tomorrow.
• Agent-native. An MCP server lets an AI agent submit, monitor and babysit jobs as tool calls (does it help?).
• Yours. One broker process on a machine you own. No accounts, no telemetry, no per-GPU-hour billing. Only the optional self-update scripts go online (runbook).

Why not just use…?

Install

pip install jobd               # broker + CLI
pip install "jobd[mcp]"        # adds the MCP server
pip install "jobd[worker]"     # adds the worker daemon (jobd-worker)

Requires Python ≥ 3.11. scripts/install-worker.sh sets up a worker under ~/jobd-worker with its own venv and a generated config.

Quickstart (single host)

# 1. start the broker (binds 127.0.0.1:8765 by default)
JOBD_ALLOW_NO_AUTH=1 jobd          # no-auth is fine for a loopback-only broker

# 2. in another shell, install + start a worker pointed at it
pip install "jobd[worker]"
JOBD_URL=http://127.0.0.1:8765 JOBD_WORKER_HOST=local jobd-worker

# 3. submit a job and wait for it
job submit --project demo --wait -- echo hello
job list
job logs <id>

For a multi-host deployment (Docker broker, systemd workers, Tailscale binding, shared auth token), see docs/security.md and the templates in docker-compose.yml and scripts/. Adding a worker to a running fleet is one command:

job fleet add user@newbox      # ssh in, install pinned to the broker's version,
                               # wire systemd units + the self-update timer,
                               # verify it registers. `job fleet status` shows drift.

Day-2 operations (health, draining a worker, upgrades, token rotation, backups) are in docs/runbook.md.

How it works

The broker (FastAPI + SQLite) holds the queue, runs the matcher and resolves per-project priorities. Workers, one per host, poll it, so nothing connects in to a worker: each heartbeats its live capacity, claims jobs it can run, streams logs back and honors preemption. The argv you submit runs as-is (shell=False); job submit --stdin and the MCP jobd_submit tool are the exceptions and run a command string as bash -c <string>. Design notes: docs/ARCHITECTURE.md.

CLI

job submit -p PROJ [options] -- CMD...       # queue a job
job list [--state STATE] [--project P]       # queue + recent jobs
job status ID | A<id> [--watch]              # one job, or an array's aggregate
job logs ID [-n BYTES]                       # tail captured output
job wait ID                                  # block until terminal
job cancel ID  /  job preempt ID             # stop a job
job adopt --pid N -p PROJ [--gpu]            # track a process started outside jobd (Linux)
job workers                                  # fleet snapshot + health
job projects list | set NAME PRI | nudge NAME DELTA
job audit [--project P] [--since 24h]        # event history

Job arrays and sweeps, --explain dry runs, job adopt, and a pueue/task-spooler command map: docs/cli.md.

MCP / agent integration

jobd-mcp exposes the queue as nine tools: jobd_submit, jobd_status, jobd_logs, jobd_list, jobd_cancel, jobd_preempt, jobd_events, jobd_workers, jobd_worker_delete. docs/agent-cookbook.md walks through babysitting, preemption, sweeps, and asking why a job won't schedule.

For Claude Code:

claude mcp add jobd \
  --env JOBD_URL=http://127.0.0.1:8765 \
  --env JOBD_API_TOKEN=<your-token> \
  -- jobd-mcp

Any other MCP client:

{
  "mcpServers": {
    "jobd": {
      "command": "jobd-mcp",
      "env": {
        "JOBD_URL": "http://127.0.0.1:8765",
        "JOBD_API_TOKEN": "<your-token>"
      }
    }
  }
}

JOBD_API_TOKEN must match the broker's token, or every call returns 401. examples/claude-code-hooks/ has optional hooks that nudge (or hard-block) an agent toward submitting heavy commands through jobd.

Does it help?

Equally accurate, and faster and cheaper: on 38 harder questions about a live broker, Claude answered 38 right in all 3 runs with jobd-mcp and 37 with the job CLI, a pre-registered tie (McNemar p = 1).

With the server the median answer cost 0.20 times as much (95% CI 0.15 to 0.28) and took 0.16 times as long (0.13 to 0.20), and it was cheaper and faster on all 38 questions. These were secondary endpoints, measured on an unreleased commit that adds a finished_at window to jobd_list. On an earlier, easier set both setups got 123 of 123.

Method, controls and limitations: benchmarks/agent_qa.

Supported platforms

CI runs on Ubuntu with no GPU runner (VRAM paths are tested against mocks), and without a systemd user instance the worker still runs jobs but drops its memory caps, process reaping and preemption guarantees.

Configuration

• Config files (projects.yaml, profiles.yaml, classifier.yaml) are all optional and read from JOBD_CONFIG_DIR; see docs/projects-yaml.md.
• Environment variables: every JOBD_* variable is in docs/configuration.md.
• Concurrency (several jobs per worker): docs/concurrency.md. Retention of logs and job records: docs/runbook.md.

Security

Run the broker on loopback or a Tailscale tailnet, never a public interface. Three stacked controls:

1. Bind: JOBD_HOST is 127.0.0.1 by default; set it to a Tailscale address (100.64.0.0/10), never 0.0.0.0. The broker does not enforce this.
2. Source IP: requests from outside loopback and 100.64.0.0/10 get 403 (JOBD_DISABLE_TAILNET_ACL=1 turns this off).
3. Token: every host sets JOBD_API_TOKEN; JOBD_ALLOW_NO_AUTH=1 is for a loopback-only broker, and on any other bind exposes unauthenticated code execution to your tailnet.

/livez, /readyz and /metrics are exempt from both the source-IP check and the token; /metrics shows every worker's hostname and version, so don't port-forward the broker. Threat model and token rotation: docs/security.md.

License

MIT; see LICENSE.