Switchboard — Complete Reference

Switchboard routes AI work to a fleet of GPU hosts. You ask for an artifact and declare what the work is for; the broker decides where it runs, loads it if needed, and hands back a URL to call. This document covers every supported feature and configuration.

Overview

Three moving parts:

• Broker — the control plane. Owns scheduling, the catalog, and a data-plane proxy. Reachable on the LAN at http://192.168.1.33:8080.
• Agents — one per GPU host. Poll desired state, launch/stop services, report health + nvidia-smi.
• Client — switchboard-client, a pip package wrapping the broker API.

The unit of work is a ticket. You POST a request, the broker queues it, schedules a host, loads the model if it isn't running, and marks the ticket ready with a proxy URL. All your data-plane calls go through that URL — the real backend address stays hidden.

Quick start

import switchboard_client as sb, httpx

url = sb.request("qwen3-30b-a3b-q4",
                 environment="production", urgency="interactive")

r = httpx.post(f"{url}/v1/chat/completions", json={
    "messages": [{"role": "user", "content": "Hello"}]})
print(r.json()["choices"][0]["message"]["content"])

Install & configuration

pip install switchboard-client --extra-index-url https://switchboard.wtray.com/wheels/

Environment variables read by the client:

Artifacts

An artifact is the concrete thing you ask for — a model id with its quantisation (qwen3-vl-32b-q4 vs qwen3-vl-32b-q8 are different artifacts), a ComfyUI workflow, or a service name (kokoro, faster-whisper, pyannote).

A run-profile is a tested way to run one artifact on a specific host/GPU with specific settings. One artifact has many profiles, and profiles of the same artifact can differ — notably in context length. Enumerate the catalog at runtime with sb.fleet() rather than hard-coding.

Workload class

Every request and session declares three independent axes. They drive scheduling, not model selection. environment and urgency are mandatory.

Rank is lexicographic: rank = environment×100 + urgency. A running service is committed at the highest rank actively using it and can only be evicted by a strictly higher rank — or unloaded once nothing needs it.

Requirements

Satisfy-or-queue hints. The broker only considers profiles that meet them, and returns 400 if no profile anywhere can. Unknown keys are ignored (forward-compatible).

url = sb.request("qwen3-vl-32b-q4", environment="production", urgency="interactive",
                 requirements={"min_context": 32768})

Placement

Constrain how a profile is laid out on the host. Pass placement= (and/or gpu= for a specific index).

gpu=1 is equivalent to placement="gpu1" and matches only single-GPU variants on that index.

Sessions

A session declares the full set of artifacts a feature will use, so the broker can plan placement across the whole fleet at once, pre-warm them, and hold them for the session's lifetime — instead of scheduling each call blind to what comes next.

REQS = [
  {"artifact":"qwen3-30b-a3b-q4",      "environment":"production","urgency":"interactive","keep_alive":120},
  {"artifact":"qwen3-vl-32b-q4",       "environment":"production","urgency":"interactive","keep_alive":120},
  {"artifact":"qwen3-embedding-4b-q8", "environment":"production","urgency":"interactive","keep_alive":120,
   "placement":"cpu"},   # frees a GPU for the VLM
]
with sb.session(REQS, priority=5.0) as s:
    text  = s.request("qwen3-30b-a3b-q4")
    vlm   = s.request("qwen3-vl-32b-q4")
    embed = s.request("qwen3-embedding-4b-q8", placement="cpu")
    # run the whole loop in here — all three stay warm

Each entry accepts artifact, environment, urgency, keep_alive (all required per entry), plus optional requirements, placement, gpu_index. requires may also be a plain list of artifact-id strings.

• s.request(artifact, …) — resolve against the warm plan; defaults to the class the artifact was declared with. Accepts the same overrides as sb.request.
• s.plan() — the broker's current placement plan (per-artifact host options and whether each is hot / planned / unavailable).
• s.close() — release (also on with-block exit).

Session semantics

Verify a session is actually in use: sb.status()["sessions"] — if that is 0 while your app is running, you are making isolated request() calls and the broker is scheduling each one blind to the next.

Artifact catalog

Live as of this document's generation. Authoritative source is sb.fleet().

Co-residency & VRAM capacity

This is the most important operational section on this page.

Hardware budgets

The large-model conflict

The big text and vision models each require a gpu0, and there are only two gpu0s in the fleet. They cannot co-reside on the same host:

The working three-model layout

A text + vision + embedding app can run fully resident — but only with embeddings pushed to CPU. This layout is verified working:

ai-150 / gpu0  ->  qwen3-30b-a3b-q4      # 23,000 of 30,000 MiB
ai-150 / cpu   ->  qwen3-embedding-4b-q8 # system RAM, no VRAM
ai-151 / gpu0  ->  qwen3-vl-32b-q4       # 22,000 of 22,000 MiB

Without placement="cpu" on the embeddings, the broker may park them on ai-151/gpu0, which blocks the VLM entirely and forces a swap on every loop turn.

Client API — core

request()

sb.request(artifact, *, environment, urgency, keep_alive=30.0, requirements=None,
           priority=0.0, placement=None, gpu=None, requester=None,
           broker=None, timeout_sec=300.0) -> str

Blocks until a healthy endpoint exists; returns the proxy URL. Raises RequestTimeout on timeout, SwitchboardError on failure/cancel, UpgradeRequired if the client is too old.

session() / Session

sb.session(requires, *, priority=0.0, requester=None, broker=None) -> Session
sb.subscribe(...)   # alias — a session IS a subscription

Introspection

Cancellation

sb.cancel(ticket, *, broker=None)         # cancel a QUEUED ticket
sb.cancel_ticket(ticket, *, broker=None)  # release a DISPATCHED ticket

Client API — speech

Text to speech

sb.tts_say(text, *, character_voice="af_bella", language="en", speed=1.0,
           chunk_min_chars=50, chunk_max_chars=300, inter_chunk_gap_ms=80,
           pronunciation_overrides=None, output_path=None, bundle=None,
           environment="production", urgency="realtime", artifact=None,
           priority=0.0, timeout_sec=300.0) -> bytes | str

sb.tts_say_stream(text, ...)  # generator of (chunk_text, wav_bytes)

Output is 24 kHz mono float32 RIFF/WAVE. tts_say buffers into one WAV (bundling on by default); tts_say_stream yields per chunk so you can start playback immediately. Chunks are silence-trimmed, with inter_chunk_gap_ms of breath prepended after the first.

Speech to text

sb.transcribe(audio, *, language="en", initial_prompt=None, model=None,
              word_timestamps=True, hotwords=None, no_speech_threshold=None,
              low_confidence_threshold=None, suppress_hallucinations=True,
              hallucination_blocklist=None, ...) -> dict

audio accepts bytes, a path, or a file-like object.

async with sb.transcribe_stream(language="en", hotwords=["Karrthûn"],
                                silence_ms=700, min_speech_ms=100,
                                max_utterance_ms=30000,
                                partial_interval_ms=300) as s:
    await s.send(pcm_int16_16k_mono)
    async for ev in s.events(): ...

Event types: ready, vad_start, partial, final, vad_stop, error. Send raw int16 16 kHz mono PCM; any frame size works.

TTS markup

tts_say* accept inline SSML-style markup:

Supported language codes: en, en-us, en-gb, es, fr, hi, it, pt, pt-br, ja, zh. pronunciation_overrides={"gaol":"jail"} applies a dict globally. sb.split_sentences(text) exposes the sentence splitter.

Client API — admin

How placement is decided

When a queued request has no running worker, the broker picks an owning host by sorting candidates on this key — lowest wins:

(evict, warmth, strength, in_flight, host)

Among already-running replicas, pick_worker_for sorts by (worker_load, strength, host, service).

Aging. A queued request's weight is priority + age_factor × wait_seconds (age_factor = 1.0), so waiting work climbs and cannot starve forever. Standing (session) demand does not age — it is steady pull at its priority, not a starvation climb.

Swapping. A host only swaps to a new config when the aged weight of the work the new config serves exceeds what's currently served plus a swap_penalty of 30.

Warmth, TTLs & eviction

Autoscaling

The broker horizontally scales replicable services under sustained load. On by default; opt out per-artifact with "autoscale": false.

Conservative by design: only a real, persistent backlog on free hardware triggers a replica, and autoscaling never evicts live work.

Broker HTTP API

Base: http://192.168.1.33:8080. Use the client unless you have a reason not to.

Data plane

Sessions

Introspection & control

Clients advertise their version via X-Switchboard-Client; below the floor the broker answers 426.

Manifest schema

Fleet catalog: ~/switchboard/manifests/artifacts.json — the source of truth, projected into per-host configs. Host budgets: ~/switchboard/manifests/hosts/<host>.json. The broker hot-reloads on mtime change; no restart needed.

Artifact

Run-profile

Operations

Troubleshooting

Requests time out repeatedly

Almost always model thrash: two models that can't co-reside are being requested alternately, and each turn pays a 15–60 s swap against a client timeout that's too short.

• Check sb.status()["sessions"] — if 0, the app isn't using sessions.
• Verify the working set can co-reside (capacity tables). Push small models to placement="cpu".
• Raise timeout_sec above cold-load time — a 32B VLM takes 53 s+; 60 s is too tight.

"Upstream unreachable" from a ready ticket

The proxy URL outlived the model (15 min TTL vs 30 s keep_alive). Re-request, and hold the model with a session or a longer keep_alive.

Everything queues and nothing dispatches

Check for a stray pin (sb.status() → pinned) and the global enable flag. A pinned host serves only its pinned config.

A model won't load

Check the per-service log on the host share. After 2 consecutive failures to reach healthy, the circuit breaker benches it for 10 minutes. Common causes: OOM (VRAM under-declared), missing model file, bad split.

Silently smaller context than expected

You landed on a profile with a smaller tested window. Pass requirements={"min_context": N} to make it fail loudly instead.

Change log

2026-07-31 (later 3)

• Big models were landing on the weaker GPU. With both hosts idle, a consumer's vision model consistently ran on the RTX 3090 while the RTX 5090 held a smaller text model. Cause: placement is greedy and was ordered by arrival — the text model was requested first, correctly won the 5090, and by the time the larger vision model was requested the 5090 was taken, so evict pushed it to the 3090. With two large models and two gpu0s one must use the 3090; the defect was that arrival order, not size, decided which. Fix: co-queued work is assigned hardest-to-place first (largest VRAM footprint), and GPU claims are now tracked during the pass so the function stops double-assigning every large model to the single strongest host. Verified live: VLM on ai-150, text on ai-151.
  Caveat: this only applies when requests are queued together — i.e. inside a session. Sequential blocking request() calls put one item in the queue at a time, so the broker never sees both and cannot co-plan.

2026-07-31 (later 2)

• Dead hosts reported stale services as "running". /status and the dashboard built their per-service view from the last status the agent POSTed, which freezes when an agent dies — so an offline host kept showing its last-known services as running: true, making it look like the fleet was still using that box. The scheduler was never affected (it reads current_config, cleared on death, and host_can_serve() rejects dead agents). running/healthy are now forced false for a dead agent and the entry is flagged stale.
  Signature: agent_alive: false next to a non-empty running list. Confirm real placement by issuing a request and seeing which host loads it.

2026-07-31 (later)

• Scheduler — warmth inversion fixed. warmth was host-level (0 if h.current_config else 1), so any host running anything out-ranked an idle host — even for unrelated work on a different GPU. Effect: an idle RTX 5090 lost to an RTX 3090 that merely had a small audio service on its second GPU. Now request-specific: warmth is 0 only if the host already runs/desires a service that serves this request. Idle and unrelated-busy now tie, so strength decides and the better GPU wins. Verified live: with ai-151 holding kokoro on gpu1 and ai-150 idle, the 32B VLM now lands on ai-150 (5090). 105/105 tests pass.

2026-07-31

• Scheduler: added a capability tie-break (strength) to host and replica selection, ranking by tested context then VRAM. Sits below evict/warmth, so anti-thrash behaviour is unchanged. 105/105 tests pass.
• ai-151 qwen3-30b-a3b-q4: context 16384 → 32768. Measured 15,390 MiB of 24,576 at the new size — fully on GPU, 9.2 GB free.
• ai-151 qwen3-embedding-4b-q8: declared VRAM 5,500 → 7,500 MiB. A clean measurement showed real cost 6,632 MiB — it was under-declared, risking overcommit.
• Fleet-wide VRAM audit of all 13 GPU profiles; results in manifests/backups/vram-audit-2026-07-31.json. Several profiles are over-declared (safe direction; costs packing density only) and were deliberately left alone.

2026-07-30

• ai-151 qwen3-vl-32b-q4: context 8192 → 32768. The 8192 cap was a stale over-declaration, not a hardware limit — measured 19,294 MiB of 24,576 at 32768.
• switchboard-client 2.0.2: fixed transcribe_stream, tts_say*, transcribe and Session.request, which requested legacy "stt"/ "tts" capability tokens and omitted the now-mandatory environment/ urgency, so they raised before making any HTTP call.