Skip to main content

Warm sessions

The problem​

Every chat message used to pay a full docker run --rm ... pi --mode json --print -- <prompt> cold start: a new container (new namespaces, cgroups, overlay mount) and a fresh Node + pi CLI boot, all before the model even saw the prompt. That's on top of a turn that already chains multiple LLM calls: the outer agent deciding to use a tool, the tool call itself, and the outer agent composing the final reply.

The fix: one warm process per conversation​

src/backend/chat/pi_runtime.py keeps one long-lived pi --mode rpc process alive per conversation, reused across messages in that conversation, instead of spawning a fresh process per message. That process is a Warm Session. The product's Session (one conversation plus its work state, adrs/0010.md) is a different thing; CONTEXT.md keeps the two names apart.

Where the process runs depends on the backend:

  • Docker (./dev.sh, PI_RUNTIME_BACKEND=docker). Each Warm Session is its own container, pi-session-<user>-<conversation>-…, started with docker run --rm as a sibling of the backend. A user with several Sessions can hold several containers.
  • Kubernetes (PI_RUNTIME_BACKEND=kubernetes). Each user has one User Pod. Every Warm Session is a pi process inside that pod, started by the in-pod daemon, not a new pod. The pod is pre-warmed on login (prewarm_user_pod) so the first reply does not wait for scheduling. Idle reaping deletes the pod only after every conversation on it has gone idle. The reasoning is Chat session lifecycle option 2, and adrs/0009.md relies on one policy file per pod.

RPC mode is a JSONL protocol over stdin/stdout designed for exactly this: a {"type": "prompt", "message": "..."} command sent in, the same AgentEvent stream as --mode json streamed back out (message_update/text_delta, tool_execution_start/end, ...), terminated by {"type": "agent_settled"} — "the full session-level run settled, nothing will auto-continue," the RPC equivalent of the old mode's process exit.

This was verified empirically against the real pi binary before being wired into Django — not just from docs, which turned out imprecise in places (e.g. the exact session-resumption flag). Confirmed live: the real event sequence for one turn, that agent_settled (not agent_end) is the correct one-turn-done signal, that a second prompt sent to the same still-running process correctly sees the first prompt's context, and that closing stdin (EOF) makes pi exit gracefully and — via --rm — the container along with it. That last point is what the idle-reaping and shutdown paths below lean on.

Session lifecycle​

  • No database state for the live process. A warm session's only source of truth is an in-memory registry (_sessions: dict[conversation_id, _WarmSession]) inside the Django worker process. It can't meaningfully survive a process restart anyway — a subprocess.Popen's piped stdin/stdout can't be reattached after the fact — so it doesn't try to. This also means the dev server's autoreload (on every file save) self-cleans: when the old process dies, the OS closes its end of each session's stdin pipe, pi sees EOF and exits, and on Docker --rm removes the container. The pi agent's own resumable log is PiSession (renamed from AgentSession, adrs/0010.md section 7), one per conversation. Keying the warm process by conversation means several Sessions stay live at once: several containers on Docker, several processes in the one User Pod on Kubernetes.
  • A background thread per session drains its stdout into a queue.Queue. A blocking readline() has no per-call timeout; queue.Queue.get(timeout=...) does, which is what makes a per-turn timeout possible without killing the whole warm process over one slow turn. A second thread drains stderr into a bounded tail buffer, purely so that pipe never fills up and blocks pi itself during long idle stretches between messages.
  • A per-conversation lock serializes turns. Two messages in the same conversation never race on the same process — simpler and safer than RPC mode's own steer/followUp queuing semantics, and it means a prompt command is only ever sent while the session is genuinely idle.
  • The concurrency semaphore now bounds concurrent turns, not concurrent containers. An idle warm session holds no more than its container; the semaphore is only acquired for the duration of an active turn, so many users can stay warm at once without competing for the same slot.
  • Idle reaping and a hard cap. A background thread (chat/apps.py, started once from AppConfig.ready(), guarded against Django's autoreload double-import) evicts sessions idle past PI_CONTAINER_IDLE_SECONDS; PI_MAX_WARM_SESSIONS LRU-evicts the oldest session if a new one would exceed the cap, regardless of idle time — since the resource cost of a warm session is count × the existing --memory=512m container cap, not just "however long everyone's been away."

The trade-off​

Reusing a container across messages means whatever a compromised or misbehaving turn did to it persists for up to PI_CONTAINER_IDLE_SECONDS, not just one message — see Agent Sandboxing. The resource caps and capability drops on the container are unchanged either way; what changed is how long one container lives before those caps get a fresh start.