💭 Thought Streaming & the Thought Bubble

Prompt Wizard 3.2.0 adds a three-state toggle for what you see while a generation runs: Off, 💭 Live thoughts (the model’s real stream), or 🌧️ Code rain (a visualization). Sibling page: Generation Pipeline & Timings.

The toggle

• Radiogroup in Settings → While generating (moved from the prompt bar in 3.2.3): Off, Live thoughts, Code rain.
• Persisted in localStorage under pwiz.thoughts; default Off.
• Full keyboard support (roving tabindex, Arrow/Home/End), targets ≥ 24 px.
• prefers-reduced-motion: the caret stops blinking, collapse is instant, and code rain degrades to a calm static frame with a slow opacity shimmer.

What omp actually streams (verified against omp 18.2.6)

• omp -p --mode=text (the previous mode) buffers the entire reply and prints it only after the message settles — it is not incremental.
• omp -p --mode=json prints one JSON event per line as the model generates. message_update events carry the provider delta in assistantMessageEvent (type: "thinking_delta" / "text_delta" with a delta string); agent_end carries the final messages.
• Thinking deltas depend on the model; when a model reasons without streaming (or doesn’t reason at all), there are no thinking_delta events. Text deltas are effectively always present.

Server changes

• ompText always spawns omp with --mode=json and reads stdout line-by-line, whether or not anyone is listening — so every text run parses the event log and rebuilds the reply the same way. Deltas go to the onStream callback immediately when a request opted in; nothing streamed is used to build the reply.
• The final reply is rebuilt by textModeStdout() to the exact bytes --mode=text would have printed (sanitizeText(text block) + "\n" for every text block of the last assistant message). Since 3.2.2 a failed reply is detected from the message itself, never from the exit code — see 3.2.2 corrections. The 256 KB output cap is applied to that rebuilt reply — never to the event stream, so a long reasoning trace cannot fail a reply that text mode would have accepted. Fence stripping, JSON parsing, local repair, rerun, timings and logs behave exactly as in 3.1.0.
• New SSE event thought (server/events.ts createThoughtRelay):

Field	Meaning
request_id	The generation request the delta belongs to
kind	thinking or text
chunk	Bounded text delta (≤ 1500 chars per event; oversized deltas are split)
ts	Server send timestamp

• Coalescing: while the run is live, at most one thought event leaves per 100 ms tick, carrying at most 1500 chars of a single kind (oldest first; deltas of another kind wait for the next tick, so no burst and no mixed-kind event). close() — always called, success or error — flushes the remainder at once in the same bounded slices, so every thought precedes the request’s done/error phase. The pending backlog is capped (24 KB); if a run outgrows it, the oldest text is dropped and the next event opens with a visible …[earlier thoughts trimmed]… marker — a truncation is never presented as model output.
• Scoping: live thoughts are opt-in per tab. Anonymous visitors share public, while optional SSO supplies signed-in identities. Each page mints a random UUID and opens /api/events?tab=<uuid>; only a generation naming thoughtsTab gets a relay to the matching identity + tab. D-479b fixes both callers so only Live creates a thought callback/subscription and sends the UUID; Off and Code rain request no real chunks. The existing relay identity/tab isolation remains unchanged.
• The relay is always closed (finally-style: success and error paths), so no timers leak.

The bubble

• Live: an auto-scrolling monospace pane near the stepper. Thinking is tinted and italic, output is tinted in the pwiz primary; a soft mask fades the top; a blinking caret marks the head. Scrolling up pauses auto-scroll. On completion it collapses smoothly and a Show thoughts button reopens the kept text until the next run. On error the bubble unmounts completely.
• Honesty rule: if no thoughts arrived by the end of a run, the bubble says “The model didn’t stream its reasoning for this run — only progress phases were available.” Nothing is ever fabricated and labeled live.
• Code rain: pure-client canvas (no model involvement) of rushing code tokens in pwiz violet/orange, 60 fps capable, device-pixel-ratio capped at 2, requestAnimationFrame cancelled on unmount, and labeled “Visualization — not model output”.
• Neither mode blocks, slows, or restyles the result view; when generation finishes, the results replace the input view and the bubble hands off cleanly.

3.2.2 corrections

Released 2026-09-26 (D-164) after an independent readback found 3.2.1 only partial.

Area	3.2.1 behavior	3.2.2 behavior
Failed reply, JSON request	omp -p --mode=json exits 0 on a provider error; the empty reply went to JSON parse → repair → a wasted rerun → misleading 502 Invalid JSON response from omp	502 model provider error: <reason> after one omp call; no repair, no rerun
Failed reply, plain request	silent 200 {content:"", timing:null}	same 502 as above
Provider errorMessage	never logged	logged once: [omp] reply failed request=… model=… attempt=… exit=…: <reason>
Thought bubble	pointer covered the title label	pointer sits on the top edge; bubble taller (live 10–28 rem, rain 22 rem)
Code rain	sparse	denser and calmer (12 px columns, 2 drops per column, slower fall, glyphs change per row)

• Failure rule: a reply failed when the final assistant message has stopReason error or aborted, or any errorMessage, whatever the exit code (replyFailure() in server/omp-print-events.ts, checked in ompText before any parse).
• Sanitized reason: terminal controls stripped, whitespace collapsed, bearer/basic credentials, key=/token:-style values, provider key prefixes and any 40+ char opaque run redacted, capped at 300 chars. With no errorMessage the reason is request aborted / request error.
• Tests: a contract test drives a fake omp that exits 0 with stopReason error (429) and aborted, JSON and plain, asserting the 502, exactly one omp invocation, the log line and no secret in response or log. The thought-relay tests take an injectable ticker instead of fake timers, so the suite runs green under the service’s preserved Bun 1.2.5.
• Reduced motion: unchanged contract — code rain paints one static (now already dense) frame with a gentle opacity shimmer; the caret stops blinking.

3.2.3 corrections

Released 2026-09-26 (D-166), from the user’s drawer complaint:

Change	Behavior
Toggle placement	The Off / Live thoughts / Code rain control moved from under the prompt bar into the Settings drawer as a normal drawer-styled row (“While generating”). Same pwiz.thoughts key, same Off default, same keyboard behaviour.
Debug tools	Closing the admin & wizard tools drawer no longer wipes the debugMode preference; the drawer’s toggle in Settings is the only writer.
Verified	Headless CDP on the live vhost: no control under the prompt bar, toggle inside the drawer persists selection, wizard tools drawer opens with all 16 wizards, preference survives drawer close, 0 console errors.

Limits

• Reasoning deltas exist only for models that stream their reasoning through omp; the toggle can’t create them.
• The final flush on close() is exempt from the 100 ms tick (it must finish before done), so the events-per-second claim holds only while a run is live.
• Opening the page before the events module loads, or a browser without crypto.randomUUID(), means Live thoughts silently fall back to phases-only for that tab; nothing else changes.
• thought events are not replayed on SSE reconnect (unlike the latest phase); a reconnect mid-run simply rejoins the live stream.
• The mermaid post-processing request runs under its own request id, sends no thoughtsTab, and is not displayed in the bubble.
• Successful text/image AI calls are still stored in llm_logs (admin-only) when thought display is Off. The bounded in-memory relay is not a persisted raw reasoning transcript; see api > Logging, retention and public storage.