Crash recovery (auto-resume sessions after a crash or restart) (part 2)
Part 2 of the Crash recovery page: the crash-loop circuit breaker that quarantines a session, the silent-death watchdog that alerts you the moment the app is killed, the states that are never auto-resumed, why revivals trickle, and where each piece lives in the code.
On this page
What it is
This is part 2 of the Crash recovery (auto-resume sessions after a crash or restart) page. It covers what happens when recovery itself goes wrong — the breaker that gives up on a session which keeps crashing the app, the watchdog that tells you the instant the app is killed, the states that are deliberately never resumed, and why sessions come back one at a time — plus where all of it lives in the code.
Where to find it
The surfaces are the ones on the parent page: the sidebar, the startup attention toast and the Bulk session manager modal. Nothing described here has a screen of its own — the alert it describes arrives as an email and, when a phone is paired, as a page on that phone.
How it behaves
When Omniscio gives up on a session (the crash-loop circuit breaker)
There's a safety limit on auto-resume itself. If a session's revival keeps crashing the app — it gets resumed on boot, the app dies before that session ever finishes a turn, and the next boot tries it again — Omniscio stops trying after 3 consecutive failed boots (within a ~15-minute window). Instead of resuming it a fourth time it quarantines the session: parks it in the orange recovery-failed state in your inbox with a message — "Auto-resume gave up after N attempts — this session kept crashing on resume. Open it and send a message to resume manually." — and lets the app boot cleanly.
This exists because one badly-behaved session (a huge history that locks the renderer, a resume that stalls the main thread until the OS kills the app) used to be able to brick every launch: each boot re-resumed it, each resume re-crashed the app, forever, with no backoff. The breaker turns that fatal loop into a one-session graceful degrade — the rest of your sessions come back normally and the poison one waits for you. The count only climbs while a session keeps getting resumed and re-crashing without ever finishing a turn. Two things clear it: it self-heals after ~15 minutes with no further failed resume, and a clean quit clears it immediately — a normal shutdown (even the interrupted quit → relaunch → quit → relaunch dance) is proof the app wasn't crash-looping, so those sessions simply come back instead of being mistaken for a poison one. So a transient hiccup, or a burst of quick relaunches, never strands a session — only a genuinely poison one that keeps crashing the app trips it.
The app-crasher is the only recovery-failed state held for a manual restart. Every other "recovery-failed" session — one that gave up for a non-crashing reason (all your logins were out of capacity, an aborted reply hit its daily retry cap, a stuck turn exhausted its retries, a transient API error — a brief 5xx server hiccup — outlasted its ~7-minute silent-retry window) — is no longer a dead end: a background safety net keeps re-driving it forever on a gently-growing interval (seconds, then minutes, capped around half an hour) until it comes back, so nothing is ever silently abandoned. Only the session that keeps crashing the app is excluded from that automatic retry (re-driving it would just re-brick the launch) — it waits in the inbox for you to restart it by hand, or for a one-click Restart stuck sessions (bulk restart). The full re-arm rules live in the session self-heal contract.
A boot that dies before it gets to your sessions never costs a strike either. The three strikes are meant to catch a session whose own revival kills the app — so a strike only counts once Omniscio has actually started bringing that session back. Revivals are deliberately gentle (one at a time, seconds apart), so if the app dies early in startup for some unrelated reason, most of the queue was never even reached. Those sessions get their strike handed back on the next launch and come back normally. Before this, three quick failed startups in a row could quarantine every session that had been running — sessions that were perfectly healthy and had never been tried once. (2026-09-06: 48 of them.)
If any session doesn't get restarted, you get told. After every launch Omniscio checks whether it started back up everything it meant to. If some sessions were never retried, it raises a single inbox card — "Some sessions have not been resumed yet" — with a Resume them now button. Clicking it asks you to confirm first (each resume uses a turn of that session's budget) and shows the live count, since the sweep may have already brought some back on its own. If everything came back, no card appears.
A missing project folder never costs a strike. If a session's folder isn't there when recovery reaches it — the drive is unplugged, the folder was moved or renamed — a resume can only fail, so Omniscio doesn't attempt it and doesn't count it: the session is skipped without burning any of its 3 quarantine attempts, and the red "Project folder is missing" banner tells you why (see project folder missing). Recovery picks the session up again by itself once the folder exists — the background safety net re-checks roughly every 90 seconds, and the next app start re-checks too — so reconnecting the drive is all it takes.
Instant alert when the app itself is killed (the silent-death watchdog)
Session recovery only runs at the next launch — so if the whole app is killed without a clean shutdown (a stray terminal-window close, an external taskkill, anything that terminates the process tree instantly), nothing above can tell you until you happen to relaunch. Since 2026-07-29 a tiny detached silent-death watchdog closes that gap: the app arms it at startup, it watches the app's 5-second heartbeat file from outside, and if the heartbeat goes silent for ~60 seconds AND the app's process is genuinely gone, it immediately pages your phone (if one is paired) and sends the crash email — except that it honours your crash-email settings exactly as the in-app crash report does, so if you have turned crash emails off it sends none — typically within 60–75 seconds of the death, even though the app itself is dead. The note the watchdog carries about your paired phone is sealed to your own Windows account, so the phone's push credential is never readable in the app's log folder — and if it cannot be sealed, the phone leg stands down and says so instead of writing it in the clear. It is alert-only by design: it never restarts the app for you. A frozen-but-alive app is deliberately NOT treated as dead (no false pages during a system-wide freeze), a clean quit makes the watchdog exit silently — and an in-app restart is told apart from a clean quit by the marker the app leaves behind, so a restart whose launcher never brings the app back pages you as an abandoned restart instead of passing as a clean shutdown — and the post-mortem email at your next launch notes when the instant alert already fired so you're never double-alarmed. Kill switch for power users: the AMC_DISABLE_SILENT_DEATH_WATCHDOG=1 environment variable. The alert works in an installed build as well as one run from source, so a silent death is paged within about a minute wherever Omniscio is running. Invariants: silent-death-watchdog contract.
When Omniscio closes ITSELF, the report now says so
A small but important honesty fix (2026-11-04). Omniscio sometimes ends its own process on purpose: a startup failure that cannot be recovered, a database that is newer than this build, a cancelled unlock, a wedge during shutdown. Those exits are deliberate, but they used to leave no record of themselves — so the next launch could only report that the app "died without a clean exit", which reads exactly like something else having killed it. If you ever went looking for the cause of one of those, you were looking for a culprit that did not exist.
Every one of those exits now writes one line to Omniscio's own termination record first, naming
the exact decision that ended it — for example hard-exit.fatal-startup-timer when a startup
error left the error dialog unanswered for ten seconds, or
hard-exit.shutdown-finalization-tail-ceiling when shutdown overran its own deadline. The next
launch's report carries that name, so a self-inflicted exit reads as one instead of as a mystery
kill. This changes nothing about how the app behaves on the way out; it only makes the app tell
the truth about itself afterwards.
What does NOT get auto-resumed
- Sessions you explicitly archived or paused — those terminal states are respected. Omniscio won't unarchive on your behalf.
- Sessions that errored on their own (a turn failure, a dead CLI process, a non-transient API error such as a context-window 400) — these are NOT crash victims (
crash_reconciled=0), so they stayerror(red) across restart, exactly as they ended, no matter how recently they erred. (A transient API 5xx that outlasts its silent retries is the exception — it lands in recovery-failed (auto-recovering, the never-fail family above), not dead red.) The crash path (listCrashRecoverableSessions) only revives genuine app-crash victims; a session preserved in the graceful-shutdown snapshot — including one force-closed intoerrorby a busy shutdown (above) — still resumes on the next launch regardless of how long the app was down. - Sessions whose CLI process exited cleanly (status
endedwithended_atpopulated) — they finished. There's nothing to resume. - Worktrees stuck in intermediate states — these are reconciled separately by
reconcileWorktreeStatesat startup (anymerging/activeworktrees that didn't finish get markedfailed), but no session re-launch comes out of that path. - Sessions Omniscio has quarantined after repeated resume crashes — see the crash-loop circuit breaker above. They wait in
recovery_failed(red — a failure — in your inbox) for a manual message instead of being resumed a fourth time. - Everything, when a second copy of Omniscio is already running on the same data — if, at startup, Omniscio detects that another live instance already owns this database (for example a test/sandbox build that got pointed at your real data, or a second launch that didn't fully take over), it skips auto-resume entirely rather than re-parking and re-spawning sessions the other copy is already running. Your sessions are left exactly as they are — open one and send a message to continue it. This is a safety backstop against two copies fighting over the same sessions; a normal single-instance restart is unaffected (the previous copy is gone, so nothing is "already running").
Why the trickle
The design exists because the alternative was bad. Before the queue (incident 2026-05-24), Omniscio tried to revive every recoverable session in parallel, before the window was created. With 13 sessions queued, the user's machine froze for 30+ seconds at every launch — no paint, no input, while every CLI child started spawning simultaneously and every keep-alive panel cold-mounted its full conversation history at the same time. The fix splits recovery into two phases: a fast DB-only phase that paints the sidebar immediately, and a slow spawn phase that runs after the window is interactive, draining one session at a time with a gap between each that follows your restart-spacing setting (Settings → the session restart spacing), stretching only while your agents are measurably slow (to twice the setting) or memory is running out (to 20 seconds). The result: you never see a frozen launch, no matter how many sessions are queued. A later refinement (2026-06-02) makes that gap weight-aware: a session revived by replaying its full conversation history — the heaviest kind of restart — gets a larger gap held to a calmer CPU level, so a cluster of those can't stack up and peg the machine. That stacking was the cause of a ~29-second main-thread stall observed during a 20-session restart recovery. Since 2026-10-17 (T-318) a merely BUSY machine no longer slows the drain: busy CPU or disk readings are produced by the drain itself, and on the 2026-10-16 restart they held every gap at 10 seconds — 178 sessions came back at 5 per minute while the agents' own tool calls stayed fast. The gap now backs off only on the measured agent slowdown, so a 3-second setting brings about 20 sessions back per minute.
Related
The parent page covers what recovery looks like from your side — which sessions come back, the trickle, and how to stop it. Dead logins and what to do when everything is out of capacity are on the account pool page; a session the app refuses to revive can be restarted by hand from bulk stop and restart; and a session whose project folder has gone missing is covered in project folder missing.
Last verified 2026-10-06