Omniscio

Cron failure alerts

What happens when a scheduled job fails for good: the Windows toast, the persistent inbox card that stays until you acknowledge, snooze, or the job recovers, the detail pane's Fix with AI / Pause / View actions, and the settings and silence rules that govern them.

On this page

What it is

Omniscio's cron scheduler runs scripts and recipes on a fixed schedule. When one of those jobs fails for the last time — meaning every retry has been exhausted — Omniscio raises a Windows toast and drops a persistent card into the unified inbox so you cannot miss the failure. The card stays in the inbox until you acknowledge it, snooze it (for a duration you pick), or the next scheduled run of that same job succeeds (in which case the card auto-dismisses without you doing anything).

This is a separate feature from cron self-healing. Self-healing is opt-in per job and spawns a Claude session to fix the failure. Failure alerts fire on every job, every time, without opting in — they're the visible breadcrumb that says "this thing broke, you should look at it." If you have self-healing turned on for a job AND that job's heal pipeline is mid-flight, the failure alert is suppressed for the same window so you don't see two cards racing for the same failure (see "Interaction with self-healing" below).

The feature is on by default (cronFailureAlertsEnabled = true). Turn it off entirely at Settings → Notifications → Cron failure alerts, or keep it on but route the OS toast around Focus Mode batching with the nested Always fire cron failure alerts toggle.

Where to find it

The two settings that control this feature live at Settings → Notifications. Everything else about it is something you receive rather than open: a failure reaches you as a Windows toast in the corner of your screen and as a persistent card in Omniscio's unified inbox, filed under the CRON JOBS section (or under the job's own project when it has one). The card is the one you act on — clicking the toast only brings the Omniscio window to the front.

How it behaves

What you see

The Windows toast

When a cron job exhausts its retries and lands in failed, Omniscio fires a system notification using the error sound profile:

  • Title: Cron Job Failed
  • Body: <job name> failed after N attempt(s) (or just <job name> failed for a job with retryCount = 0)

Click the toast and the Omniscio window comes to the foreground — that's the same focus-the-window behavior every other Omniscio notification uses.

The inbox card

A cron.failure_alert row lands in the unified inbox under the CRON JOBS section (a failure whose job has an originating project groups under that project instead). Layout:

  • A red status dot on the left (no other card in that section uses red — this is the visual cue that it's a failure card).
  • The job name on top, truncated if long.
  • The relative timestamp of the most recent failure on the right.
  • Below the name: Failed N time(s) — the cumulative count of failures captured by this card.
  • An optional amber Self-heal pending… chip when a heal is mid-flight for this job.
  • A monospace one-line preview of the most recent stderr / error output.

The whole row is the click target — there is no inline button. Clicking opens the inline detail pane described below (same surface that arrow-key navigation, swipe activation, and archive auto-advance route to). The failure-count display caps at 50+ once it crosses 50 so a runaway job that's failed 4,000 times doesn't blow out the row width.

A checkout-recovery refusal is the one failure whose preview names files. Before a scheduled script starts, Omniscio proves that every file the script imports is intact. When one cannot be proven — edited, unreadable, or missing — that fire is refused and nothing on disk is touched, so the preview reads, for example, Scheduled script dependencies need attention; its files were preserved: scripts/ops/check-queue-dispatcher.mjs (index-differs-from-head). Up to ten files are named with the reason each was refused, and anything beyond that is summarised as +N more; the reason, not only the path, is what tells you whether the file was edited or simply cannot be read. A refusal that has no file to name keeps the plain sentence. The rule the refusal follows — including the states that abstain rather than refuse — is the ops scheduler contract.

Card identity: one card per job

The producer is insert-or-bump, not insert-per-failure. The first failure of <job X> inserts a new pending row. The second failure of the same job updates the existing row's failureCount, refreshes lastErrorText and lastFailureAt, and leaves created_at and firstFailureAt untouched — so the inbox shows one card per failing job, not one per failure event. The total failureCount is the cumulative number of failures the card represents.

Auto-dismiss on success

When the job's next scheduled run succeeds, Omniscio marks the open card approved automatically (per-kind status semantics: approved = acknowledged) and emits a push so the inbox UI removes it without any user click. Auto-dismiss only fires once per row — if the row was already non-pending (you'd already acknowledged or dismissed it), the success branch skips the push. A row you snoozed is still pending (just hidden), so a later success still auto-dismisses it.

Auto-resolve on job deletion

Deleting the job also closes its open failure card. The card carries the job id in target_id, but deleteCronJob is a HARD DELETE FROM cron_jobs WHERE id=? with no FK cascade to cli_pending_actions — so a deleted job used to leave its card behind, and every action on it then resolved a job that no longer existed: Fix it (startManualHeal → getCronJob → null) bounced with "This job no longer exists.", Pause job failed the same way, and only Dismiss worked (with nothing telling you that). An AFTER DELETE ON cron_jobs trigger (trg_cron_jobs_failure_alert_resolve_delete) now marks the deleted job's PENDING cron.failure_alert / cron.failure_heal rows approved on every delete path — the manual delete, the plugin-bridge upsert/unschedule, and Nighty-Tidy unsubscribe — and the CRON_JOB_DELETE handler emits CLI_PENDING_CHANGED so the inbox drops the closed card live. A one-time backfill in the same migration cleared any cards already orphaned before the trigger shipped. Only pending rows are touched, so an already-acknowledged/dismissed/snoozed card is left as-is. Full invariant: the cron-failures inbox contract § I9.

The detail pane

Clicking the inbox card opens an inline detail pane (ApprovalPaneShell) showing the full failure context — the same shell every other CLI Pending approval kind uses. The activation path is the standard cli-pending one: setActiveApproval({ kind: 'cli-pending', id }), dispatched to CronFailureAlertPane by CliPendingApprovalModal based on actionKind === 'cron.failure_alert'. This means click, arrow-key nav (J/K), swipe activation, and archive auto-advance all converge on the same surface.

Footer action buttons, each with its keyboard shortcut advertised on hover — a key-cap in the button's tooltip (from its title + aria-keyshortcuts), NOT an always-visible footer hint line: N Fix · P Pause · O View (plus H Snooze on the header clock, and S Auto-fix whenever the Auto-fix row is shown — the job has loaded and no fix is mid-flight). The bare keys are routed through the global approval-keyboard intercept via a per-pane registry (approval-pane-hotkeys.ts) so they act on the open card — H snoozes this card, not a background session, and the pane's N beats the global New-Session N because the approval intercept runs first (its dispatcher sits ahead of the general binding lookup and consumes the key). Full invariants in the frontend-inbox-row contract § "Specialized cron-pane hotkeys".

The three footer actions are built around actually resolving the failure, not just hiding it (the older set — Snooze / Run Now / View job / Dismiss — only hid it, re-failed it, or navigated away):

  • Fix with AI (N, secondary, with the new-session MessageSquarePlus icon — the same icon Omniscio uses elsewhere for "start a session") — the primary useful action: starts a Claude session to debug and fix the failing job. It keys off N, Omniscio's standard "new session" shortcut, and opens the app's normal Start-session dialog — the same shared ItemStartSessionDialog the digest / scheduled-message cards use, with no Base branch picker — defaulted to the job's own repo. Your own Message box starts empty; the job name and its captured error ride in a collapsed, editable Failed job details box. The session receives your message, then an "About this item" note Omniscio builds from the stored card and job by id (the job's name and schedule, whether it is on or paused, how many runs failed and when, the latest run status, the card and job ids, and the routes to read the card and job, turn the job on or off and run it now, plus links to open both in the app), then the failure details. The dialog IS the confirm — a stray tap can't spawn a session; you still hit "Start session". On a successful launch the pane pauses the job (toggleJob(jobId, false)) so it can't keep re-failing while you work, toasts, and resolves the alert so the inbox advances. Disabled while a heal is already running (payload.healPending) and for job types a heal can't repair (session / one-off). (This replaced the earlier bare confirm-dialog + healNow server-heal spawn — the escalation card still uses that manual-heal path, see cron-self-healing.md § "Manual Fix it". The launched session is a normal session; its note names the job's /toggle and /run routes, but you resume the paused schedule from the app once the fix is confirmed.)
  • Pause job (P, secondary) — useCronStore.toggleJob(jobId, false): stops the job firing (and re-failing / re-alerting) until you deal with it. Toasts Paused "<job>" on success; leaves the card open so you can still Fix / View / Archive.
  • View job (O, secondary) — switches the active project to the Cron Jobs virtual project (CRON_PROJECT_ID) and selects the failing job so you can inspect its schedule, env vars, command, and full run history. Closes the pane, leaves the pending row for later resolution. (Renamed from "View details" — it just opens the job.)

There is no footer Dismiss button — the top-right Archive button (below) already removes the card (a side-effect-free reject-based dismiss), so a separate acknowledge-style Dismiss was redundant.

Snooze is no longer a footer button — the shell header's clock icon already snoozes this row, so the duplicate footer Snooze was removed. H still opens Omniscio's universal snooze picker (snooze-entity CustomEvent → SnoozePalette → inbox_snoozes, the same picker the header clock and every other inbox item use). Because the universal snooze only HIDES the row (it does not resolve the cli_pending action), the pane subscribes to the snooze store and auto-closes once isInboxItemSnoozed('cli-pending-approval', row.id) is true.

Archive — the shell header's top-right corner carries the shared Archive button (the same one every inbox item has; on mobile it's pinned at the fixed top-right corner via MobileInboxArchivePin, exactly where you'd tap the X). It removes the card from the inbox and advances to the next item. Because a cron-failure alert is acknowledge-only (not a consent request), the shared button's default click resolves it through the standard inbox-dismiss chokepoint — a side-effect-free reject (Dismissed from inbox); reject never runs the dispatch, so nothing is spawned or re-fired. This is why an acknowledge-only card gets an Archive button while a true approve/reject card must not (archiving one would silently reject a real request). The header's close ✗ still just closes the pane without removing the row. The sibling cron self-heal escalation card and the nighty-tidy summary card carry the same Archive button for the same reason; the escalation card's Legacy (consent) variant deliberately does not. See mobile-archive-placement-contract.md.

Auto-fix (S) — a first-class row inside the details block (see the pane body below), NOT a footer button and no longer a loose inline "Turn on" link. The row is an Auto-fix label + an honest one-line state + a trailing <ToggleSwitch> bound to the job's healingEnabled. The toggle works both ways — flipping it calls useCronStore.updateJob(jobId, { healingEnabled: next }) (→ IPC.CRON_JOB_UPDATE) in place with an Auto-fix on/off for "<job>" toast, and is disabled while the update is in flight so a rapid double-tap can't double-fire. The S shortcut flips it too, and is registered whenever the row is shown (the job has loaded and no automatic fix is mid-flight, i.e. !payload.healPending; while a fix runs the amber heal-pending banner covers it and the row is hidden).

The pane body leads with a shared ApprovalDetailsBlock — the same bordered, hairline-divided icon + UPPERCASE-label + value block today's other CLI approval cards use — so this card reads consistently with them. The heal-pending banner sits above the block; Auto-fix is now a ROW inside the block (with a trailing on/off toggle — the one interactive control allowed inside the card, per the approval-standard-ui-contract I3 trailing-slot carve-out); the error output + recent runs stay below it. The body shows:

  • Heal-pending banner (only when payload.healPending === true, above the block) — amber callout reading A fix is already running automatically.
  • Job (the block's first row) — the full job name via humanizeCronJobName(payload.jobName): a machine slug (enable-show-all-final-blocks) shows as "Enable Show All Final Blocks" with the raw slug on title= hover, and a name already written with spaces/capitals renders verbatim. This is the ONE place the COMPLETE name always lives — the ApprovalPaneShell header title is truncated, so on a narrow (mobile) viewport it clips to enable-sh…. The same humanized label is reused in the pane title, the Fix-it confirm, the toasts, and the snooze label so a raw slug never shows in one place while the clean name shows in another (mirrors the cron-approval card's Job row — cron-approval-display-contract I7).
  • Failures (the block's amber emphasis row) — a truthful summary derived from the job's REAL recent-run history via summarizeRecentFailures(recentRuns, payload.failureCount): Failed the last N runs in a row · last succeeded <date>, Failed the latest run, or Failed N of the last M runs. It falls back to Failed N time(s) (the card's bump count) only when no run history is available — the bump count alone is misleading because it resets every time the card is dismissed, so a job that failed every week showed "Failed 1 time".
  • Last failed (a block row, only when payload.lastFailureAt is set) — the most-recent-failure time via formatInboxDateTime (full July 11, 1:48 PM). Replaces the old inline · Last failure <date> append the loose summary text used to carry.
  • Auto-fix (a block row with a trailing toggle, shown when the job has loaded and !healPending) — an Auto-fix label, a trailing on/off ToggleSwitch reflecting healingEnabled, and an HONEST one-line state as the value: OFF → Off. Turn on to auto-fix future failures.; ON → On, but it hasn't fixed this failure yet. The ON copy deliberately does NOT use a green ✓ handled check: healing being enabled is not the same as it having fixed anything (heal attempts can be zero, or it can't fix this failure type), and a green "handled" check on a still-failing job reads as a lie. A fix genuinely in progress surfaces via the amber heal-pending banner instead (and this row is hidden then). This one row replaces BOTH the old separate "self-healing status" note and the loose inline "Turn on" control.
  • Last error output — the job's real stderr/error, in a monospace, scroll-on-overflow <pre> block (capped at 1024 chars at the producer side). When the run genuinely captured no output, an honest empty-state (No error output was captured for this run.) replaces the blank box. Historically this box was ALWAYS empty for every card: notifyJobResult built the alert from the pre-update run snapshot (whose errorOutput was still null), not the committed DB row. It now refetches the run (getCronJobRun) before building the alert, so every card carries the real error.
  • Recent runs — the last 5 runs of this job pulled from the database via IPC.CRON_JOB_RUNS, rendered by the shared CronRecentRunsList (cron-run-status.tsx): each row is a status pill (a colored dot + the Title-Case status label, Failed / Success) plus a clean Mon D, h:mm AM run time — the same status style the job's own run-history screen (RunHistoryList) uses. Both this card and the sibling escalation card share the one strip; the old hand-rolled bare-lowercase-word + raw toLocaleString() (seconds-precision) strip, which had been duplicated verbatim across the two panes, is gone. The Failures summary's "last succeeded" date and the run times now route through the shared date-utils formatters (formatInboxDate / a clean short date+time), so the card no longer shows three different date styles at once.

If the payload JSON is malformed (should be impossible — the producer validates against cronFailureAlertPayloadSchema at insert time), the pane falls back to a degraded body: Unable to parse this alert. The pending row may be corrupt.

Why a pane and not a centered dialog?

The pane replaces an earlier centered DialogShell modal that bypassed the inbox approval producer→consumer dispatcher (it called CLI_PENDING_APPROVE directly), which meant Acknowledge did not auto-advance the inbox to the next approval and broke under push races. It also split activation across two surfaces — click opened the dialog, arrow-key nav opened the generic JSON-payload pane — so the experience drifted between input methods. The pane fixes both: one surface, dispatcher routing, auto-advance.

Skip conditions

The producer in cron-engine-service.ts walks the following gates in order. The first matching condition short-circuits — no card, no toast — and the order matters because the heal-pending and per-kind-cap checks happen inside the same db.transaction() that does the insert:

  1. AMC_INSTANCE_ID is set — e2e and Claude-sandbox isolation. Both the toast (in notification-service.fireSystemNotification) and the card insert (in insertOrBumpFailureAlert) bail out instantly when the env var is non-empty. This is why you never see cron-failure alerts in the sandbox or in npm run test:e2e:prod.
  2. cronFailureAlertsEnabled === false — the user opt-out at Settings → Notifications. Suppresses both the toast and the card insert.
  3. run.systemMarkedFailure === true — the startup reconciler flips this column on any unfinished runs after a crash. Those aren't real failures — they're the app falling over mid-flight — so we never alert on them.
  4. An open cron_heal_attempts row exists for this job (status pending or spawned) — the self-healing pipeline owns this failure for now, and surfacing both a heal card and an alert card for the same failure would be noisy. Read inside the same transaction as the insert so a heal that lands a microsecond before this insert is still seen.
  5. Per-kind cap reached — countPendingByKind('cron.failure_alert') >= 5. Five concurrent failed jobs in the queue is already a "your environment is broken" signal; capping at 5 leaves the rest of the global queue free for other approval kinds (cron approvals, automations, recipe runs, etc.). When the cap fires, the producer logs cron.failure_alert per-kind cap (5) reached; skipping insert for job <id> to electron-log and skips the insert.
  6. Daily idempotency — the clientRequestId is <jobId>-failure-<YYYY-MM-DD>. Once you've cleared a card today — acknowledged it (→ approved) or dismissed it with the X key (→ rejected) — a same-day re-failure re-enters the producer because findOpenAlertByTarget only sees pending rows, but that tombstone still occupies the daily key in the (client_request_id, action_kind) UNIQUE index. The producer probes findIdempotent first and skips re-insert with an info log: cron.failure_alert daily key already used for job <id> (status=<status>); skipping re-insert. So a cleared card won't re-fire until tomorrow's date. Snoozing (the header clock icon or H) is NOT a clear — it leaves the row pending and hidden via inbox_snoozes for the duration you pick, so same-day re-failures bump the still-hidden row instead of being idempotency-skipped.

The Windows toast respects two additional gates inside notificationService.fireSystemNotification:

  • Focus Mode — the toast goes through focusModeService.evaluate like every other notification-eligible item. If Focus Mode would suppress (rule not met), the toast is suppressed too. The user can override with the Always fire cron failure alerts setting (cronFailureAlertsAlwaysFire = true), which sets bypassFocusMode: true on the notification call and pierces the gate.
  • Global silence and behavior settings — notificationService.shouldNotify() covers the standard "Silence until" window, "Silence forever" toggle, and the cross-platform muted/idle gates. Same surface every other Omniscio notification uses; nothing special for cron-failure alerts.

The inbox card is independent of Focus Mode and silence — those settings only suppress the toast, not the persistent inbox card. The card always lands as long as the producer-side gates (1–6 above) all pass.

When multiple suppressors apply (e.g. cronFailureAlertsEnabled = false AND a heal is also pending), the earliest in the order wins; later checks never run. So if you flip the master setting off, no toast and no card, regardless of heal state.

A deferred run is not a failure — and never reaches here

A scheduled job can finish a run having done none of its work with nothing broken at all: the box was momentarily busy and refused it, so the job backed off and left the work for its next fire. That run is a deferral, and you get no toast and no card for it.

A script says so by exiting 75 (EX_TEMPFAIL). The engine records the run as skipped — not failed — with the script's own reason kept on the run row, and a skip never reaches the producer above: no card, no toast, no error chime, and no paid self-heal session spent debugging a script that behaved exactly as designed. An in-app executor says the same thing in-band instead of through an exit code.

A checkout that is being rewritten defers the fire before the script is even asked. When the master sync is refreshing the folder a script runs from, or a git operation is parked in it — a hand catch-up merge being the common one — the fire waits for the next one instead of failing: the run is skipped with the reason on the row, and nothing is dispatched. The alternative is what used to happen: every script whose imports lived in that folder failed for the length of the merge, because their files were mid-merge rather than committed. Refreshing and rewriting are bounded, so a checkout left in that state for hours stops deferring and refuses the fire instead, which does reach the machinery above.

A deferral never takes a card away. It is not a recovery either: if the job was already failing, its card stays up through any number of deferred fires and comes down only when a run really succeeds, because a fire that never got as far as asking the script says nothing about the job. A job that only ever defers is caught by its own "has not completed" alarm rather than by a failure count.

75 is the only exit code that means this. Every other non-zero exit, and every deadline kill, is still a real failure and still alerts you — including a 503 from the spawn route that carries no retry-later code, which is a broken route rather than a busy moment. So if you expected a card for a job and did not get one, open its run history: a skipped row names the reason, and anything reading failed went through the normal gates above.

See deferred-run-exit.mjs for the contract, and cron-script-executor.ts for where the exit code is read.

A run whose exit code the app lost is not a failure either

Scripts are started by a small helper inside the app. If that helper restarts while a run is still going, the script keeps running and usually finishes fine, but the app never learns its exit code: the run ends with an exit code of "unknown". That is the app losing the answer, not the job failing, so the engine records the run as skipped with the note "the app's spawn helper restarted while this run was in flight, so its exit code was lost". You get no card, no toast, and no self-heal session for it.

On Windows the hidden window a script runs in also writes down how the script ended, and the app reads that note first. So a run the helper lost track of is usually recorded with its REAL ending: a success as a success, and a script that really failed still as a failure, with its card and its self-heal session. Only when that note cannot be read either is the run recorded skipped as above. If a job keeps showing that skip, the computer itself is overloaded, not the job.

This applies only when the helper itself reports the exit as unknown. A run the scheduler killed on purpose (its timeout, a cancel, or a long machine sleep) keeps its own handling above, and an exit code the helper did report, 0 or anything else, is judged exactly as before. The check lives in cron-script-executor.ts (exitLost, which reads the spawn helper's exitOutcomeKnown flag).

A busy database is not a job failing to start

A scheduled job can fail to get as far as starting because of the app's own storage layer rather than anything about the job: the database was momentarily too busy to answer in time, its worker process restarted under the request, or a run for that job was already in flight. None of those is the job failing, so none of them counts toward the "keeps failing to start" card, and none of them clears a card that a real failure earned.

When a fire is interrupted after the database may already have saved its run, the app now tidies that half-saved run up on the spot instead of leaving it to block the next fire, so a single busy moment cannot turn into two skipped runs. A job whose own settings are broken still cards as before, and a successful fire still clears the card.

A fire that waits for a busy box is late, never lost

Omniscio runs at most four scheduled fires at once. A fire that finds all four busy waits its turn, and once it has waited the whole overdue window (15 minutes) it runs anyway rather than being dropped — a job that is late is a nuisance, a job that never happened is a hole. Either way it ends as a normal run with its next-run time moved on, so the job stays visible and keeps its place. The alternative is what an earlier build did: a fire parked on that wait could create no run row at all and leave the job's next-run time frozen, which is a job silently stopping while still looking scheduled.

A failed WAKE is carded only once the outage is sustained

A wake schedule — a session job that wakes an existing session rather than starting a fresh one — is the one failure that does not card on its first miss. A wake whose fire was refused records its run failed like any other, and that run row is what the outage is measured from, but the card waits for BOTH three consecutive failed fires AND thirty minutes dark.

Either half alone would still page you for a blip: three misses fit inside one bad minute on a five-minute heartbeat, and a single miss of an hourly schedule sits dark for a full hour. Together they mean consecutive misses spread over real time.

Below that bar the wake is transient — no toast, no card, no chime; the failure is still on the run row and in the log, so nothing is lost, only the interruption is withheld. At or above it the ordinary cron-failure card appears once, under the same identity as any other job, so a recovery still auto-dismisses it and later fires of the same outage simply bump that one card without re-chiming. Once the wake is sustained, the crew that owns the woken session is told as well — its lead, or the crew's own board when the lead is the session that went dark — and told again only every six hours while the outage lasts, so a long outage does not become a chorus.

A non-wake job is unaffected: it still cards on its very first failure. And a wake whose run history cannot be read stays silent — a missed card is the cheaper mistake.

See wake-outage.ts for the predicate and cron-failure-alert.ts for where it gates the card.

Last verified 2026-10-27