Omniscio

Main-process heartbeat (silent-crash forensic trail)

A log file recording a continuous five-second tape of Omniscio's main-process health — event-loop lag, memory, live resource counts and the last operation that ran — so that a window which simply disappears with no crash dialog and no error report can still be explained after the fact.

On this page

What it is

A dedicated log file — heartbeat.log — that captures a continuous 5-second-resolution tape of main-process health while Omniscio is running: event-loop lag P50/P99, RSS/heap memory, active-resource counts, and the last IPC channel that ran before each tick. It exists for the case where the Omniscio window simply disappears with no crash dialog, no Sentry error, no Windows Event Log entry, and no Crashpad dump — silent main-process death. With a heartbeat tape on disk, the next-launch post-mortem can answer:

  1. Was main healthy in the 0–5 s before death? (capacity question — was lag climbing? was RSS bloating?)
  2. What was main DOING when it died? (causation question — lastOp names the IPC handler that was still running, or (no-tracked-op-in-flight) when none was)
  3. Was this an internal hang or an external kill? (presence vs. absence of [CLEAN-EXIT] / [GOODBYE] on shutdown; correlation with [POST-MORTEM] markers on next launch)

Two sibling files hold the last 10 ticks (the ring tail) captured at the moment of an event, so the dump survives even if rotation dropped the live heartbeat.log before you could read it. heartbeat-snapshot.log is written for an event the app survived — a dead renderer or GPU helper, a recovered blank screen, or an uncaught exception / unhandled rejection (whose handlers deliberately do not exit). heartbeat-final.log is written ONLY from the process.on('exit') hook, and only when that exit was not marked expected by an intentional-quit hook — it means "the process was going away and nobody planned it". A graceful quit or an update relaunch writes neither file. Both carry the same content shape (an AMC HEARTBEAT FINAL DUMP header, then the last ticks).

The file lives next to the other diagnostics logs:

  • Windows: %APPDATA%/omniscio/logs/heartbeat.log
  • macOS: ~/Library/Application Support/omniscio/logs/heartbeat.log
  • Linux: ~/.config/omniscio/logs/heartbeat.log

Like interaction.log (and unlike startup.log), heartbeat.log is a continuous tape across launches — the same file grows every time you open the app, capped at 1 MB with rotation to .1 → … → .5. This is intentional: a silent crash that only repeats every few days needs a tape that spans those days.

Where to find it

heartbeat.log sits in the app's machine-level logs folder, beside the other liveness tapes and outside any one account's folder — on Windows under %APPDATA%\omniscio\logs\. The per-account session log (main.log, startup.log, interaction.log) lives one level down, under profiles\<your account>\logs\. There is no in-app viewer; heartbeat.log is read after the fact, most often as the evidence attached to a crash report.

How it behaves

How to use it

After a silent app death

  1. Don't reopen Omniscio immediately. The post-mortem analyser runs on the NEXT launch and reads the on-disk tape — that's good — but if you have other diagnostics to capture first (Task Manager, dxdiag, Reliability Monitor), do those before relaunching, since relaunch is also what writes the [POST-MORTEM] markers.

  2. Relaunch Omniscio normally. During init, the heartbeat:

    • Walks back through heartbeat.log looking for the prior launch's [HEARTBEAT-START] banner.
    • Checks whether [CLEAN-EXIT] or [GOODBYE] appears in the window between that banner and the new one.
    • If neither appears, writes [POST-MORTEM] lines into the new launch's section flagging the silent death, the last known tick (with its lastOp), and the operating system’s own record of the exit: Windows Event Log entries from ±60 s around that tick, or on macOS launchd’s exit line for the dead process (its signal or exit code, and who sent it).
  3. Open Settings → Diagnostics → Main-Process Heartbeat → Reveal Heartbeat Log. The file manager opens with heartbeat.log highlighted.

  4. Read in this order:

    • First, heartbeat-final.log in the same folder, if it exists — that's the emergency ring-buffer dump written when the process reached an exit nobody planned. If it isn't there, open heartbeat-snapshot.log instead: it holds the same dump for an event the app survived (an uncaught exception, a render/child-process-gone, or a blank-screen recovery) and is the fastest path to a stack trace or to the surrounding ticks for a silent renderer death.
    • Next, the tail of heartbeat.log: scan up from the bottom for the most recent [POST-MORTEM] lines. They sit between the prior and current launch banners. If a renderGone=N or childGone=N count appears, the analyser also surfaces the last [RENDER-GONE] / [CHILD-GONE] line — that names the suspect (renderer vs. GPU/utility).
    • Then the last few [tick] / [STALL] / [BOOT-STALL] lines BEFORE the silent boundary — that's the 5–25 s window before death.
    • Finally, the Windows Event Log scrape lines (only on Windows): the analyser shells out to PowerShell Get-WinEvent for Application-Error / Application-Hang / Windows-Error-Reporting events near the last known tick. If something shows up here, the kill came from outside Omniscio (Windows OOM, security software, driver crash, etc.). On macOS the same os-event= line carries launchd’s record instead, e.g. exited due to SIGBUS | sent by exc handler[96840] — a crash signal from the exception handler means Omniscio itself crashed (Crashpad caught it), while SIGKILL | sent by <process>[pid] names the process that killed it.
    • On macOS the crash dump itself does not stay on disk. Crashpad writes it to Crashpad/pending, and at the next launch Sentry's minidump loader uploads it and deletes it — before Omniscio's own crash sweep looks — so the code location of a crash lives only in Sentry. Open follow-up: keep a local copy first.
  5. Attach heartbeat.log (and heartbeat-snapshot.log and/or heartbeat-final.log if present) to the bug report.

  6. If the window froze and never came back, also attach main-freeze-stack.json from the same folder. A separate watcher thread writes it after about 8 seconds of frozen heartbeat. It names the code the main thread was stuck in, or says the thread was inside a system call that could not be interrupted. It gains a resumedAtIso field if the app recovered. Settings → Diagnostics → Export diagnostic bundle already includes it, under logs/; the plain Export Logs zip does not.

    When no JavaScript stack can be taken, the file still says what is known. stack is null and stackAbsentReason names why the capture came up empty — pause-timeout (the inspector pause was never answered within the deadline), evaluate-refused (the runtime answered with an error), inspector-unreachable (the inspector session could not be opened at all), no-app-frames (a stack came back holding only Node-internal frames), or capture-threw (the capture itself raised). Beside it sits evidence, which is ALWAYS an object and never a bare null:

    • evidence.nativeOp — the traced synchronous fs/db/proc call holding the main thread at that instant (inFlight.label + heldMs), plus which tracer families were armed. A family that was not armed reports armed: false, so its silence is never read as "nothing was happening".
    • evidence.osThread — the Windows thread state across the sample window (stateName, waitReasonName, cpuMsGained, gapMs, stuckThreads), or { available: false, reason } naming why the reader was dark (not-windows-x64, koffi-path-missing, stderr-is-nul-device, read-failed: …, capture-threw).
    • evidence.beatDuringSample — measured ACROSS the thread sample, not before it. true means the heartbeat advanced while the sample was being taken, so the main thread ran again before the record was written: the pause was refused rather than the thread being dead, and the record describes the moment it was still stuck.

    The record also carries inFlight — what the app was carrying at the last heartbeat tick: count, maxAgeMs, the oldest handlers by name (up to IN_FLIGHT_PEEK_LIMIT, oldest first), asOfIso (the tick that list came from) and stale/staleMs. A list from a tick more than one 5 s interval old is stale: true: on a frozen main thread it always is, because the tick that produced it is the last one before the freeze. Read count and maxAgeMs off the tick, never off the clock at capture time.

What you'll see in the file

═══════════════════════════════════════════════════════════════
 Omniscio MAIN-PROCESS HEARTBEAT
 …self-describing header (purpose, Version, Electron, Mode, PID,
  tick interval, stall threshold)…
═══════════════════════════════════════════════════════════════

── Launch resumed at 2026-05-22T18:00:00.123Z (PID 24680) ──
2026-05-22T18:00:00.124Z [HEARTBEAT-START] pid=24680 version=1.2.3 mode=development
2026-05-22T18:00:05.130Z [tick] uptime=5s lagP50=2ms lagP99=4ms rss=180MB heap=72MB lastOp="(no-tracked-op-in-flight)" resourceCount=18
2026-05-22T18:00:10.131Z [tick] uptime=10s lagP50=2ms lagP99=3ms rss=185MB heap=74MB lastOp="session:list" resourceCount=22
2026-05-22T18:00:15.137Z [STALL] uptime=15s lagP50=180ms lagP99=1240ms rss=195MB heap=78MB lastOp="recipe:run" resourceCount=24 resources={Timeout:12,TCPWRAP:6,FSReqCallback:3,Immediate:1,TTYWRAP:2}
… more ticks …
2026-05-22T18:02:01.211Z [CLEAN-EXIT] pid=24680 ticksWritten=24 uptime=121s
2026-05-22T18:02:01.213Z [GOODBYE] pid=24680 code=0 ticksWritten=24

Line legend:

  • [HEARTBEAT-START] — banner stamped at every init. Marks a new launch.
  • [tick] — normal 5-second sample. P99 lag below the stall threshold (1000 ms).
  • [STALL] — same shape as [tick] but P99 ≥ 1000 ms AFTER the first IPC arrived. Suggests a CPU-bound operation in main (heavy synchronous SQLite, JSON parse on a huge buffer, NDJSON parse spike).
  • [BOOT-STALL] — same as STALL but before any IPC ran. Suggests an early-startup hang (migration, native module load, sync I/O during bootstrap).
  • [STALL-CAUGHT] — emitted by a separate 200 ms fast-tick watchdog whenever the gap between two consecutive watchdog fires is ≥ 2 000 ms (the loop was just unblocked after a multi-second pause). Line shape: durationMs=<actual gap> lastOp="<channel>" inFlightCount=<n> inFlight=[…]. lastOp here means "an IPC handler was still running inside this frozen window" — a handler that finished before the window opened is never named, and when nothing was in flight the field reads (no-tracked-op-in-flight) and no lastOpLead is emitted. The inFlight array lists every IPC that was mid-await at the moment of unblock, sorted DESC by ageMs (oldest first — most likely the culprit) and capped at 10 entries. Companion to [STALL]: the 5 s [STALL] says "lagP99 was high somewhere in this window," [STALL-CAUGHT] says "the loop was pinned for exactly N ms and these IPCs were waiting." A 2–5 s stall that ends mid-window can be missed by [STALL] (dilutes below the p99 threshold) but always shows up here. See postmortem.
  • blockedOp= on that line names what held the thread. A single op is its own label; agg:<op>x<count> is a run of spans of ONE op summed to the gap; burst:<op>+<op>+<op>x<count> is one contiguous run of DIFFERENT ops that together covered at least half the freeze — it names the three biggest and then how many spans the run held, e.g. five different DB reads back to back. A parenthesised value ((no-op-covered-gap), (none-instrumented), (off-cpu-no-blocker)) is the tracker saying it could not name one, and each of those carries its own next move.
  • [stall-stack] — emitted right AFTER a [STALL-CAUGHT] when the V8 CPU profiler was armed and captured samples. Pair the two by stallId=<epochMs> — the freeze's unblock timestamp, carried by every record of one freeze ([STALL-CAUGHT], [stall-stack], [STALL-GC], and the [GC-PAUSE] warn that lands in main.log instead of heartbeat.log). Pairing by durationMs was the old join and it does not work: a storm puts dozens of stalls in the same 2–4 s band, and [GC-PAUSE] never carried it at all. Line shape: stallId=<epochMs> durationMs=<gap> top: <fn> (<file>:<line>) <n>% | … ← caller: <fn> (<file>:<line>). top: is the top-5 frames by SELF time — on a multi-second synchronous block the blocking frame dominates every other sample, so the first entry IS the blocking op (frequently a native leaf like existsSync (native) 81%). ← caller: is the answer to "which of OUR code asked for that": the profiler walks UP the call tree from the hottest leaf to the first APP frame. Node built-in frames are deliberately SKIPPED — they carry a source url (node:fs, node:internal/…) but are only Node's own wrapper around the native leaf, so returning one would merely restate what top: already said. Before that skip existed, 22 of 26 attributed callers on a live box read existsSync (node:fs:278) / statSync (node:fs:1733) and named nothing useful. A node: frame still appears as a last-resort fallback when the path holds no app frame at all; ← caller: is omitted entirely for a pure native / off-CPU wait, and the line reads top: (no samples — likely native/off-CPU wait) when the window had no samples. NOTE — this is a DIFFERENT field from blockedOp= on the [STALL-CAUGHT] line, which is fed by trackMainOp breadcrumbs — the DB layer, plus (since 2026-08-29) EVERY periodic background service, named service:<label>, because createPeriodicTask wraps each tick's synchronous slice. So a freeze inside a service tick now reads e.g. blockedOp=service:agent-status-board rather than (none-instrumented); that fallback still appears for a synchronous block in code no breadcrumb covers, and for an ASYNC tick, which correctly registers no block at all. The profiler's answer cannot be folded into this field because the capture is asynchronous and resolves after that line is already written. Read the two lines together. Kill switch AMC_DISABLE_MAIN_STALL_PROFILER=1; armed from the first tick and kept armed by default (AMC_STALL_PROFILER_ALWAYS_ARMED, 0 = armed only under box load — see the T-423 follow-up below). Arm/disarm/rotate leave their own [stall-profiler] transition lines on this tape (armed intervalUs=… maxWindowMs=… / disarmed windowMs=… / rotated windowMs=… samplesDiscarded=… / arm-failed reason=… / arm-deferred reason=…), so "was the instrument running during the 13:20Z storm?" is answerable after the fact. A [stall-profiler] arm-failed line means the profiler is DARK, and it can be dark while armed reads true to nobody but itself — V8 refuses setSamplingInterval while a profile is live, so if a command's reply is lost under load the profiler keeps sampling while the module believes it is off. The arm path is therefore its own breaker: three consecutive failures go quiet for 60 s (logging once per episode, never once per 200 ms tick, and the tape keeps the token every time), and the next arm stops the suspect profiler before it re-enables one. This is the same trap the renderer twin hit on 2026-09-25 (bug 9acddc97), where the retry loop itself measured ~250 log lines a minute and 46.5% of a core. And starting the profiler is itself expensive on a large process: Profiler.start runs V8's StartProfiling on the main thread, which at a ~370 MB heap measured 2–4 s (Help Desk 6069a960, 2026-09-29). Because that exceeds the 2 s stall threshold the watchdog recorded the instrument's own start-up as an app freeze → the freeze asked for a capture → the capture stopped the profile and re-armed → arming paid it again; [stall-profiler] armed went from ~20/hour to 500–980/hour while the UI lagged ~2 s per click. So the profiler now times its own start and disarms itself for the rest of the run when it exceeds AMC_STALL_PROFILER_MAX_ARM_COST_MS (1000 ms — set ABOVE the measured start cost, whose 28 recorded defeats run 500–807 ms, so a healthy run does not latch the instrument off; and set well below the 2 s stall threshold — not exactly 2× under it, because the watchdog's gap also carries the arm's own pre-start round trips — so the profiler's own work cannot be counted as a stall. It was 500 ms until 2026-10-04, which sat below the very costs it was bounding and took the instrument out on essentially every run; re-derive it from the armCostMs= now printed on the armed line). The tape says [stall-profiler] self-defeated armCostMs=…, and a later [stall-stack] from that process reads unavailable=self-defeated. The self-defeat latch catches an expensive start only AFTER paying it, so the arm is now PREDICTED before it is paid and a calm tick no longer disarms instantly (2026-10-09). One gate at the top of armStallProfiler covers every caller — the 200 ms tick, the capture's finally re-arm, and the rotation re-arm: prospectiveArmDecline() predicts the start cost from the live heap against AMC_STALL_PROFILER_ARM_BUDGET_MS (default 950 ms — 50 ms under the 1000 ms latch and 1050 ms under the 2 s stall threshold) and declines when a start would reach it. The prediction is fitted to the instrument's own measured arms, not to one reading. 221 successful main-process arms cost a median 730 ms / p90 920 ms of main-isolate block (and 28 recorded defeats under a 500 ms bound ran 500–807 ms). The earlier 10.8 ms/MB zero-intercept slope came from ONE Help Desk reading — 2–4 s at a ~370 MB heap on a box at 75–90% CPU with 16 live sessions — and predicted 4.3–14.2 s at the heaps this process actually runs at, 6–15× above what the same instrument measured. That is why it declined every arm between 2026-10-15 and 2026-10-16 (15 arm-deferred reason=heap-too-large lines, heap 401–1310 MB, zero armed lines, while /perf/status read busyStacks=0 and worstStallStack.unavailable=disarmed). The fit now anchors the measured median at 400 MB and the measured p90 at 2000 MB (2050 MB was observed live on this box), i.e. 0.11875 ms/MB with a 682.5 ms intercept — deliberately conservative at the top of the band. The ceiling stays DERIVED from the budget and the fit (MAX_ARM_HEAP_MB ≈ 2253 MB, was ~46 MB), so a heap whose start is predicted to reach 950 ms is still refused; the trade-off is that a real arm now blocks the main thread for the measured ~0.7–0.92 s, which the 1000 ms latch already permits and which is the cost of having an instrument at all. A decline prints ONE [stall-profiler] arm-deferred reason=heap-too-large heapMB=… predictedStartCostMs=… budgetMs=… line per episode (never one per 200 ms tick) and is re-decided on every attempt, so the profiler re-arms by itself once the heap drops — it is NOT a latch. A capture also ends with a AMC_STALL_PROFILER_POST_CAPTURE_HOLD_MS calm window (default 10 s, line arm-deferred reason=post-capture-calm remainingMs=… holdMs=…) before the re-arm, so the capture's own aftermath is not reopened by the very next arm. Separately, a calm tick now HOLDS the arm while the last load evidence is inside AMC_STALL_PROFILER_CALM_HOLD_MS (five minutes, mirrored from the renderer twin's ARM_HOLD_MS) — a DEBOUNCE, not a latch: re-read against the last evidence every tick, so a genuinely calm box still disarms by itself, and a backward clock jump holds (failing toward the instrument being on). All three knobs default ON and route through resolveEnvNumber; 0 is the off switch on each and restores the prior behaviour. The kill switch must be set BEFORE the app launches: setting AMC_DISABLE_MAIN_STALL_PROFILER=1 and then using the in-app restart does NOT pick it up — app.relaunch inherits the old process environment, so it takes a full quit and relaunch. See src/main/services/diagnostics/main-stall-profiler.ts.
  • T-423 (2026-10-17) — the self-defeat above is now a bounded back-off, not a run-long latch. The self-defeated tape line now reads self-defeated armCostMs=… boundMs=… backing-off backoffMs=… until=<iso> heapMB=… cheapHeapMB=…, and the [STALL-CAUGHT] line of a dark stall adds profilerDarkUntilIso=. The back-off is bounded, never a run-long latch (T-423, 2026-10-17). It used to latch off for the life of the process, and on 2026-10-17 it did so on 2 of 2 boots about two minutes in (1237 ms at a ~1.65 GB heap, 1359 ms at ~1.6–1.9 GB), right on the post-capture re-arm after the first [busy-stack], so the 20–25 s freezes later that morning carried no stack. Now a too-costly arm keeps the profiler off for 30 min, doubling on each consecutive costly arm up to a 4 h cap, and a cheap arm resets the ladder ([stall-profiler] backoff-reset armCostMs=… previousBackoffMs=…). The costly arm also teaches a heap ceiling: the start cost on this box tracks the main heap roughly in proportion (318 ms at ~600 MB, 744 ms at ~1 GB, 1237 ms at ~1.65 GB on 2026-10-17), and the heap saw-tooths ~0.8 → 2.4 GB every ~25 s, so cheapHeapMB = heapAtCostlyArm × min(AMC_STALL_PROFILER_ARM_BUDGET_MS, AMC_STALL_PROFILER_MAX_ARM_COST_MS) / armCostMs (≈1260 MB for the 1237 ms arm at ~1.64 GB). Every later arm waits for the heap to fall under that ceiling (arm-deferred reason=heap-above-learned-ceiling heapMB=… ceilingMB=… maxWaitMs=300000), so it lands on a GC trough rather than a peak, but never for more than 5 min — only a paid arm restarts that clock, so a load verdict flapping calm/busy cannot hold it open — past that it arms anyway, so a heap that never troughs cannot keep the profiler dark. A cheap arm at or above the ceiling disproves it and clears it. Both clocks treat a backward clock jump as elapsed, like the module's other gates, so it ends a back-off or a wait rather than lengthening it. The kill switch AMC_DISABLE_MAIN_STALL_PROFILER=1 remains the only way to keep it off for the whole run. Current state: GET /diag/heartbeat (the tape tail) — the latest [stall-profiler] line is armed …, self-defeated … backing-off … until=<iso>, or a deferral; a dark [STALL-CAUGHT] names profilerDarkUntilIso.
  • T-423 follow-up (2026-10-18): the profiler stays armed, and a capture never restarts it from scratch. Measured live after the 21:48 ET boot on 2026-10-17: 17 of 18 freezes of 2 s or more carried no stack. The cause was not the heap. Every capture, rotation and busy capture stopped the only running profile, so the re-arm paid V8's whole cold StartProfiling on a box at 100% CPU. That cost 1,634 ms at a 2,080 MB heap and 1,702 ms at a 1,031 MB heap; each of those starts sat inside the 2.0 s and 2.1 s freezes it caused, tripped the 1,000 ms bound, and backed the profiler off for 30 and then 60 minutes. A probe on Node 24 showed CPU contention drives the size: a cold start cost 2,274 ms calm and 11,763 ms with the cores oversubscribed. The fix is a bridged hand-off. V8's inspector shares one CPU profiler between the Profiler.start profile and any console.profile profile, and pays the cold start only when the live-profile count goes from 0 to 1. So a capture, rotation or busy capture now opens a short bridge profile (inspector.console.profile('amc-stall-bridge')), stops and reads the window, starts the next window, and closes the bridge. Measured: 0–1 ms per hand-off under 2x CPU oversubscription, against 846 ms for a cold start in the same run. The bridge is always closed in a finally. If it cannot be closed, bridging is switched off for the run and the next arm sends Profiler.disable before Profiler.enable, because a leftover bridge makes setSamplingInterval refuse. If inspector.console.profile is missing or throws, the capture falls back to the old stop-then-cold-arm path, so the instrument is never left dark. A bridged restart that still costs over AMC_STALL_PROFILER_MAX_ARM_COST_MS is treated as a cold-arm defeat. Always armed: AMC_STALL_PROFILER_ALWAYS_ARMED defaults ON, so a calm tick arms the profiler and keeps it armed (rotating every 20 min like a loaded tick). Only the kill switch or a back-off disarms it, so the one cold start lands at boot (348–372 ms live). 0 restores the load-gated arm/disarm. Its own cost is on the tape: [stall-stack] carries costMs= (stop plus restart, before top: so perf-status still reads the frames), rotated carries costMs= and bridged=yes|no, and every 10 minutes armed the tape gets [stall-profiler] overhead windowMs=… profilerMainMs=… handoffs=… coldArms=… samplesRead=… boundMs=…. profilerMainMs is the measured main-thread time of every profiler command in the window that completed. A round trip that hit the 5 s inspector deadline is not counted, because its wall time is waiting rather than profiler work. Above 1% of the window it backs off through the same 30 min → 4 h ladder with its own reason (self-defeated reason=overhead …, stallProfilerDefeat().reason = overhead-<ms>ms) and leaves the learned heap ceiling alone. A cheap restart does not reset that ladder; only an overhead window that closes under its bound does. A dark [stall-stack] during either kind of back-off reads profiler dark — backing off: it exceeded its own cost bound (start cost or overhead …), and the self-defeated line names which one. The sampler thread's own CPU runs off the main thread and cannot be separated in-process. Measured offline on Node 24 it was 343 ms of process CPU per 10 s idle armed (about 3.4% of one core); it cannot freeze the UI, and the live check reads it from process CPU.
  • T-224 (2026-11-10) — every silent instrument on a [STALL-CAUGHT] line now says WHY it is silent. Three signals used to go absent for several unrelated reasons each, and the row carried no token saying which, so an absence read exactly like an instrument that looked and found nothing. On the four worst freezes of the 01:31–02:31 window (13.3–16.7 s, all blockedOp=(no-op-covered-gap)) none of the three could be interpreted. Now:
    • stackCapture= is written from the SAME predicate the capture itself asks (stallProfilerBlindReason(), i.e. the capture's own blind-reason check), not from the profiler's armed flag. armed therefore means "a capture will really run for this freeze"; a profiler that is armed but kill-switched, breaking its overhead bound, or backing off reads disarmed, which is the state whose [stall-stack] line carries unavailable=<reason>. Before this, such a stall stamped armed and the capture then produced no stack — a claim and its evidence disagreeing on the same tape.
    • syncFsGap= names why syncFsInGapMs is absent: none (the ops tracker AND the sync-fs tracer were both live, and no fs.*Sync span completed inside the gap — a positive claim, which is what the off-CPU freezes of 2026-11-10 actually were: cpuBusyPct 1–2, stack top (idle) 99–100%), tracker-off (AMC_DISABLE_MAIN_OP_TRACK, checked first because recordCompletedMainOp returns on it before it ever looks at the label), or tracer-off (AMC_DISABLE_SYNC_FS_TRACE). Absent whenever syncFsInGapMs is on the line.
    • culprits= names why no [stall-cpu-culprits] line accompanies this stall: not-starved (a BLOCKED/MIXED freeze — the CPU was burned on the thread, so a process list names nobody) or box-calm (STARVED with neither CPU nor disk pressure on the box). The sampler's own (not-sampled: …) line only exists when the caller reaches it; the watchdog's gate declines first on most stalls, so this field covers the silence that line cannot. Absent when the gate said sample, where the culprit line speaks for itself. Both new fields are omitted on a stall whose instruments reported, so a healthy line and every pre-existing line is byte-identical. See src/main/services/diagnostics/heartbeat-stall-tape.ts and heartbeat-stall-classify.ts.
  • Help Desk ddd34b4a (2026-11-11) — the line's own CLASS now reaches the card. Every field the class needs was already here, and the reader that raises the single-freeze card turned none of it into an answer. worstStallClassOf in perf-status/stall-stack-parse.ts reads the same longest-freeze line worstStallOp picks and publishes two new fields on the main-thread block: worstStallClass = sync-filesystem when blockedIn=syscall is present or syncFsInGapMs covers at least BLOCKED_OP_MIN_COVERAGE_FRACTION (the same half-gap bound isBlockedInSyscall uses), memory-faults when isMemoryFaultStorm holds (the predicate is reused, never re-derived), and null when neither was measured; and worstStallSyncFsOp, the topSyncFsOp token, for a developer build. On the reporter's tape 13 of 29 caught freezes were sync-fs dominated (86.0 s of 89.0 s) and the worst two were one blocking openSync on C: with cpuBusyPct=0 — a freeze the card described with a memory remedy. null is "not measured", never "the box was calm".
  • [CLEAN-EXIT] — written at the top of gracefulShutdown. Disk is healthy, shutdown reached the heartbeat-stop path.
  • [GOODBYE] — written from a process.on('exit') hook. Confirms the runtime actually unwound, not just that gracefulShutdown started.
  • [FATAL] — written from uncaughtException / unhandledRejection handlers. Includes lastOp and a 6-line stack snippet.
  • [RENDER-GONE] — written from the render-process-gone handler in src/main/index.ts when details.reason !== 'clean-exit'. Stamps the current lastOp plus reason and exitCode, then dumps the ring buffer into heartbeat-snapshot.log (the app survived — only the renderer died) so the surrounding ticks are preserved even if rotation later trims the live tape. Tells you whether a silent death was the renderer (the BrowserWindow's web content) dying out from under main.
  • [CHILD-GONE] — same mechanism as [RENDER-GONE] but from the app-level child-process-gone handler. Fires for GPU process / utility process / sandbox helper exits. Includes type, reason, exitCode, and (when present) name / service. A GPU-process death is a common cause of "the window disappeared" with no main-process exception.
  • [POST-MORTEM] — written on the NEXT launch when the analyser sees a silent death pattern. Lines include previousLaunchEndedSilently=true, lastKnownTick=…, os-event=…. When [RENDER-GONE] or [CHILD-GONE] lines fell inside the prior-launch window, the analyser also surfaces renderGone=N / childGone=N counts and the last line of each, so the post-mortem names a suspect even after the live tape is gone.

resources={Timeout:12,TCPWRAP:6,…} shows the libuv active-resource histogram (counts of pending timers, sockets, fs requests, etc.) — present on STALL/BOOT-STALL lines because the typical silent-death pattern is "resource count climbing without bound."

The watch-handle part of that count does not come from libuv alone. process.getActiveResourcesInfo() lists an FSEventWrap only for persistent: true watches, and the app's directory watchers are armed non-persistent (every flat git-status watch) or live in the off-thread watch worker — neither of which that accessor reports, so a tape read FSEventWrap: 2 on a box holding 128 handles and a freeze postmortem ruled the handle count out on that zero. The count now adds the handles the app arms through its watch chokepoint (src/main/utils/watch-directory.ts) and through the off-thread watch host, under the same FSEventWrap key, so resourceCount and the histogram cover every handle the app holds. A [tick] line from a busy box therefore reads higher than one recorded before this change; the step is the instrument being completed, not new load. The same numbers feed the once-a-day watch-handle fan-out card.

The silent-death sentinel

Alongside the tape, each tick rewrites <userData>/logs/heartbeat-sentinel.json — a small JSON carrying iso, pid, the run's stall totals and the last few ring entries. A clean exit DELETES it, so the file surviving to the next launch is the evidence of a silent death, and the boot sweep reports it retroactively.

It is written through ONE asynchronous, single-in-flight fs/promises write that the tick ISSUES — never by a worker or a timer of its own. That dependency is the measurement, not an implementation detail: a writer that did not depend on the tick would keep the file advancing while the main thread was frozen, and the watchdog would read a hung app as healthy. A frozen loop issues no tick, so the file simply stops advancing and the 30 s freshness bound trips. The write path holds no blocking fs call — a synchronous arm here measured 2495 ms on tick #1 (2026-10-10), the freeze recorder freezing the app — and a write still in flight across an intentional exit neither re-arms the sentinel nor leaves its file behind.

[heartbeat-sentinel] … lines go to main.log, not to this tape. Two of them matter:

  • armed asynchronously in <N>ms after it was issued from the tick (metric=sentinel_armed_async) — once per process, the first time the arming write lands. It is the positive control that the sentinel's async path ran, and <N> is the write's real wall time.
  • sentinel write failed: <error> - <consequence> — the escalation ladder. The FIRST failure of a streak warns; a failure every 100th consecutive time escalates to log.error. The consequence names which half is broken: silent-death detection is dark when the file could never be armed (a kill from there is reported as a clean shutdown), the armed sentinel can no longer refresh when it is armed but its contents have frozen at arm time. A write that lands clears the streak, so an intermittent fault never climbs to the error rung.

Knobs

  • Master kill switch (env): AMC_DISABLE_MAIN_HEARTBEAT=1 skips init entirely. Use for perf measurements where you want zero heartbeat overhead. (Steady-state cost is ~1 ms per 5 s = ~0.02 % CPU — practically free — but the switch exists for the principled case.)
  • Master kill switch (config): config.json field mainHeartbeatEnabled: false. Read synchronously at init, before the config-store module is loaded. Defaults to true.
  • Reveal button: Settings → Diagnostics → Main-Process Heartbeat → Reveal Heartbeat Log.

Why it's its own file (not main.log)

main.log is a rolling firehose — every log.info from every service. After a silent crash, opening main.log means reading thousands of lines hoping the last few survived the buffer flush. heartbeat.log is focused: just the periodic samples and the lifecycle markers. The self-describing header at the top means an LLM reading a pasted copy with no other context understands every line.

The other diagnostic tapes Omniscio writes — the performance-sensitive interaction timeline and the general runtime log — are described on the Logs and debugging page, and what happens after a crash on Crash recovery.

Last verified 2026-11-07