Main-process heartbeat (silent-crash forensic trail)
A log file recording a continuous five-second tape of Omniscio's main-process health — event-loop lag, memory, live resource counts and the last operation that ran — so that a window which simply disappears with no crash dialog and no error report can still be explained after the fact.
On this page
What it is
A dedicated log file — heartbeat.log — that captures a continuous 5-second-resolution tape of main-process health while Omniscio is running: event-loop lag P50/P99, RSS/heap memory, active-resource counts, and the last IPC channel that ran before each tick. It exists for the case where the Omniscio window simply disappears with no crash dialog, no Sentry error, no Windows Event Log entry, and no Crashpad dump — silent main-process death. With a heartbeat tape on disk, the next-launch post-mortem can answer:
- Was main healthy in the 0–5 s before death? (capacity question — was lag climbing? was RSS bloating?)
- What was main DOING when it died? (causation question —
lastOpnames the IPC handler that was still running, or(no-tracked-op-in-flight)when none was) - Was this an internal hang or an external kill? (presence vs. absence of
[CLEAN-EXIT]/[GOODBYE]on shutdown; correlation with[POST-MORTEM]markers on next launch)
Two sibling files hold the last 10 ticks (the ring tail) captured at the moment of an event, so the dump survives even if rotation dropped the live heartbeat.log before you could read it. heartbeat-snapshot.log is written for an event the app survived — a dead renderer or GPU helper, a recovered blank screen, or an uncaught exception / unhandled rejection (whose handlers deliberately do not exit). heartbeat-final.log is written ONLY from the process.on('exit') hook, and only when that exit was not marked expected by an intentional-quit hook — it means "the process was going away and nobody planned it". A graceful quit or an update relaunch writes neither file. Both carry the same content shape (an AMC HEARTBEAT FINAL DUMP header, then the last ticks).
The file lives next to the other diagnostics logs:
- Windows:
%APPDATA%/omniscio/logs/heartbeat.log - macOS:
~/Library/Application Support/omniscio/logs/heartbeat.log - Linux:
~/.config/omniscio/logs/heartbeat.log
Like interaction.log (and unlike startup.log), heartbeat.log is a continuous tape across launches — the same file grows every time you open the app, capped at 1 MB with rotation to .1 → … → .5. This is intentional: a silent crash that only repeats every few days needs a tape that spans those days.
Where to find it
heartbeat.log sits in the app's machine-level logs folder, beside the other liveness tapes and outside any one account's folder — on Windows under %APPDATA%\omniscio\logs\. The per-account session log (main.log, startup.log, interaction.log) lives one level down, under profiles\<your account>\logs\. There is no in-app viewer; heartbeat.log is read after the fact, most often as the evidence attached to a crash report.
How it behaves
How to use it
After a silent app death
Don't reopen Omniscio immediately. The post-mortem analyser runs on the NEXT launch and reads the on-disk tape — that's good — but if you have other diagnostics to capture first (Task Manager, dxdiag, Reliability Monitor), do those before relaunching, since relaunch is also what writes the
[POST-MORTEM]markers.Relaunch Omniscio normally. During init, the heartbeat:
- Walks back through
heartbeat.loglooking for the prior launch's[HEARTBEAT-START]banner. - Checks whether
[CLEAN-EXIT]or[GOODBYE]appears in the window between that banner and the new one. - If neither appears, writes
[POST-MORTEM]lines into the new launch's section flagging the silent death, the last known tick (with itslastOp), and the operating system’s own record of the exit: Windows Event Log entries from ±60 s around that tick, or on macOS launchd’s exit line for the dead process (its signal or exit code, and who sent it).
- Walks back through
Open Settings → Diagnostics → Main-Process Heartbeat → Reveal Heartbeat Log. The file manager opens with
heartbeat.loghighlighted.Read in this order:
- First,
heartbeat-final.login the same folder, if it exists — that's the emergency ring-buffer dump written when the process reached an exit nobody planned. If it isn't there, openheartbeat-snapshot.loginstead: it holds the same dump for an event the app survived (an uncaught exception, a render/child-process-gone, or a blank-screen recovery) and is the fastest path to a stack trace or to the surrounding ticks for a silent renderer death. - Next, the tail of
heartbeat.log: scan up from the bottom for the most recent[POST-MORTEM]lines. They sit between the prior and current launch banners. If arenderGone=NorchildGone=Ncount appears, the analyser also surfaces the last[RENDER-GONE]/[CHILD-GONE]line — that names the suspect (renderer vs. GPU/utility). - Then the last few
[tick]/[STALL]/[BOOT-STALL]lines BEFORE the silent boundary — that's the 5–25 s window before death. - Finally, the Windows Event Log scrape lines (only on Windows): the analyser shells out to PowerShell
Get-WinEventfor Application-Error / Application-Hang / Windows-Error-Reporting events near the last known tick. If something shows up here, the kill came from outside Omniscio (Windows OOM, security software, driver crash, etc.). On macOS the sameos-event=line carries launchd’s record instead, e.g.exited due to SIGBUS | sent by exc handler[96840]— a crash signal from the exception handler means Omniscio itself crashed (Crashpad caught it), whileSIGKILL | sent by <process>[pid]names the process that killed it. - On macOS the crash dump itself does not stay on disk. Crashpad writes it to
Crashpad/pending, and at the next launch Sentry's minidump loader uploads it and deletes it — before Omniscio's own crash sweep looks — so the code location of a crash lives only in Sentry. Open follow-up: keep a local copy first.
- First,
Attach
heartbeat.log(andheartbeat-snapshot.logand/orheartbeat-final.logif present) to the bug report.If the window froze and never came back, also attach
main-freeze-stack.jsonfrom the same folder. A separate watcher thread writes it after about 8 seconds of frozen heartbeat. It names the code the main thread was stuck in, or says the thread was inside a system call that could not be interrupted. It gains aresumedAtIsofield if the app recovered. Settings → Diagnostics → Export diagnostic bundle already includes it, underlogs/; the plain Export Logs zip does not.When no JavaScript stack can be taken, the file still says what is known.
stackisnullandstackAbsentReasonnames why the capture came up empty —pause-timeout(the inspector pause was never answered within the deadline),evaluate-refused(the runtime answered with an error),inspector-unreachable(the inspector session could not be opened at all),no-app-frames(a stack came back holding only Node-internal frames), orcapture-threw(the capture itself raised). Beside it sitsevidence, which is ALWAYS an object and never a barenull:evidence.nativeOp— the traced synchronousfs/db/proccall holding the main thread at that instant (inFlight.label+heldMs), plus which tracer families were armed. A family that was not armed reportsarmed: false, so its silence is never read as "nothing was happening".evidence.osThread— the Windows thread state across the sample window (stateName,waitReasonName,cpuMsGained,gapMs,stuckThreads), or{ available: false, reason }naming why the reader was dark (not-windows-x64,koffi-path-missing,stderr-is-nul-device,read-failed: …,capture-threw).evidence.beatDuringSample— measured ACROSS the thread sample, not before it.truemeans the heartbeat advanced while the sample was being taken, so the main thread ran again before the record was written: the pause was refused rather than the thread being dead, and the record describes the moment it was still stuck.
The record also carries
inFlight— what the app was carrying at the last heartbeat tick:count,maxAgeMs, theoldesthandlers by name (up toIN_FLIGHT_PEEK_LIMIT, oldest first),asOfIso(the tick that list came from) andstale/staleMs. A list from a tick more than one 5 s interval old isstale: true: on a frozen main thread it always is, because the tick that produced it is the last one before the freeze. ReadcountandmaxAgeMsoff the tick, never off the clock at capture time.
What you'll see in the file
═══════════════════════════════════════════════════════════════
Omniscio MAIN-PROCESS HEARTBEAT
…self-describing header (purpose, Version, Electron, Mode, PID,
tick interval, stall threshold)…
═══════════════════════════════════════════════════════════════
── Launch resumed at 2026-05-22T18:00:00.123Z (PID 24680) ──
2026-05-22T18:00:00.124Z [HEARTBEAT-START] pid=24680 version=1.2.3 mode=development
2026-05-22T18:00:05.130Z [tick] uptime=5s lagP50=2ms lagP99=4ms rss=180MB heap=72MB lastOp="(no-tracked-op-in-flight)" resourceCount=18
2026-05-22T18:00:10.131Z [tick] uptime=10s lagP50=2ms lagP99=3ms rss=185MB heap=74MB lastOp="session:list" resourceCount=22
2026-05-22T18:00:15.137Z [STALL] uptime=15s lagP50=180ms lagP99=1240ms rss=195MB heap=78MB lastOp="recipe:run" resourceCount=24 resources={Timeout:12,TCPWRAP:6,FSReqCallback:3,Immediate:1,TTYWRAP:2}
… more ticks …
2026-05-22T18:02:01.211Z [CLEAN-EXIT] pid=24680 ticksWritten=24 uptime=121s
2026-05-22T18:02:01.213Z [GOODBYE] pid=24680 code=0 ticksWritten=24
Line legend:
[HEARTBEAT-START]— banner stamped at every init. Marks a new launch.[tick]— normal 5-second sample. P99 lag below the stall threshold (1000 ms).[STALL]— same shape as[tick]but P99 ≥ 1000 ms AFTER the first IPC arrived. Suggests a CPU-bound operation in main (heavy synchronous SQLite, JSON parse on a huge buffer, NDJSON parse spike).[BOOT-STALL]— same as STALL but before any IPC ran. Suggests an early-startup hang (migration, native module load, sync I/O during bootstrap).[STALL-CAUGHT]— emitted by a separate 200 ms fast-tick watchdog whenever the gap between two consecutive watchdog fires is ≥ 2 000 ms (the loop was just unblocked after a multi-second pause). Line shape:durationMs=<actual gap> lastOp="<channel>" inFlightCount=<n> inFlight=[…].lastOphere means "an IPC handler was still running inside this frozen window" — a handler that finished before the window opened is never named, and when nothing was in flight the field reads(no-tracked-op-in-flight)and nolastOpLeadis emitted. TheinFlightarray lists every IPC that was mid-await at the moment of unblock, sorted DESC byageMs(oldest first — most likely the culprit) and capped at 10 entries. Companion to[STALL]: the 5 s[STALL]says "lagP99 was high somewhere in this window,"[STALL-CAUGHT]says "the loop was pinned for exactly N ms and these IPCs were waiting." A 2–5 s stall that ends mid-window can be missed by[STALL](dilutes below the p99 threshold) but always shows up here. Seepostmortem.blockedOp=on that line names what held the thread. A single op is its own label;agg:<op>x<count>is a run of spans of ONE op summed to the gap;burst:<op>+<op>+<op>x<count>is one contiguous run of DIFFERENT ops that together covered at least half the freeze — it names the three biggest and then how many spans the run held, e.g. five different DB reads back to back. A parenthesised value ((no-op-covered-gap),(none-instrumented),(off-cpu-no-blocker)) is the tracker saying it could not name one, and each of those carries its own next move.[stall-stack]— emitted right AFTER a[STALL-CAUGHT]when the V8 CPU profiler was armed and captured samples. Pair the two bystallId=<epochMs>— the freeze's unblock timestamp, carried by every record of one freeze ([STALL-CAUGHT],[stall-stack],[STALL-GC], and the[GC-PAUSE]warn that lands inmain.loginstead ofheartbeat.log). Pairing bydurationMswas the old join and it does not work: a storm puts dozens of stalls in the same 2–4 s band, and[GC-PAUSE]never carried it at all. Line shape:stallId=<epochMs> durationMs=<gap> top: <fn> (<file>:<line>) <n>% | … ← caller: <fn> (<file>:<line>).top:is the top-5 frames by SELF time — on a multi-second synchronous block the blocking frame dominates every other sample, so the first entry IS the blocking op (frequently a native leaf likeexistsSync (native) 81%).← caller:is the answer to "which of OUR code asked for that": the profiler walks UP the call tree from the hottest leaf to the first APP frame. Node built-in frames are deliberately SKIPPED — they carry a source url (node:fs,node:internal/…) but are only Node's own wrapper around the native leaf, so returning one would merely restate whattop:already said. Before that skip existed, 22 of 26 attributed callers on a live box readexistsSync (node:fs:278)/statSync (node:fs:1733)and named nothing useful. Anode:frame still appears as a last-resort fallback when the path holds no app frame at all;← caller:is omitted entirely for a pure native / off-CPU wait, and the line readstop: (no samples — likely native/off-CPU wait)when the window had no samples. NOTE — this is a DIFFERENT field fromblockedOp=on the[STALL-CAUGHT]line, which is fed bytrackMainOpbreadcrumbs — the DB layer, plus (since 2026-08-29) EVERY periodic background service, namedservice:<label>, becausecreatePeriodicTaskwraps each tick's synchronous slice. So a freeze inside a service tick now reads e.g.blockedOp=service:agent-status-boardrather than(none-instrumented); that fallback still appears for a synchronous block in code no breadcrumb covers, and for an ASYNC tick, which correctly registers no block at all. The profiler's answer cannot be folded into this field because the capture is asynchronous and resolves after that line is already written. Read the two lines together. Kill switchAMC_DISABLE_MAIN_STALL_PROFILER=1; armed from the first tick and kept armed by default (AMC_STALL_PROFILER_ALWAYS_ARMED,0= armed only under box load — see the T-423 follow-up below). Arm/disarm/rotate leave their own[stall-profiler]transition lines on this tape (armed intervalUs=… maxWindowMs=…/disarmed windowMs=…/rotated windowMs=… samplesDiscarded=…/arm-failed reason=…/arm-deferred reason=…), so "was the instrument running during the 13:20Z storm?" is answerable after the fact. A[stall-profiler] arm-failedline means the profiler is DARK, and it can be dark whilearmedreads true to nobody but itself — V8 refusessetSamplingIntervalwhile a profile is live, so if a command's reply is lost under load the profiler keeps sampling while the module believes it is off. The arm path is therefore its own breaker: three consecutive failures go quiet for 60 s (logging once per episode, never once per 200 ms tick, and the tape keeps the token every time), and the next arm stops the suspect profiler before it re-enables one. This is the same trap the renderer twin hit on 2026-09-25 (bug 9acddc97), where the retry loop itself measured ~250 log lines a minute and 46.5% of a core. And starting the profiler is itself expensive on a large process:Profiler.startruns V8'sStartProfilingon the main thread, which at a ~370 MB heap measured 2–4 s (Help Desk 6069a960, 2026-09-29). Because that exceeds the 2 s stall threshold the watchdog recorded the instrument's own start-up as an app freeze → the freeze asked for a capture → the capture stopped the profile and re-armed → arming paid it again;[stall-profiler] armedwent from ~20/hour to 500–980/hour while the UI lagged ~2 s per click. So the profiler now times its own start and disarms itself for the rest of the run when it exceedsAMC_STALL_PROFILER_MAX_ARM_COST_MS(1000 ms — set ABOVE the measured start cost, whose 28 recorded defeats run 500–807 ms, so a healthy run does not latch the instrument off; and set well below the 2 s stall threshold — not exactly 2× under it, because the watchdog's gap also carries the arm's own pre-start round trips — so the profiler's own work cannot be counted as a stall. It was 500 ms until 2026-10-04, which sat below the very costs it was bounding and took the instrument out on essentially every run; re-derive it from thearmCostMs=now printed on thearmedline). The tape says[stall-profiler] self-defeated armCostMs=…, and a later[stall-stack]from that process readsunavailable=self-defeated. The self-defeat latch catches an expensive start only AFTER paying it, so the arm is now PREDICTED before it is paid and a calm tick no longer disarms instantly (2026-10-09). One gate at the top ofarmStallProfilercovers every caller — the 200 ms tick, the capture'sfinallyre-arm, and the rotation re-arm:prospectiveArmDecline()predicts the start cost from the live heap againstAMC_STALL_PROFILER_ARM_BUDGET_MS(default 950 ms — 50 ms under the 1000 ms latch and 1050 ms under the 2 s stall threshold) and declines when a start would reach it. The prediction is fitted to the instrument's own measured arms, not to one reading. 221 successful main-process arms cost a median 730 ms / p90 920 ms of main-isolate block (and 28 recorded defeats under a 500 ms bound ran 500–807 ms). The earlier10.8 ms/MBzero-intercept slope came from ONE Help Desk reading — 2–4 s at a ~370 MB heap on a box at 75–90% CPU with 16 live sessions — and predicted 4.3–14.2 s at the heaps this process actually runs at, 6–15× above what the same instrument measured. That is why it declined every arm between 2026-10-15 and 2026-10-16 (15arm-deferred reason=heap-too-largelines, heap 401–1310 MB, zeroarmedlines, while/perf/statusreadbusyStacks=0andworstStallStack.unavailable=disarmed). The fit now anchors the measured median at 400 MB and the measured p90 at 2000 MB (2050 MB was observed live on this box), i.e. 0.11875 ms/MB with a 682.5 ms intercept — deliberately conservative at the top of the band. The ceiling stays DERIVED from the budget and the fit (MAX_ARM_HEAP_MB≈ 2253 MB, was ~46 MB), so a heap whose start is predicted to reach 950 ms is still refused; the trade-off is that a real arm now blocks the main thread for the measured ~0.7–0.92 s, which the 1000 ms latch already permits and which is the cost of having an instrument at all. A decline prints ONE[stall-profiler] arm-deferred reason=heap-too-large heapMB=… predictedStartCostMs=… budgetMs=…line per episode (never one per 200 ms tick) and is re-decided on every attempt, so the profiler re-arms by itself once the heap drops — it is NOT a latch. A capture also ends with aAMC_STALL_PROFILER_POST_CAPTURE_HOLD_MScalm window (default 10 s, linearm-deferred reason=post-capture-calm remainingMs=… holdMs=…) before the re-arm, so the capture's own aftermath is not reopened by the very next arm. Separately, a calm tick now HOLDS the arm while the last load evidence is insideAMC_STALL_PROFILER_CALM_HOLD_MS(five minutes, mirrored from the renderer twin'sARM_HOLD_MS) — a DEBOUNCE, not a latch: re-read against the last evidence every tick, so a genuinely calm box still disarms by itself, and a backward clock jump holds (failing toward the instrument being on). All three knobs default ON and route throughresolveEnvNumber;0is the off switch on each and restores the prior behaviour. The kill switch must be set BEFORE the app launches: settingAMC_DISABLE_MAIN_STALL_PROFILER=1and then using the in-app restart does NOT pick it up —app.relaunchinherits the old process environment, so it takes a full quit and relaunch. Seesrc/main/services/diagnostics/main-stall-profiler.ts.- T-423 (2026-10-17) — the self-defeat above is now a bounded back-off, not a run-long latch. The
self-defeatedtape line now readsself-defeated armCostMs=… boundMs=… backing-off backoffMs=… until=<iso> heapMB=… cheapHeapMB=…, and the[STALL-CAUGHT]line of a dark stall addsprofilerDarkUntilIso=. The back-off is bounded, never a run-long latch (T-423, 2026-10-17). It used to latch off for the life of the process, and on 2026-10-17 it did so on 2 of 2 boots about two minutes in (1237 ms at a ~1.65 GB heap, 1359 ms at ~1.6–1.9 GB), right on the post-capture re-arm after the first[busy-stack], so the 20–25 s freezes later that morning carried no stack. Now a too-costly arm keeps the profiler off for 30 min, doubling on each consecutive costly arm up to a 4 h cap, and a cheap arm resets the ladder ([stall-profiler] backoff-reset armCostMs=… previousBackoffMs=…). The costly arm also teaches a heap ceiling: the start cost on this box tracks the main heap roughly in proportion (318 ms at ~600 MB, 744 ms at ~1 GB, 1237 ms at ~1.65 GB on 2026-10-17), and the heap saw-tooths ~0.8 → 2.4 GB every ~25 s, socheapHeapMB = heapAtCostlyArm × min(AMC_STALL_PROFILER_ARM_BUDGET_MS, AMC_STALL_PROFILER_MAX_ARM_COST_MS) / armCostMs(≈1260 MB for the 1237 ms arm at ~1.64 GB). Every later arm waits for the heap to fall under that ceiling (arm-deferred reason=heap-above-learned-ceiling heapMB=… ceilingMB=… maxWaitMs=300000), so it lands on a GC trough rather than a peak, but never for more than 5 min — only a paid arm restarts that clock, so a load verdict flapping calm/busy cannot hold it open — past that it arms anyway, so a heap that never troughs cannot keep the profiler dark. A cheap arm at or above the ceiling disproves it and clears it. Both clocks treat a backward clock jump as elapsed, like the module's other gates, so it ends a back-off or a wait rather than lengthening it. The kill switchAMC_DISABLE_MAIN_STALL_PROFILER=1remains the only way to keep it off for the whole run. Current state:GET /diag/heartbeat(the tape tail) — the latest[stall-profiler]line isarmed …,self-defeated … backing-off … until=<iso>, or a deferral; a dark[STALL-CAUGHT]namesprofilerDarkUntilIso. - T-423 follow-up (2026-10-18): the profiler stays armed, and a capture never restarts it from scratch. Measured live after the 21:48 ET boot on 2026-10-17: 17 of 18 freezes of 2 s or more carried no stack. The cause was not the heap. Every capture, rotation and busy capture stopped the only running profile, so the re-arm paid V8's whole cold
StartProfilingon a box at 100% CPU. That cost 1,634 ms at a 2,080 MB heap and 1,702 ms at a 1,031 MB heap; each of those starts sat inside the 2.0 s and 2.1 s freezes it caused, tripped the 1,000 ms bound, and backed the profiler off for 30 and then 60 minutes. A probe on Node 24 showed CPU contention drives the size: a cold start cost 2,274 ms calm and 11,763 ms with the cores oversubscribed. The fix is a bridged hand-off. V8's inspector shares one CPU profiler between theProfiler.startprofile and anyconsole.profileprofile, and pays the cold start only when the live-profile count goes from 0 to 1. So a capture, rotation or busy capture now opens a short bridge profile (inspector.console.profile('amc-stall-bridge')), stops and reads the window, starts the next window, and closes the bridge. Measured: 0–1 ms per hand-off under 2x CPU oversubscription, against 846 ms for a cold start in the same run. The bridge is always closed in afinally. If it cannot be closed, bridging is switched off for the run and the next arm sendsProfiler.disablebeforeProfiler.enable, because a leftover bridge makessetSamplingIntervalrefuse. Ifinspector.console.profileis missing or throws, the capture falls back to the old stop-then-cold-arm path, so the instrument is never left dark. A bridged restart that still costs overAMC_STALL_PROFILER_MAX_ARM_COST_MSis treated as a cold-arm defeat. Always armed:AMC_STALL_PROFILER_ALWAYS_ARMEDdefaults ON, so a calm tick arms the profiler and keeps it armed (rotating every 20 min like a loaded tick). Only the kill switch or a back-off disarms it, so the one cold start lands at boot (348–372 ms live).0restores the load-gated arm/disarm. Its own cost is on the tape:[stall-stack]carriescostMs=(stop plus restart, beforetop:so perf-status still reads the frames),rotatedcarriescostMs=andbridged=yes|no, and every 10 minutes armed the tape gets[stall-profiler] overhead windowMs=… profilerMainMs=… handoffs=… coldArms=… samplesRead=… boundMs=….profilerMainMsis the measured main-thread time of every profiler command in the window that completed. A round trip that hit the 5 s inspector deadline is not counted, because its wall time is waiting rather than profiler work. Above 1% of the window it backs off through the same 30 min → 4 h ladder with its own reason (self-defeated reason=overhead …,stallProfilerDefeat().reason = overhead-<ms>ms) and leaves the learned heap ceiling alone. A cheap restart does not reset that ladder; only an overhead window that closes under its bound does. A dark[stall-stack]during either kind of back-off readsprofiler dark — backing off: it exceeded its own cost bound (start cost or overhead …), and theself-defeatedline names which one. The sampler thread's own CPU runs off the main thread and cannot be separated in-process. Measured offline on Node 24 it was 343 ms of process CPU per 10 s idle armed (about 3.4% of one core); it cannot freeze the UI, and the live check reads it from process CPU. - T-224 (2026-11-10) — every silent instrument on a
[STALL-CAUGHT]line now says WHY it is silent. Three signals used to go absent for several unrelated reasons each, and the row carried no token saying which, so an absence read exactly like an instrument that looked and found nothing. On the four worst freezes of the 01:31–02:31 window (13.3–16.7 s, allblockedOp=(no-op-covered-gap)) none of the three could be interpreted. Now:stackCapture=is written from the SAME predicate the capture itself asks (stallProfilerBlindReason(), i.e. the capture's own blind-reason check), not from the profiler'sarmedflag.armedtherefore means "a capture will really run for this freeze"; a profiler that is armed but kill-switched, breaking its overhead bound, or backing off readsdisarmed, which is the state whose[stall-stack]line carriesunavailable=<reason>. Before this, such a stall stampedarmedand the capture then produced no stack — a claim and its evidence disagreeing on the same tape.syncFsGap=names whysyncFsInGapMsis absent:none(the ops tracker AND the sync-fs tracer were both live, and nofs.*Syncspan completed inside the gap — a positive claim, which is what the off-CPU freezes of 2026-11-10 actually were:cpuBusyPct1–2, stack top(idle)99–100%),tracker-off(AMC_DISABLE_MAIN_OP_TRACK, checked first becauserecordCompletedMainOpreturns on it before it ever looks at the label), ortracer-off(AMC_DISABLE_SYNC_FS_TRACE). Absent wheneversyncFsInGapMsis on the line.culprits=names why no[stall-cpu-culprits]line accompanies this stall:not-starved(a BLOCKED/MIXED freeze — the CPU was burned on the thread, so a process list names nobody) orbox-calm(STARVED with neither CPU nor disk pressure on the box). The sampler's own(not-sampled: …)line only exists when the caller reaches it; the watchdog's gate declines first on most stalls, so this field covers the silence that line cannot. Absent when the gate said sample, where the culprit line speaks for itself. Both new fields are omitted on a stall whose instruments reported, so a healthy line and every pre-existing line is byte-identical. Seesrc/main/services/diagnostics/heartbeat-stall-tape.tsandheartbeat-stall-classify.ts.
- Help Desk ddd34b4a (2026-11-11) — the line's own CLASS now reaches the card. Every field the class needs was already here, and the reader that raises the single-freeze card turned none of it into an answer.
worstStallClassOfinperf-status/stall-stack-parse.tsreads the same longest-freeze lineworstStallOppicks and publishes two new fields on the main-thread block:worstStallClass=sync-filesystemwhenblockedIn=syscallis present orsyncFsInGapMscovers at leastBLOCKED_OP_MIN_COVERAGE_FRACTION(the same half-gap boundisBlockedInSyscalluses),memory-faultswhenisMemoryFaultStormholds (the predicate is reused, never re-derived), andnullwhen neither was measured; andworstStallSyncFsOp, thetopSyncFsOptoken, for a developer build. On the reporter's tape 13 of 29 caught freezes were sync-fs dominated (86.0 s of 89.0 s) and the worst two were one blockingopenSynconC:withcpuBusyPct=0— a freeze the card described with a memory remedy.nullis "not measured", never "the box was calm". [CLEAN-EXIT]— written at the top ofgracefulShutdown. Disk is healthy, shutdown reached the heartbeat-stop path.[GOODBYE]— written from aprocess.on('exit')hook. Confirms the runtime actually unwound, not just that gracefulShutdown started.[FATAL]— written fromuncaughtException/unhandledRejectionhandlers. IncludeslastOpand a 6-line stack snippet.[RENDER-GONE]— written from therender-process-gonehandler insrc/main/index.tswhendetails.reason !== 'clean-exit'. Stamps the currentlastOpplusreasonandexitCode, then dumps the ring buffer intoheartbeat-snapshot.log(the app survived — only the renderer died) so the surrounding ticks are preserved even if rotation later trims the live tape. Tells you whether a silent death was the renderer (the BrowserWindow's web content) dying out from under main.[CHILD-GONE]— same mechanism as[RENDER-GONE]but from the app-levelchild-process-gonehandler. Fires for GPU process / utility process / sandbox helper exits. Includestype,reason,exitCode, and (when present)name/service. A GPU-process death is a common cause of "the window disappeared" with no main-process exception.[POST-MORTEM]— written on the NEXT launch when the analyser sees a silent death pattern. Lines includepreviousLaunchEndedSilently=true,lastKnownTick=…,os-event=…. When[RENDER-GONE]or[CHILD-GONE]lines fell inside the prior-launch window, the analyser also surfacesrenderGone=N/childGone=Ncounts and the last line of each, so the post-mortem names a suspect even after the live tape is gone.
resources={Timeout:12,TCPWRAP:6,…} shows the libuv active-resource histogram (counts of pending timers, sockets, fs requests, etc.) — present on STALL/BOOT-STALL lines because the typical silent-death pattern is "resource count climbing without bound."
The watch-handle part of that count does not come from libuv alone. process.getActiveResourcesInfo() lists an FSEventWrap only for persistent: true watches, and the app's directory watchers are armed non-persistent (every flat git-status watch) or live in the off-thread watch worker — neither of which that accessor reports, so a tape read FSEventWrap: 2 on a box holding 128 handles and a freeze postmortem ruled the handle count out on that zero. The count now adds the handles the app arms through its watch chokepoint (src/main/utils/watch-directory.ts) and through the off-thread watch host, under the same FSEventWrap key, so resourceCount and the histogram cover every handle the app holds. A [tick] line from a busy box therefore reads higher than one recorded before this change; the step is the instrument being completed, not new load. The same numbers feed the once-a-day watch-handle fan-out card.
The silent-death sentinel
Alongside the tape, each tick rewrites <userData>/logs/heartbeat-sentinel.json — a small JSON
carrying iso, pid, the run's stall totals and the last few ring entries. A clean exit DELETES it,
so the file surviving to the next launch is the evidence of a silent death, and the boot sweep
reports it retroactively.
It is written through ONE asynchronous, single-in-flight fs/promises write that the tick ISSUES —
never by a worker or a timer of its own. That dependency is the measurement, not an implementation
detail: a writer that did not depend on the tick would keep the file advancing while the main thread
was frozen, and the watchdog would read a hung app as healthy. A frozen loop issues no tick, so the
file simply stops advancing and the 30 s freshness bound trips. The write path holds no blocking fs
call — a synchronous arm here measured 2495 ms on tick #1 (2026-10-10), the freeze recorder freezing
the app — and a write still in flight across an intentional exit neither re-arms the sentinel nor
leaves its file behind.
[heartbeat-sentinel] … lines go to main.log, not to this tape. Two of them matter:
armed asynchronously in <N>ms after it was issued from the tick (metric=sentinel_armed_async)— once per process, the first time the arming write lands. It is the positive control that the sentinel's async path ran, and<N>is the write's real wall time.sentinel write failed: <error> - <consequence>— the escalation ladder. The FIRST failure of a streak warns; a failure every 100th consecutive time escalates tolog.error. The consequence names which half is broken:silent-death detection is darkwhen the file could never be armed (a kill from there is reported as a clean shutdown),the armed sentinel can no longer refreshwhen it is armed but its contents have frozen at arm time. A write that lands clears the streak, so an intermittent fault never climbs to the error rung.
Knobs
- Master kill switch (env):
AMC_DISABLE_MAIN_HEARTBEAT=1skips init entirely. Use for perf measurements where you want zero heartbeat overhead. (Steady-state cost is ~1 ms per 5 s = ~0.02 % CPU — practically free — but the switch exists for the principled case.) - Master kill switch (config):
config.jsonfieldmainHeartbeatEnabled: false. Read synchronously at init, before the config-store module is loaded. Defaults totrue. - Reveal button: Settings → Diagnostics → Main-Process Heartbeat → Reveal Heartbeat Log.
Why it's its own file (not main.log)
main.log is a rolling firehose — every log.info from every service. After a silent crash, opening main.log means reading thousands of lines hoping the last few survived the buffer flush. heartbeat.log is focused: just the periodic samples and the lifecycle markers. The self-describing header at the top means an LLM reading a pasted copy with no other context understands every line.
Related
The other diagnostic tapes Omniscio writes — the performance-sensitive interaction timeline and the general runtime log — are described on the Logs and debugging page, and what happens after a crash on Crash recovery.
Last verified 2026-11-07