Reporting a bad Plain Speak rewrite

What it is

What it is

A "Report bad overlay" item in the 3-dot menu on agent messages. While a message is showing its Plain Speak rewrite, opening the menu reveals the item; clicking it opens a small notes-only dialog with an optional textarea. Type a short note (or leave it blank), hit Submit, and Omniscio packages the agent's original message, the rewritten overlay text, the surrounding conversation, your note, and ships the bundle as an email to the team. The point is to make it nearly zero-friction for the user to flag a Plain Speak rewrite that misled, mistranslated, or dropped something important — so the prompt + pipeline can be tuned against real bad rewrites instead of synthetic ones. The feedback channel is email, not IPC — the team triages reports in an inbox just like any other bug intake, and any individual JSON bundle can be dropped back into the offline plain-speak harness to replay the failure deterministically.

This is one of two reports that route through Omniscio's unified bug-report channel (bug-report:submit, kind: 'overlay' | 'qw-miss'). The sibling flow is qw-miss-reporting.md. Both share the same BugReportDialog component, the same IPC handler, and the same dual-delivery transport — only the per-kind bundle contents and subject-line slug differ.

Where to find it

The 3-dot menu on an agent message while it is showing its Plain Speak rewrite. The switch that hides the item is Settings → Plain Speak → Show 'Report bad overlay' menu item.

How it behaves

When the menu item is visible

The "Report bad overlay" menu item appears in an agent-message 3-dot menu only when ALL of the following are true:

1. The message is an agent message (operator/user messages don't get the item — there's no overlay to flag on them).
2. The message has a Plain Speak overlay rewrite attached (no rewrite = nothing to report).
3. The bubble is currently showing the Plain Speak view. Flipping the Plain Speak / Original pill to Original hides the item too — the report is "this rewrite is bad", not "this raw agent text is bad", so it only makes sense while you're looking at the rewrite. Flip back to Plain Speak and the item reappears.
4. The global setting Show 'Report bad overlay' menu item (Settings → Plain Speak, default on) is enabled. Flipping the setting off hides the item from every overlay-bearing message live — useful for screenshots, demos, or if you don't want the option in the menu at all.
5. The message is not on cooldown for the overlay kind. After a successful submission, the same (message id, overlay) pair is hidden for 5 minutes so a stray double-click can't fire a duplicate report. Cooldown is per-(message, kind) — submitting overlay does NOT put the sibling "Report missed question widget" item on cooldown for the same message, and vice versa.

The cooldown is in-memory only. Refreshing the Omniscio window wipes it. The intent is "stop me from immediately re-clicking", not "remember across sessions" — five minutes is plenty to cover the email/disk write round-trip without being so long it locks the user out of legitimate re-reports.

What the dialog shows

Clicking "Report bad overlay" opens a centered modal dialog titled Report bad overlay:

• A short description paragraph: "Flag the Plain Speak rewrite for this message. The original message and the overlay output are attached automatically — leave a note if you want to tell us what it got wrong."
• A Notes (optional) textarea, autofocused, capped at 2,000 characters, with the placeholder "What did the overlay get wrong? (optional — leave blank to just flag it)".
• Cancel and Submit buttons in the footer. Ctrl+Enter / Cmd+Enter inside the dialog also submits (per the Omniscio useCtrlEnterSubmit cross-modal contract).

The dialog deliberately does not preview the rewrite text. The user just saw the message in the chat — re-showing it inside the dialog adds visual noise without adding information, and the server-side bundle is the authoritative copy. The dialog's job is to capture the optional note and confirm intent.

The dialog uses the standard DialogShell primitive — Esc closes it, clicking outside closes it, and the body is keyboard-focus-trapped. Submit is fire-and-forget: clicking Submit closes the dialog immediately so the user can keep working, then ships the report in the background. A per-(message, kind) cooldown is marked optimistically at Submit-time so re-opening the menu can't double-fire; on background failure the cooldown is cleared so the user can re-flag without waiting out the 5-minute window.

What happens on Submit

The button fires an IPC call (bug-report:submit) carrying kind: 'overlay', the message id, and the user's notes. The main process:

1. Looks up the target message and joins conversation_messages with the latest ai_manager_decisions row that has feature='overlay' and outcome='applied' for this message. If no overlay was actually applied (e.g. the user clicked the item on a message whose overlay was cancelled, skipped, or never produced), the handler returns a friendly error toast ("No overlay found for this message.") and the dialog stays open so the user can copy their note.

2. Builds a JSON report bundle containing: a fresh UUID, the report timestamp, the Omniscio app version, the user's notes, the session id and title, the project name, the session's pendingAction at the time of the click, the agent message id + content + timestamp, the array of preceding conversation messages (oldest-first), a lengthBucket classifier ('short' / 'medium' / 'long' computed from the agent message length), the plain-speak rewrite text, the active overlay pipeline id (e.g. 'cascade', 'sonnet-single', 'qwen-single'), and a checkFor: 'user-reported-bad-overlay' tag the evaluator harness uses to filter.

3. Fans the bundle out across two destinations in parallel — Resend (an email, subject [BUG: amc-overlay] <session title>, with the full bundle and Omniscio logs attached, addressed so a reply reaches the signed-in reporter — their account email is set as the reply-to and shown as a "Reported by" line in the body) and Firestore (the developer's bug-report database). Both are governed by the bugReportTransport setting (default both) and gated by the same Send bug reports & feedback switch (RT-F024) — both stop together when it's off. The amc-overlay slug routes Resend reports to the plain-speak / evaluator harness instead of the generic Omniscio bug queue. The report counts as delivered if either destination succeeds.

   Delivery (2026-05-20, ErrPort added): The report fans out across up to three destinations in parallel — Resend, Firestore, and ErrPort (the structured error dashboard) — all three gated by the one Send bug reports & feedback switch (RT-F024). The report counts as delivered if any destination succeeds. Only if every leg fails is the report kept in the <userData>/bug-reports/ outbox, so it is never silently lost, and retried later by the hourly outbox drain. (ErrPort was retired 2026-09-28; delivery is now Resend + Firestore only.)

   Very long conversations (2026-09-26): the report is fitted to what the relay accepts instead of being refused. The oldest thread messages are left out first — the report says how many it kept out of how many ("Thread: 120/200 (truncated)") — and only if that is still not enough is the one long message itself cut short, with a note saying so. A normal-sized report is sent unchanged.

4. The dialog closes the instant Submit is pressed (the cooldown is marked first, so the menu item is hidden); the IPC then resolves in the background. On send success a green Feedback sent — thanks! toast fires — the user has already returned to the chat by then, so the toast confirms a completed background action rather than gating a wait.

5. Only if every leg (Resend, Firestore) fails: the bundle is written to disk at <userData>/bug-reports/overlay-<sessionId>-<msgIdShort>-<bundleId>.json (a path-traversal-safe filename — every interpolated id is stripped of any character outside [a-zA-Z0-9._-], runs of two-or-more dots are collapsed, and parts are capped at 64 chars). The user still sees a success toast, but the IPC response carries delivery: 'disk-fallback' for telemetry. The file can be mailed manually later if the user wants to escalate. If even the disk fallback fails (read-only volume, permission error), the IPC response carries delivery: 'failed', the per-(message, kind) cooldown is cleared so the user can re-flag from the menu immediately, and a red error toast fires with the underlying error message; the bundle is lost.

The dialog is React.lazy()-loaded — its weight (DialogShell, Button, useToast, framer-motion) stays out of the entry chunk and only downloads when the user actually opens one of the bug-report menu items.

How reports get back to the evaluator

The JSON bundle is intentionally shaped to be drop-in compatible with the offline plain-speak fixtures format. To replay a reported failure against any pipeline:

1. Detach the .json attachment from the report email (or pick it up from <userData>/bug-reports/ if it disk-fellback).
2. Drop the bundle into tools/plain-speak/fixtures.json — the field names match the existing fixtures schema.
3. Re-run the harness (tools/plain-speak/backtest-run.js) against the current pipelines. The checkFor: 'user-reported-bad-overlay' tag is the filter the evaluator uses to surface "this is a user-reported failure" cases in its scoreboard.

This loop is the entire point of the feature: synthetic test fixtures are easy to write but rarely match the failure modes users actually notice. A menu click → emailed JSON → fixtures.json drop closes the gap with a single user-side click.

How to disable the menu item

Settings → Plain Speak → flip Show 'Report bad overlay' menu item off. The setting is overlayFeedbackEnabled in AppSettings (default true), persisted in config.json like any other setting. With the toggle off, the item disappears from every overlay-bearing message's 3-dot menu live — no reload needed, the visibility is computed reactively from the settings store.

Turning the toggle off does NOT disable Plain Speak itself — overlays still render, the Plain Speak / Original pill still works. It only hides the menu item.

Telemetry

Every successful submission fires the plain_speak_overlay_report_submitted feature event with metadata { delivery: 'email' | 'firestore' | 'disk-fallback' | 'failed' } — the only allow-listed metadata key. The delivery value names exactly which leg won: 'email' or 'firestore'; 'disk-fallback' means every remote leg failed and the report landed in the outbox instead (retried later); 'failed' means even that kept-record write failed. The event surfaces under Stats → Feature usage as Plain Speak Bad-Overlay Reports, with the Top breakdown showing whether reports are reaching a remote endpoint, piling up on disk after both remotes fail, or hard-failing entirely (a signal the team uses to detect provider outages quickly). No raw message text, no user notes, no session ids leak into telemetry — only the delivery outcome.

Privacy

What the team receives in the email bundle: the agent message text, the Plain Speak rewrite text, the preceding conversation messages (operator + agent + system rows alike), the user's free-form notes, the active pipeline id, session and project names, and Omniscio's collected logs. The free-form userNotes field is the only place where unrestricted user-typed text lands in the bundle — every other field is either reproduced verbatim from the existing session transcript or a small enum (pipeline id, length bucket, pending action).

What it does NOT include: API keys, OAuth tokens, secrets from config.json, content from other sessions, content from soft-deleted sessions (the SQL filter excludes any session or project row with is_deleted = 1 — a flagged report against a since-archived session returns "No overlay found" rather than leaking content the user thought was discarded).

For agents

Implementation references

For agents working in the repo:

• 3-dot menu item + dialog mount: agent-message 3-dot menu inside src/renderer/src/components/ui/MessageBubble/MessageBubble.tsx — the menu item lives next to the sibling "Report missed question widget" item; both are gated on the per-(message, kind) cooldown from src/renderer/src/stores/bug-report-cooldown-store.ts. The shared BugReportDialog is mounted lazily once bugReportKind flips to 'overlay' or 'qw-miss'.
• Shared dialog: src/renderer/src/components/ui/BugReportDialog.tsx. Notes-only, parameterized by kind: 'overlay' | 'qw-miss'; uses DialogShell with initialFocusRef={textareaRef} (the trap's 50 ms timer would otherwise pull focus to the header close X), wires useCtrlEnterSubmit scoped to the dialog body (invariant I5). Submit is fire-and-forget: calls markSubmitted(messageId, kind) optimistically, closes the dialog, runs the IPC in the background, then toasts success or — on a success: false envelope or a rejected promise — calls clearSubmitted(messageId, kind) and toasts the underlying error. A submittedRef single-fire guard absorbs key-repeat double-fires before the close completes.
• Settings toggle: in src/renderer/src/features/plain-speak/PlainSpeakSettings.tsx with data-setting-id="overlay-feedback-enabled"; matching index entry in src/renderer/src/features/settings/settings-search-index.ts. The AppSettings field + DEFAULT_SETTINGS entry live in src/shared/types.ts, and the matching Zod field in updateSettingsSchema is in src/shared/ipc-schemas.ts.
• IPC channel: bug-report:submit registered as IPC.BUG_REPORT_SUBMIT in src/shared/ipc-channels/index.ts. Payload schema bugReportSubmitSchema in src/shared/ipc-schemas.ts (kind: 'overlay' | 'qw-miss', messageId, userNotes).
• Bundle builder: buildOverlayReportBundle() in src/main/services/overlay-report-builder.ts — joins conversation_messages with ai_manager_decisions, returns null when no applied overlay exists or when the session/project is soft-deleted. Pulls the active pipeline id from aiManager.overlayPipeline settings (falls back to DEFAULT_PIPELINE_ID).
• Delivery transport: sendBugReportEmail() in src/main/services/email/resend-service.ts (Resend leg — subject prefix [BUG: amc-overlay], JSON bundle attached, Omniscio log attachments appended via collectLogAttachments() under the shared 40 MB Resend cap) + submitBugReportBundleToFirestore() (Firestore leg). Both run through the one shared chain, deliverReport in src/main/services/bug/report-delivery.ts. Kept-record fallback <userData>/bug-reports/overlay-<sessionId>-<msgIdShort>-<bundleId>.json written by sendReport only when nothing delivered — retried automatically by the hourly outbox drain.
• IPC handler: src/main/ipc/bug-report-handlers.ts — single handler dispatches to the per-kind builder, then hands the scrubbed bundle to sendReport({ kind: 'bundle', ... }) (report-delivery.ts), which runs every leg and writes the kept record only when nothing delivered (sanitized filename parts, resolved-path containment check), returns { success: true, data: { delivery: 'email' | 'firestore' | 'disk-fallback', correlationId? } } on the delivered/kept paths and { success: false, error, data: { delivery: 'failed' } } only when even the kept-record write fails — so the renderer surfaces a real error toast on a genuinely lost bundle rather than the cheerful success copy. Telemetry fires on every outcome regardless of envelope shape.
• Telemetry: feature id plain_speak_overlay_report_submitted in src/shared/feature-registry/index.ts with metadataAllowList: ['delivery'].

Related

Related

• plain-speak.md — the parent feature the menu item attaches to; describes the rewrite pipelines, daily cost cap, per-session toggle, and the ## Questions answer-submit format the overlay-feedback report complements.
• qw-miss-reporting.md — the sibling bug-report flow; same dialog, same IPC channel, different bundle shape and subject-line slug. The menu items live side-by-side in the agent-message 3-dot menu.
• bug-report-intake.md — the parent bug-routing system Omniscio uses; the [BUG: amc-overlay] subject prefix is a sibling slug that the plain-speak harness routes off of.
• stats.md — where the Plain Speak Bad-Overlay Reports feature event shows up; Top metadata breakdown shows email vs disk-fallback vs failed delivery split.