Supermail Video Summaries (Loom & Vimeo)

What it is

Video summaries are an optional AI helper in Supermail. When an incoming message contains a Loom or Vimeo video link, it reads the video for you and shows a short, AI-written summary card inside the message — so you can tell what a video is about before deciding whether to watch it.

It is in development and off by default — hidden until you turn it on in Settings → Features (the supermailVideoSummariesEnabled Lab toggle). When off, nothing runs and no card appears. It is the video-reading sibling of Supermail's other AI helpers, AI Summary (summarizes a thread) and AI Filtering (triages new mail).

Where to find it

In development and off by default — reveal it from Settings → Features. The summary card then appears inside the message that carries the link.

How it behaves

How it works

Shaped by Supermail's architecture (the mailbox UI lives in the vendored plugin; the AI keys + spend live only in the main process):

• The paid AI runs in the main process. The renderer sends only {threadId, host, videoId} — it never sends the message body or the video URL. Main orchestrates the whole pipeline: fetch the video's captions → if none, download the audio and auto-transcribe it → summarize the resulting text with the LLM → return the summary. The LLM key and the transcription key never reach the renderer.
• Captions first, transcription as the fallback. Loom and Vimeo both expose captions for most videos; when they exist, they are used (the card notes "Captions"). When a video has no usable captions, the audio is transcribed instead via the same Groq Whisper engine the app's voice features use (the card notes "Audio transcription"). If transcription isn't available either (no key, download failed, audio too big), the card shows an honest note — it never hangs and never silently shows nothing.
• Joined by a desktop-only bridge/IPC channel (SUPERMAIL_VIDEO_SUMMARY), blocked over the mobile web bridge: it spends AI credit and works off the desktop app's keys.
• The summary renders as a dismissible card at the top of the message reading area, with a "Watch video" link to the source.

Code: src/main/services/supermail/supermail-video-summary-*.ts (main) and src/plugins/supermail/ui/src/features/conversation/ (use-video-summary.ts + video-summary-card.tsx + detect-video-links.ts).

When it runs — automatic, incoming only

• The card appears automatically when you open a message that contains a Loom or Vimeo link and the feature is on. There is no button to press.
• It only runs for incoming messages (sent to you) — outgoing and draft messages never trigger a paid call.
• It summarizes one video per message (the first Loom/Vimeo link); other links are left untouched.
• To avoid re-billing, the result is cached in memory per (message, video) for the app session — re-opening the same message reuses the summary instead of re-running.

Cost + safety

• Cost-capped. Every summary call is cost-tracked (source: supermail-video-summary) and refused with a friendly "daily limit reached" state once a main-side daily cap (a constant, not renderer-tunable) is hit.
• Video content is untrusted and fenced. Captions/transcripts are third-party content, not instructions: they are wrapped in sentinel fences (forged fence markers neutralized), scrubbed of contact details (email / phone / SSN patterns), truncated to a bounded window, and the model is told to output only a summary. A prompt-injection attempt inside a video can at most alter the summary's prose — it executes nothing.
• It fetches only the video id, from the fixed host. The fetcher never follows the raw message URL. It extracts the video id (Loom 32-char hex / Vimeo numeric) and pulls the transcript only from Loom's GraphQL transcript endpoint or Vimeo's player config — with a bounded timeout and one retry.
• Where your video's text goes. The caption/transcript text is sent to the app's AI providers (LLM + transcription) to produce the summary. This only happens on a message you open, with the feature on.
• Fails honest. Feature off / no key / fetch failed / too big → a clear "unavailable" state, never a silent blank.

Related

The other Supermail AI helpers each have their own page — AI Filtering, Catch me up, Assistants and the AI Digest.