--- title: PM Import Framework (import data into Mission Control boards) --- # PM Import Framework (import data into Mission Control boards) ## What it is Omniscio's PM system includes a **universal import framework** that converts data from external sources (CSV, Excel, Google Sheets, Trello, Jira, Asana, Monday, Linear, Notion, ClickUp, and Airtable) into local PM boards. The framework includes an import wizard UI, CLI routes for programmatic access, and connected source refresh. ## Where to find it The import wizard is the user-facing entry point: a five-step flow that walks you through selecting a source type, configuring the connection or file, previewing the data (with column type overrides and reorder/rename/exclude controls), executing the import, and reading the report. It is gated behind the `pm-import` unreleased feature flag. A batch run of several sources at once shows its own progress screen — one row per platform with a progress bar and stage label — which transitions to a completion summary when every job reaches a terminal state. Connected sources (Google Sheets and the API-based adapters) refresh on a schedule with no UI at all, and the same pipeline is reachable over the CLI routes under `/pm/import/` for agents. ## How it behaves ### The 7-stage pipeline The import framework is a **7-stage pipeline**: extract, map, validate, execute, report, landing board, and preset application. Every source adapter plugs into the same pipeline by implementing the `ImportSourcePlugin` interface — the adapter converts its native data into a **canonical Intermediate Representation (IR)**, and the pipeline handles the rest: column type mapping, validation, atomically writing to SQLite, tracking the job, and recording entity mappings for re-import dedup. **Stages:** 1. **Extract** — the adapter reads the source (a file, an API, etc.) and produces an `ImportManifest` containing one or more `ImportBoard`s, each with groups, columns, items, and column values. 2. **Map** — columns are mapped to PM column types. Known sources (Monday, Jira) have explicit type maps; unknown sources fall back to the column-type inference engine (the same one used by the existing data-transfer CSV import). 3. **Validate** — the manifest is checked for: non-empty board names, no duplicate sourceIds, items reference valid groups, column values reference valid columns, item count within the 5,000 per-board limit. 4. **Execute** — the entire import runs inside a single SQLite transaction. Boards, groups, columns, and items are created via the standard `pm-queries-board/group/column/item.ts` upsert functions. Entity mappings are recorded for dedup. Failed items are logged but do not abort the batch. 5. **Report** — an `ImportReport` is returned with counts (boards/groups/columns/items created, skipped, failed), warnings, errors, the entity source-to-local ID map, and an optional `contentSummary` array describing what content types were found in the source and which were imported vs. skipped (e.g., "Databases: 3 imported; Pages: 2 imported"). 6. **Landing board** — selects the best board to navigate to after import and ensures it has a table view. 7. **Preset application** (non-fatal) — for Linear and Jira imports, auto-applies the matching engineering preset (`engineering-linear` / `engineering-jira`), sets status column settings to the engineering workflow, and remaps imported status values from text to indexed format using state type metadata from the adapter. ### Conflict strategies When re-importing data that overlaps with previously imported entities: - **Skip** (default) — if the entity was already imported (found via entity mapping + liveness check), skip it entirely. - **Overwrite** — reuse the existing local ID and upsert over it. Changed items count as `itemsUpdated`. - **Merge** — reuse the existing local ID but compare name, group, and all column values against existing rows before writing. Unchanged items are skipped (`itemsUnchanged`); changed items are upserted (`itemsUpdated`). The import wizard shows a SegmentedControl strategy picker when existing items are detected, with a per-board preview of new vs. existing item counts. ### Error recovery The pipeline has four error recovery mechanisms: - **Retry with backoff** — extraction (`extractToIR`) is wrapped in `retryWithBackoff` (3 attempts, 1s base, 16s max, full jitter). Permanent errors (400/401/403/404/422) fail immediately; transient errors (5xx, 429, network) are retried. Per-board extraction in `buildPmManifest` also retries each board independently. - **Per-board resilience** — if one board fails extraction after retries, the pipeline continues with the remaining boards. Failed boards appear as `extractionWarnings` on the manifest and in the report. If ALL boards fail with the same error class (e.g. 401), the error surfaces as a single primary message. - **Checkpointing** — after successful extraction, the manifest is serialized and saved as `checkpoint_data` on the job row. If the import fails or the app crashes during execution, a subsequent import from the same source will detect the checkpoint and skip extraction. Checkpoints are cleared on success and auto-cleaned after 7 days. - **Humanized errors** — raw API/HTTP error strings in the report are mapped to user-friendly messages with recovery hints via `import-error-humanizer.ts`. Raw errors are preserved in the job's `error_log` for debugging. The batch progress screen shows a **rate-limited** stage (ember/orange) when the pipeline retries after a transient error, so the user knows the import is waiting, not stalled. ### Job tracking Every pipeline run creates a row in `pm_import_jobs` tracking source, status (pending / extracting / mapping / validating / importing / completed / partial / failed / cancelled), conflict strategy, entity counts, error log, and timestamps. Entity mappings (`pm_import_entity_map`) persist across jobs so re-imports of the same source can detect previously imported entities. ### Connected sources and scheduled refresh Google Sheets and API-based adapters can be saved as **connected sources** that refresh automatically. Sources are stored in `pm_import_sources` with connection config, refresh schedule (hourly/daily/weekly/manual), and sync timestamps. The refresh scheduler checks every 5 minutes and re-runs the pipeline with `overwrite` strategy for due sources. A refresh runs unattended, so it inherits the third-party PII consent recorded on the source row rather than asking again — see below. ### Third-party PII consent (F005) Importing from a **platform** — Notion, Trello, Asana, Jira, Monday, ClickUp, Linear, Airtable — copies other people's names, email addresses and phone numbers out of that account and into the local database, alongside the tasks. The executor refuses such an import unless the caller passes `thirdPartyPiiConsent`. Importing a **document the user already owns** is exempt: `csv`, `excel` and `google-sheets` (their own sheet, read with their own credentials) carry no account directory. The exempt set lives in `PII_CONSENT_EXEMPT_SOURCES` (`src/shared/types/pm-import.ts`) and is checked with `importSourceRequiresPiiConsent()`, which both the executor and the wizard call, so the UI and the gate always agree. It is an allow-list on purpose — fail-CLOSED, so a newly added adapter needs consent by default. Where the acknowledgment comes from: | Caller | Source of consent | | --- | --- | | Import wizard | A toggle on the **preview** step, shown only for a non-exempt source. Start Import stays disabled until it is on. | | `POST /pm/import/execute`, `/pm/import/batch`, the IPC handlers | `thirdPartyPiiConsent` in the request — an API caller opts in explicitly. | | Scheduled refresh + `/pm/import/sources/:id/refresh` | Inherited from `pm_import_sources.third_party_pii_consent_at`, recorded when the source was created with `thirdPartyPiiConsent: true`. A source nobody acknowledged stays blocked. | A blocked import returns a **failed report** naming consent as the reason — not a thrown error. The refusal has to carry the required `warnings` and `entityMap` fields, because the pipeline spreads `...report.warnings` when it merges warning sources; omitting them threw `report.warnings is not iterable` and replaced the real reason with that raw text. ### Batch import and progress screen Multiple sources can be imported simultaneously via the `PM_IMPORT_BATCH_EXECUTE` IPC channel, which processes adapters in chunks of 3 (`IMPORT_CONCURRENCY`) via `Promise.allSettled`, with a 5-second cooldown between batch calls. Real-time progress is pushed to the renderer via two push channels: - `PM_IMPORT_PROGRESS` — stage updates per adapter (connecting/fetching/rate-limited/mapping/importing/completed/failed) - `PM_IMPORT_DISAMBIGUATION_NEEDED` — column mapping ambiguity that requires user input The progress screen (`ImportProgressScreen.tsx`) renders a row per platform with a progress bar and stage label. When all jobs reach a terminal state the screen auto-transitions to a completion summary (3-second delay, skipped when all failed). **Disambiguation** uses a deferred-promise pattern: the batch handler pauses the affected adapter, emits a push event, and waits up to 5 minutes for the user to resolve the column mapping via `PM_IMPORT_RESOLVE_DISAMBIGUATION`. Other adapters continue unblocked. ### Content-type detection API-based adapters report what content types exist in the source workspace via a `contentSummary` array on the `ImportManifest`. Each entry has a type, label, count, and supported flag. The wizard uses this to: - Show an info banner when unsupported content exists ("3 items can be imported. Some content is not available for import.") - Dim unsupported entities in the entity list (unselectable, with "Not available for import" label and invisible checkbox) - Only include supported entities in select all/none - Show a content summary section on the completion screen Notion uses live API detection (Search API for pages, parallel with database listing) and lists each page and database as an individually selectable, supported entity. Other adapters declare known unsupported types statically (e.g., Jira: Confluence pages, roadmaps, sprints). File-based adapters (CSV, Excel, Google Sheets) do not set `contentSummary`. ### Column mapping (reorder, rename, exclude) The preview step exposes per-column controls beyond type overrides: - **Reorder** — drag-and-drop (via `useDragReorder`) to change the column order before import. The pipeline sorts columns by their `position` field. - **Rename** — inline text input sets a `displayName` that overrides the source column title in the imported board. - **Exclude** — checkbox toggles a column's `excluded` flag; excluded columns are filtered out before import. The Start Import button is disabled when all columns are excluded. These preferences are stored as `ColumnMappingOverrides` (a `Record` with `targetType`, `position`, `displayName`, and `excluded` fields). On successful import with a connected source, mappings are persisted to `pm_import_mappings` (position and display_name columns) so they are restored on re-import. The `PM_IMPORT_MAPPINGS_LIST` IPC channel loads saved mappings for a source. The backend applies column mappings in `applyColumnMappings()` (Stage 2 of the pipeline, after `applyColumnOverrides`): filter excluded → apply overrides → sort by position → normalize to 0-based sequential positions. ### Import wizard UI A 5-step wizard (`src/renderer/src/features/pm/ImportWizard.tsx`) walks users through: select source type, configure connection/file, preview data with column type overrides and reorder/rename/exclude controls, execute import, and view report. For a third-party platform the preview step also carries the PII consent toggle, which blocks Start Import until the user acknowledges it. Gated behind the `pm-import` unreleased feature flag. ## For agents ### Available adapters **CSV** (`csvAdapter`) — the reference adapter. Reads RFC 4180 CSV, strips UTF-8 BOM, deduplicates headers, detects groups from a "Group"/"Section" column, infers column types. File size cap: 50 MB. Cell length cap: 20,000 characters. **Google Sheets** (`googleSheetsAdapter`) — SOP-aware adapter using the Google Sheets API. Tab color classification: green hue (80-160 degrees HSL) = import, others = skip. Header background color maps to column types (8 HSL hue ranges: red=status, orange=dropdown, yellow=date, green=checkbox, blue=text, purple=people, brown=text, gray=numbers). Supports connected refresh (re-runs with `overwrite` strategy on schedule). **Excel** (`excelAdapter`) — ExcelJS-based .xlsx parser. Filters hidden worksheets, infers column types from headers, handles merged cells. File size cap: 50 MB. **Trello** (`trelloAdapter`) — API-key authenticated. Imports boards with lists as groups, cards as items, and labels/due dates/checklists as columns. **Jira** (`jiraAdapter`) — API-key authenticated. Imports projects with issues as items, sprints as groups, and field mappings for status, priority, and story points. Auto-applies the `engineering-jira` preset. **Asana** (`asanaAdapter`) — API-key authenticated. Imports projects with sections as groups and tasks as items, including assignees, due dates, and custom fields. **Monday** (`mondayAdapter`) — API-key authenticated. Imports boards with groups and items, mapping Monday column types to PM column types. **Linear** (`linearAdapter`) — API-key authenticated. Imports teams/projects with cycles as groups and issues as items. Auto-applies the `engineering-linear` preset. **Notion** (`notionAdapter`) — API-key authenticated. Imports databases as projects with properties as columns, and standalone pages as project items with their text content extracted via the Notion blocks API. Each page and database appears as an individually selectable entity in the import wizard. **ClickUp** (`clickupAdapter`) — API-key authenticated. Imports spaces/lists with folders as groups and tasks as items, mapping ClickUp field types to PM columns. **Airtable** (`airtableAdapter`) — API-key authenticated. Imports bases/tables as boards with views as groups and records as items, mapping Airtable field types to PM columns. ### How an agent uses it **Via CLI routes** (recommended): | Method | Route | Purpose | | ------ | -------------------------------- | -------------------------------------------------- | | GET | `/pm/import/adapters` | List available adapters | | POST | `/pm/import/preview` | Parse source into a manifest preview (no DB write) | | POST | `/pm/import/execute` | Run the full import pipeline | | GET | `/pm/import/sources` | List connected sources | | POST | `/pm/import/sources` | Create a connected source | | PATCH | `/pm/import/sources/:id` | Update a source | | DELETE | `/pm/import/sources/:id` | Soft-delete a source | | POST | `/pm/import/sources/:id/refresh` | Trigger manual refresh | **Via code** (direct): ```typescript import { runImportPipeline } from 'src/main/services/pm/import/import-pipeline' import { csvAdapter } from 'src/main/services/pm/import/adapters/csv-adapter' const report = await runImportPipeline(db, { source: csvAdapter, input: { filePath: '/path/to/data.csv' }, conflictStrategy: 'skip' }) ``` ### Writing a new source adapter Implement `ImportSourcePlugin` from `src/shared/types/pm-import.ts`: ```typescript interface ImportSourcePlugin { id: string // e.g. 'jira' displayName: string // e.g. 'Jira Cloud' inputMethod: 'file-drop' | 'oauth' | 'api-key' | 'google-auth' supportedTargets: ('pm' | 'vault')[] extractToIR(input: ImportSourceInput): Promise validateCredentials?(credentials: Record): Promise listImportableEntities?(credentials: Record): Promise supportsRefresh?: boolean // true if the adapter supports connected-source refresh fetchComments?( credentials: Record, boardId: string, since?: string ): Promise } ``` The adapter only needs to implement `extractToIR` — produce a valid `ImportManifest` and the pipeline handles mapping, validation, execution, and reporting. Optionally add a source type map entry in `import-column-mapper.ts` for better column type fidelity. API-based adapters validate responses at the fetch boundary using Zod schemas with `.passthrough()` (validates fields the adapter reads, ignores extras). Each adapter defines a curried parse factory — e.g. `parseClickup(schema, resource)` returns `(raw: unknown) => T` — and passes it as the transport layer's `parse` callback. ### Key files - `src/shared/types/pm-import.ts` — all shared types and interfaces - `src/main/services/pm/import/import-pipeline.ts` — the 7-stage orchestrator - `src/main/services/pm/import/import-executor.ts` — transaction-wrapped SQLite writer - `src/main/services/pm/import/import-validator.ts` — manifest validation - `src/main/services/pm/import/import-column-mapper.ts` — column type mapping - `src/main/services/pm/import/import-job-tracker.ts` — job + entity mapping + checkpoint CRUD - `src/main/services/pm/import/import-error-humanizer.ts` — raw error to friendly message mapping - `src/main/services/pm/import/adapter-registry.ts` — lazy adapter discovery - `src/main/services/pm/import/import-source-queries.ts` — connected source CRUD - `src/main/services/pm/import/import-refresh-scheduler.ts` — scheduled refresh - `src/main/services/pm/import/adapters/csv-adapter.ts` — CSV adapter - `src/main/services/pm/import/adapters/google-sheets-adapter.ts` — Google Sheets SOP-aware adapter - `src/main/services/pm/import/adapters/excel-adapter.ts` — Excel (.xlsx) adapter - `src/main/services/pm/import/adapters/trello-adapter.ts` — Trello adapter - `src/main/services/pm/import/adapters/jira-adapter.ts` — Jira adapter - `src/main/services/pm/import/adapters/asana-adapter.ts` — Asana adapter - `src/main/services/pm/import/adapters/monday-adapter.ts` — Monday.com adapter - `src/main/services/pm/import/adapters/linear-adapter.ts` — Linear adapter - `src/main/services/pm/import/adapters/notion-adapter.ts` — Notion adapter - `src/main/services/pm/import/adapters/clickup-adapter.ts` — ClickUp adapter - `src/main/services/pm/import/adapters/airtable-adapter.ts` — Airtable adapter - `src/renderer/src/features/pm/ImportProgressScreen.tsx` — batch import progress screen - `src/renderer/src/features/pm/useImportProgress.ts` — push-driven progress hook - `src/main/ipc/pm-import-handlers.ts` — IPC handlers (20 channels: 16 in pm-import.ts + 4 in pm.ts) - `src/main/services/cli/cli-server-pm-import-routes.ts` — 8 CLI routes - `src/renderer/src/features/pm/ImportWizard.tsx` — import wizard UI - `src/main/db/migrations/20260721181746-pm-import-tracking.ts` — job tracking tables - `src/main/db/migrations/20260722191922-pm-import-sources-and-mappings.ts` — connected source tables - `src/main/db/migrations/20260813023345-add-position-and-display-name-to-pm-import-mappings.ts` — column position + display name ## Related The import framework lands everything into Mission Control boards, so [mission-control.md](mission-control.md) is the parent page for what happens to the data once it arrives, and [data-transfer.md](data-transfer.md) covers the older CSV import path whose column-type inference engine this pipeline reuses. When the data is coming from a platform someone is migrating off, [pm-onboarding-features.md](pm-onboarding-features.md) describes the first-run import entry screen and the presets it applies to each imported board. The full list of library pages is in [INDEX.md](INDEX.md).