PM Import Framework (import data into Mission Control boards)

What it is

Omniscio's PM system includes a universal import framework that converts data from external sources (CSV, Excel, Google Sheets, Trello, Jira, Asana, Monday, Linear, Notion, ClickUp, and Airtable) into local PM boards. The framework includes an import wizard UI, CLI routes for programmatic access, and connected source refresh.

Where to find it

The import wizard is the user-facing entry point: a five-step flow that walks you through selecting a source type, configuring the connection or file, previewing the data (with column type overrides and reorder/rename/exclude controls), executing the import, and reading the report. It is gated behind the pm-import unreleased feature flag.

A batch run of several sources at once shows its own progress screen — one row per platform with a progress bar and stage label — which transitions to a completion summary when every job reaches a terminal state.

Connected sources (Google Sheets and the API-based adapters) refresh on a schedule with no UI at all, and the same pipeline is reachable over the CLI routes under /pm/import/ for agents.

How it behaves

The 7-stage pipeline

The import framework is a 7-stage pipeline: extract, map, validate, execute, report, landing board, and preset application. Every source adapter plugs into the same pipeline by implementing the ImportSourcePlugin interface — the adapter converts its native data into a canonical Intermediate Representation (IR), and the pipeline handles the rest: column type mapping, validation, atomically writing to SQLite, tracking the job, and recording entity mappings for re-import dedup.

Stages:

1. Extract — the adapter reads the source (a file, an API, etc.) and produces an ImportManifest containing one or more ImportBoards, each with groups, columns, items, and column values.
2. Map — columns are mapped to PM column types. Known sources (Monday, Jira) have explicit type maps; unknown sources fall back to the column-type inference engine (the same one used by the existing data-transfer CSV import).
3. Validate — the manifest is checked for: non-empty board names, no duplicate sourceIds, items reference valid groups, column values reference valid columns, item count within the 5,000 per-board limit.
4. Execute — each board is imported in its own SQLite transaction, with items batched in per-batch transactions (50 items each). Between batches the event loop yields so IPC progress pushes reach the renderer. Boards, groups, columns, and items are created via the standard pm-queries-board/group/column/item.ts upsert functions. Entity mappings are recorded for dedup. A board-level failure isolates to that board (per-board atomicity); previously committed boards survive.
5. Report — an ImportReport is returned with counts (boards/groups/columns/items created, skipped, failed), warnings, errors, the entity source-to-local ID map, and an optional contentSummary array describing what content types were found in the source and which were imported vs. skipped (e.g., "Databases: 3 imported; Pages: 2 imported").
6. Landing board — selects the best board to navigate to after import and ensures it has a table view.
7. Preset application (non-fatal) — for Linear and Jira imports, auto-applies the matching engineering preset (engineering-linear / engineering-jira), sets status column settings to the engineering workflow, and remaps imported status values from text to indexed format using state type metadata from the adapter.

Conflict strategies

When re-importing data that overlaps with previously imported entities:

• Skip (default) — if the entity was already imported (found via entity mapping + liveness check), skip it entirely.
• Overwrite — reuse the existing local ID and upsert over it. Changed items count as itemsUpdated.
• Merge — reuse the existing local ID but compare name, group, and all column values against existing rows before writing. Unchanged items are skipped (itemsUnchanged); changed items are upserted (itemsUpdated). The import wizard shows a SegmentedControl strategy picker when existing items are detected, with a per-board preview of new vs. existing item counts.

Error recovery

The pipeline has four error recovery mechanisms:

• Retry with backoff — extraction (extractToIR) is wrapped in retryWithBackoff (3 attempts, 1s base, 16s max, full jitter). Permanent errors (400/401/403/404/422) fail immediately; transient errors (5xx, 429, network) are retried. Per-board extraction in buildPmManifest also retries each board independently.
• Per-board resilience — if one board fails extraction after retries, the pipeline continues with the remaining boards. Failed boards appear as extractionWarnings on the manifest and in the report. If ALL boards fail with the same error class (e.g. 401), the error surfaces as a single primary message.
• Checkpointing — after successful extraction, the manifest is serialized and saved as checkpoint_data on the job row. If the import fails or the app crashes during execution, a subsequent import from the same source will detect the checkpoint and skip extraction. Checkpoints are cleared on success and auto-cleaned after 7 days.
• Humanized errors — raw API/HTTP error strings in the report are mapped to user-friendly messages with recovery hints via import-error-humanizer.ts. Raw errors are preserved in the job's error_log for debugging.

The batch progress screen shows a rate-limited stage (ember/orange) when the pipeline retries after a transient error, so the user knows the import is waiting, not stalled.

Job tracking

Every pipeline run creates a row in pm_import_jobs tracking source, status (pending / extracting / mapping / validating / importing / completed / partial / failed / cancelled), conflict strategy, entity counts, error log, and timestamps. Entity mappings (pm_import_entity_map) persist across jobs so re-imports of the same source can detect previously imported entities.

Connected sources and scheduled refresh

Google Sheets and API-based adapters can be saved as connected sources that refresh automatically. Sources are stored in pm_import_sources with connection config, refresh schedule (hourly/daily/weekly/manual), and sync timestamps. The refresh scheduler checks every 5 minutes and re-runs the pipeline with overwrite strategy for due sources. A refresh runs unattended, so it inherits the third-party PII consent recorded on the source row rather than asking again — see below.

Third-party PII consent (F005)

Importing from a platform — Notion, Trello, Asana, Jira, Monday, ClickUp, Linear, Airtable — copies other people's names, email addresses and phone numbers out of that account and into the local database, alongside the tasks. The executor refuses such an import unless the caller passes thirdPartyPiiConsent.

Importing a document the user already owns is exempt: csv, excel and google-sheets (their own sheet, read with their own credentials) carry no account directory. The exempt set lives in PII_CONSENT_EXEMPT_SOURCES (src/shared/types/pm-import.ts) and is checked with importSourceRequiresPiiConsent(), which both the executor and the wizard call, so the UI and the gate always agree. It is an allow-list on purpose — fail-CLOSED, so a newly added adapter needs consent by default.

Where the acknowledgment comes from:

Caller	Source of consent
Import wizard	A toggle on the preview step, shown only for a non-exempt source. Start Import stays disabled until it is on.
POST /pm/import/execute, /pm/import/batch, the IPC handlers	thirdPartyPiiConsent in the request — an API caller opts in explicitly.
Scheduled refresh + /pm/import/sources/:id/refresh	Inherited from pm_import_sources.third_party_pii_consent_at, recorded when the source was created with thirdPartyPiiConsent: true. A source nobody acknowledged stays blocked.

A blocked import returns a failed report naming consent as the reason — not a thrown error. The refusal has to carry the required warnings and entityMap fields, because the pipeline spreads ...report.warnings when it merges warning sources; omitting them threw report.warnings is not iterable and replaced the real reason with that raw text.

Batch import and progress screen

Multiple sources can be imported simultaneously via the PM_IMPORT_BATCH_EXECUTE IPC channel, which processes adapters in chunks of 3 (IMPORT_CONCURRENCY) via Promise.allSettled, with a 5-second cooldown between batch calls. Real-time progress is pushed to the renderer via two push channels:

• PM_IMPORT_PROGRESS — stage updates per adapter (connecting/fetching/rate-limited/mapping/importing/completed/failed)
• PM_IMPORT_DISAMBIGUATION_NEEDED — column mapping ambiguity that requires user input

The progress screen (ImportProgressScreen.tsx) renders a row per platform with a progress bar and stage label. When all jobs reach a terminal state the screen auto-transitions to a completion summary (3-second delay, skipped when all failed).

Disambiguation uses a deferred-promise pattern: the batch handler pauses the affected adapter, emits a push event, and waits up to 5 minutes for the user to resolve the column mapping via PM_IMPORT_RESOLVE_DISAMBIGUATION. Other adapters continue unblocked.

Content-type detection

API-based adapters report what content types exist in the source workspace via a contentSummary array on the ImportManifest. Each entry has a type, label, count, and supported flag. The wizard uses this to:

• Show an info banner when unsupported content exists ("3 items can be imported. Some content is not available for import.")
• Dim unsupported entities in the entity list (unselectable, with "Not available for import" label and invisible checkbox)
• Only include supported entities in select all/none
• Show a content summary section on the completion screen

Notion uses live API detection (Search API for pages, parallel with database listing) and lists each page and database as an individually selectable, supported entity. Other adapters declare known unsupported types statically (e.g., Jira: Confluence pages, roadmaps, sprints). File-based adapters (CSV, Excel, Google Sheets) do not set contentSummary.

Column mapping (reorder, rename, exclude)

The preview step exposes per-column controls beyond type overrides:

• Reorder — drag-and-drop (via useDragReorder) to change the column order before import. The pipeline sorts columns by their position field.
• Rename — inline text input sets a displayName that overrides the source column title in the imported board.
• Exclude — checkbox toggles a column's excluded flag; excluded columns are filtered out before import. The Start Import button is disabled when all columns are excluded.

These preferences are stored as ColumnMappingOverrides (a Record<sourceId, ColumnMappingEntry> with targetType, position, displayName, and excluded fields). On successful import with a connected source, mappings are persisted to pm_import_mappings (position and display_name columns) so they are restored on re-import. The PM_IMPORT_MAPPINGS_LIST IPC channel loads saved mappings for a source.

The backend applies column mappings in applyColumnMappings() (Stage 2 of the pipeline, after applyColumnOverrides): filter excluded → apply overrides → sort by position → normalize to 0-based sequential positions.

Import wizard UI

The wizard is the Mission Control plugin's own — mounted from src/plugins/mission-control/web/features/import/ (reached via the plugin shell's routes.tsx → ImportPage.tsx) and bridged to the main process through import-bridge.ts. It walks users through: select source type, configure connection/file, preview data with column type overrides and reorder/rename/exclude controls, execute import, and view report. For a third-party platform the preview step also carries the PII consent toggle, which blocks Start Import until the user acknowledges it. Gated behind the pm-import unreleased feature flag.

The renderer's own copy of this wizard (src/renderer/src/features/pm/ImportWizard.tsx and its cluster) was unreachable and was deleted 2026-09-29; only ImportPiiConsentNotice.tsx survives, still used by the Asana wizard in src/renderer/src/features/mission-control/setup/AsanaImportWizard.tsx.

For agents

Available adapters

CSV (csvAdapter) — the reference adapter. Reads RFC 4180 CSV, strips UTF-8 BOM, deduplicates headers, detects groups from a "Group"/"Section" column, infers column types. File size cap: 50 MB. Cell length cap: 20,000 characters.

Google Sheets (googleSheetsAdapter) — SOP-aware adapter using the Google Sheets API. Tab color classification: green hue (80-160 degrees HSL) = import, others = skip. Header background color maps to column types (8 HSL hue ranges: red=status, orange=dropdown, yellow=date, green=checkbox, blue=text, purple=people, brown=text, gray=numbers). Supports connected refresh (re-runs with overwrite strategy on schedule).

Excel (excelAdapter) — ExcelJS-based .xlsx parser. Filters hidden worksheets, infers column types from headers, handles merged cells. File size cap: 50 MB.

Trello (trelloAdapter) — API-key authenticated. Imports boards with lists as groups, cards as items, and labels/due dates/checklists as columns.

Jira (jiraAdapter) — API-key authenticated. Imports projects with issues as items, sprints as groups, and field mappings for status, priority, and story points. Auto-applies the engineering-jira preset.

Asana (asanaAdapter) — API-key authenticated. Imports projects with sections as groups and tasks as items, including assignees, due dates, and custom fields.

Monday (mondayAdapter) — API-key authenticated. Imports boards with groups and items, mapping Monday column types to PM column types.

Linear (linearAdapter) — API-key authenticated. Imports teams/projects with cycles as groups and issues as items. Auto-applies the engineering-linear preset.

Notion (notionAdapter) — API-key authenticated. Imports databases as projects with properties as columns, and standalone pages as project items with their text content extracted via the Notion blocks API. Each page and database appears as an individually selectable entity in the import wizard.

ClickUp (clickupAdapter) — API-key authenticated. Imports spaces/lists with folders as groups and tasks as items, mapping ClickUp field types to PM columns.

Airtable (airtableAdapter) — API-key authenticated. Imports bases/tables as boards with views as groups and records as items, mapping Airtable field types to PM columns.

How an agent uses it

Via CLI routes (recommended):

Method	Route	Purpose
GET	/pm/import/adapters	List available adapters
POST	/pm/import/preview	Parse source into a manifest preview (no DB write)
POST	/pm/import/execute	Run the full import pipeline
GET	/pm/import/sources	List connected sources
POST	/pm/import/sources	Create a connected source
PATCH	/pm/import/sources/:id	Update a source
DELETE	/pm/import/sources/:id	Soft-delete a source
POST	/pm/import/sources/:id/refresh	Trigger manual refresh

Via code (direct):

import { runImportPipeline } from 'src/main/services/pm/import/import-pipeline'
import { csvAdapter } from 'src/main/services/pm/import/adapters/csv-adapter'

const report = await runImportPipeline(db, {
  source: csvAdapter,
  input: { filePath: '/path/to/data.csv' },
  conflictStrategy: 'skip'
})

Writing a new source adapter

Implement ImportSourcePlugin from src/shared/types/pm-import.ts:

interface ImportSourcePlugin {
  id: string // e.g. 'jira'
  displayName: string // e.g. 'Jira Cloud'
  inputMethod: 'file-drop' | 'oauth' | 'api-key' | 'google-auth'
  supportedTargets: ('pm' | 'vault')[]
  extractToIR(input: ImportSourceInput): Promise<ImportManifest>
  validateCredentials?(credentials: Record<string, string>): Promise<boolean>
  listImportableEntities?(credentials: Record<string, string>): Promise<ImportableEntity[]>
  supportsRefresh?: boolean // true if the adapter supports connected-source refresh
  fetchComments?(
    credentials: Record<string, string>,
    boardId: string,
    since?: string
  ): Promise<ImportComment[]>
}

The adapter only needs to implement extractToIR — produce a valid ImportManifest and the pipeline handles mapping, validation, execution, and reporting. Optionally add a source type map entry in import-column-mapper.ts for better column type fidelity.

API-based adapters validate responses at the fetch boundary using Zod schemas with .passthrough() (validates fields the adapter reads, ignores extras). Each adapter defines a curried parse factory — e.g. parseClickup(schema, resource) returns (raw: unknown) => T — and passes it as the transport layer's parse callback.

Key files

• src/shared/types/pm-import.ts — all shared types and interfaces
• src/main/services/pm/import/import-pipeline.ts — the 7-stage orchestrator
• src/main/services/pm/import/import-executor.ts — per-board/per-batch transactional SQLite writer
• src/main/services/pm/import/import-validator.ts — manifest validation
• src/main/services/pm/import/import-column-mapper.ts — column type mapping
• src/main/services/pm/import/import-job-tracker.ts — job + entity mapping + checkpoint CRUD
• src/main/services/pm/import/import-error-humanizer.ts — raw error to friendly message mapping
• src/main/services/pm/import/adapter-registry.ts — lazy adapter discovery
• src/main/services/pm/import/import-source-queries.ts — connected source CRUD
• src/main/services/pm/import/import-refresh-scheduler.ts — scheduled refresh
• src/main/services/pm/import/adapters/csv-adapter.ts — CSV adapter
• src/main/services/pm/import/adapters/google-sheets-adapter.ts — Google Sheets SOP-aware adapter
• src/main/services/pm/import/adapters/excel-adapter.ts — Excel (.xlsx) adapter
• src/main/services/pm/import/adapters/trello-adapter.ts — Trello adapter
• src/main/services/pm/import/adapters/jira-adapter.ts — Jira adapter
• src/main/services/pm/import/adapters/asana-adapter.ts — Asana adapter
• src/main/services/pm/import/adapters/monday-adapter.ts — Monday.com adapter
• src/main/services/pm/import/adapters/linear-adapter.ts — Linear adapter
• src/main/services/pm/import/adapters/notion-adapter.ts — Notion adapter
• src/main/services/pm/import/adapters/clickup-adapter.ts — ClickUp adapter
• src/main/services/pm/import/adapters/airtable-adapter.ts — Airtable adapter
• src/main/ipc/pm-import-handlers.ts — IPC handlers (20 channels: 16 in pm-import.ts + 4 in pm.ts)
• src/main/services/cli/cli-server-pm-import-routes.ts — 8 CLI routes
• src/plugins/mission-control/web/features/import/ — the import wizard UI (in the plugin)
• src/main/db/migrations/20260721181746-pm-import-tracking.ts — job tracking tables
• src/main/db/migrations/20260722191922-pm-import-sources-and-mappings.ts — connected source tables
• src/main/db/migrations/20260813023345-add-position-and-display-name-to-pm-import-mappings.ts — column position + display name

Related

The import framework lands everything into Mission Control boards, so mission-control.md is the parent page for what happens to the data once it arrives, and data-transfer.md covers the older CSV import path whose column-type inference engine this pipeline reuses. When the data is coming from a platform someone is migrating off, pm-onboarding-features.md describes the first-run import entry screen and the presets it applies to each imported board. The full list of library pages is in INDEX.md.