Automate issuer overlay creation from ticker searches

doc/issuer-overlay-automation.md

@@ -0,0 +1,63 @@
+# Issuer Overlay Automation
+
+## Overview
+
+Issuer overlays are now runtime data, not just checked-in JSON files.
+
+The system creates or updates a global overlay for a ticker after an explicit user ticker submit triggers filing sync. Overlay generation is additive-only:
+
+- it extends existing canonical surfaces
+- it never synthesizes brand-new surfaces
+- it never auto-deletes prior mappings
+
+## Trigger Flow
+
+1. An explicit ticker submit calls `POST /api/tickers/ensure`.
+2. The endpoint ensures an `issuer_overlay` row exists for the ticker.
+3. If the ticker has no ready taxonomy snapshots, no overlay build history, or its latest ready snapshot was hydrated with an older overlay revision, the endpoint queues `sync_filings`.
+4. `sync_filings` hydrates taxonomy snapshots with the current active overlay revision.
+5. The task then generates a new overlay candidate from stored residual extension concepts.
+6. If a new overlay revision is published, the task immediately rehydrates recent filings with that revision before queueing search indexing.
+
+## Storage
+
+Two tables back the feature:
+
+- `issuer_overlay`: one row per ticker with status, active revision, error state, and summary stats
+- `issuer_overlay_revision`: immutable overlay revisions with normalized definition JSON and diagnostics
+
+Each `filing_taxonomy_snapshot` also stores `issuer_overlay_revision_id` so the sync pipeline can detect when a filing needs rehydration because the overlay changed.
+
+## Generator Rules
+
+The generator samples up to 12 ready `10-K`/`10-Q` snapshots from the last 3 years and chooses the most common detected fiscal pack in that window.
+
+It only promotes a concept when all of the following are true:
+
+- the concept is an extension concept
+- the concept is still residual/unmapped in the stored taxonomy output
+- the concept is not abstract
+- the concept has a consistent statement kind across filings
+- the concept appears in at least 2 distinct filings
+- the concept can be mapped uniquely to an existing canonical surface by:
+  - authoritative concept local-name match, or
+  - exact local-name match against canonical source/authoritative concepts
+
+Accepted concepts are appended to `allowed_source_concepts` for the resolved surface in the runtime overlay definition.
+
+## Merge Order
+
+The hydrator merges surfaces in this order:
+
+1. pack primary/disclosure
+2. core primary/disclosure
+3. optional static issuer overlay file
+4. optional runtime issuer overlay
+
+Runtime overlays therefore override static issuer overlays for the same surface when both are present.
+
+## Failure Handling
+
+- Generator failures set the overlay status to `error` and preserve the prior active revision.
+- Empty builds set the status to `empty` only when no active revision exists.
+- Search indexing is queued only after the final hydration pass, so downstream search reflects the overlay-adjusted snapshot state.