Neon-Desk/doc/issuer-overlay-automation.md

Issuer Overlay Automation

Overview

Issuer overlays are now runtime data, not just checked-in JSON files.

The system creates or updates a global overlay for a ticker after an explicit user ticker submit triggers filing sync. Overlay generation is additive-only:

• it extends existing canonical surfaces
• it never synthesizes brand-new surfaces
• it never auto-deletes prior mappings

Trigger Flow

1. An explicit ticker submit calls POST /api/tickers/ensure.
2. The endpoint ensures an issuer_overlay row exists for the ticker.
3. If the ticker has no ready taxonomy snapshots, no overlay build history, or its latest ready snapshot was hydrated with an older overlay revision, the endpoint queues sync_filings.
4. sync_filings hydrates taxonomy snapshots with the current active overlay revision.
5. The task then generates a new overlay candidate from stored residual extension concepts.
6. If a new overlay revision is published, the task immediately rehydrates recent filings with that revision before queueing search indexing.

Storage

Two tables back the feature:

• issuer_overlay: one row per ticker with status, active revision, error state, and summary stats
• issuer_overlay_revision: immutable overlay revisions with normalized definition JSON and diagnostics

Each filing_taxonomy_snapshot also stores issuer_overlay_revision_id so the sync pipeline can detect when a filing needs rehydration because the overlay changed.

Generator Rules

The generator samples up to 12 ready 10-K/10-Q snapshots from the last 3 years and chooses the most common detected fiscal pack in that window.

It only promotes a concept when all of the following are true:

• the concept is an extension concept
• the concept is still residual/unmapped in the stored taxonomy output
• the concept is not abstract
• the concept has a consistent statement kind across filings
• the concept appears in at least 2 distinct filings
• the concept can be mapped uniquely to an existing canonical surface by:
  • authoritative concept local-name match, or
  • exact local-name match against canonical source/authoritative concepts

Accepted concepts are appended to allowed_source_concepts for the resolved surface in the runtime overlay definition.

Merge Order

The hydrator merges surfaces in this order:

1. pack primary/disclosure
2. core primary/disclosure
3. optional static issuer overlay file
4. optional runtime issuer overlay

Runtime overlays therefore override static issuer overlays for the same surface when both are present.

Failure Handling

• Generator failures set the overlay status to error and preserve the prior active revision.
• Empty builds set the status to empty only when no active revision exists.
• Search indexing is queued only after the final hydration pass, so downstream search reflects the overlay-adjusted snapshot state.