64 lines
2.8 KiB
Markdown
64 lines
2.8 KiB
Markdown
# Issuer Overlay Automation
|
|
|
|
## Overview
|
|
|
|
Issuer overlays are now runtime data, not just checked-in JSON files.
|
|
|
|
The system creates or updates a global overlay for a ticker after an explicit user ticker submit triggers filing sync. Overlay generation is additive-only:
|
|
|
|
- it extends existing canonical surfaces
|
|
- it never synthesizes brand-new surfaces
|
|
- it never auto-deletes prior mappings
|
|
|
|
## Trigger Flow
|
|
|
|
1. An explicit ticker submit calls `POST /api/tickers/ensure`.
|
|
2. The endpoint ensures an `issuer_overlay` row exists for the ticker.
|
|
3. If the ticker has no ready taxonomy snapshots, no overlay build history, or its latest ready snapshot was hydrated with an older overlay revision, the endpoint queues `sync_filings`.
|
|
4. `sync_filings` hydrates taxonomy snapshots with the current active overlay revision.
|
|
5. The task then generates a new overlay candidate from stored residual extension concepts.
|
|
6. If a new overlay revision is published, the task immediately rehydrates recent filings with that revision before queueing search indexing.
|
|
|
|
## Storage
|
|
|
|
Two tables back the feature:
|
|
|
|
- `issuer_overlay`: one row per ticker with status, active revision, error state, and summary stats
|
|
- `issuer_overlay_revision`: immutable overlay revisions with normalized definition JSON and diagnostics
|
|
|
|
Each `filing_taxonomy_snapshot` also stores `issuer_overlay_revision_id` so the sync pipeline can detect when a filing needs rehydration because the overlay changed.
|
|
|
|
## Generator Rules
|
|
|
|
The generator samples up to 12 ready `10-K`/`10-Q` snapshots from the last 3 years and chooses the most common detected fiscal pack in that window.
|
|
|
|
It only promotes a concept when all of the following are true:
|
|
|
|
- the concept is an extension concept
|
|
- the concept is still residual/unmapped in the stored taxonomy output
|
|
- the concept is not abstract
|
|
- the concept has a consistent statement kind across filings
|
|
- the concept appears in at least 2 distinct filings
|
|
- the concept can be mapped uniquely to an existing canonical surface by:
|
|
- authoritative concept local-name match, or
|
|
- exact local-name match against canonical source/authoritative concepts
|
|
|
|
Accepted concepts are appended to `allowed_source_concepts` for the resolved surface in the runtime overlay definition.
|
|
|
|
## Merge Order
|
|
|
|
The hydrator merges surfaces in this order:
|
|
|
|
1. pack primary/disclosure
|
|
2. core primary/disclosure
|
|
3. optional static issuer overlay file
|
|
4. optional runtime issuer overlay
|
|
|
|
Runtime overlays therefore override static issuer overlays for the same surface when both are present.
|
|
|
|
## Failure Handling
|
|
|
|
- Generator failures set the overlay status to `error` and preserve the prior active revision.
|
|
- Empty builds set the status to `empty` only when no active revision exists.
|
|
- Search indexing is queued only after the final hydration pass, so downstream search reflects the overlay-adjusted snapshot state.
|