Fix P0 issues in financial ingestion architecture #27

Francy51 · 2026-03-15T21:15:23Z

Francy51 commented

2026-03-15 21:15:23 +00:00

Summary

This PR addresses critical (P0) issues identified during architecture review of the financial ingestion system.

Changes

P0-1: Transaction-wrapped snapshot updates

Wrapped upsertFilingTaxonomySnapshot in db.transaction() to prevent data corruption
Added error context for each child table operation for better debugging

P0-2: Sidecar retry with exponential backoff

Created lib/server/utils/retry.ts with withRetry() utility
Configured: 3 attempts, 2s base delay, 10s max, 30% jitter
Updated lib/server/taxonomy/parser-client.ts to use retry wrapper

P0-3: HTTP timeout + SEC rate limiting in Rust

Added rate limiter with configurable SEC_RATE_LIMIT_MS (default: 100ms = 10 req/s)
Added 30s HTTP timeout per request to fetch_text and fetch_json
Added 120s overall timeout to HTTP client builder

P0-3b: XBRL validation with status reporting

Created XbrlValidationResult struct with status and message fields
Created validate_xbrl_structure() function (checks root element, tag balance)
Validation warns but does not fail - status included in response for TypeScript to decide
Added xbrl_validation field to HydrateFilingResponse and TaxonomyHydrationResult

Verification

✅ Rust sidecar builds successfully
✅ bun run lint passes
✅ bun run fmt passes
✅ bun x tsc --noEmit passes

Closes #4 (ticker-scoped filings - already fixed in 61b072d)
Closes #3 (auth handoff - already fixed in ac3b036)

## Summary This PR addresses critical (P0) issues identified during architecture review of the financial ingestion system. ## Changes ### P0-1: Transaction-wrapped snapshot updates - Wrapped `upsertFilingTaxonomySnapshot` in `db.transaction()` to prevent data corruption - Added error context for each child table operation for better debugging ### P0-2: Sidecar retry with exponential backoff - Created `lib/server/utils/retry.ts` with `withRetry()` utility - Configured: 3 attempts, 2s base delay, 10s max, 30% jitter - Updated `lib/server/taxonomy/parser-client.ts` to use retry wrapper ### P0-3: HTTP timeout + SEC rate limiting in Rust - Added rate limiter with configurable `SEC_RATE_LIMIT_MS` (default: 100ms = 10 req/s) - Added 30s HTTP timeout per request to `fetch_text` and `fetch_json` - Added 120s overall timeout to HTTP client builder ### P0-3b: XBRL validation with status reporting - Created `XbrlValidationResult` struct with `status` and `message` fields - Created `validate_xbrl_structure()` function (checks root element, tag balance) - Validation warns but does not fail - status included in response for TypeScript to decide - Added `xbrl_validation` field to `HydrateFilingResponse` and `TaxonomyHydrationResult` ## Verification - ✅ Rust sidecar builds successfully - ✅ `bun run lint` passes - ✅ `bun run fmt` passes - ✅ `bun x tsc --noEmit` passes ## Related Issues - Closes #4 (ticker-scoped filings - already fixed in 61b072d) - Closes #3 (auth handoff - already fixed in ac3b036)

Francy51 added 5 commits 2026-03-15 21:15:25 +00:00

Remove legacy TypeScript financial surface mapping, make Rust JSON single source of truth a7f7be50b4

- Delete standard-template.ts, surface.ts, materialize.ts (dead code)
- Delete financial-taxonomy.test.ts (relied on removed code)
- Add missing income statement surfaces to core.surface.json
- Add cost_of_revenue mapping to core.income-bridge.json
- Refactor standardize.ts to remove template dependency
- Simplify financial-taxonomy.ts to use only DB snapshots
- Add architecture documentation

Add atomic task deduplication with partial unique index ed4420b8db

- Add partial unique index for active resource-scoped tasks
- Implement createTaskRunRecordAtomic for race-free task creation
- Update findOrEnqueueTask to use atomic insert first
- Add tests for concurrent task creation deduplication

Consolidate metric definitions with Rust JSON as single source of truth 24aa8e33d4

- Add core.computed.json with 32 ratio definitions (filing + market derived)
- Add Rust types for ComputedDefinition and ComputationSpec
- Create generate-taxonomy.ts to generate TypeScript from Rust JSON
- Generate lib/generated/ (gitignored) with surfaces, computed, kpis
- Update financial-metrics.ts to use generated definitions
- Add build-time generation via 'bun run generate'
- Add taxonomy architecture documentation

Two-phase ratio computation:
- Filing-derived: margins, returns, per-share, growth (Rust computes)
- Market-derived: valuation ratios (TypeScript computes with price data)

All 32 ratios defined in core.computed.json:
- Margins: gross, operating, ebitda, net, fcf
- Returns: roa, roe, roic, roce
- Financial health: debt_to_equity, net_debt_to_ebitda, cash_to_debt, current_ratio
- Per-share: revenue, fcf, book_value
- Growth: yoy metrics + 3y/5y cagr
- Valuation: market_cap, ev, p/e, p/fcf, p/b, ev/sales, ev/ebitda, ev/fcf

Format surface_mapper tests edf1cfb421

Fix P0 issues in financial ingestion architecture

PR Checks / typecheck-and-build (pull_request) Has been cancelled

Details

Taxonomy Sidecar / taxonomy-sidecar (pull_request) Has been cancelled

Details

4313058d65

- Wrap snapshot updates in transactions with error context for each child table
- Add sidecar retry with exponential backoff (3 attempts, 2s base, 10s max, 30% jitter)
- Add HTTP timeout (30s per request) and SEC rate limiting (10 req/s) in Rust
- Add XBRL validation with status reporting (checks root element, tag balance)

Francy51 closed this pull request

2026-03-16 19:13:46 +00:00

PR Checks / typecheck-and-build (pull_request) Has been cancelled

Details

Taxonomy Sidecar / taxonomy-sidecar (pull_request) Has been cancelled

Details

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Francy51/Neon-Desk#27