diff --git a/E2E_UX_REPORT_2026-03-14.md b/E2E_UX_REPORT_2026-03-14.md new file mode 100644 index 0000000..c8e0df2 --- /dev/null +++ b/E2E_UX_REPORT_2026-03-14.md @@ -0,0 +1,135 @@ +# E2E and UX Report + +Date: 2026-03-14 +Repo: `fiscal clone` +Command run: `bun run test:e2e` + +## Executive Summary + +The full Playwright suite did not pass. Result: 4 passed, 6 failed, 1 did not run. + +The most important product issues surfaced by the run and manual pass were: + +1. The overview workflow can land in a non-recovering loading state with no actionable guidance. +2. Auth/session reliability appears unstable under realistic navigation and parallel activity. +3. Filing sync creates noisy, duplicated notifications that follow the user across pages. +4. The filings workflow communicates one filter state while showing data from multiple tickers. + +## Automated E2E Outcome + +Artifacts are under `output/playwright/`. + +Failed specs: + +- `e2e/analysis.spec.ts`: `shows the overview skeleton while analysis is loading` + - Failure: timed out waiting for `Microsoft Corporation`. + - Observed state: app chrome rendered, but the page remained on `Loading company overview`. + - Artifacts: + - `output/playwright/test-results/analysis-shows-the-overvie-6b9b1-n-while-analysis-is-loading/error-context.md` + - `output/playwright/test-results/analysis-shows-the-overvie-6b9b1-n-while-analysis-is-loading/test-failed-1.png` + +- `e2e/financials.spec.ts`: `renders the standardized operating expense tree and inspector details` + - Failure: timed out waiting for `Normalization Summary`. + - Observed state: page stayed in a loading shell in the failing run. + - Artifacts: + - `output/playwright/test-results/financials-renders-the-sta-7a51f--tree-and-inspector-details/error-context.md` + - `output/playwright/test-results/financials-renders-the-sta-7a51f--tree-and-inspector-details/test-failed-1.png` + +- `e2e/financials.spec.ts`: `shows not meaningful expense breakdown rows for bank pack filings` + - Failure: sign-up flow never reached `Command Center`. + - Observed state: browser ended on `Secure Sign In`. + - Inference: auth/session creation or post-signup redirect is intermittently failing. + - Artifacts: + - `output/playwright/test-results/financials-shows-not-meani-c9d8d--rows-for-bank-pack-filings/error-context.md` + - `output/playwright/test-results/financials-shows-not-meani-c9d8d--rows-for-bank-pack-filings/test-failed-1.png` + +- `e2e/graphing.spec.ts`: `supports graphing compare controls and partial failures` + - Failure: sign-up flow never reached `Command Center`. + - Observed state: browser ended on `Secure Sign In`. + +- `e2e/graphing.spec.ts`: `distinguishes not meaningful metrics from missing data in the latest values table` + - Failure: sign-up flow never reached `Command Center`. + - Observed state: browser ended on `Secure Sign In`. + +- `e2e/research-mvp.spec.ts`: `supports the core coverage-to-research workflow` + - Failure: strict-mode locator conflict on `NVDA status`. + - Observed state: the page exposed two matching controls for the same accessible name. + - Inference: duplicated DOM or duplicated accessible controls in the coverage row. + - Artifacts: + - `output/playwright/test-results/research-mvp-supports-the-core-coverage-to-research-workflow/error-context.md` + - `output/playwright/test-results/research-mvp-supports-the-core-coverage-to-research-workflow/test-failed-1.png` + +Runtime notes from the run: + +- `sqlite-vec` fell back to table-backed storage repeatedly. +- workflow-local reported `concurrency limit (1) reached`. +- The suite completed in about 1.2 minutes before failing. + +## Manual Workflow Findings + +### Critical + +- Overview page does not distinguish loading vs failure vs empty data. + - Visiting `/analysis` rendered the full page shell but stayed on `Loading company overview`. + - The user gets no timeout, error explanation, retry guidance, or indication that background filing tasks failed. + - This is a broken primary workflow, not just a cosmetic state issue. + +- Auth confidence is too low. + - Three failed specs fell back to `Secure Sign In` immediately after supposed account creation. + - A user-facing version of this would feel like silent account creation failure or dropped session state. + +- Filings filter messaging is misleading. + - The page header said `100 records loaded for NVDA`, but the ledger visibly contained both `NVDA` and `MSFT` rows. + - This breaks user trust because the UI claims a scoped result set while showing cross-ticker data. + +### High + +- Notification spam degrades every workflow. + - Coverage creation and navigation produced stacked `Filing sync` toasts. + - Multiple toasts showed nearly identical states and remained visible across pages. + - The notifications panel becomes ambient noise instead of actionable feedback. + +- Coverage actions can trigger expensive side effects without clear consent. + - Adding one coverage record immediately started sync-related background activity. + - There is no up-front explanation of what will be fetched, how long it may take, or whether it can be deferred. + +- Research workspace has high cognitive load on first entry. + - Filters, note capture, uploads, memo editing, and packet review all appear at once. + - The page is powerful, but the first-time path is unclear and there is no obvious recommended sequence. + +### Medium + +- Empty states on Financials and Graphing are under-instructive. + - Financials: `No rows available`, `No trend data`, and zeroed normalization metrics do not tell the user whether they need to sync, change ticker, or wait. + - Graphing: `No chart data available` appears after opening a prefilled compare set, which makes the default state feel broken. + +- Upload interaction in Research is confusing. + - The page exposes two controls with the label `Upload file`, one active and one disabled. + - That is confusing visually and poor for keyboard/screen-reader users. + +- Navigation context is inconsistent. + - Some links carry ticker context and others drop back to untickered routes such as `/analysis`. + - The app often compensates by defaulting to `MSFT`, but that is implicit and can feel arbitrary. + +- Filings table density is too high by default. + - Loading 100 records into a single ledger creates a long scan path before the user can find the next action. + - Pagination, grouped sections, or stronger filter summaries would reduce cognitive load. + +### Low + +- `favicon.ico` returns 404. +- The boot splash and auth handoff add a short delay before the real screen appears, but do not communicate whether the app is loading, redirecting, or checking session state. + +## Suggested Fix Order + +1. Stabilize post-signup session creation and redirect behavior. +2. Make `/analysis` fail loudly and recoverably instead of hanging in a loader. +3. Fix filings filter truthfulness so the page never claims `NVDA` while rendering mixed results. +4. Deduplicate or batch filing sync notifications. +5. Add next-step empty states to Financials and Graphing. +6. Reduce first-run complexity in Research with a guided progression or collapsed sections. +7. Remove duplicate accessible controls and labels in Coverage and Research upload flows. + +## Notes + +The manual pass used the repo's e2e server entrypoint: `bun run e2e:webserver`.