136 lines
7.0 KiB
Markdown
136 lines
7.0 KiB
Markdown
# E2E and UX Report
|
|
|
|
Date: 2026-03-14
|
|
Repo: `fiscal clone`
|
|
Command run: `bun run test:e2e`
|
|
|
|
## Executive Summary
|
|
|
|
The full Playwright suite did not pass. Result: 4 passed, 6 failed, 1 did not run.
|
|
|
|
The most important product issues surfaced by the run and manual pass were:
|
|
|
|
1. The overview workflow can land in a non-recovering loading state with no actionable guidance.
|
|
2. Auth/session reliability appears unstable under realistic navigation and parallel activity.
|
|
3. Filing sync creates noisy, duplicated notifications that follow the user across pages.
|
|
4. The filings workflow communicates one filter state while showing data from multiple tickers.
|
|
|
|
## Automated E2E Outcome
|
|
|
|
Artifacts are under `output/playwright/`.
|
|
|
|
Failed specs:
|
|
|
|
- `e2e/analysis.spec.ts`: `shows the overview skeleton while analysis is loading`
|
|
- Failure: timed out waiting for `Microsoft Corporation`.
|
|
- Observed state: app chrome rendered, but the page remained on `Loading company overview`.
|
|
- Artifacts:
|
|
- `output/playwright/test-results/analysis-shows-the-overvie-6b9b1-n-while-analysis-is-loading/error-context.md`
|
|
- `output/playwright/test-results/analysis-shows-the-overvie-6b9b1-n-while-analysis-is-loading/test-failed-1.png`
|
|
|
|
- `e2e/financials.spec.ts`: `renders the standardized operating expense tree and inspector details`
|
|
- Failure: timed out waiting for `Normalization Summary`.
|
|
- Observed state: page stayed in a loading shell in the failing run.
|
|
- Artifacts:
|
|
- `output/playwright/test-results/financials-renders-the-sta-7a51f--tree-and-inspector-details/error-context.md`
|
|
- `output/playwright/test-results/financials-renders-the-sta-7a51f--tree-and-inspector-details/test-failed-1.png`
|
|
|
|
- `e2e/financials.spec.ts`: `shows not meaningful expense breakdown rows for bank pack filings`
|
|
- Failure: sign-up flow never reached `Command Center`.
|
|
- Observed state: browser ended on `Secure Sign In`.
|
|
- Inference: auth/session creation or post-signup redirect is intermittently failing.
|
|
- Artifacts:
|
|
- `output/playwright/test-results/financials-shows-not-meani-c9d8d--rows-for-bank-pack-filings/error-context.md`
|
|
- `output/playwright/test-results/financials-shows-not-meani-c9d8d--rows-for-bank-pack-filings/test-failed-1.png`
|
|
|
|
- `e2e/graphing.spec.ts`: `supports graphing compare controls and partial failures`
|
|
- Failure: sign-up flow never reached `Command Center`.
|
|
- Observed state: browser ended on `Secure Sign In`.
|
|
|
|
- `e2e/graphing.spec.ts`: `distinguishes not meaningful metrics from missing data in the latest values table`
|
|
- Failure: sign-up flow never reached `Command Center`.
|
|
- Observed state: browser ended on `Secure Sign In`.
|
|
|
|
- `e2e/research-mvp.spec.ts`: `supports the core coverage-to-research workflow`
|
|
- Failure: strict-mode locator conflict on `NVDA status`.
|
|
- Observed state: the page exposed two matching controls for the same accessible name.
|
|
- Inference: duplicated DOM or duplicated accessible controls in the coverage row.
|
|
- Artifacts:
|
|
- `output/playwright/test-results/research-mvp-supports-the-core-coverage-to-research-workflow/error-context.md`
|
|
- `output/playwright/test-results/research-mvp-supports-the-core-coverage-to-research-workflow/test-failed-1.png`
|
|
|
|
Runtime notes from the run:
|
|
|
|
- `sqlite-vec` fell back to table-backed storage repeatedly.
|
|
- workflow-local reported `concurrency limit (1) reached`.
|
|
- The suite completed in about 1.2 minutes before failing.
|
|
|
|
## Manual Workflow Findings
|
|
|
|
### Critical
|
|
|
|
- Overview page does not distinguish loading vs failure vs empty data.
|
|
- Visiting `/analysis` rendered the full page shell but stayed on `Loading company overview`.
|
|
- The user gets no timeout, error explanation, retry guidance, or indication that background filing tasks failed.
|
|
- This is a broken primary workflow, not just a cosmetic state issue.
|
|
|
|
- Auth confidence is too low.
|
|
- Three failed specs fell back to `Secure Sign In` immediately after supposed account creation.
|
|
- A user-facing version of this would feel like silent account creation failure or dropped session state.
|
|
|
|
- Filings filter messaging is misleading.
|
|
- The page header said `100 records loaded for NVDA`, but the ledger visibly contained both `NVDA` and `MSFT` rows.
|
|
- This breaks user trust because the UI claims a scoped result set while showing cross-ticker data.
|
|
|
|
### High
|
|
|
|
- Notification spam degrades every workflow.
|
|
- Coverage creation and navigation produced stacked `Filing sync` toasts.
|
|
- Multiple toasts showed nearly identical states and remained visible across pages.
|
|
- The notifications panel becomes ambient noise instead of actionable feedback.
|
|
|
|
- Coverage actions can trigger expensive side effects without clear consent.
|
|
- Adding one coverage record immediately started sync-related background activity.
|
|
- There is no up-front explanation of what will be fetched, how long it may take, or whether it can be deferred.
|
|
|
|
- Research workspace has high cognitive load on first entry.
|
|
- Filters, note capture, uploads, memo editing, and packet review all appear at once.
|
|
- The page is powerful, but the first-time path is unclear and there is no obvious recommended sequence.
|
|
|
|
### Medium
|
|
|
|
- Empty states on Financials and Graphing are under-instructive.
|
|
- Financials: `No rows available`, `No trend data`, and zeroed normalization metrics do not tell the user whether they need to sync, change ticker, or wait.
|
|
- Graphing: `No chart data available` appears after opening a prefilled compare set, which makes the default state feel broken.
|
|
|
|
- Upload interaction in Research is confusing.
|
|
- The page exposes two controls with the label `Upload file`, one active and one disabled.
|
|
- That is confusing visually and poor for keyboard/screen-reader users.
|
|
|
|
- Navigation context is inconsistent.
|
|
- Some links carry ticker context and others drop back to untickered routes such as `/analysis`.
|
|
- The app often compensates by defaulting to `MSFT`, but that is implicit and can feel arbitrary.
|
|
|
|
- Filings table density is too high by default.
|
|
- Loading 100 records into a single ledger creates a long scan path before the user can find the next action.
|
|
- Pagination, grouped sections, or stronger filter summaries would reduce cognitive load.
|
|
|
|
### Low
|
|
|
|
- `favicon.ico` returns 404.
|
|
- The boot splash and auth handoff add a short delay before the real screen appears, but do not communicate whether the app is loading, redirecting, or checking session state.
|
|
|
|
## Suggested Fix Order
|
|
|
|
1. Stabilize post-signup session creation and redirect behavior.
|
|
2. Make `/analysis` fail loudly and recoverably instead of hanging in a loader.
|
|
3. Fix filings filter truthfulness so the page never claims `NVDA` while rendering mixed results.
|
|
4. Deduplicate or batch filing sync notifications.
|
|
5. Add next-step empty states to Financials and Graphing.
|
|
6. Reduce first-run complexity in Research with a guided progression or collapsed sections.
|
|
7. Remove duplicate accessible controls and labels in Coverage and Research upload flows.
|
|
|
|
## Notes
|
|
|
|
The manual pass used the repo's e2e server entrypoint: `bun run e2e:webserver`.
|