Manual Test Plan — Multi-store scenarios
The PS#2 reporter's environment was multi-store. PS#2 closed via #329 / #333 / #334 / #335; this plan keeps the operator-facing surface regression-gated.
Substrate
MultiStoreHost (added in #318). Two modes:
| Mode | What it registers | Aspire profile |
|---|---|---|
MainPlusAncillary (default) | Main IDocumentStore (Trips) + ancillaries: IIncidentsStore, ITelehealthStore | Full |
AncillaryOnly | Three typed ancillaries: ITripsStore, IIncidentsStore, ITelehealthStore — no main IDocumentStore | multi-store (dedicated launch profile) |
Both modes register three independent event-store databases. MultiStorePublisher drives synthetic events into all three so projections advance independently.
MS-1 — All three event stores surface in the BFF UI
| Field | Value |
|---|---|
| Setup | dotnet run --project src/BffHost (Full scenario — uses MainPlusAncillary by default). Wait for MultiStoreHost and MultiStorePublisher to be Running. |
| Action | Open the Storage tab on the MultiStoreHost service detail page (/service/MultiStoreHost). |
| Expected observation | The Event Stores card lists three entries — one for the main Trips store, one each for the Incidents and Telehealth ancillaries. Each carries its own database identity and projection list. |
| How to verify | UI: at least three rows under the Event Stores card, each with a distinct subjectUri (e.g. marten://store, marten://iincidentsstore, marten://itelehealthstore). API: service.eventStores.length === 3. The Playwright spec src/FrontEnd/e2e/projections/multi-store-projections.spec.ts covers an adjacent assertion. |
MS-2 — Projections page renders one HWM row per store (PS#2 regression gate)
| Field | Value |
|---|---|
| Setup | Same as MS-1. Navigate to the Projections page; filter by MultiStoreHost. |
| Action | Inspect the HWM rows. |
| Expected observation | Three distinct HighWaterMark rows — one per store. Each row carries its store URI in the Store column and an independent sequence number reflecting that store's HWM. Pre-PS#2 this rendered a single collapsed HWM row tracking whichever store wrote last — a regression of that would be a one-row render here. |
| How to verify | UI: await page.locator('.el-table__row', { hasText: 'HighWaterMark' }).count() >= 2 (the Playwright spec asserts this directly). API: service.shardStatesByStore has three outer keys, each with a HighWaterMark entry, sequences distinct. The PS#2 PR-A unit test multi_store_shard_state_projection_tests.writes_to_one_store_do_not_overwrite_another_stores_hwm covers the wire-side invariant. |
MS-3 — Projection "behind" is computed against the projection's owning-store HWM
| Field | Value |
|---|---|
| Setup | Same as MS-1. Let the publisher drive synthetic traffic for at least 30 seconds so each store has events spread out. |
| Action | On the Projections page, inspect the Behind column for each projection. |
| Expected observation | Each projection's "behind" value is hwm_of_its_owning_store - projection_sequence. The Itinerary projection (on the Trips store) does not appear thousands of events behind just because the TelehealthComposite store's HWM is way ahead. Each "behind" stays small under steady-state publisher traffic. |
| How to verify | For each projection row: cross-check behindBy value against service.shardStatesByStore[<projection's storeUri>]['HighWaterMark'].sequence - <projection's sequence>. Should be a small non-negative number. PS#2 PR-B's frontend test gapForProjection uses per-store HWM covers this in isolation. |
MS-4 — AncillaryOnly mode boots cleanly with no main IDocumentStore
| Field | Value |
|---|---|
| Setup | Set CRITTERWATCH_SCENARIO=multi-store (or use the multi-store launch profile). This flips MultiStoreHost to AncillaryOnly mode via MultiStore__Mode=AncillaryOnly env passthrough. Wait for MultiStoreHost to be Running. |
| Action | Open the service detail page. |
| Expected observation | Three event stores surface (same shape as MS-1) — Trips, Incidents, Telehealth, all as typed ancillaries. The page must not crash on the missing main IDocumentStore; capability-discovery in CritterWatch must surface the typed stores via GetServices<IEventStore>() (the contract PS#2 PR-A formalised). Storage tab renders correctly with no IDocumentStore row but three event-store rows. |
| How to verify | UI: same as MS-1 but the Documents tab is empty (no main IDocumentStore). API: service.documentStores.length === 0, service.eventStores.length === 3. The pre-existing integration test Tests.Integration.multi_store_host_tests.ancillary_only_mode_registers_three_typed_stores_no_main covers the registration contract; this scenario adds the UI gate. |
MS-5 — Per-store admin command (pause / restart / rebuild) hits the right store's daemon
| Field | Value |
|---|---|
| Setup | Full scenario at MainPlusAncillary mode. Verify all three stores' projections show Updated. |
| Action | Pause IncidentsByCategory on the Incidents ancillary. Wait 2 seconds. Restart it. |
| Expected observation | Only the IncidentsByCategory:All shard transitions through Updated → Paused → Updated. The main store's Itinerary projection and the Telehealth ancillary's TelehealthComposite projection stay Updated throughout with no interruption. The state-transition timeline on the IncidentsByCategory detail page shows both transitions; the other two projections' timelines show no transitions. |
| How to verify | SQL across schemas: SELECT name, agent_status FROM multistore_incidents.mt_event_progression; — paused row visible during the pause window. SELECT name, agent_status FROM multistore_trips.mt_event_progression; — all rows Running throughout. SELECT name, agent_status FROM multistore_telehealth.mt_event_progression; — all rows Running throughout. PS#2 PR-D's automated test Tests.Integration.multi_store_admin_commands covers exactly this flow against the BFF wire path. |
MS-6 — Rebuild an ancillary projection without disturbing the main store
| Field | Value |
|---|---|
| Setup | Full scenario, wait for publisher to drive sequences ≥ 100 on each projection. |
| Action | Trigger Rebuild on TelehealthComposite (lives on the ITelehealthStore ancillary). |
| Expected observation | TelehealthComposite:All sequence resets to 0 and climbs back. Main store's Itinerary projection and the Incidents ancillary's IncidentsByCategory projection sequences keep advancing under publisher pressure with no interruption. |
| How to verify | UI: state-transition timeline on TelehealthComposite shows Rebuild transitions; the other two projections' timelines show no Rebuild transitions during the same window. SQL: last_seq_id on TelehealthComposite:All resets and climbs; on Itinerary:All and IncidentsByCategory:All it advances monotonically without reset. |
MS-7 — Per-store dead-letter rendering (PS#3 + PS#2 interaction)
| Field | Value |
|---|---|
| Setup | Full scenario. Note that each store may have a different Projections.Errors.SkipApplyErrors policy — MultiStoreHost-Incidents has SkipApplyErrors = false (stop-on-error); MultiStoreHost-Trips and MultiStoreHost-Telehealth keep the JasperFx 2.0 default true (skip-and-DLQ). |
| Action | Open the Projection Detail page for IncidentsByCategory (stop-on-error policy). Then open Itinerary (skip-and-DLQ). |
| Expected observation | IncidentsByCategory renders the PS#3 stop-on-error indicator (apply-error-policy-stop); Itinerary renders the skip-and-DLQ section with the View Related Dead Letters button (apply-error-policy-skip). The policy is per-store, sourced from service.eventStores[<storeIndex>].projectionErrors.skipApplyErrors. |
| How to verify | UI: the data-testid for the Apply Errors card differs between the two pages. API: each store's eventStores[i].projectionErrors.skipApplyErrors is independent. PS#3 PR-A's frontend test covers each branch in isolation; this scenario gates the per-store independence in the same service. |
Cross-reference
- PS#2 closure: #329, #333, #334, #335
- Substrate:
src/Samples/MultiStoreHost/(added in #318) - Pre-existing automated coverage:
Tests.Integration.multi_store_host_tests(registration + per-store HWM independence at the Marten layer),Tests.Integration.multi_store_progression_poller_tests(per-store snapshot emission),Tests.Integration.multi_store_admin_commands(admin commands against ancillary projections) - PS#3 interaction (per-store error policy): #326
