Skip to content

Manual Test Plan — Multi-store scenarios

The PS#2 reporter's environment was multi-store. PS#2 closed via #329 / #333 / #334 / #335; this plan keeps the operator-facing surface regression-gated.

Substrate

MultiStoreHost (added in #318). Two modes:

ModeWhat it registersAspire profile
MainPlusAncillary (default)Main IDocumentStore (Trips) + ancillaries: IIncidentsStore, ITelehealthStoreFull
AncillaryOnlyThree typed ancillaries: ITripsStore, IIncidentsStore, ITelehealthStoreno main IDocumentStoremulti-store (dedicated launch profile)

Both modes register three independent event-store databases. MultiStorePublisher drives synthetic events into all three so projections advance independently.

MS-1 — All three event stores surface in the BFF UI

FieldValue
Setupdotnet run --project src/BffHost (Full scenario — uses MainPlusAncillary by default). Wait for MultiStoreHost and MultiStorePublisher to be Running.
ActionOpen the Storage tab on the MultiStoreHost service detail page (/service/MultiStoreHost).
Expected observationThe Event Stores card lists three entries — one for the main Trips store, one each for the Incidents and Telehealth ancillaries. Each carries its own database identity and projection list.
How to verifyUI: at least three rows under the Event Stores card, each with a distinct subjectUri (e.g. marten://store, marten://iincidentsstore, marten://itelehealthstore). API: service.eventStores.length === 3. The Playwright spec src/FrontEnd/e2e/projections/multi-store-projections.spec.ts covers an adjacent assertion.

MS-2 — Projections page renders one HWM row per store (PS#2 regression gate)

FieldValue
SetupSame as MS-1. Navigate to the Projections page; filter by MultiStoreHost.
ActionInspect the HWM rows.
Expected observationThree distinct HighWaterMark rows — one per store. Each row carries its store URI in the Store column and an independent sequence number reflecting that store's HWM. Pre-PS#2 this rendered a single collapsed HWM row tracking whichever store wrote last — a regression of that would be a one-row render here.
How to verifyUI: await page.locator('.el-table__row', { hasText: 'HighWaterMark' }).count() >= 2 (the Playwright spec asserts this directly). API: service.shardStatesByStore has three outer keys, each with a HighWaterMark entry, sequences distinct. The PS#2 PR-A unit test multi_store_shard_state_projection_tests.writes_to_one_store_do_not_overwrite_another_stores_hwm covers the wire-side invariant.

MS-3 — Projection "behind" is computed against the projection's owning-store HWM

FieldValue
SetupSame as MS-1. Let the publisher drive synthetic traffic for at least 30 seconds so each store has events spread out.
ActionOn the Projections page, inspect the Behind column for each projection.
Expected observationEach projection's "behind" value is hwm_of_its_owning_store - projection_sequence. The Itinerary projection (on the Trips store) does not appear thousands of events behind just because the TelehealthComposite store's HWM is way ahead. Each "behind" stays small under steady-state publisher traffic.
How to verifyFor each projection row: cross-check behindBy value against service.shardStatesByStore[<projection's storeUri>]['HighWaterMark'].sequence - <projection's sequence>. Should be a small non-negative number. PS#2 PR-B's frontend test gapForProjection uses per-store HWM covers this in isolation.

MS-4 — AncillaryOnly mode boots cleanly with no main IDocumentStore

FieldValue
SetupSet CRITTERWATCH_SCENARIO=multi-store (or use the multi-store launch profile). This flips MultiStoreHost to AncillaryOnly mode via MultiStore__Mode=AncillaryOnly env passthrough. Wait for MultiStoreHost to be Running.
ActionOpen the service detail page.
Expected observationThree event stores surface (same shape as MS-1) — Trips, Incidents, Telehealth, all as typed ancillaries. The page must not crash on the missing main IDocumentStore; capability-discovery in CritterWatch must surface the typed stores via GetServices<IEventStore>() (the contract PS#2 PR-A formalised). Storage tab renders correctly with no IDocumentStore row but three event-store rows.
How to verifyUI: same as MS-1 but the Documents tab is empty (no main IDocumentStore). API: service.documentStores.length === 0, service.eventStores.length === 3. The pre-existing integration test Tests.Integration.multi_store_host_tests.ancillary_only_mode_registers_three_typed_stores_no_main covers the registration contract; this scenario adds the UI gate.

MS-5 — Per-store admin command (pause / restart / rebuild) hits the right store's daemon

FieldValue
SetupFull scenario at MainPlusAncillary mode. Verify all three stores' projections show Updated.
ActionPause IncidentsByCategory on the Incidents ancillary. Wait 2 seconds. Restart it.
Expected observationOnly the IncidentsByCategory:All shard transitions through Updated → Paused → Updated. The main store's Itinerary projection and the Telehealth ancillary's TelehealthComposite projection stay Updated throughout with no interruption. The state-transition timeline on the IncidentsByCategory detail page shows both transitions; the other two projections' timelines show no transitions.
How to verifySQL across schemas:
SELECT name, agent_status FROM multistore_incidents.mt_event_progression; — paused row visible during the pause window.
SELECT name, agent_status FROM multistore_trips.mt_event_progression; — all rows Running throughout.
SELECT name, agent_status FROM multistore_telehealth.mt_event_progression; — all rows Running throughout. PS#2 PR-D's automated test Tests.Integration.multi_store_admin_commands covers exactly this flow against the BFF wire path.

MS-6 — Rebuild an ancillary projection without disturbing the main store

FieldValue
SetupFull scenario, wait for publisher to drive sequences ≥ 100 on each projection.
ActionTrigger Rebuild on TelehealthComposite (lives on the ITelehealthStore ancillary).
Expected observationTelehealthComposite:All sequence resets to 0 and climbs back. Main store's Itinerary projection and the Incidents ancillary's IncidentsByCategory projection sequences keep advancing under publisher pressure with no interruption.
How to verifyUI: state-transition timeline on TelehealthComposite shows Rebuild transitions; the other two projections' timelines show no Rebuild transitions during the same window. SQL: last_seq_id on TelehealthComposite:All resets and climbs; on Itinerary:All and IncidentsByCategory:All it advances monotonically without reset.

MS-7 — Per-store dead-letter rendering (PS#3 + PS#2 interaction)

FieldValue
SetupFull scenario. Note that each store may have a different Projections.Errors.SkipApplyErrors policy — MultiStoreHost-Incidents has SkipApplyErrors = false (stop-on-error); MultiStoreHost-Trips and MultiStoreHost-Telehealth keep the JasperFx 2.0 default true (skip-and-DLQ).
ActionOpen the Projection Detail page for IncidentsByCategory (stop-on-error policy). Then open Itinerary (skip-and-DLQ).
Expected observationIncidentsByCategory renders the PS#3 stop-on-error indicator (apply-error-policy-stop); Itinerary renders the skip-and-DLQ section with the View Related Dead Letters button (apply-error-policy-skip). The policy is per-store, sourced from service.eventStores[<storeIndex>].projectionErrors.skipApplyErrors.
How to verifyUI: the data-testid for the Apply Errors card differs between the two pages. API: each store's eventStores[i].projectionErrors.skipApplyErrors is independent. PS#3 PR-A's frontend test covers each branch in isolation; this scenario gates the per-store independence in the same service.

Cross-reference

  • PS#2 closure: #329, #333, #334, #335
  • Substrate: src/Samples/MultiStoreHost/ (added in #318)
  • Pre-existing automated coverage: Tests.Integration.multi_store_host_tests (registration + per-store HWM independence at the Marten layer), Tests.Integration.multi_store_progression_poller_tests (per-store snapshot emission), Tests.Integration.multi_store_admin_commands (admin commands against ancillary projections)
  • PS#3 interaction (per-store error policy): #326

Released under the MIT License.