Manual Test Plan — Multi-tenancy scenarios
Covers dynamic tenant lifecycle (add / disable / enable / remove / hard-delete), per-tenant projection actions (rebuild a single tenant's projection without disturbing others), and the per-tenant view selector on the Projections page.
Substrates
| Sample | Tenancy model | Scope |
|---|---|---|
MTTrips (master-table dynamic tenancy) | Marten's MultiTenantedDatabasesWithMasterDatabaseTable — one DB per tenant, tenants added at runtime via the master table | Dynamic lifecycle (the operator-UI surface from #103) |
TeleHealth (conjoined + hash-partitioning) | Conjoined multi-tenancy + 8-bucket hash-partitioned event table (b_1 through b_8) | Per-tenant projection ops at scale (the long-running rebuild substrate's tenancy dimension) |
The MTTrips family is the primary substrate for tenant-lifecycle tests (it's what the tenancy Aspire scenario is for). TeleHealth is the substrate for per-tenant projection actions at realistic scale.
MT-1 — Dynamic tenant add via the Tenant Management UI
| Field | Value |
|---|---|
| Setup | dotnet run --project src/BffHost with CRITTERWATCH_SCENARIO=tenancy. This loads the MTTrip family (one MTTripService plus optionally extra MTTripService-nodeN replicas marked with WithExplicitStart()). Wait for the service to be Running and for the seed tenant (typically tenant1) to appear in the Tenant Management UI on /service/MTTripService → Tenants tab. |
| Action | On the Tenants tab, click Add Tenant. Enter tenant id tenant_dyn and a connection string pointing to a new database. Submit. |
| Expected observation | Within ~5 seconds: a new row appears in the Tenants list with id tenant_dyn. The tenant's database is provisioned (the BFF-side AddTenantHandler ensures it physically exists, then writes the registry row via IDynamicTenantSource<string>.AddTenantAsync). Subsequent publisher activity on tenant_dyn produces events that appear on the Projections page under the per-tenant view selector. |
| How to verify | UI: tenant row visible in the Tenants tab. SQL: SELECT tenant_id, connection_string FROM mtmaster.tenants; — tenant_dyn row present. Subsequent publisher activity surfaces MTTrip:All:tenant_dyn shard in mt_event_progression for the tenant's database. The pre-existing test Tests.Integration.MultiTenancy.single_server_provision_on_demand_tests covers the provisioning flow. |
MT-2 — Dynamic tenant disable → projections stop advancing for that tenant only
| Field | Value |
|---|---|
| Setup | Continue from MT-1 with tenant_dyn provisioned + receiving traffic. Confirm MTTrip:All:tenant_dyn shard is advancing. |
| Action | On the Tenants tab, click Disable on the tenant_dyn row. |
| Expected observation | The tenant's projections stop advancing within ~5 seconds. Other tenants (tenant1, tenant2, …) keep advancing unaffected. The Tenants list row shows a Disabled badge. |
| How to verify | SQL: last_seq_id on MTTrip:All:tenant_dyn freezes; other per-tenant shards keep climbing. The Tenants table's disabled column flips true for tenant_dyn. The pre-existing test Tests.Integration.MultiTenancy.tenant_lifecycle_disable_resume (if present; otherwise the integration coverage in TenantManagementHandler's units) gates this. |
MT-3 — Re-enable a disabled tenant → projections resume
| Field | Value |
|---|---|
| Setup | Continue from MT-2 with tenant_dyn disabled. |
| Action | Click Enable on the tenant_dyn row. |
| Expected observation | Tenant's projections resume advancing within ~5 seconds. The Disabled badge disappears. Crucially, the projection resumes from its pre-disable sequence (no replay), so the publisher pressure that built up during the disable window is processed in order. |
| How to verify | SQL: last_seq_id on MTTrip:All:tenant_dyn resumes climbing from where it froze. The Tenants table's disabled column flips back to false. |
MT-4 — Hard-delete a tenant → database dropped + registry row removed
| Field | Value |
|---|---|
| Setup | Continue from MT-3. Enable tenant_dyn and let it accumulate some events. |
| Action | On the Tenants tab, click Hard Delete on tenant_dyn. Confirm the operator dialog (it's a heavy action — irreversible). |
| Expected observation | The tenant's database is physically dropped (DROP DATABASE WITH (FORCE)). The registry row is removed. The tenant's row disappears from the Tenants list. Future publisher activity scoped to this tenant fails with UnknownTenantIdException. |
| How to verify | SQL: tenant_dyn row no longer in the tenants table. psql -l (or equivalent) confirms the database no longer exists. The pre-existing test Tests.Services.TenantManagementResolverTests.HardDeleteTenant_drops_database (or equivalent) gates the handler logic; this scenario adds the UI gate. |
| Why it matters | Hard delete is the only operator surface that drops a physical DB. The flow has to confirm explicitly (operator-confirmable, two-step at minimum); a fat-finger here is unrecoverable. |
MT-5 — Per-tenant projection rebuild (TeleHealth scale)
| Field | Value |
|---|---|
| Setup | Full scenario at TeleHealth default scale (5 tenants × 50k events). Wait for all 5 tenants' shards to be at Updated with the full per-tenant HWM. |
| Action | On the Projection Detail page for TelehealthComposite, expand the per-tenant view selector. Pick tenant_0001. Trigger Rebuild scoped to that tenant. |
| Expected observation | Only tenant_0001's shard resets to 0 and climbs back. The other 4 tenants' shards stay at their current sequence — no interruption, no rebuild activity. |
| How to verify | SQL: SELECT name, last_seq_id, agent_status FROM telehealth.mt_event_progression WHERE name LIKE 'TelehealthComposite:All:%'; — only tenant_0001's row resets and climbs; the other tenants' rows stay frozen at their pre-rebuild values throughout the run. The pre-existing test Tests.Integration.MultiTenancy.PerTenantDaemonRebuildTests covers this at the daemon layer (currently parked per project task #303 — restore once unblocked). |
MT-6 — Per-tenant view selector on the Projections page
| Field | Value |
|---|---|
| Setup | Full scenario at TeleHealth default scale. Navigate to the Projections page, filter to TeleHealthService. |
| Action | Expand the per-tenant view selector. |
| Expected observation | The Projections page shifts to a per-tenant grouping — each tenant appears as its own row group with the tenant's per-shard state laid out beneath. The HWM rows update to per-(tenant, store) granularity. Switching back to the store-global view collapses everything back to the per-store rows from MS-2. |
| How to verify | UI: the per-tenant view has a tenant label in the row group header. Selectors in projections-store.ts — shardStatesByTenant, tenantsWithProjectionActivity, perTenantGap, STORE_GLOBAL_TENANT_BUCKET — drive the rendering. Frontend unit tests in projections-store-tenant.test.ts cover the selectors in isolation. |
MT-7 — Master-table tenancy resolution across multiple sources (#280)
| Field | Value |
|---|---|
| Setup | Run a monitored service that registers two IDynamicTenantSource<string> implementations — e.g. Marten's MasterTableTenancy and Wolverine's MasterTenantSource for the message store. (Substrate: Tests.Common.HostDefinitions.MultiTenantedMartenProjectionService or similar). |
| Action | On the Tenants tab, add a tenant. |
| Expected observation | The Tenant Management handler fans out across both sources — each source's registry row is written, each source's AddTenantAsync is invoked. The Tenants list aggregates rows across both sources; rows are labelled with the source name (MasterTableTenancy vs MasterTenantSource) so the operator can disambiguate. Per-source failures are surfaced independently (one source can fail without blocking the other). |
| How to verify | UI: each tenant row in the table carries a SourceName column. API: TenantListResponse.tenants has rows from both sources, each with sourceName populated. The handler-level unit tests in Tests.Services.TenantManagementResolverTests cover the fan-out shape; this scenario gates the UI rendering. |
Open items / not-yet-implemented
- Sharded tenancy (auto-assign, no connection value) — jasperfx#413's conformance work is needed before sharded tenancy supports the Tenant Management UI. Until it does, an operator on a sharded host sees the Tenants tab empty.
- Polecat dynamic tenancy — polecat#165 tracks
MasterTableTenancyparity. Until it lands, dynamic-tenant UI is Marten-only.
Cross-reference these gaps in the issues above when they come up during a real test run.
Cross-reference
- Tenant Management UI: #103
- Multi-source dynamic tenancy fan-out: #280 (consolidated into Wolverine.CritterWatch's
TenantManagementHandler) - Per-tenant async daemon: #209, JasperFx#407
- Substrate:
src/Samples/MTTrips/(master-table dynamic tenancy),src/Samples/TeleHealth/(conjoined + hash-partitioning at scale) - Pre-existing automated coverage:
Tests.Integration.MultiTenancy.single_server_provision_on_demand_tests,Tests.Integration.MultiTenancy.PerTenantDaemonRebuildTests(parked),Tests.Services.TenantManagementResolverTests
