Multi-Tenancy

CritterWatch supports services that run multiple tenants on separate databases — a common pattern for SaaS applications using Marten / Wolverine. This page covers what you'll see, what you can do, and the gotchas that come with managing tenants at runtime.

For step-by-step UI walkthrough, see Services → Tenants tab. For the integration code in your monitored service, see Wolverine.CritterWatch → Registration.

What CritterWatch sees

When a multi-tenant service starts up, it advertises its tenancy mode. The console picks this up automatically — there's nothing for the operator to configure on the console side.

The tenancy modes you'll see in the UI:

Mode	Meaning	Tenants tab behavior
None / Single	Not multi-tenant.	Tenants tab not shown.
ConjoinedTenancy	All tenants share one database, partitioned by tenant id.	Tenants tab shown read-only — tenants are configured at startup.
StaticMultiple	Tenants pinned to specific databases at startup.	Tenants tab shown read-only with the "Read-only" hint.
DynamicMultiple	Tenants and databases added/removed at runtime.	Full lifecycle controls (Add / Disable / Enable / Remove / Hard delete).

Per-tenant scoping

Most operational surfaces have a tenant filter for multi-tenant services:

• Dead Letter Queue — tenant filter scopes queries and replay/discard operations to one tenant's message store. Replay against acme-corp won't touch globex-inc.
• Projections — each tenant has its own projection shards with independent sequence tracking. The Projections page renders one row per (shard, tenant) pair. Pause / Restart / Rebuild buttons are per-row: a rebuild of TripSummary:All for acme-corp only rebuilds that tenant's read model. Other tenants' shards keep advancing untouched.
• Scheduled Messages — tenant filter scopes the list and per-message edits/cancels.
• Durability Monitor — each tenant's inbox/outbox shows as a separate row.

This is enforced server-side by routing the operation through the target service's command handler with the tenant id attached. Operators can't accidentally cross-pollute data between tenants. The PauseProjection / RestartProjection / RebuildProjection / RewindSubscription commands all carry an optional TenantId for the same reason.

Per-tenant traffic columns

The Tenants tab includes per-tenant traffic columns when the service emits per-tenant Prometheus labels:

• Executions (1h) — handler executions in the last hour
• Failures (1h) — handler failures in the last hour (reds when non-zero)
• DLQ depth — current dead-letter depth (reds when non-zero)

These let you spot the noisy-neighbour pattern at a glance: a tenant whose DLQ depth is climbing while the rest are flat is the one to investigate. See Services → Tenants → When to drill in for the patterns.

If the columns aren't showing, your service either has zero tenants reporting metrics or doesn't yet emit tenant_id labels in its OpenTelemetry / Prometheus config.

Adding a tenant at runtime

For DynamicMultiple services, the Add Tenant button on the Tenants tab opens a dialog with two fields: tenant id and connection string. Submitting it sends the AddTenant command to the service.

The service handler:

1. Adds the database to its multi-tenant store via Wolverine's tenant management API.
2. Applies the Marten / EF Core schema to the new database (table creation, etc.).
3. Reports the new tenant in the next telemetry batch — the row appears in the Tenants tab with Active status.

End-to-end this is typically 1–3 seconds, dominated by the schema-apply step on the new database.

Connection-string security

The connection string travels over the transport (RabbitMQ) from the console to the target service. The console does not persist it — only the database URI (host + database name) for identification. Two practical consequences:

• Use TLS on your transport (amqps://) in production. The connection string is a credential.
• If you want a clean audit trail, rotate the database password after cutover. The audit log captures the click but not the secret.

Tenant ID normalization

If your Marten store sets TenantIdStyle to ForceLowerCase or ForceUpperCase, both AddTenant and session-open silently rewrite the tenant id before doing anything with it. The Add Tenant dialog warns the operator inline if the typed id would be rewritten — see Services → Add Tenant dialog.

What this means in practice: you cannot have two tenants whose ids differ only in case. The second AddTenant will resolve to the same row as the first.

The mode the service is using is shown on the Service Overview tab under "Tenant ID Style".

Disable / Re-enable / Remove / Hard delete

The four destructive operations — in increasing severity:

Action	Effect	Reversible?
Disable	Soft toggle. Sessions against the tenant throw UnknownTenantIdException. Data preserved.	Yes — re-enable.
Remove	Drops only the master-table row. Per-tenant database left intact.	Yes — re-add with the same id + connection string.
Hard delete	Drops the per-tenant database and removes the master row. Permanent.	No.

Remove is only reversible on a database-per-tenant source

The table above describes database-per-tenant tenancy, where each tenant has its own database and Remove really does drop only the master-table row.

On a shared-database (conjoined) source with partitioning — Wolverine's ConjoinedTenantSource over partitioned EF Core — RemoveTenantAsync deletes the wolverine_tenants registry record and then drops the tenant's data partition with its data (DropTenantAsync(tenantId, deleteData: true)). There is no per-tenant database to leave intact, so there is nothing to re-attach by re-adding the id.

⚠️ On that source shape Remove and Hard delete do the same thing. Hard delete detects a shared-database source by shape and delegates to the very same RemoveTenantAsync rather than running its DROP DATABASE branch — which would otherwise destroy every tenant plus the application's own storage. The severity ladder above collapses to one rung, and only the hard-delete rung carries the typed-id confirmation.

Before using Remove, know which shape your source is. If it is conjoined and partitioned, treat Remove as permanent.

The lifecycle full picture is in Services → Tenants → Lifecycle.

⚠️ Hard delete's database-dropping branch is PostgreSQL-only. It parses the tenant's connection string with Npgsql and issues DROP DATABASE IF EXISTS … WITH (FORCE) against a maintenance connection. SQL Server / Polecat support is tracked on #68 — the equivalent flow needs ALTER DATABASE … SET SINGLE_USER WITH ROLLBACK IMMEDIATE, different connection-string parsing and different administrative privileges. (The shared-database path described above is engine-agnostic: it never touches the database directly.)

Hard delete is gated behind a typed-id confirmation modal — the operator must type the exact tenant id before the Confirm button enables. The intent is friction proportional to the blast radius. Every hard delete records an extra-verbose audit entry with the typed id, database URI, and confirmation time.

What if I add a tenant outside the console?

If your service code adds tenants programmatically (e.g. during onboarding, from your own app's admin UI), CritterWatch will see the new tenant on the next telemetry batch — within ~1 second.

If the new tenant doesn't appear, click the Refresh button on the Tenants tab. That dispatches a RequestTenantList which forces the service to re-publish its current tenant list.

Multiple tenant sources

A service can register more than one IDynamicTenantSource<string> — for example one source backed by your Marten master-table store and a second backed by a Wolverine-managed config table. When that happens, the Tenants tab disambiguates by showing the source name in a column next to the tenant id. The same tenant id can appear in multiple sources without colliding; CritterWatch keys per-tenant state by (tenantId, sourceName).

This matters operationally for two reasons:

• Lifecycle commands FAN OUT across every registered source. Adding acme-corp from the console adds it to every IDynamicTenantSource<string> the service registers, and the same is true of disable, enable and remove; the response carries a per-source result so a partial failure is visible rather than silent. ⚠️ This page previously said the opposite — "lifecycle commands target one source at a time" — which was true before #280 and has not been since. Hard delete fans out too: a tenant with a Marten event-store database and a Wolverine message-store database has both physical databases dropped.
• Per-tenant projection rows multiply by source. A tenant that exists in two sources shows up in two (shard, tenant) rows on the Projections page. This is correct — the two sources back distinct projection shards — but it's worth knowing before you go counting rows.

Snapshot size on very large tenant counts

The per-second telemetry batch (ServiceUpdates) carries one ShardStateSnapshot row per (shard, tenant) plus a PersistenceCounts entry per tenant. On services with hundreds of tenants this can push the compressed payload close to the broker / SQS payload cap (256 KiB on SQS).

The integration ships with a defensive guard: if a snapshot exceeds 240 KiB after compression, the service logs a warning:

warn: Wolverine.CritterWatch[0]
  ServiceUpdates payload exceeded 240 KiB after compression
  (compressed: 244832 bytes, raw: 1834291 bytes).
  Snapshot is at risk of being dropped by size-limited transports.

What to do:

• If you only see one or two of these. Likely a transient: a churning rebuild that briefly fanned shard rows out, a one-off snapshot during a tenant migration. The next batch will be back under the cap.
• If you see them sustained. The service has grown past the heartbeat's comfortable working set. The fix is to split it: either federate the monitored service into smaller per-region services, or move very-low-traffic tenants to a separate service with its own telemetry queue. The 256 KiB SQS cap is the hard ceiling — at that point batches will be silently dropped, and the console will show gaps in your shard / persistence-count history.

This guard runs inside the service. It doesn't fire on the console side.

Rebuilding a projection across every tenant

The per-tenant cards rebuild one tenant's partition each. To rebuild every tenant's partition of a projection, use Rebuild all tenants on the flat projections table's row menu — that row is the projection across tenants, where a per-tenant card is one tenant by construction (GH-821).

It fans out one rebuild per tenant rather than sending a single store-global rebuild, and that distinction matters: on a per-tenant partitioned store each tenant owns its own shard, so a rebuild with a null tenant id addresses the store-global shard and would leave every tenant partition untouched — a console reporting work it did not do.

The confirmation names the count for a reason

At fleet scale this is hundreds of simultaneous rebuilds, and an operator who cannot see the number cannot judge the blast radius. The dialog also points at Rebuild later… (scheduling a rebuild), which is usually the safer route on a large fleet. Each tenant gets its own request id and pending entry, so a partial failure is visible per tenant rather than collapsed into one toast.

What CritterWatch does not do

• No tenant onboarding workflow. The Add Tenant button is a low-level admin tool. Production tenant onboarding (with billing, provisioning, SLA assignment, etc.) belongs in your application's own admin UI; use CritterWatch for ad-hoc additions and incident-time fixes.
• No bulk operations. Each Add / Remove / Hard delete is one tenant at a time. This is deliberate — bulk operations on tenant databases are usually a red flag.
• No per-tenant alert thresholds. Thresholds still cascade global → service → message-type, and not per-tenant. Per-tenant noise is a solved problem though — see Per-tenant alerting for the ceiling, the watch list and per-tenant suppression (GH-1232).