Deploying CritterWatch as a cluster

Single-node is the default, and clustering is opt-in — what you actually have to configure depends entirely on your deployment shape, and for the common single-node case the answer is "nothing."

Which applies to me?

• One BFF node? Do nothing — AddCritterWatch(connectionString, configureWolverine) works out of the box. Clustering stays off, and on a single node there's nothing to distribute. See Single node below.
• Two or more BFF nodes behind a load balancer? Pick a partition count N and wire a matching sharded topology on both the BFF and every monitored service, with the same N — supplying that topology is what turns clustering on. See Enabling global partitioning. You'll also want the Redis SignalR backplane.

Single node (no extra configuration)

A single-node CritterWatch needs no extra configuration — clustering is off by default, so there is nothing to distribute: ICritterWatchMessage traffic routes through the single critterwatch queue and per-service single-writer ordering is handled in-process. You don't supply configureClusterShardedTopology, and you don't need a sharded transport:

builder.AddCritterWatch(connectionString, configureWolverine: opts =>
{
    opts.UseRabbitMq(/* ... */).AutoProvision();
    opts.ListenToRabbitQueue("critterwatch").UseCritterWatchSerializer();
});

Clustering turns on the moment you supply a sharded topology (or set enableClusterPartitioning: true explicitly). Scaling beyond one node is opt-in and additive — the rest of this page covers it.

What changes when you cluster

Concern	Single node	Clustered
Per-service single-writer	local (in-process slots)	GlobalPartitioned distributes by service id; no two nodes process the same service's updates concurrently
Periodic singleton jobs (alert evaluators, retention prunes, rollups, rebuild dispatch)	the sole node runs one agent per responsibility	CritterWatchSingletonAgentFamily agents are distributed across nodes by Wolverine node coordination, with automatic failover — exactly one loop runs cluster-wide per responsibility, and no periodic work piles onto the leader
External metrics scraping	the sole node runs one agent per configured data source	MetricsScrapingAgentFamily agents are distributed across nodes by Wolverine node coordination, with automatic failover — each data source is scraped by exactly one node
SignalR fan-out across browsers	in-process hub	Redis backplane fans out every server-pushed message to all connected clients across nodes
Marten async daemon	self-distributes via Wolverine-managed subscription distribution (unchanged)	same

Enabling the Redis SignalR backplane

The backplane is config-driven: set a redis connection string and CritterWatchHostingExtensions.AddCritterWatch automatically chains AddStackExchangeRedis() onto the SignalR builder. Absent the connection string, single-node SignalR (the default) keeps working.

Aspire (dev / test)

The Aspire BffHost declares a redis resource and references it from the BFF — Aspire injects ConnectionStrings__redis automatically, so a clustered dev run is dotnet run from src/BffHost with no extra knobs.

Docker Compose

docker-compose.yml ships a redis:7-alpine service on the default port. Hosts running outside Aspire set the connection string in appsettings.json (or via ConnectionStrings__redis):

{
  "ConnectionStrings": {
    "redis": "localhost:6379"
  }
}

Azure SignalR (opt-in, documented-and-supported)

Wolverine.SignalR uses the standard ASP.NET Core Hub / IHubContext, so any scale-out provider that hooks into the SignalR DI builder fans out cross-node with no broadcast-code changes. To use Azure SignalR Service instead of the Redis backplane, do not set the redis connection string and add Azure SignalR alongside AddCritterWatch:

builder.AddCritterWatch(connectionString);
builder.Services.AddSignalR().AddAzureSignalR();

Exactly one backplane per deployment. CritterWatch only ships the Redis integration out of the box; Azure SignalR is documented but not bundled or CI-exercised.

Enabling global partitioning (per-service single-writer)

The Redis backplane handles fan-out of outbound SignalR traffic. Global partitioning is the matching story on the inbound side: it guarantees that all updates for a given monitored service land on a single BFF node, cluster-wide, so two BFF nodes never race to project the same ServiceSummary aggregate. It's opt-in and additive — single-node deployments don't need it.

Pick N = your expected BFF node count

The integer N you pass to UseSharded…Queues(...) is the partition count: that many physical sharded queues are declared on the transport, and Wolverine hashes each message's group id (the monitored service's ServiceName / Id) mod N to decide which slot it lands on. Each slot is owned by exactly one BFF node.

Set N to the number of BFF nodes you expect to run. With N = 5 and 3 nodes, two nodes carry two slots each and one carries one — load skews slightly but every node has work. With N smaller than your node count, some BFF nodes sit idle (Wolverine assigns each slot to a single node). With N much larger than your node count, you pay overhead for queues you don't need.

N must agree exactly between the producer side and the consumer side — mismatched hashes route to slots the consumer isn't listening on. Pick one value and centralise it (a shared constant, environment variable, or the service-handshake mechanism the BFF already uses for capability negotiation).

Wire the consumer side (BFF)

// #953 — the entry point IS the decision. The sharded topology is a required
// argument here, so a clustered console with no topology cannot be written;
// for a single console instance call AddCritterWatchForSoloHost instead.
opts.AddCritterWatchForClusteredHost(
    NpgsqlDataSource.Create(connectionString),
    configureShardedTopology: topology =>
    {
        // Mix and match per the transports this BFF actually uses.
        topology.UseShardedRabbitQueues("critterwatch", 5);
        // topology.UseShardedAmazonSqsQueues("critterwatch", 5);
        // topology.UseShardedAzureServiceBusQueues("critterwatch", 5);
    },
    // Everything else is optional and lives on CritterWatchOptions.
    configure: cw => cw.ConfigureHub = hub => hub.EnableDetailedErrors = true);

configureShardedTopology is a required positional argument, so "clustered with no topology" is not something you can write — it is a compile error rather than a startup one. (Wolverine's GlobalPartitionedMessageTopology.AssertValidity() requires a sharded external topology be registered alongside the message subscription; the old AddCritterWatchServices could only catch that at startup, with an ArgumentNullException.)

Which entry point (#953)

Console instances	Call	Needs a sharded topology	Needs a SignalR backplane
1	AddCritterWatchForSoloHost	no	no
2+	AddCritterWatchForClusteredHost	yes (required argument)	yes

The decision used to be spread across enableClusterPartitioning, whether a topology happened to be supplied, and whether a redis connection string happened to be set — three implicit signals for one decision, each of which failed silently in the wrong direction. AddCritterWatchServices is now [Obsolete] and forwards to the same core, so existing deployments keep working unchanged.

Declaring a sharded topology is not the same as running multiple nodes. The dev BFF runs five shard slots on a single node, which is a normal and supported state. So the backplane requirement is checked against the observed node count at runtime, not assumed from the call:

• registered solo, but >1 node running — logged at Critical. This is the corrupting one: a solo console is the single writer to every ServiceSummary stream by construction, and N nodes means N concurrent writers to the same streams.
• >1 node running with DefaultHubLifetimeManager<> — logged at Critical. No backplane means a browser connected to node B renders nothing for work that happened on node A, which looks exactly like the console going blind and is almost never diagnosed as a missing backplane.

The probe is on the resolved HubLifetimeManager<WolverineHub> type rather than on a redis connection string, which is what makes it work for Azure SignalR and anything else, and makes it independent of registration order. Neither condition kills the process — a monitoring console in a crashloop takes away the one tool that would have shown you what is wrong.

Wire the producer side (every monitored service)

opts.AddCritterWatchMonitoring(
    critterWatchUri: new Uri("rabbitmq://queue/critterwatch"),
    systemControlUri: new Uri("rabbitmq://queue/my_service_control"),
    configureShardedTopology: topology =>
    {
        // Same value of N. Same transport-specific call.
        topology.UseShardedRabbitQueues("critterwatch", 5);
    });

Azure Service Bus variant

The same shape with UseShardedAzureServiceBusQueues — sample BFF + producer pair:

// BFF
opts.AddCritterWatchServices(
    NpgsqlDataSource.Create(connectionString),
    configureClusterShardedTopology: topology =>
    {
        topology.UseShardedAzureServiceBusQueues("critterwatch", 5);
    });

// Each monitored service
opts.AddCritterWatchMonitoring(
    critterWatchUri: new Uri("azureservicebus://queue/critterwatch"),
    systemControlUri: new Uri("azureservicebus://queue/my_service_control"),
    configureShardedTopology: topology =>
    {
        topology.UseShardedAzureServiceBusQueues("critterwatch", 5);
    });

The N-matching constraint is identical to the RabbitMQ case (5 here must match on both sides). The same rollout-order rule applies — BFF first, then monitored services — so the consumer is on the matching N before any producer starts publishing onto the sharded slots.

UseShardedAmazonSqsQueues follows the same pattern. The transport-specific call resolves the shard naming and provisioning to whatever the underlying broker convention is (queue per shard on RabbitMQ / SQS, subscription per shard on ASB).

Once both sides ship

ICritterWatchMessage traffic (ServiceUpdates, AgentHealthReport, ShardStatesChanged, …) flows over the sharded slots. Heartbeats (WolverineHeartbeat) and MessageHandlingMetrics keep flowing over the unsharded critterwatch URI you've always passed — the BFF deliberately doesn't shard those, and a sharded slot with no listener would dead-letter them.

What if I only ship one side?

The producer and consumer hooks are independent rollouts. Both have a default-off path so half-finished migrations are graceful:

Producer	Consumer	What happens
sharded	sharded	Full per-service single-writer. Recommended for multi-BFF deployments.
sharded	single (default)	Producer's ICritterWatchMessage lands on the sharded slots but no BFF is listening on them — messages stall on the broker. Don't roll out the producer side until the BFF is on the matching N.
single (default)	sharded	BFF still listens on the legacy single critterwatch queue alongside the sharded slots. Older monitored services keep working untouched. Roll out the consumer side first.
single (default)	single (default)	Single-queue legacy path. The BFF's listener is ListenOnlyAtLeader()-pinned (see below) so multi-node BFFs don't split-brain on it.

Legacy single-queue listener is leader-pinned

Even without partitioning, the BFF's ListenToRabbitQueue("critterwatch") and ListenToSqsQueue("critterwatch") call .ListenOnlyAtLeader(). In a single-node deployment that's identical to the pre-leader-aware default (the sole node is the leader). In a multi-node deployment, only one node consumes the legacy queue at a time — preventing the optimistic-concurrency retry storms and split-brain ServiceSummary processing that competing consumers on a single queue would otherwise cause. The sharded slots stay leader-agnostic; only this back-compat queue is leader-pinned.

Load balancer requirements

Health endpoints are LB-appropriate (each node serves /health); the boot-smoke CI gate asserts the same endpoint reports Healthy.

No sticky sessions required. The Redis backplane fans every SignalR send to every node, so a client that connects to node B receives updates produced on node A. The same property holds for Azure SignalR. Configure the LB for plain round-robin (or least-connections) over WebSocket — sticky sessions add no value and can mask backplane misconfiguration.

Cluster correctness audit

The periodic background responsibilities in CritterWatch.Services are classified as follows:

Responsibility	Classification	Why
Metrics alert evaluation, projection alert evaluation, idle rollup re-eval, metrics retention prune, minute→hour rollup, timeline retention sweep, rebuild dispatch, rebuild stale-cell sweep	agent family (CritterWatchSingletonAgentFamily, one agent each)	single-owner work: alert records + notification side effects must not duplicate, table-wide sweeps must not race, rebuild cells must not be double-claimed. Node coordination distributes the eight agents across the cluster and restarts them on a surviving node on failover — exactly one loop per responsibility cluster-wide, and periodic work no longer piles onto the leader.
External metrics scraping	agent family (MetricsScrapingAgentFamily)	one agent per configured data source (Prometheus / VictoriaMetrics / AppInsights / Datadog); same distribution + failover. Different sources scrape concurrently on different nodes.
MetricsSampleFlushService	per-node OK	flushes the samples THIS node accumulated as a writer; every node must flush its own
StateRefreshService	per-node OK	refreshes its own connected clients; backplane fans out
AlertBatchAccumulator	per-node OK	batches what this node received; backplane fans out
SignalRBatchAccumulator	per-node OK	same shape

The #217 leader-pinned tick queues (every node publishing a tick, only the elected leader listening) are gone as of #955 — the agent shape replaced them wholesale. The one remaining leader pin is the legacy single critterwatch telemetry queue described above, which is a transport concern, not a periodic job.