Configuration Reference

Monitored Service Configuration

AddCritterWatchMonitoring()

Call this inside UseWolverine() in each service you want to monitor.

Simple form — the telemetry + control queue URIs (the service identity is the Wolverine ServiceName, not a parameter):

builder.Host.UseWolverine(opts =>
{
    // CritterWatch keys this service by its Wolverine ServiceName — there is no
    // separate service-name parameter on AddCritterWatchMonitoring.
    opts.ServiceName = "my-service";

    opts.AddCritterWatchMonitoring(
        critterWatchUri: new Uri("rabbitmq://queue/critterwatch"),
        systemControlUri: new Uri("rabbitmq://queue/my-service-control"));
});

Full options form:

builder.Host.UseWolverine(opts =>
{
    opts.AddCritterWatchMonitoring(
        // URI of the queue CritterWatch listens on for telemetry
        critterWatchUri: new Uri("rabbitmq://queue/critterwatch"),

        // URI of the queue this service listens on for incoming commands
        systemControlUri: new Uri("rabbitmq://queue/trip-service-control"),

        // How this service exports metrics (default: Hybrid)
        metricsMode: WolverineMetricsMode.Hybrid
    );
});

Options Reference

Parameters of AddCritterWatchMonitoring():

Parameter	Default	Description
critterWatchUri	—	URI of the queue CritterWatch listens on for this service's telemetry
systemControlUri	(none)	URI of the queue this service listens on for commands from CritterWatch. Null opts out of operator commands.
metricsMode	Hybrid	How this service exports metrics — see metrics modes
heartbeatInterval	30 seconds	Cadence of the WolverineHeartbeat liveness ping each node sends. The dashboard's liveness dot turns amber after one missed beat and red after five. TimeSpan.Zero disables heartbeats, and the service's nodes then render grey/unknown.
configureBaselines	(none)	Optional callback to declare expected throughput/execution-time baselines for this service. See Alerts › Editing Thresholds.
configureShardedTopology	(none)	Optional callback declaring sharded telemetry queue slots. Supplying it is what turns clustering on — see Clustering.

The service's identity is the Wolverine ServiceName, not a parameter here.

Properties on the returned CritterWatchOptions:

Property	Default	Description
PublishInterval	2 seconds	How long the observer accumulates telemetry before publishing it as one ServiceUpdates. See Publish cadence below.
CaptureConversations	true	Capture per-envelope causality (conversation/correlation/parent-span) for the conversation graph.
ConversationCaptureBufferSize	5000	Bounded client-side buffer of captured hops between flushes; oldest are dropped when full.
EnableEventStoreExplorer	(off)	Opt in to the event-store explorer diagnostic handlers.
TraceProvider(name)	(none)	Declare the preferred CritterWatch-side trace provider.
MetricsDataSource(name)	(none)	Declare the preferred CritterWatch-side metrics data source.

Agent health polling is a process-wide setting rather than a per-service option: CritterWatchObserver.HealthCheckInterval (default 60 seconds) and CritterWatchObserver.StateSnapshotInterval (default 60 seconds).

Publish cadence and console write volume

Every publish is one write of this service's ServiceSummary document in the console's store, and that document grows with the number of endpoints, handlers and message types the service reports — hundreds of kilobytes is normal for a large service. Because the projection is Inline, each write is a full document rewrite, so PostgreSQL write volume, TOAST churn and autovacuum work all scale linearly with the publish rate. A single-service deployment publishing every second was measured holding a one-row mt_doc_servicesummary table at 370 MB with 19,266 autovacuums.

Widening PublishInterval never drops telemetry — updates merge into the same batch, so the only effect is up to that much extra latency before the console sees them. The default is far inside the console's own cadences (30 s alert evaluation, 60 s health-check and state-snapshot intervals), so raising it is cheap:

builder.Host.UseWolverine(opts =>
{
    var critterWatch = opts.AddCritterWatchMonitoring(
        critterWatchUri: new Uri("rabbitmq://queue/critterwatch"),
        systemControlUri: new Uri("rabbitmq://queue/trip-service-control"));

    // Each publish is a full rewrite of this service's ServiceSummary document in the
    // console's store, so write volume scales linearly with this cadence. Nothing is
    // dropped by widening it — updates merge into the same batch.
    critterWatch.PublishInterval = TimeSpan.FromSeconds(10);
});

Large deployments — many endpoints, many projection agents, or a console store shared with the monitored application's database — should raise it. Values below 250 ms are clamped.

Service Name Must Be Unique

The ServiceName is used as the Marten event stream key. Two services with the same name will overwrite each other's state. Use a name that uniquely identifies the service across your entire deployment.

CritterWatch Server Configuration

AddCritterWatch()

var builder = WebApplication.CreateBuilder(args);

builder.AddCritterWatch(
    builder.Configuration.GetConnectionString("critterwatch")!,
    opts =>
    {
        opts.UseRabbitMq(new Uri("amqp://localhost")).AutoProvision();
        opts.ListenToRabbitQueue("critterwatch").UseCritterWatchSerializer();
    });
// Single-node is the default — nothing else to configure. For a multi-node
// cluster, supply configureClusterShardedTopology (see Deployment › Clustering).

var app = builder.Build();
app.UseCritterWatch();
app.Run();

UseCritterWatch()

Maps all CritterWatch middleware into the ASP.NET Core pipeline:

app.UseCritterWatch();

// With a custom SignalR route:
app.UseCritterWatch(signalRRoute: "/my-hub");

UseCritterWatch() registers:

• Wolverine HTTP endpoints under /api/critterwatch/*
• SignalR hub at /api/messages (configurable)
• Static file serving for the embedded Vue SPA
• Client-side routing fallback for the SPA

Storage schema

All CritterWatch data — its documents (service summaries, alerts, metrics rollups) and its event store — is isolated to a dedicated database schema, so it never collides with the host application's own tables. The default schema is critterwatch, and the name is configurable via the schemaName parameter:

builder.AddCritterWatch(
    builder.Configuration.GetConnectionString("critterwatch")!,
    schemaName: "monitoring");   // default: "critterwatch"

The same schemaName parameter is available on the lower-level opts.AddCritterWatchServices(...) registration (both the Marten/PostgreSQL and Polecat/SQL Server flavors) and on opts.AddCritterWatchEmbedded(...). CritterWatch only ever creates and migrates tables inside that one schema.

Why a dedicated schema matters

This is what lets CritterWatch share a database with the application it monitors without stepping on it — most important for embedded CritterWatch (CritterWatch.Embedded), where the console runs inside your app's own process and database. Point schemaName at any schema you like; CritterWatch keeps all of its storage there.

Docker Compose

A complete docker-compose.yml for local development:

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: critterwatch
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

  rabbitmq:
    image: rabbitmq:3-management
    ports:
      - "5672:5672"    # AMQP
      - "15672:15672"  # Management UI
    environment:
      RABBITMQ_DEFAULT_USER: guest
      RABBITMQ_DEFAULT_PASS: guest

volumes:
  postgres_data:

Connection String Formats

PostgreSQL

Host=localhost;Port=5432;Database=critterwatch;Username=postgres;Password=postgres

For cloud providers:

Host=my-postgres.postgres.database.azure.com;Database=critterwatch;Username=app@my-postgres;Password=secret;SSL Mode=Require

Connection pool ceiling

Npgsql defaults Max Pool Size to 100 per process. The console runs a number of independent readers (projection progression, rebuild batches and cells, table-size estimates, envelope counts, metrics) plus Wolverine's durability agent; a single console pod monitoring one service was measured holding ~25 connections, most of them idle.

CritterWatch therefore applies its own default ceiling of 50 when your connection string doesn't specify one. An explicit Max Pool Size in the connection string always wins — including a larger one. To change the default without touching the connection string:

{
  "CritterWatch": {
    "Postgres": {
      "MaxPoolSize": 25
    }
  }
}

Consoles sharing a database with the monitored application

If the console's store lives on the same instance as the application it monitors, size this deliberately. Exhausting max_connections there takes down the monitored application, not just the console — the failure mode is FATAL: remaining connection slots are reserved for roles with privileges of the "pg_use_reserved_connections" role in the application's logs.

RabbitMQ

amqp://guest:guest@localhost:5672/
amqps://user:pass@my-rabbit.cloud:5671/  # TLS

Amazon SQS

AddCritterWatchMonitoring works identically against the SQS transport. Two queues are involved per monitored service:

Queue	Direction	Purpose
critterWatchUri	service → CritterWatch	Metrics, heartbeats, capability snapshots
systemControlUri	CritterWatch → service	Pause / restart listeners, rebuild projections, DLQ ops

opts.UseAmazonSqsTransport();
opts.AddCritterWatchMonitoring(
    critterWatchUri:  SqsEndpointUri.Queue("critterwatch"),
    systemControlUri: SqsEndpointUri.Queue("critterwatch-control-trip-service"));

Dead letter queues with AutoProvision() off

By default the WolverineFx.AmazonSqs transport attaches every listener to its DefaultDeadLetterQueueName (wolverine-dead-letter-queue). With AutoProvision() enabled the broker creates that queue on startup, so the control listener AddCritterWatchMonitoring installs is wired up cleanly.

In production environments with AutoProvision() off, that default DLQ typically isn't pre-provisioned. The broker startup then fails:

Wolverine.AmazonSqs.WolverineSqsTransportException: Error while trying to
initialize Amazon SQS queue 'wolverine-dead-letter-queue'
 ---> Amazon.SQS.Model.QueueDoesNotExistException

Three ways to handle it, pick whichever fits your infra automation:

1. Provision the default DLQ alongside your application queues (CDK / Terraform / etc.). Nothing to change in code.

2. Point the control listener at an existing DLQ you already provision. Re-open the same endpoint after AddCritterWatchMonitoring and chain DeadLetterQueueName:

opts.AddCritterWatchMonitoring(critterWatchUri, systemControlUri);
opts.ListenToSqsQueue("critterwatch-control-trip-service", q =>
{
    q.DeadLetterQueueName = "trip-service-dlq";
});

3. Disable native DLQs on this transport entirely (rely on Wolverine's durability instead):

opts.UseAmazonSqsTransport(t => t.DisableAllNativeDeadLetterQueues());
opts.AddCritterWatchMonitoring(critterWatchUri, systemControlUri);

Same shape applies to other native-DLQ transports

Azure Service Bus and the other transports with a default DLQ behave the same way. Whichever DLQ strategy you pick for the rest of your application's listeners, the queues AddCritterWatchMonitoring installs inherit the same contract — there's no special CritterWatch DLQ to provision.

Native DLQs and CritterWatch management

CritterWatch's Dead Letter Queue explorer manages the durable (database) DLQ only — the failed messages Wolverine persists to its message store. It has no visibility into broker-native dead-letter queues (Amazon SQS DLQ, Azure Service Bus $DeadLetterQueue, RabbitMQ DLX). A message that dead-letters natively stays at the broker and won't appear in CritterWatch until it's forwarded into the Wolverine database.

So for CritterWatch to manage a service's dead letters, those failures need to land in — or be propagated into — the durable store. Two approaches:

A. Send failures straight to the durable store (no native DLQ). Simplest for CritterWatch; it changes where your dead letters live, so weigh it against any existing native-DLQ tooling or alarms you rely on.

// Amazon SQS — disable native DLQs transport-wide; failures go to the
// Wolverine durability database instead.
opts.UseAmazonSqsTransport(t => t.DisableAllNativeDeadLetterQueues());

On RabbitMQ the per-endpoint equivalent is the WolverineStorage dead-letter mode (failures bypass the native DLX and go to the durable store).

B. Keep the native DLQ and forward it into the durable store. Preserves your existing native dead-lettering and surfaces those messages in CritterWatch — the better fit when retrofitting onto a running system.

• RabbitMQ has this built in — Wolverine listens on the native DLQ, reconstructs each envelope, and writes it to the durability database where CritterWatch can replay or discard it:

  opts.UseRabbitMq(/* ... */).EnableDeadLetterQueueRecovery();

• Amazon SQS / Azure Service Bus don't yet have a built-in equivalent (tracked upstream in wolverine#3103). Until it lands, either use approach A, or stand up a listener on the native DLQ that forwards each message to the store via IMessageInbox.MoveToDeadLetterStorageAsync(...).

Retrofitting without disrupting the host

Adding CritterWatch shouldn't change queues your application already owns. Keep AutoProvision() scoped (or pre-provision CritterWatch's critterwatch and control queues through your infrastructure automation), and use per-endpoint DLQ settings so CritterWatch never re-declares or alters the host's existing dead-letter infrastructure.

appsettings.json

The recommended approach for managing connection strings:

{
  "ConnectionStrings": {
    "critterwatch": "Host=localhost;Database=critterwatch;Username=postgres;Password=postgres",
    "rabbitmq": "amqp://localhost"
  }
}

builder.AddCritterWatch(
    builder.Configuration.GetConnectionString("critterwatch")!,
    opts =>
    {
        var rabbitUri = new Uri(
            builder.Configuration.GetConnectionString("rabbitmq")!);
        opts.UseRabbitMq(rabbitUri).AutoProvision();
        opts.ListenToRabbitQueue("critterwatch").UseCritterWatchSerializer();
    });

Timeline retention

The Health Timeline (and the dashboard's Recent Events widget) is backed by an append-only TimelineEntry document per lifecycle fact. CritterWatch sweeps that table on a background timer — hourly by default, on the cluster leader only — so it stays bounded without an operator ever having to run a cleanup script.

The sweep makes three passes, each independently configurable under CritterWatch:Timeline:

{
  "CritterWatch": {
    "Timeline": {
      "RetentionPeriod": "30.00:00:00",
      "PruneInterval": "01:00:00",
      "CompactRedundantEntries": true,
      "CompactableEventTypes": [ "AgentStarted" ],
      "CompactionGracePeriod": "00:15:00",
      "MaxEntriesPerService": 0,
      "DeleteBatchSize": 500,
      "MaxDeletesPerSweep": 50000
    }
  }
}

Key	Default	Meaning
RetentionPeriod	30.00:00:00 (30 days)	Entries older than this are deleted. 00:00:00 keeps entries forever.
PruneInterval	01:00:00 (1 hour)	How often the sweep runs. Every pass is idempotent, so the cadence isn't load-bearing.
CompactRedundantEntries	true	Collapse a run of consecutive identical entries — same service, event type, subject, title, severity, description — down to the first entry of the run.
CompactableEventTypes	[ "AgentStarted" ]	Which event types compaction applies to. Configuring this replaces the default list.
CompactionGracePeriod	00:15:00	Entries newer than this are never compacted, so the sweep can't race the live feed.
MaxEntriesPerService	0 (off)	Opt-in hard cap: keep only the newest N entries per service, regardless of age.
DeleteBatchSize	500	Documents deleted per transaction. Pruning a large backlog stays chunked instead of taking one long table-wide lock.
MaxDeletesPerSweep	50000	Ceiling on deletions per sweep. A big accumulated backlog drains over successive sweeps rather than in one transaction storm. 0 = no ceiling.

Why compaction matters more than the age window. Before 1.0.0-beta.4, every keep-alive from a monitored service re-reported its running agents, and each re-report was materialized as a fresh "Agent started" entry — thousands per hour for a modest fleet, burying the events an operator actually cares about. That emit path is fixed (an AgentStarted is now written only when the agent → node assignment really changed), but existing deployments still carry the accumulated rows, and those rows are all inside any sane retention window — no age policy would ever remove them. Compaction is what reclaims them: it keeps the entry that recorded the transition and deletes the re-reports behind it. A genuine change (an agent moving to a different node) breaks the run and is always preserved.

Keeping long history

Set RetentionPeriod to 00:00:00 to keep timeline entries indefinitely. Compaction still runs, so the redundant re-reports go away while every real transition is kept.

Metrics storage and sizing

Read this before running the console against a system under load

The built-in metrics collection is the largest thing the console writes, by roughly two orders of magnitude. A production deployment measured mt_doc_metricssample at 2,590 MB / 2.65 million rows in two and a half hours — about 1 GB/hour — against 20 MB for the next-largest document table. On that fleet it accounted for ~99% of the console's database traffic.

The default is the right choice for getting started: zero extra infrastructure, works out of the box. It is not the right choice for a system under real load. See Use an external metrics stack under load below, and size the knobs in this section deliberately if you stay on the built-in mode.

What gets stored

The console persists one MetricsSample row per (service, message type, destination, tenant, bucket). Multiply those out before estimating — the tenant axis is what surprises people:

• Each 1-minute bucket writes a row per (message type × destination) per service.
• On a multi-tenant service, the console also writes one row per real tenant alongside the store-global aggregate, so per-tenant evaluation cannot be averaged away by a busy neighbour. Since 1.0 the tenant rows are capped at PerTenantSampleCeiling (default 25) per series-bucket — the busiest tenants keep their own rows, the remainder folds into one "*OTHER*" row — so a service with 850 active tenants writes bounded per-tenant rows instead of 850 per series.
• Only active combinations materialize rows, so the table grows with tenant activity, not with tenant count — quiet tenants becoming active push it up without any fleet change.

Rows fold to hourly granularity past SampleHotWindow, and whole monthly partitions drop past SampleRetentionPeriod — but both act on data you have already written and paid to store, and the hourly fold preserves every dimension of the key, tenant included. The hourly tail is smaller only by the 60:1 bucket fold; it still carries the full tenant multiplier.

Steady-state row count, using per-bucket figures you can measure on your own deployment (select count(*) from mt_doc_metricssample group by bucket_start on a recent minute):

rows ≈ active (message type × destination × (capped tenants + 1)) combos
       × (hot-window minutes  +  retained-tail hours)

A field data point for scale: one monitored service with 512 shard databases and ~530 active tenants measured ~4,300 rows per minute-bucket (~912 bytes each) — ~12.5M resident rows for a 2-day hot window plus a 43-day hourly tail of similar order, before the per-tenant cap shipped.

Retention and sampling knobs

Metric samples are bucketed, rolled up and pruned on background timers. The knobs bind from CritterWatch:Metrics:

{
  "CritterWatch": {
    "Metrics": {
      "SampleRetentionPeriod": "45.00:00:00",
      "SampleHotWindow": "2.00:00:00",
      "SampleBucketWidth": "00:01:00",
      "BaselineLookback": "28.00:00:00"
    }
  }
}

Key	Default	Description
SampleRetentionPeriod	45.00:00:00	How long MetricsSample rows are kept. The table is range-partitioned by month, so retention drops whole partitions rather than deleting rows. 00:00:00 disables pruning.
SampleHotWindow	2.00:00:00	How long samples stay at 1-minute granularity. Past this, the leader folds each hour into one hourly row. Baselines read across both granularities, so totals are unaffected — you lose only within-hour shape for older data.
SampleBucketWidth	00:01:00	Quantization width for incoming samples — one row per window instead of ~60 one-second inserts. Widening it thins the throughput evaluator's 15-minute read window (2–3 closed buckets at 00:05:00 instead of ~15) and can under-report throughput enough to trip low-throughput alerts falsely — prefer the hot window and tenant ceiling as sizing levers.
PerTenantSampleCeiling	25	Cap on distinct per-tenant rows per (service × message type × destination × bucket) series. The busiest tenants keep their own rows; the remainder folds into one "*OTHER*" row, so totals stay exact while the tenant multiplier is bounded. Per-tenant metrics alerting only evaluates tenants with their own rows. 0 removes the cap.
SampleFlushInterval	00:00:15	How often the accumulator flushes closed in-memory buckets to the store.
PersistOpenBuckets	false	Write still-accumulating buckets on every flush. Off by default: at the defaults it wrote each row four times per minute, three of them an intermediate total superseded before anything read it. Turn it on only if you need the open bucket visible to readers within the bucket width.
SamplePruneInterval	01:00:00	How often the retention prune runs (leader only).
SampleRollupInterval	01:00:00	How often aged minute rows are folded to hourly (leader only).
MaxRollupHoursPerPass	24	Cap on hours folded per rollup pass, so a large backlog drains over successive passes instead of in one long tick.
BaselineLookback	28.00:00:00	How far back the hour-of-day / day-of-week baseline lookup reads. Four weeks gives every (hour, weekday) cell four observations. 00:00:00 reads all retained history.

Shrinking SampleHotWindow is the highest-leverage knob

Minute rows outnumber their hourly fold 60:1, and the hot window decides how many of them are resident. Dropping it from 7 days to 2 removes roughly 70% of the resident minute rows at no cost to any shipped view — GetThroughputSeriesAsync reads both granularities and slots by bucket end, so totals are identical either way.

Sizing BaselineLookback

Keep it comfortably above the 10-day baseline minimum and below SampleRetentionPeriod — reading past retention can only find rows that are about to be pruned. Widening it directly widens the amount of data the baseline query scans on every alert evaluation, and on a busy multi-month table that read is expensive even with the (ServiceName, HourOfDay, DayOfWeek) index in place.

Use an external metrics stack under load

If you are tuning SampleRetentionPeriod because the table is too big, that is the signal to change mode rather than the knob. A purpose-built time-series store does this job better and cheaper, and keeps the load off the database your monitored application is using.

On the monitored service, stop persisting samples in the console's store:

// SystemDiagnosticsMeter: the monitored service persists NOTHING in the
// console's store — metrics flow only through .NET's System.Diagnostics.Metrics
// for an external scraper (Prometheus, VictoriaMetrics, Datadog agent, ...).
// Bind the service to a metrics data source on the console and every metrics
// view + alert evaluator reads through the external store instead.
var builder = WebApplication.CreateBuilder(args);

builder.Host.UseWolverine(opts =>
{
    opts.UseRabbitMq(new Uri("amqp://localhost")).AutoProvision();

    opts.AddCritterWatchMonitoring(
        critterWatchUri: new Uri("rabbitmq://queue/critterwatch"),
        systemControlUri: new Uri("rabbitmq://queue/trip-service-control"),
        metricsMode: WolverineMetricsMode.SystemDiagnosticsMeter
    );
});

builder.Build().Run();

Then bind that service to a metrics data source on the console (Prometheus, VictoriaMetrics, Datadog, or Application Insights) — see Settings → Metrics Data Sources, Datadog metrics, and Application Insights metrics. The console polls the external store instead of accumulating its own samples.

What you keep: every metrics view, and alerting — the evaluators read through the bound data source. What stops applying: the retention/sampling knobs above, because the console is no longer the one storing the data. Its retention becomes your time-series store's retention.

A service in this posture — SystemDiagnosticsMeter mode and bound to an external source — defaults to live-only persistence: the console writes no MetricsSample / MessagingMetricsBucket rows for it at all. An explicit External metrics persistence override on the service (Settings, or the alert-overrides API) wins in either direction; Persist same as internal restores the old always-persist behaviour. Before this defaulting (#938), live-only was a buried per-service opt-in and the console silently paid the ~1 GB/hour storage cost for data nothing read.

Alert thresholds

Most alert thresholds are tuned in the UI rather than in configuration files — see Alert Configuration for the live preview, history tab, and three-level cascade (global → per-service → per-message-type).

Defaults that ship with the console:

Threshold	Default
DLQ count Warning / Critical	10 / 100
Projection lag Warning / Critical	30s / 300s
Agent unhealthy Warning / Critical	2 / 5 consecutive checks
DLQ rate / hour Warning / Critical	10 / 50
Failure rate Warning / Critical	5% / 20%
Throughput multiplier Warning / Critical	3× / 10× of baseline
Exec time Warning / Critical	+50% / +200% over baseline

For services that need different defaults baked in (rather than tuned post-deploy), declare baselines from AddCritterWatchMonitoring — see Registration → Declared Baselines.