CritterWatch

Appearance

Inbound Commands ​

Inbound commands are messages sent from the CritterWatch server to monitored services. They implement either IServiceMessage (fire-and-forget control commands) or IServiceQuery (request/response queries that return data) and are routed directly to the target service's CritterWatch listener.

All commands are automatically handled when AddCritterWatchMonitoring() is called — no additional handler registration is needed.

The signatures shown below are the live record declarations from Wolverine.CritterWatch/Messages/Inbound/*.cs. The companion await bus.SendAsync(...) line for each is the minimal call you'd issue from a custom integration; in CritterWatch's own UI flows these are emitted from the SignalR hub on operator action.

Operations gating: every mutating command respects the global "Operations Enabled" flag. When disabled, the BFF refuses to send and the matching UI button renders disabled. Queries (IServiceQuery) are always allowed.

DLQ Commands ​

ReplayMessages ​

Replay specific dead-lettered messages back through the processing pipeline. The IDs are the envelope ids returned from a DLQ summary or query.

csharp
public record ReplayMessages(
    string ServiceName,
    Uri StoreUri,
    Guid[] IdList) : IServiceMessage;
csharp
await bus.SendAsync(new ReplayMessages(
    "trip-service",
    new Uri("postgresql://localhost/trip-db"),
    [envelopeId]));

DiscardMessages ​

Permanently remove dead-lettered messages from the queue.

csharp
public record DiscardMessages(
    string ServiceName,
    Uri StoreUri,
    Guid[] IdList) : IServiceMessage;
csharp
await bus.SendAsync(new DiscardMessages(
    "trip-service",
    new Uri("postgresql://localhost/trip-db"),
    [envelopeId]));

EditScheduledMessage ​

Edit a scheduled message's body and/or scheduled time before it fires. Either edit may be null. The MessageId is the envelope id of the scheduled message.

csharp
public record EditScheduledMessage(
    Guid MessageId,
    string? EditedBodyJson,
    DateTimeOffset? NewScheduledTime) : IServiceMessage
{
    public string ServiceName { get; init; } = "";
}
csharp
await bus.SendAsync(new EditScheduledMessage(
    messageId,
    EditedBodyJson: """{"amount":42}""",
    NewScheduledTime: DateTimeOffset.UtcNow.AddHours(1))
    { ServiceName = "trip-service" });

Listener Commands ​

PauseListener ​

Hard-stop processing on a single listener endpoint. In-flight messages are abandoned. Use DrainListener for graceful shutdown.

csharp
public record PauseListener(string ServiceName, string EndpointUri) : IServiceMessage;
csharp
await bus.SendAsync(new PauseListener("trip-service", "rabbitmq://exchange/trips/"));

RestartListener ​

Resume a paused or drained listener.

csharp
public record RestartListener(string ServiceName, string EndpointUri) : IServiceMessage;
csharp
await bus.SendAsync(new RestartListener("trip-service", "rabbitmq://exchange/trips/"));

DrainListener (#67) ​

Graceful shutdown of a single listener — stops accepting new messages, lets queued and in-flight messages finish, then halts. The right call before a deploy or a planned-maintenance window. Distinct from PauseListener which abandons in-flight work.

csharp
public record DrainListener(string ServiceName, string EndpointUri) : IServiceMessage;
csharp
await bus.SendAsync(new DrainListener("trip-service", "rabbitmq://exchange/trips/"));

PauseAllListeners ​

Pause every listener in the service. Useful before an emergency maintenance window. Mirrors the per-listener PauseListener but applies to all endpoints in one round trip.

csharp
public record PauseAllListeners(string ServiceName) : IServiceMessage;
csharp
await bus.SendAsync(new PauseAllListeners("trip-service"));

RestartAllListeners ​

Resume every listener in the service.

csharp
public record RestartAllListeners(string ServiceName) : IServiceMessage;
csharp
await bus.SendAsync(new RestartAllListeners("trip-service"));

Endpoint Configuration Commands ​

UpdateEndpointBufferingLimits ​

Adjust the in-memory buffering thresholds on a single endpoint at runtime. Maximum is MaximumMessagesToReceive; Restart is the recovery threshold below which buffering resumes.

csharp
public record UpdateEndpointBufferingLimits(
    string EndpointUri,
    int Maximum,
    int Restart) : IServiceMessage
{
    public string ServiceName { get; init; } = "";
}
csharp
await bus.SendAsync(new UpdateEndpointBufferingLimits(
    "rabbitmq://exchange/trips/",
    Maximum: 1000,
    Restart: 200)
    { ServiceName = "trip-service" });

UpdateEndpointCircuitBreaker ​

Adjust the circuit-breaker configuration on a single endpoint at runtime.

csharp
public record UpdateEndpointCircuitBreaker(
    string EndpointUri,
    double FailurePercentageThreshold,
    TimeSpan PauseTime) : IServiceMessage
{
    public string ServiceName { get; init; } = "";
}
csharp
await bus.SendAsync(new UpdateEndpointCircuitBreaker(
    "rabbitmq://exchange/trips/",
    FailurePercentageThreshold: 0.25,
    PauseTime: TimeSpan.FromMinutes(2))
    { ServiceName = "trip-service" });

Projection Commands ​

All four projection commands accept an optional TenantId. Leave it null (the default) to act on the store-global shard — the existing single-tenant behavior, byte-for-byte. Set it to a tenant id to scope the operation to that one tenant's shard for the projection. The CritterWatch UI fills it in automatically based on the Tenants-tab row you act from; integrations sending these commands directly (custom UIs, ops scripts, the HTTP API) need to set it themselves.

Per-tenant operations respect tenant-scoped RBAC — a user authorized for only some tenants can't rebuild another tenant's projection. See Multi-Tenancy → Per-tenant scoping for the bigger picture.

PauseProjection ​

Pause a single projection shard. The shard stops advancing until restarted.

csharp
public record PauseProjection(string ServiceName, Uri AgentUri) : IServiceMessage
{
    public string? TenantId { get; init; }
}
csharp
await bus.SendAsync(new PauseProjection(
    "trip-service",
    new Uri("marten://projection/TripSummary:All")));

// Per-tenant variant
await bus.SendAsync(new PauseProjection(
    "trip-service",
    new Uri("marten://projection/TripSummary:All"))
    { TenantId = "acme-corp" });

RestartProjection ​

Resume a paused projection shard.

csharp
public record RestartProjection(string ServiceName, Uri AgentUri) : IServiceMessage
{
    public string? TenantId { get; init; }
}
csharp
await bus.SendAsync(new RestartProjection(
    "trip-service",
    new Uri("marten://projection/TripSummary:All")));

RebuildProjection ​

Reset a projection's state and rebuild from the beginning of its source stream. Long-running on big stores; while it runs, the shard's threshold-based alerts should be suppressed via Alert Configuration → Projections → Per-Shard Overrides.

TenantId rebuilds only that tenant's read model — useful when a single tenant's projection has drifted (a code bug, a manual data fix) and you don't want to spend the rebuild cost on every tenant. There is no "rebuild for all tenants" button today; each tenant is one click. See the gap on #286.

csharp
public record RebuildProjection(string ServiceName, Uri AgentUri) : IServiceMessage
{
    public string? TenantId { get; init; }
}
csharp
// Single-tenant service — or a global rebuild on a multi-tenant store.
await bus.SendAsync(new RebuildProjection(
    "trip-service",
    new Uri("marten://projection/TripSummary:All")));

// Per-tenant rebuild
await bus.SendAsync(new RebuildProjection(
    "trip-service",
    new Uri("marten://projection/TripSummary:All"))
    { TenantId = "acme-corp" });

RewindSubscription ​

Move a subscription's position back to a chosen point. The mode controls which target field applies.

csharp
public record RewindSubscription(
    string AgentUri,
    RewindMode Mode,
    long? TargetSequence,
    DateTimeOffset? TargetTimestamp) : IServiceMessage
{
    public string ServiceName { get; init; } = "";
    public string? TenantId { get; init; }
}

public enum RewindMode
{
    ToBeginning,
    ToSequence,
    ToTimestamp
}
csharp
await bus.SendAsync(new RewindSubscription(
    "marten://subscription/AnalyticsExporter",
    RewindMode.ToTimestamp,
    TargetSequence: null,
    TargetTimestamp: DateTimeOffset.UtcNow.AddHours(-1))
    { ServiceName = "trip-service" });

Agent Commands ​

PinAgentToNode ​

Force an agent to run on a specific node and prevent the leader from rebalancing it elsewhere. Useful when a particular node has hardware or licensing affinity (a GPU, a license-restricted IP, etc.).

csharp
public record PinAgentToNode(string ServiceName, Uri AgentUri, int NodeNumber) : IServiceMessage;
csharp
await bus.SendAsync(new PinAgentToNode(
    "trip-service",
    new Uri("marten://projection/TripSummary:All"),
    NodeNumber: 2));

UnpinAgent ​

Remove a pin so the leader can rebalance the agent again.

csharp
public record UnpinAgent(string ServiceName, Uri AgentUri) : IServiceMessage;
csharp
await bus.SendAsync(new UnpinAgent(
    "trip-service",
    new Uri("marten://projection/TripSummary:All")));

PushAgentThresholds ​

Push updated behind-high-water-mark warning + critical thresholds to a specific shard for client-side validation. Mirrors the values stored in Alert Configuration → Projections → Per-Shard Overrides so the service can refuse work that would never satisfy the operator's stated SLO.

csharp
public record PushAgentThresholds(
    string ShardName,
    long? WarningBehind,
    long? CriticalBehind) : IServiceMessage
{
    public string ServiceName { get; init; } = "";
}
csharp
await bus.SendAsync(new PushAgentThresholds(
    "TripSummary:All",
    WarningBehind: 1000,
    CriticalBehind: 10000)
    { ServiceName = "trip-service" });

RequestAgentHealthReport (query) ​

Ask the service to immediately publish an agent-health snapshot, bypassing the periodic health-report timer. Implements IServiceQuery.

csharp
public record RequestAgentHealthReport : IServiceQuery
{
    public string ServiceName { get; init; } = "";
}
csharp
await bus.InvokeAsync(new RequestAgentHealthReport { ServiceName = "trip-service" });

RequestHandlerSourceCode (query) ​

Ask the service to send the generated handler source code for a given message type. The optional EndpointUri selects the endpoint-specific specialisation when sticky routing applies.

csharp
public record RequestHandlerSourceCode(string HandlerMessageType, string? EndpointUri) : IServiceQuery
{
    public string ServiceName { get; init; } = "";
}
csharp
await bus.InvokeAsync(new RequestHandlerSourceCode(
    "Trips.CompleteTrip",
    EndpointUri: null)
    { ServiceName = "trip-service" });

RequestEndpointProperties (query) ​

Lazy-fetch the Properties + Children configuration tree for one Wolverine endpoint. The Pipeline tab issues this when the operator opens an endpoint row; the response is cached on the console keyed by service version. See Observer → Lazy-fetched detail panes for the full caching shape.

csharp
public record RequestEndpointProperties(string EndpointUri) : IServiceQuery
{
    public string ServiceName { get; init; } = "";
}
csharp
await bus.InvokeAsync(new RequestEndpointProperties(
    "rabbitmq://exchange/trips/")
    { ServiceName = "trip-service" });

RequestMessageHandlerProperties (query) ​

Lazy-fetch the per-handler Properties rows for one message type. The handler-chain detail page issues this when opened. Returns one row per handler chain registered against the message type.

csharp
public record RequestMessageHandlerProperties(string MessageType) : IServiceQuery
{
    public string ServiceName { get; init; } = "";
}
csharp
await bus.InvokeAsync(new RequestMessageHandlerProperties(
    "Trips.CompleteTrip")
    { ServiceName = "trip-service" });

Node Commands ​

EjectNode ​

Remove one or more stale nodes from the cluster. The leader redistributes their assigned agents.

csharp
public record EjectNode(string ServiceName, int[] NodeNumbers) : IServiceMessage;
csharp
await bus.SendAsync(new EjectNode("trip-service", [3]));

TriggerElection ​

Force a new leader election. The current leader's lease is revoked and the cluster votes again. Disruptive — only fire when the cluster is genuinely stuck.

csharp
public record TriggerElection(string ServiceName) : IServiceMessage;
csharp
await bus.SendAsync(new TriggerElection("trip-service"));

ClearNodeHistory ​

Trim the historical-node log, retaining the most recent RetainRecords entries. The optional Timestamp is the cutoff before which records are eligible for removal.

csharp
public record ClearNodeHistory(
    string ServiceName,
    int RetainRecords,
    DateTimeOffset? Timestamp) : IServiceMessage;
csharp
await bus.SendAsync(new ClearNodeHistory(
    "trip-service",
    RetainRecords: 10,
    Timestamp: null));

Tenant Commands ​

All tenant commands carry an implicit ServiceName via IServiceMessage's ServiceName property; the explicit parameters are just the tenant fields.

AddTenant ​

Add a new tenant database to a multi-tenant service. The connection string is sent over SignalR; rotate credentials after the cutover if you want a clean audit trail.

csharp
public record AddTenant(string TenantId, string ConnectionString) : IServiceMessage
{
    public string ServiceName { get; init; } = "";
}
csharp
await bus.SendAsync(new AddTenant(
    "acme-corp",
    "Host=db;Database=acme;Username=...;Password=...")
    { ServiceName = "trip-service" });

DisableTenant ​

Soft-disable a tenant. Messages addressed to the tenant queue without losing data; can be re-enabled with EnableTenant.

csharp
public record DisableTenant(string TenantId) : IServiceMessage
{
    public string ServiceName { get; init; } = "";
}
csharp
await bus.SendAsync(new DisableTenant("acme-corp") { ServiceName = "trip-service" });

EnableTenant ​

Re-enable a previously disabled tenant.

csharp
public record EnableTenant(string TenantId) : IServiceMessage
{
    public string ServiceName { get; init; } = "";
}
csharp
await bus.SendAsync(new EnableTenant("acme-corp") { ServiceName = "trip-service" });

RemoveTenant ​

Remove a tenant's master-table record. The per-tenant database itself is not dropped — it stays around for any forensic work or restore scenario.

csharp
public record RemoveTenant(string TenantId) : IServiceMessage
{
    public string ServiceName { get; init; } = "";
}
csharp
await bus.SendAsync(new RemoveTenant("acme-corp") { ServiceName = "trip-service" });

HardDeleteTenant (#68) ​

Drop the tenant database and remove the master-table record. Permanent. The CritterWatch UI gates this behind a typed-tenant-id confirmation modal — see Services → Tenants tab. License-gated to Professional+ on multi-tenant deployments.

csharp
public record HardDeleteTenant(string TenantId) : IServiceMessage
{
    public string ServiceName { get; init; } = "";
}
csharp
await bus.SendAsync(new HardDeleteTenant("acme-corp") { ServiceName = "trip-service" });

RequestTenantList (query) ​

Ask the service to publish its current tenant list. Used by the Tenants tab's Refresh button. Implements IServiceQuery.

csharp
public record RequestTenantList : IServiceQuery
{
    public string ServiceName { get; init; } = "";
}
csharp
await bus.InvokeAsync(new RequestTenantList { ServiceName = "trip-service" });

See Also ​

  • Outbound Events — events services publish to CritterWatch
  • Registration — installing the integration in your service
  • Multi-Tenancy — full tenant management flow
  • Audit Log — every command emits an audit entry; use the log to reconstruct what was sent and when

Released under the MIT License.

Copyright © 2024-present JasperFx Software