Application Insights Metrics

CritterWatch can read throughput, failure-rate, baseline, and dead-letter rate from the Log Analytics workspace behind an Application Insights resource, instead of Prometheus, VictoriaMetrics, Datadog — or its own Postgres store. The alert evaluator, the per-service summary, the Dashboard's 24h throughput trend, and the trend visualizations all work transparently against whichever metrics backend the operator has configured.

This is the bring-your-own-observability posture: your Wolverine services keep exporting OTel metrics to Azure Monitor exactly as they already do, CritterWatch queries the workspace on demand with KQL, and — combined with SystemDiagnosticsMeter mode and live-only persistence below — CritterWatch stores no metrics data at all.

What CritterWatch needs from Azure

• The Log Analytics workspace id (a GUID) backing your Application Insights resource. This is the workspace id — not the instrumentation key, not the Application Insights app id. Workspace-based Application Insights resources (the default since 2021) all have one.
• An identity with the Log Analytics Reader role on that workspace. Auth uses DefaultAzureCredential: a managed identity when the console runs in Azure, your developer sign-in locally. No keys or secrets are configured, stored, or persisted anywhere — consistent with the rule that bindings persist source names only.
• Your Wolverine services already exporting via the Azure Monitor OpenTelemetry distro (Azure.Monitor.OpenTelemetry.AspNetCore). The distro lands Wolverine's instruments in the workspace AppMetrics table with their dimensions (message.type, message.destination, tenant.id, exception.type) intact.

How to wire it up

In your CritterWatch host startup:

services.AddCritterWatchMetricsDataSource<AppInsightsMetricsDataSource, AppInsightsMetricsDataSourceOptions>(
    "appinsights",
    opts =>
    {
        // The Log Analytics WORKSPACE id backing the Application Insights resource —
        // not the instrumentation key, not the Application Insights app id.
        opts.WorkspaceId = builder.Configuration["AppInsights:WorkspaceId"]!;

        // Auth defaults to DefaultAzureCredential (managed identity in Azure,
        // developer sign-in locally). Assign opts.Credential to override.

        // AppInsights ingest lag is routinely 1-3 minutes; every query window ends
        // this far behind "now" so the un-ingested tail never reads as starvation.
        opts.IngestLagAllowance = TimeSpan.FromMinutes(3);
    });

Per-service routing

// In the monitored service's UseWolverine — point it at the named source the console
// registered, and stop pushing metrics to CritterWatch at all: AppInsights is the
// system of record on both the write and the read side.
opts.AddCritterWatchMonitoring(
        critterWatchUri, systemControlUri,
        metricsMode: WolverineMetricsMode.SystemDiagnosticsMeter)
    .MetricsDataSource("appinsights");

WolverineMetricsMode.SystemDiagnosticsMeter stops the service pushing metrics to CritterWatch entirely — Azure Monitor is the only metrics pipeline. The operator can override the source binding per-service via Settings → Metrics Data Sources.

A service that is both SystemDiagnosticsMeter and bound to an external source defaults to live-only persistence — CritterWatch writes no MetricsSample rows for it (see Metrics storage and sizing). An explicit Persist same as internal override in Settings restores console-side persistence per service.

Which metrics CritterWatch reads

The workspace AppMetrics table, under the instrument names Wolverine emits:

What	Wolverine instrument	KQL shape
Executions	wolverine-messages-received	sum(Sum)
Failures	wolverine-execution-failure	sum(Sum)
Mean execution time	wolverine-execution-time	sum(Sum) / sum(ItemCount)
Mean effective time	wolverine-effective-time	sum(Sum) / sum(ItemCount)
Dead letters	wolverine-dead-letter-queue	sum(Sum), grouped by message.type / exception.type

Every interface call is one batched KQL query (a single summarize ... by Name), not one query per metric — the Log Analytics query API is rate-limited and the alert evaluator polls per service.

Cloud role mapping: CritterWatch filters on AppRoleName == '<service name>'. If your services report under a different cloud role name than their CritterWatch service name, map them with opts.ServiceLabelMapping["ServiceName"] = "cloud-role-name".

Ingest lag

Application Insights ingestion routinely runs 1–3 minutes behind. Every query window this source issues ends IngestLagAllowance (default 3 minutes) behind now, so the not-yet-ingested tail never reads as a phantom throughput drop. Consequences to expect:

• Metrics views trail live traffic by roughly the allowance plus the actual ingest lag.
• Short-window alert checks evaluate a window that ended a few minutes ago — a real incident is detected a few minutes later than it would be with pushed metrics. That is inherent to the posture, not a configuration problem.
• If your workspace's observed lag exceeds 3 minutes, raise IngestLagAllowance rather than chasing false low-throughput alerts.

Measured on a live South Central US workspace during the #938 end-to-end verification (a Wolverine service exporting via the Azure Monitor exporter at default 60-second intervals): average lag 39.7 s, p95 2 m 34 s (ingestion_time() - TimeGenerated over the wolverine-* instruments). The 3-minute default allowance covered the p95 with margin; the source, message.type, and message.destination dimensions all arrived intact in AppMetrics.Properties.

Baselines

Unlike the other external sources, the AppInsights source implements the oldest-sample probe (min(TimeGenerated) over a bounded lookback), so observed baselines mature normally once the workspace holds enough history — deviation-style alerts work without declaring baselines by hand. Absolute-threshold alerts work from day one either way.

When a metric isn't available

Query failures, missing tables, and empty results all degrade to zero-populated snapshots — blank cells in the affected widgets, never error pages. Same contract as every other external source. Verify your instruments are actually landing with a workspace query like:

AppMetrics | where Name startswith "wolverine-" | summarize count() by Name

If that returns nothing, the exporter side is the problem — confirm the Azure Monitor OTel distro is registered in the monitored service and that custom metrics aren't being filtered.