Upgrade Notes

Version-to-version changes that affect a deployed console's database schema or stored data — plus compatibility notes for monitored hosts — what CritterWatch does about them automatically, and the manual path when it can't.

rc.8 → next: the standalone MessageHandlingMetrics publish route is gone (GH-955)

AddCritterWatchMonitoring no longer declares a publish route for Wolverine's MessageHandlingMetrics type. The route was dead wiring: the CritterWatch observer folds every exported metrics batch into ServiceUpdates (and deliberately never publishes the batches individually), so the route carried zero traffic on every deployment — but it declared a contract that repeatedly misled analysis about what the console's leader-pinned legacy queue carries.

Who is affected: nobody, behaviorally. Metrics keep arriving inside ServiceUpdates exactly as before, in every metrics mode. If you had custom Wolverine routing that keyed off the MessageHandlingMetrics message type on the monitored side, that type no longer has a CritterWatch-declared route — but nothing publishes it standalone, so such routing was already inert.

rc.6 → next: external-source services default to live-only metrics persistence (GH-938)

A service that reports WolverineMetricsMode.SystemDiagnosticsMeter and is bound to a non-Postgres metrics data source (Prometheus, VictoriaMetrics, Datadog, Application Insights) now defaults to ExternalMetricsPersistence.LiveOnly: the console stops writing MetricsSample / MessagingMetricsBucket rows for it. Previously live-only was a per-service opt-in buried in the alert overrides, so this posture silently kept paying the console-side storage cost (~1 GB/hour measured in the field) for data nothing read — the metrics views and alert evaluators for such a service read the external store.

Who is affected: only deployments that already run services in SystemDiagnosticsMeter mode with an external source binding and did NOT set the live-only override. After upgrading, those services' new samples stop accumulating; existing rows age out through normal retention.

To keep the old behaviour for a specific service, set its External metrics persistence override to PersistSameAsInternal (Settings → the service's alert overrides, or the alert-config API) — an explicit override always wins over the default, in either direction.

rc.6 → next: SampleRetentionPeriod now actually drives the metrics partition window (GH-935)

Before this fix, both store registrations built the MetricsSample rolling partition window from the hardcoded 45-day default, ignoring a configured CritterWatch:Metrics:SampleRetentionPeriod. Consequences on earlier versions:

• Shortening retention below 45 days reclaimed space by row deletes only (dead tuples + vacuum churn) — the O(1) monthly partition drops still waited for the 45-day floor.
• Lengthening retention past ~56 days was destructive: the drop pass retired month-partitions that still held rows inside the configured window.

After upgrading, the window follows the configured value on both PostgreSQL and SQL Server, and the startup partition pass is additive only — provisioning happens at boot, while partition drops run solely from the hourly prune, which skips them when retention is 00:00:00 (keep forever) or when any per-service retention override outlives the global window. Existing partitions are adopted in place; no data migration runs. If you overrode SampleRetentionPeriod on an earlier version expecting partition-level reclaim, the first prune after upgrade will retire every partition past your configured window — verify that window is what you intend before deploying the upgrade.

Monitored hosts using AddOpenApi(): upgrade Wolverine to 6.17.2+ (GH-689)

The problem

On WolverineFx / WolverineFx.Http 6.17.1 and earlier, any read of ASP.NET Core's ApiExplorer that lands before the web server starts permanently freezes every OpenAPI document as empty ("paths": {}) for the lifetime of the host — ASP.NET's version-keyed description cache serves the first (empty) answer forever (wolverine#3371).

CritterWatch's monitoring satellite is an in-box early reader: the capability snapshot (ServiceCapabilities.ReadFrom) consults ApiExplorer from the observer's boot-time batching loop, which can win the race against web-server startup — reported at roughly 40% incidence on 2-core CI runners, rarer but possible anywhere. The result is that a plain Wolverine + CritterWatch + AddOpenApi() host can break its own OpenAPI documents just by being monitored, with no user code involved. HTTP routing keeps working; only the OpenAPI documents (Microsoft.AspNetCore.OpenApi, Swashbuckle, versioned documents) are affected, and only until the process restarts and wins the race.

Why the read happens at all

CritterWatch describes the monitored host's non-Wolverine endpoints — minimal API, MVC, Razor Pages, SignalR — by matching each RouteEndpoint against ASP.NET's ApiExplorer (IApiDescriptionGroupCollectionProvider), which is what populates the ASP.NET Endpoints page. That lookup is what touches ApiExplorer, and the capability snapshot runs it during the observer's boot handshake. This matters for what the fix below does and does not cover.

The fix

Upgrade the monitored service to Wolverine 6.17.2 or later. Wolverine's ApiDescription provider now enumerates its own HttpGraph — complete as soon as MapWolverineEndpoints() returns — instead of the DI EndpointDataSource that ASP.NET only populates at server start, so a pre-start read returns the same complete answer as a post-start one (wolverine#3373).

No CritterWatch configuration change is needed, and no CritterWatch version change is involved — the fix is entirely on the monitored host's Wolverine version.

Hybrid hosts: 6.17.2 does not close this completely

If your OpenAPI document also contains minimal-API, MVC, or Razor Pages endpoints, upgrading is not the whole answer. 6.17.2 makes only Wolverine's descriptions start-independent. ASP.NET's own providers still compose their descriptions at server start, and the ApiExplorer read is still happening pre-start, and ASP.NET still caches the whole collection for the lifetime of the host.

So on a hybrid host the pre-start read now freezes a document that has the Wolverine endpoints but none of the minimal-API / MVC ones — where before 6.17.2 it froze fully empty. That is a better failure, and a more deceptive one: the document is populated, so it looks fine until someone notices their minimal-API routes are missing. This is called out in the fix's own behavior notes.

A pure Wolverine.Http host (every described endpoint comes from MapWolverineEndpoints()) is fully covered by 6.17.2 and needs nothing further. A hybrid host should defer its boot-time ApiExplorer reads as well — see below.

How to tell you've been bitten

Hit the OpenAPI document on a running host (/openapi/v1.json, or your configured route):

• "paths": {} — the fully-empty freeze. Monitored host on Wolverine ≤ 6.17.1.
• Paths present, but every minimal-API / MVC route missing while Wolverine routes are there — the hybrid-host freeze described above.

In both cases HTTP routing itself keeps working — only the documents are wrong — and both clear on a process restart that wins the race, which is exactly what makes this easy to dismiss as a flake. It reproduced at roughly 40% on 2-core CI runners and far more rarely on developer machines, so a document that is fine locally and empty in CI is the signature.

Interim mitigation

For a host pinned to Wolverine ≤ 6.17.1, or a hybrid host on any version that wants to be certain:

• Preferred — make sure nothing reads ApiExplorer before ApplicationStarted. Wrap the registered IApiDescriptionGroupCollectionProvider so that pre-start reads are served from a throwaway provider instance (a fresh instance has a fresh cache, so ASP.NET's lifetime cache is never poisoned) and post-start reads delegate to the real one.
• Simplest — don't attach CritterWatch monitoring to an OpenAPI-exposing host until it is on 6.17.2+ (and, if hybrid, until its boot-time ApiExplorer reads are deferred).

Either way this is a property of the monitored host, not of the console: no CritterWatch setting changes it, because the read is intrinsic to describing non-Wolverine endpoints at snapshot time.

beta.1 → beta.2+: mt_doc_metricssample becomes a partitioned table (GH-685)

What changed

beta.1 created the metrics history table (<schema>.mt_doc_metricssample, PostgreSQL/Marten consoles) as a plain heap table. beta.2 redesigned it for retention at scale: the table is now a range-partitioned parent (monthly child partitions plus a DEFAULT catch-all) with duplicated, indexed service_name / bucket_end columns that the metrics reads filter on.

PostgreSQL cannot ALTER a plain table into a partitioned one, so a console database first provisioned on beta.1 and then upgraded is stuck in the old shape. On beta.2 the symptoms were:

• GET /api/critterwatch/throughput-series returns HTTP 500 for every parameter combination, so the dashboard's System Throughput chart is permanently empty (the generated SQL references the bucket_end column, which the old shape doesn't have).
• Retention leaks: partition-based pruning has no partitions to drop and the row-level prune fails on the same missing column, so mt_doc_metricssample grows without bound (a single low-volume service accumulated 1.9M rows in the original report).
• Both failures were silent: writes kept working, the host booted cleanly, and the only trace was a startup warning.

What happens automatically now

On startup (and again on every retention prune cycle), the metrics partitioner inspects the table shape:

• Current partitioned shape — nothing to do.

• beta.1 heap shape (an ordinary table with Marten's id + data columns) — it is auto-migrated, with row counts logged at each step:

  1. The legacy table (and its constraints/indexes) is renamed aside to mt_doc_metricssample_pre_partition.
  2. The partitioned parent — and the matching upsert functions and indexes — are rebuilt from Marten's configured DDL. This works under any AutoCreate policy, including None.
  3. The monthly + DEFAULT child partitions are created.
  4. Rows are backfilled from the renamed table, bounded by the retention horizon (CritterWatch:Metrics:SampleRetentionPeriod, default 45 days). Rows already past retention — including the leaked backlog the drift itself caused — are deliberately left behind: they would be pruned immediately anyway, and the bound keeps the migration a one-time bounded cost.
  5. The renamed table is dropped.

  Startup waits only for the schema phase (steps 1–3 — fast catalog work). The backfill and drop (steps 4–5) run in the background after the console is up and serving, with the metrics-store component on GET /api/critterwatch/system-health reporting degraded until they complete. This matters on a large legacy heap: earlier builds ran the whole copy inside host startup, so on a multi-GB table orchestrator liveness probes killed the "hung" console before the copy finished and every restart began it again (#928). Don't restart the console while metrics-store reports degraded unless you have to — the migration resumes where it left off, but each interruption re-scans.

  The migration is idempotent and resumable: each step keys off catalog state and the backfill uses ON CONFLICT DO NOTHING, so an interruption (crash, deploy, connection loss) between any two steps resumes cleanly on the next startup.

• Anything else — a shape that is neither the beta.1 heap nor the current partitioned parent — CritterWatch refuses to touch the table and fails loudly instead of warn-and-continue: an error log, plus a degraded metrics-store component on GET /api/critterwatch/system-health and on the critterwatch-system ASP.NET Core health check. The rest of the console keeps working.

During the (one-time) migration window, an in-flight metrics flush can log a transient write error between the rename and the function rebuild; it retries on the next flush interval and succeeds.

Manual migration path

If you'd rather migrate by hand — or your table is in a shape the auto-migration refuses — run the following against the console database (adjust critterwatch to your configured schema name). Metrics history is statistical/ephemeral data, so the simplest safe path is drop-and-recreate:

-- 1. Move the old table aside (or DROP TABLE it if you don't care about recent history)
ALTER TABLE critterwatch.mt_doc_metricssample RENAME TO mt_doc_metricssample_old;
ALTER TABLE critterwatch.mt_doc_metricssample_old
    RENAME CONSTRAINT pkey_mt_doc_metricssample TO pkey_mt_doc_metricssample_old;

Then restart the console. On boot it recreates mt_doc_metricssample in the correct partitioned shape (this is the same "resume from the renamed table" path the auto-migration uses when the leftover table is named mt_doc_metricssample_pre_partition; with any other name the fresh table is simply created empty). Optionally backfill recent history:

-- 2. (Optional) copy recent rows into the new partitioned table
INSERT INTO critterwatch.mt_doc_metricssample
    (id, data, mt_last_modified, mt_version, mt_dotnet_type, service_name, bucket_end)
SELECT id, data, mt_last_modified, mt_version, mt_dotnet_type,
       data ->> 'serviceName',
       (data ->> 'bucketEnd')::timestamptz
FROM critterwatch.mt_doc_metricssample_old
WHERE (data ->> 'bucketEnd')::timestamptz >= now() - interval '45 days'
ON CONFLICT DO NOTHING;

-- 3. Clean up
DROP TABLE critterwatch.mt_doc_metricssample_old;

How to tell it worked

• GET /api/critterwatch/throughput-series?hours=24&bucketMinutes=60 returns 200 with a bucket array, and the dashboard's System Throughput chart renders.
• GET /api/critterwatch/system-health reports "status": "healthy".
• SELECT relkind FROM pg_class WHERE relname = 'mt_doc_metricssample' returns p (partitioned), and \d+ critterwatch.mt_doc_metricssample shows monthly partitions plus mt_doc_metricssample_default.
• The leaked backlog is gone after the migration (out-of-retention rows are not carried over), and from then on normal retention keeps the table bounded by dropping aged monthly partitions.