Metrics & logs

Three signals on the daemon, three places to look — /metrics for numbers, the query log for who-asked-what, the audit log for who-changed-what.

What this is

Warden exposes day-2 visibility through three independent surfaces. None of them are on by default — every one is opt-in via the config TOML, deliberately, so a fresh install does nothing more than answer DNS.

This page is the map: which surface answers which question, how to turn each one on, and what to scrape or tail to get a useful signal out the other end. The Prometheus endpoint gets the deepest treatment here because it has no dedicated page elsewhere; the two logs each have their own deep-dive (linked at the bottom).

The three surfaces

They are independent. You can run Prometheus without the query log, or both logs without the metrics endpoint. Most operators eventually want Prometheus for “is the box healthy?” and the query log for “why was that domain blocked?”; the audit log is mostly useful when more than one person edits the config.

Enabling Prometheus

The /metrics endpoint is gated at router-registration time — when metrics_enabled = false the route is not registered at all (you get a 404 from the fallback, not from a disabled handler). This avoids exposing an enumeration surface on a daemon that wasn’t supposed to be scraped.

Two TOML keys gate it:

[api]
enabled         = true              # required — turns the HTTP server on
metrics_enabled = true              # required — registers the /metrics route
listen          = "127.0.0.1:8053"  # default — loopback only

Both must be true. enabled controls the whole [api] server (it’s also what /healthz and /api/* depend on). metrics_enabled is the per-route switch for /metrics specifically.

After saving, run sudo warden reload to pick up the new config. Confirm with:

curl -s http://127.0.0.1:8053/metrics | head

A successful enable yields a body starting with # HELP purge_shield_uptime_seconds …. A 404 means either the daemon hasn’t reloaded or one of the two keys is still false.

The endpoint

The endpoint is hand-rolled OpenMetrics text, not the prometheus crate. The tradeoff is zero extra dependencies and zero labels / histograms — every metric is a scalar.

Metric reference

Every metric Warden exposes today. No labels on any series.

Source: src/api/handlers.rs:1124-1206.

Scraping

Minimal Prometheus job for a daemon on the same host:

scrape_configs:
  - job_name: purge_shield
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets: ["127.0.0.1:8053"]

For a remote scraper, expose the endpoint over TLS:

[api]
enabled         = true
metrics_enabled = true
listen          = "10.0.0.20:8053"
tls_cert        = "/etc/purge-warden/tls/api.crt"
tls_key         = "/etc/purge-warden/tls/api.key"

The validator refuses a non-loopback listen without both tls_cert and tls_key. The same TLS material covers /healthz, /metrics, and /api/*.

A 15-second scrape interval is plenty — most series move on the seconds-to-minutes timescale, not sub-second.

Grafana dashboard

No dashboard ships with Warden today. There is no dashboards/*.json in the upstream repo. A reasonable starting panel set:

• Top row (overview): uptime, total queries / sec (rate), blocked ratio, cache hit ratio.
• Filter health: purge_shield_domains_loaded (gauge), purge_shield_blocked_total rate per minute, purge_shield_refused_acl_total rate per minute.
• Cache: purge_shield_cache_entries, purge_shield_cache_hits_total rate.
• Topology: purge_shield_tracked_devices.

Useful PromQL fragments:

# Queries per second over the last minute
rate(purge_shield_queries_total[1m])

# Per-minute block rate
rate(purge_shield_blocked_total[1m]) * 60

# Live blocked ratio (uses the precomputed gauge)
purge_shield_blocked_ratio

A community-contributed dashboard JSON would be welcome — open an issue if you want to publish one.

Security of the metrics endpoint

A /metrics endpoint on a DNS resolver leaks how busy the box is, how many clients it serves, how much it blocks, and what it caches. Treat the body as sensitive operational telemetry.

• Default listen = "127.0.0.1:8053". Only loopback can scrape. A Prometheus on the same host works; anything else needs a deliberate config change.
• No bearer token on /metrics. The bearer-token middleware wraps /api/* only — /metrics is in public_routes alongside /healthz. If you bind the API to a non-loopback address, every scraper inside the TLS trust boundary can read the body.
• Validator forces TLS off-loopback. Binding [api] listen to 0.0.0.0:8053 or a LAN IP without tls_cert / tls_key is refused. The compromise is “TLS yes, auth no” — fine for a Prometheus sitting behind your perimeter, not fine for a public internet bind.
• If you need auth. Today: reverse-proxy /metrics behind nginx / caddy / Traefik with basic auth or mTLS. Bind Warden’s [api] to loopback only and let the proxy listen externally. Roadmap item: optional bearer-token gating on /metrics matching /api/*.

Source: src/api/routes.rs:32-82, src/api/server.rs:21.

Logs

This page covers metrics in depth; the two log surfaces each have their own page.

• Query log — one JSONL line per resolved DNS query with client_ip, client_name, queried domain, RCODE, the resolved profile name, and (where applicable) which rule fired. Rotated and capped via [query_log]. MAC address is never recorded. See query log for schema, rotation, and warden log tail.
• Audit log — tracing::info!(target = "audit", action = "...") events emitted on every config-mutating CLI / API call. Examples: profile.blocklists.add, profile.blocklists.remove, rule.add, rule.undo. Mirrored to a file via [audit_log]. Inspect with warden audit tail [-n N]. See audit log.

Both logs default to off (enabled = false) — opt-in just like /metrics.

Troubleshooting

curl /metrics returns 404.

Most common cause: metrics_enabled = false (the route is genuinely not registered). Check:

grep -A 5 '^\[api\]' /etc/purge-warden/config.toml
# Both enabled = true AND metrics_enabled = true must be present.

journalctl -u purge-warden -n 30 --no-pager | grep -i 'api\|listen\|metrics'
# Confirms the server bound and which routes are live.

If both flags are true but the body is still missing, the daemon may not have reloaded after the edit — sudo warden reload, then check the journal timestamp.

The body has only three lines (uptime_seconds, domains_loaded, cache_entries).

The stats engine is not initialised in this daemon’s configuration. Counters and ratios require runtime tracking. Check the device-tracking section of the config; for most installs this is on automatically.

Prometheus reports up == 0 even though curl from the host succeeds.

The scraper is on a different host than the daemon and the bind is 127.0.0.1. Either move the scraper to localhost or rebind the daemon to a routable address with TLS (see scraping above).

warden reload is fine but the /metrics body still shows old gauges.

Some gauges (uptime_seconds, domains_loaded) update on the next scrape; counters do not reset on reload, by design — total counters keep rolling across reloads so derivative-based dashboards stay sane. If you need a reset, restart the daemon: sudo systemctl restart purge-warden.

The /metrics Content-Type is text/plain; version=0.0.4 — is that right?

Yes. That’s the OpenMetrics text version Warden speaks. Both Prometheus and modern Grafana ingest it without configuration.

For broader diagnosis, see troubleshooting.

See also

• Query log — per-query JSONL.
• Audit log — config-change audit trail.
• TUI dashboard — interactive view of the same counters, useful when SSH-ing in.
• Server globals — where [api] lives alongside [server] and friends.
• TOML reference — every field, every constraint.