Metrics & logs

Three signals on the daemon, three places to look — /metrics for numbers, the query log for who-asked-what, the audit log for who-changed-what.

What this is

Warden exposes day-2 visibility through three independent surfaces. Only /metrics is off by default. The query log is on by default ([tracking] enabled and query_log_enabled default to true), and the audit log is always written.

This page is the map: which surface answers which question, how to turn each one on, and what to scrape or tail to get a useful signal out the other end. The Prometheus endpoint gets the deepest treatment here because it has no dedicated page elsewhere; the query log has its own page (linked at the bottom).

The three surfaces

They are independent. You can run Prometheus without the query log, or both logs without the metrics endpoint. Most operators eventually want Prometheus for “is the box healthy?” and the query log for “why was that domain blocked?”; the audit log is mostly useful when more than one person edits the config.

Enabling Prometheus

The /metrics endpoint is gated at router-registration time — when metrics_enabled = false the route is not registered at all (you get a 404 from the fallback, not from a disabled handler). This avoids exposing an enumeration surface on a daemon that wasn’t supposed to be scraped.

Two TOML keys gate it:

[api]
enabled         = true              # required — turns the HTTP server on
metrics_enabled = true              # required — registers the /metrics route
listen          = "127.0.0.1:8053"  # default — loopback only

Both must be true. enabled controls the whole [api] server (it’s also what /healthz and /api/* depend on). metrics_enabled is the per-route switch for /metrics specifically.

After saving, restart the daemon with sudo systemctl restart purge-warden. A reload is refused for [api] changes. Confirm with:

curl -s http://127.0.0.1:8053/metrics | head

A successful enable yields a body starting with # TYPE purge_warden_uptime_seconds gauge. A 404 means either the daemon has not restarted or one of the two keys is still false.

The endpoint

The endpoint is hand-rolled Prometheus text, not the prometheus crate. The tradeoff is zero extra dependencies and no histograms.

Metric reference

Every metric Warden exposes today. One series carries a label: purge_warden_list_truncated_entries{source="…"}. Every other series is a plain scalar.

Source: metrics() in src/api/handlers.rs.

Scraping

Minimal Prometheus job for a daemon on the same host:

scrape_configs:
  - job_name: purge_warden
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets: ["127.0.0.1:8053"]

For a remote scraper, expose the endpoint over TLS:

[api]
enabled         = true
metrics_enabled = true
listen          = "10.0.0.20:8053"
tls_cert        = "/etc/purge-warden/tls/api.crt"
tls_key         = "/etc/purge-warden/tls/api.key"

The validator refuses a non-loopback listen without both tls_cert and tls_key. The same TLS material covers /healthz, /metrics, and /api/*.

A 15-second scrape interval is plenty — most series move on the seconds-to-minutes timescale, not sub-second.

Grafana dashboard

No dashboard ships with Warden today. There is no dashboards/*.json in the upstream repo. A reasonable starting panel set:

• Top row (overview): uptime, total queries / sec (rate), blocked ratio, cache hit ratio.
• Filter health: purge_warden_domains_loaded (gauge), purge_warden_blocked_total rate per minute, purge_warden_refused_acl_total rate per minute.
• Cache: purge_warden_cache_entries, purge_warden_cache_hits_total rate.
• Topology: purge_warden_tracked_devices.

Useful PromQL fragments:

# Queries per second over the last minute
rate(purge_warden_queries_total[1m])

# Per-minute block rate
rate(purge_warden_blocked_total[1m]) * 60

# Live blocked ratio (uses the precomputed gauge)
purge_warden_blocked_ratio

A community-contributed dashboard JSON would be welcome — open an issue if you want to publish one.

Security of the metrics endpoint

A /metrics endpoint on a DNS resolver leaks how busy the box is, how many clients it serves, how much it blocks, and what it caches. Treat the body as sensitive operational telemetry.

• Default listen = "127.0.0.1:8053". Only loopback can scrape. A Prometheus on the same host works; anything else needs a deliberate config change.
• No bearer token on /metrics. The bearer-token middleware wraps /api/* only — /metrics is in public_routes alongside /healthz. If you bind the API to a non-loopback address, every scraper inside the TLS trust boundary can read the body.
• Validator forces TLS off-loopback. Binding [api] listen to 0.0.0.0:8053 or a LAN IP without tls_cert / tls_key is refused. The compromise is “TLS yes, auth no” — fine for a Prometheus sitting behind your perimeter, not fine for a public internet bind.
• If you need auth. Today: reverse-proxy /metrics behind nginx / caddy / Traefik with basic auth or mTLS. Bind Warden’s [api] to loopback only and let the proxy listen externally.

Source: build_router() in src/api/routes.rs, spawn_api_server() in src/api/server.rs.

Logs

This page covers metrics in depth; the query log has its own page.

• Query log — one JSON line per query with timestamp, client_ip, client_name, domain, query_type, result and response_time_us, plus cname_chain_via or rewrote_from when they apply. It rotates daily at UTC midnight and keeps [tracking] retention_days days; query_log_max_size_mb and query_log_max_files cap one busy day, at 800 MiB with the defaults: past that, the oldest lines of the day are deleted (how much the log keeps). MAC address is never recorded. Read it with warden logs. See query log for the details.
• Audit log — JSON lines in /var/lib/purge-warden/audit/audit.log. Records include daemon boot, reload and shutdown, and configuration changes made with the warden CLI. Example actions: blocklist.add, device.add, local_records.add, subnet.set. Inspect with warden audit tail [-n N].

The query log is on by default; turn it off with [tracking] query_log_enabled = false. The audit log is always on.

Troubleshooting

curl /metrics returns 404.

Most common cause: metrics_enabled = false (the route is genuinely not registered). Check:

grep -A 5 '^\[api\]' /etc/purge-warden/config.toml
# Both enabled = true AND metrics_enabled = true must be present.

journalctl -u purge-warden -n 30 --no-pager | grep -i 'api\|listen\|metrics'
# Look for "REST API listening" with the bound address.

If both flags are true but the body is still missing, the daemon may not have restarted after the edit. Run sudo systemctl restart purge-warden, then check the journal timestamp.

The body has no counters, ratios or tracked_devices.

The stats engine is not initialised in this daemon’s configuration. Counters and ratios require runtime tracking. Check [tracking] enabled in the config; it is on by default.

Prometheus reports up == 0 even though curl from the host succeeds.

The scraper is on a different host than the daemon and the bind is 127.0.0.1. Either move the scraper to localhost or rebind the daemon to a routable address with TLS (see scraping above).

warden reload is fine but the /metrics body still shows old gauges.

Some gauges (uptime_seconds, domains_loaded) update on the next scrape; counters do not reset on reload, by design — total counters keep rolling across reloads so derivative-based dashboards stay sane. If you need a reset, restart the daemon: sudo systemctl restart purge-warden.

The /metrics Content-Type is text/plain; version=0.0.4 — is that right?

Yes. That is the Prometheus text exposition format, which Prometheus and Grafana read without configuration.

For broader diagnosis, see troubleshooting.

See also

• Query log — per-query JSONL.
• Basic configuration — where the config file lives, and how to validate and reload it.