Troubleshooting

Find the symptom, read the cause, run the fix.

What this is

Problems with DNS, installing, upgrading, backing up and starting Warden, with the fix for each. Start with these three commands. They answer most questions:

systemctl status purge-warden
journalctl -u purge-warden --since "1 hour ago"
sudo warden config lint

warden config lint loads the configuration the way the daemon does. It prints config is valid or every error with its file and line, a bad line in a Custom List pack included.

The installer also writes a log of every run to /var/log/purge-warden-install-<date>-<time>.log.

To keep most of these problems from turning into an outage, see Prevent an outage.

DNS stops working

The daemon does not start after a manual edit

A file of the configuration has an error: a typo in config.toml or in an included file, a bad line in a Custom List pack, or a pack file that was deleted while the configuration still declares it. The running daemon keeps the policy it has, so nothing looks wrong at first. At the next restart or reboot the daemon refuses the configuration, and this node serves no DNS. Systemd retries every 5 seconds and gives up after a few failed starts.

To see the cause:

journalctl -u purge-warden -b
sudo warden config lint

For a deleted pack, the journal shows something like this:

Error: the daemon does not start until the config loads, so this node serves no DNS. Run `warden config lint` to list every error with its file and line, and fix the file. To go back instead, list the archives with `warden config restore --list` and restore one from before the change with `warden config restore ARCHIVE`. Then run `systemctl reset-failed purge-warden` and `systemctl start purge-warden`.

Caused by:
    node bootstrap config rejected: validation failed: custom list "kids-extra": custom list file /etc/purge-warden/packs/kids-extra.txt does not exist ...

The sentence about archives appears only when the backup directory holds some.

Fix it in one of two ways:

• Correct the file. Edit the file that warden config lint names, and run warden config lint again until it prints config is valid. For a deleted pack, put the file back, or remove the [[custom_lists]] entry that names it and remove its id from the custom_lists of every profile. In 0.44.5, warden custom-list delete does not work while the pack file is missing.
• Go back to a backup. List the archives, then restore one taken before the change. The newest archive can be a copy of the broken configuration, so check its date. Restore works while the daemon is stopped, and it repairs a configuration that does not load. See Backup & restore.

sudo warden config restore --list
sudo warden config restore /var/lib/purge-warden/backups/config-20260505T091322Z.tar.gz

Then start the daemon again:

sudo systemctl reset-failed purge-warden
sudo systemctl start purge-warden

Safe mode does not bring DNS back. warden start --safe-mode answers every query with REFUSED, and only on 127.0.0.1:5335, not on port 53. On an installed host it does not start through the warden command: it stops with Error: PID file /run/purge-warden/purge-warden.pid: Permission denied (os error 13). You do not need it for the fix: warden config lint and warden config restore work while the daemon is stopped.

packs/ holds other files

An editor or a backup tool can leave files next to a pack, such as .kids.txt.swp, kids.txt~ or kids.txt.bak. From 0.44.5, Warden ignores every entry in packs/ whose name is not a pack name. It does not read them, back them up or restore them, and it warns:

packs/ holds entries that are not packs and are ignored: packs/.kids.txt.swp. Warden reads, backs up and restores only <id>.txt files directly in packs/

The warning appears in the journal, in warden config lint and in the output of warden config backup. warden status counts the entries on an ignored: line. Move the files out of packs/ to clear it.

An entry with a pack name (kids.txt) is not ignored. If it is a symbolic link, a hard link, a folder or a special file, the configuration does not load: unsupported flat Custom List pack tree: ...; every packs/<id>.txt must be a single-link regular file. Replace it with a plain file.

On 0.44.0, any other file in packs/ stops the next start the same way, and backup and restore refuse the folder.

No upstream answers

Warden answers from its blocklists and its cache, and asks the upstream resolvers for everything else. When they do not answer, clients get SERVFAIL, and the journal shows request failed: upstream request failed. Some names can still get an old answer from the cache. The journal then shows upstream failed, serving stale cache.

To check, ask Warden, then ask the upstream directly from the Warden host. The addresses are in upstream.servers:

dig @127.0.0.1 example.com
warden config show
dig @192.0.2.53 example.com
journalctl -u purge-warden -n 30

If the upstream does not answer the Warden host either, check the network and the firewall of that host: it must be allowed to reach the upstream address and port. If the upstream answers but Warden does not, read the journal for the error.

Clients do not use Warden

Warden only filters the queries it receives. Check from the client first:

nslookup doubleclick.net 192.168.1.10   # asks Warden: expect 0.0.0.0
nslookup doubleclick.net                # asks the client's own DNS server

If the first answer is 0.0.0.0 and the second is a real address, the client asks another server. On the Warden host, warden logs --client 192.168.1.23 --since 10m lists the queries Warden received from that client. An empty result means the client does not ask Warden. The filter matches part of a name or address, so 192.168.1.23 also lists 192.168.1.230 to 192.168.1.239: check the client column.

The usual causes:

A list does not load

When a list does not load, warden status shows source rows instead of sources active on the lists: line, and adds a SOURCE COVERAGE INCOMPLETE line. For example:

lists:      3/4 source rows
            SOURCE COVERAGE INCOMPLETE — partial corpus installed

The lists that loaded keep filtering. After a restart, lists: 0/3 sources active, 3 loaded from cache (next fetch pending) is normal: the lists filter from their cache until the next download.

The journal names the list and the error. A failed download shows download failed; retained-body fallback unavailable. To see both:

warden status
journalctl -u purge-warden --since "1 hour ago" | grep -i list

The usual causes are a download that fails (network, a wrong URL, a server that is down) and a list that exceeds a limit. Fix the cause, then download again:

warden lists refresh

A list you no longer want can go with warden lists remove. Then run sudo systemctl reload purge-warden: warden lists refresh keeps downloading the removed list. See Import lists.

Clients lose DNS during a restart

A restart of purge-warden stops DNS on that host until the lists are loaded again. With large lists that takes a minute or two. A reboot, an upgrade, a crash and systemctl restart all cause it. A reload (warden reload or systemctl reload purge-warden) does not stop DNS.

During the gap, a client with a second DNS server asks that one. A client without one has no DNS. Which second server you hand out decides whether the filter still applies: see Prevent an outage.

The node corpus store cannot open

Warden keeps a node corpus in a folder named .warden-node-corpus. On a standard install it is /var/lib/purge-warden/.warden-node-corpus. A host that already has one beside its master config.toml keeps using that one.

From 0.44.5, a host without a cluster keeps serving DNS when the corpus cannot open, and logs this error with the cause:

node corpus store unusable; DNS continues without it: list bodies come only from the lists directory, the IP filter is built from the config, and this host cannot pair until the store opens

The usual cause is the owner or the mode. The folder and the folders inside it must be real folders, not symbolic links, owned by the user the daemon runs as (purge-warden), with no permissions for group or others (mode 0700). A mode of 0750 is refused. To fix it:

sudo chown -R purge-warden:purge-warden /var/lib/purge-warden/.warden-node-corpus
sudo find /var/lib/purge-warden/.warden-node-corpus -type d -exec chmod 0700 {} +
sudo systemctl restart purge-warden

On a host that keeps the corpus beside its master, use the .warden-node-corpus folder next to config.toml in these commands instead, for example /etc/purge-warden/.warden-node-corpus.

With cluster.enabled = true, the same error still stops the start.

Install

What the installer’s exit status means

Quick start lists every exit status of the installer. Three of them have an entry on this page:

• 3: Warden is installed and still loading its lists. Nothing failed: see status 3.
• 5: a DNS check failed, or the daemon stopped while it was starting: see status 5.
• 6: a configuration already on the machine, for example one kept by the uninstaller, does not load with the new version. The installer stopped before it installed the daemon or touched a service. The lint error above the message names what to repair: see a manual edit.

An upgrade has statuses of its own: see Upgrade.

With curl -fsSL https://get.purge.cc | sudo sh, the exit status is the installer’s, unless the download step fails first: see The download step stops early. If curl or wget itself fails, the command exits with 0 and installs nothing.

The installer exits with status 3

On first start, Warden downloads and builds its blocklists before it answers DNS. On a slow link that takes longer than the installer waits, which is 600 seconds by default. Everything is installed. The installer only skipped the DNS checks and the final summary. Watch the start finish, then check that Warden answers and blocks:

journalctl -u purge-warden -f
dig @127.0.0.1 example.com
dig @127.0.0.1 doubleclick.net   # expect 0.0.0.0

To wait longer in a future run, set PURGE_LISTENER_TIMEOUT to a number of seconds. See Environment variables.

The installer exits with status 5

There are two cases:

• The daemon stopped while it was starting. The installer prints purge-warden stopped while starting up and points to journalctl -u purge-warden -n 50. Read the log for the cause. A refused configuration is covered in The daemon does not start after a manual edit.
• The daemon runs, but the final check failed. The installer asks Warden for a name that must resolve (google.com) and one that must be blocked (doubleclick.net). If google.com does not resolve, the upstream does not answer: see No upstream answers. If doubleclick.net is not blocked after 600 seconds, the first list download did not finish.

Check by hand:

dig @127.0.0.1 google.com
dig @127.0.0.1 doubleclick.net

If the second name is not blocked, read the daemon log. To wait longer in a future run, set BLOCK_VERIFY_TIMEOUT_SECS. See Environment variables.

Port 53 is already in use

Another program listens on port 53. On Debian, Ubuntu and Fedora this is often systemd-resolved. The installer detects it and offers to free the port, and install.sh --dry-run shows the plan first. See Quick start. To see who holds the port:

sudo ss -lntup 'sport = :53'

The installer refuses because a package is installed

If a purge-warden package is already installed, the installer stops, because the package and the bundle would both own the same files. Remove the package, or set PURGE_ALLOW_PACKAGE_OVERLAP=1 and accept that you own the result. See Debian & Ubuntu and Fedora.

The download step stops early

get.sh checks the SHA-256 of the bundle before it unpacks anything. If a download or a check fails, nothing is installed, and get.sh exits with status 1. Run the command again. If you set PURGE_BASE_URL, it must start with https://.

If a Warden bundle is already unpacked, get.sh first asks the new installer whether it would refuse. If it would, get.sh prints the installer’s message, exits with the installer’s status and keeps the bundle you have. With PURGE_VERSION set to a release before 0.44.5, get.sh cannot ask that installer first: it replaces the bundle, and the older installer decides.

Upgrade

The upgrade exits with 130, 75 or 4

Steps for each case are in Upgrade.

The upgrade refuses because of .s6-old files

A killed or crashed upgrade leaves the old binary and unit as /usr/local/libexec/purge-warden/warden.s6-old and /etc/systemd/system/purge-warden.service.s6-old. While either file exists, install.sh --upgrade refuses to start and changes nothing. Follow “Killed or crashed upgrades” in /usr/local/lib/purge-warden/docs/UPGRADE_RECOVERY.md. Do not delete the files first. They are your way back.

uninstall.sh removes both files together with the binary and the unit, and prints each path it removed. After that, the interrupted upgrade can no longer be recovered.

A command names a .warden-migration fence

An interrupted policy write or schema migration leaves a .warden-migration directory in the config directory. Commands refuse to run while it exists, and their error names the step that finishes it. To see it, run:

sudo warden config lint

For an interrupted policy write, the error says unfinished policy transaction and asks for a restart. The restart finishes or removes the fence:

sudo systemctl restart purge-warden

Never delete .warden-migration by hand.

A fence that the error calls an unfinished v3-to-v4 migration does not clear on a restart. Each start then fails, and the journal shows:

Error: an interrupted v3-to-v4 migration requires `warden migrate v3-to-v4` recovery before the daemon can start

Run the warden migrate v3-to-v4 --from-config ... command that the error prints, with sudo. On a schema 5 configuration, that command removes the fence, then prints cannot migrate schema_version = 5 and exits with status 1. Run sudo warden config lint until it prints config is valid. Then start the daemon:

sudo systemctl reset-failed purge-warden
sudo systemctl start purge-warden

uninstall.sh without --purge refuses while this fence exists, and changes nothing, because a later install would refuse the configuration it keeps. Run sudo warden config lint to see the step that finishes it, then uninstall again. With --purge, the uninstall goes on and deletes the configuration too.

The daemon fails after a reboot

On a 0.42.0 or 0.43.0 host, the daemon can stop with receipt identity mismatch. install.sh --upgrade to 0.44.0 or later accepts a failed daemon and fixes this. See Upgrade.

The service keeps restarting

After a few failed starts systemd gives up, and systemctl status purge-warden shows Start request repeated too quickly. Read the log for the cause. Then:

sudo systemctl reset-failed purge-warden
sudo systemctl start purge-warden

See systemd service.

Configuration changes

Changes stop taking effect after a Local DNS edit

This was a known issue in 0.44.0, and 0.44.5 fixes it. From 0.44.5, a global Local DNS record applies on reload, like a per-profile one, and later reloads are not blocked. The rest of this entry is for hosts that still run 0.44.0.

On 0.44.0, after you add or remove a global Local DNS record (warden local-dns add or remove without --profile, or the Local DNS tab of the TUI), Warden does not serve the change. It also refuses every later configuration reload until it restarts, so later changes to devices, profiles, Custom Lists and the rest are saved but not applied. The commands still report success.

After a global Local DNS change on 0.44.0, restart the daemon:

sudo systemctl restart purge-warden

systemctl reload and warden reload do not help. During the restart DNS stops answering until the lists are loaded again, which can take a minute or two with large lists. Make all your Local DNS changes first, then restart once.

To avoid the restart on 0.44.0:

• If you do not want the change, remove the record you just added. The next reload succeeds and applies the changes that were waiting.
• Add the record to a profile with --profile. It applies on reload, but only while no global change is waiting, and it answers only the devices that use that profile.

To check whether reloads are being refused, run sudo warden audit tail -n 20. A refused reload shows reload requires daemon restart for runtime sections.

In 0.44.5 too, the same happens after warden security set rrl.enabled or warden security set rate_limit.enabled, and after you change enabled or tunneling.enabled under [security] in config.toml. These settings take effect only after a restart, and until then the daemon refuses later reloads. If you do not want the change, set the value back. The next reload then applies again.

On a cluster, restart the primary first. Then restart each secondary, one at a time, once warden cluster status on it shows that the new policy has arrived.

Move the config to /etc

warden config relocate prints refused (<name>): <reason>. Most refusals end with Nothing was changed. Look the name up in the refusal table. The most common ones:

• not-root: use sudo /usr/local/libexec/purge-warden/warden config relocate. The /usr/local/bin/warden wrapper runs as purge-warden and is refused.
• daemon-running: run sudo systemctl stop purge-warden-backup.timer purge-warden first.
• unknown-entry: the old directory holds a .warden- entry that Warden does not know. Move it away. Other files are not refused: they stay in the old directory, and the dry run lists them.

A refusal that tells you to roll back first means a previous run did not finish. Run the command it prints, which is relocate --reverse.

The daemon does not start: RelocateInProgress

An interrupted warden config relocate run leaves /var/lib/purge-warden/.warden-relocate.json. While that file exists, the node is fenced: the daemon does not start, and this node serves no DNS. The journal shows a message like this:

Error: the daemon does not start until the interrupted `config relocate` run is rolled back or finished, so this node serves no DNS. As root, run the `config relocate --dry-run` command named below, then the command it prints. Then run `systemctl reset-failed purge-warden` and `systemctl start purge-warden`.

Caused by:
    RelocateInProgress: /var/lib/purge-warden/.warden-relocate.json exists, so a `config relocate` run is incomplete and this node stays fenced; run `/usr/local/libexec/purge-warden/warden config relocate --dry-run` as root to see the rollback command

Run the dry run, then the command it prints, then start the daemon:

sudo /usr/local/libexec/purge-warden/warden config relocate --dry-run
sudo systemctl reset-failed purge-warden
sudo systemctl start purge-warden

Do not delete .warden-relocate.json by hand. If the dry run answers refused (receipt), the file is damaged: follow the manual steps it prints instead. uninstall.sh without --purge also refuses while the file exists.

Backup and restore

A backup fails with a permission error

The backup runs as purge-warden and writes only to /etc/purge-warden and /var/lib/purge-warden. Keep archives under /var/lib/purge-warden/backups, or give the backup service the extra path with a drop-in. See systemd service.

A backup refuses a file

A backup refuses symbolic links, hard links and special files in the configuration directory. Replace the link with a regular file.

Restore refuses an archive

More in Backup & restore.

Prevent an outage

When Warden stops answering, the whole network loses DNS. These habits keep an outage short, or stop it from happening.

Hand out a second Warden, never a public resolver. Give clients two DNS servers by DHCP: this host and a second Warden host. Clients move to the second one while the first restarts or fails to start. Do not use a public resolver such as 1.1.1.1 as the second server: clients that use it skip the filter. With only one Warden host, leave the second field empty. Clients then have no DNS while Warden restarts, but nothing skips the filter.

Size the second host for the full load, and give it the same policy. Some clients use the second server even while the first one is up. Clients that moved during an outage do not all come back when the first host answers again: many stay for minutes, and some stay for a day or more. A cluster secondary serves the policy of its primary. A standalone second host serves only its own configuration, so keep its profiles, lists and devices the same by hand.

Change the configuration through Warden. Use warden custom-list, warden config edit or the TUI. They check a change before they write it. Do not delete a pack in packs/ by hand, and keep other files out of that folder. See Custom rules.

Run warden config lint after every manual change. A running daemon does not show the error until its next start:

sudo warden config lint

Check that a reload applied. The change commands print daemon accepted the reload request — activation is not correlated, and warden reload prints daemon reloaded — config and lists are live. Neither proves that the daemon applied the change. The audit log records the result:

sudo warden audit tail -n 20

A refused reload is a reload row with the result rejected, and its errors=[...] gives the reason. Do not rely on warden status for this: it shows CONFIG REJECTED only until the next list cycle.

Keep automatic backups on. They are off by default. A fresh install already has a [backup] table: add auto_interval to it with sudo warden config edit for a daily archive. A second [backup] table makes the file fail to load.

[backup]
dir = "/var/lib/purge-warden/backups"
auto_interval = "24h"

See Automatic backups.

See also

• Upgrade
• systemd service
• File paths