Hostveil / Docs / Measured results

Measured results

Hostveil’s score going up after Hostveil’s fixes proves nothing: the same code decides what a finding is, what fixing it means, and what the number should be afterwards. So this page reports what other people’s tools saw on the same host, before and after — including the ones that did not move, and one that moved for the wrong reason. Every figure here comes out of a committed run you can reproduce.

Why not just report the score

A self-consistent scanner is easy to build and worthless. If Hostveil declares a port binding a finding, edits the Compose file, re-reads the file it just wrote, and awards itself points, nothing about that loop touches your server’s security. Every step could be internally correct and the whole thing still be theatre.

The only way out is instruments that were not written by us and do not share our opinions. Three of the four below were written by other people for other reasons; the fourth is the kernel’s own list of listening sockets, which has no opinions at all.

What follows is one run against one deliberately misconfigured host. It is evidence, not a benchmark: read what this does not show before quoting any of it.

The host

A server seeded with the misconfigurations self-hosters actually ship, running the services rather than stubbing them: three Compose stacks — Nextcloud with PostgreSQL, Jellyfin with Redis, Portainer with Watchtower — publishing every port on 0.0.0.0, with no no-new-privileges, no restart policy and no memory limits; an SSH drop-in permitting empty passwords, root login and password authentication; ufw installed and switched off; automatic security updates off; a natively installed Redis bound to 0.0.0.0; a second UID 0 account; a world-readable /etc/shadow; two AI agent runtimes with their config beside their API keys; two sysctl values relaxed the way a debugging or profiling guide tells you to, and left in /etc/sysctl.d to survive a reboot; and a hand-written backup unit with none of systemd’s sandboxing directives set.

The instruments

Five phases, and why three of them are separate

Every Auto fix Hostveil applies is a file edit. That is what makes it reversible, and it is also why none of it is in force the moment it lands: a running container keeps the port mapping it was created with, and a daemon keeps serving from the config it already loaded.

So the host is measured five times.

Reporting only after would say the independent instruments saw nothing, which is true and reads as though the fixes did nothing. Reporting only restarted would quietly credit Hostveil with a restart it does not perform and deliberately does not offer as an Auto fix — restarting a service is not reversible and cannot be checkpointed. The gap between those two columns is the single most useful thing on this page.

And reporting only the unattended fixes would measure half the tool. fix --all applies Auto fixes only; every SSH hardening option is Review, because turning off root login or password authentication can end the session you are issuing the command from. The reviewed column is what an operator who reads them and accepts them gets, and it is where the SSH domain finally moves.

The results

hostveil fix --all --yes applied 55 fixes and failed none. fix --all --review --yes then offered 42 more, of which 24 left a checkpoint and 18 did not — the ones that are not file edits, each marked [not reversible] in Hostveil’s own history, and none of them undone by the rollback below.

What moved

What answers from off the host. Restarting the services put four ports out of reach of a TCP connect from another network namespace: 5432, 6380, 8080, 8096, 9000 — PostgreSQL, the Redis published beside Jellyfin, Nextcloud and Jellyfin itself. This is the least arguable result on the page: nothing interpreted anything, the bind address changed, and the service stopped being reachable.

Accepting the Review fixes took two more, and those are the interesting ones. What still answers is 22, and nothing else. The two that went quiet are 2375, the Docker daemon’s own API — unauthenticated root for anyone who can reach it — and 6379, a natively installed Redis, which Hostveil reports and declines to fix, because binding that one to loopback means editing a config file whose path and syntax differ per datastore and which the finding does not carry. Neither was edited shut. Hostveil turned the firewall on, and a fix it refused to make was resolved by one it did.

So the kernel and the scanner disagree here, and the disagreement is worth having. 2 sockets are still bound to a routable address after the Review fixes, and 1 of them answers from off the host. A bind address and a packet filter are different claims about a port, both are true, and this page reports both rather than the flattering one.

The CIS Docker Benchmark. Three checks cleared at the restart — 5.14, 5.26, 5.3 — and a fourth once the Review fixes were accepted: 5.11, the memory usage for containers is limited, which is compose.ds010. 5.14 is incoming container traffic is bound to a specific host interface and 5.26 is the container is restricted from acquiring additional privileges — the port bindings and no-new-privileges, confirmed by a tool that has never heard of hostveil. 5.3 is a different story; see below.

SSH, but only on the reviewed path. Hostveil’s SSH domain goes 10 → 31 with the unattended fixes, and 10 → 100 once the Review ones are accepted. That gap is the whole reason the reviewed column exists: root login and password authentication are the two options that matter most and the two Hostveil will not switch off while you are not looking, because getting them wrong ends the session the command was issued from.

Lynis, at last. The hardening index went 57 → 80, and it moved without the lists moving: this run ends on the same 3 warnings and 20 suggestions it started with. Lynis’s index counts hardening tests passed, not entries struck off, so the three points are spread across tests no single suggestion names. On the unattended path the same index moves a single point — measuring only that path is what made an earlier version of this page report that Lynis never moves.

What did not move

Hostveil’s container axis went 0 → 2 out of 100. Every fix it has for a Compose file was applied and the axis stayed on the floor, because what is left is Manual by design: the Docker socket mounted into Portainer and Watchtower, host networking, secrets in the environment. Removing the socket mount breaks the two tools that need it, and Hostveil will not make that call for you.

Two of Lynis’s 3 warnings are a finding Hostveil found and declined. AUTH-9204 is check users with an UID of zero and AUTH-9208 is check non-unique accounts in passwd file — both the seeded second root, which Hostveil reports as accounts.uid0 and leaves alone. Deleting an account is not reversible from a checkpoint: userdel takes the account, its group and its mail spool, and orphans every file it owned. The overlap between the two tools is real, and Hostveil is choosing not to take it.

The CVE axis did not move at all: 16 → 16. Image updates are among the Review fixes accepted here, and pulling a current tag does not make the axis go up, because a newer image is not a patched one — each current tag carries its own published vulnerabilities. An earlier run on this seed watched the same axis go down for exactly that reason. It is on the page because a page that only showed the figures that moved the right way would be the thing this page exists to argue against.

The AI-agent axis went 0 → 13. The two file-mode findings were fixed; every finding about the config itself was not, and each is on the deliberately-unfixed list for the same reason: OpenClaw’s config is JSON5, users comment it heavily, and Hostveil has no round-tripper — re-encoding it would silently delete the operator’s comments. This is the domain that costs this host the most after containers, and it is the clearest statement on the page of where Hostveil detects and cannot yet fix.

Nothing moved in the after column except Hostveil’s own numbers. 55 fixes had been applied and 28 files on disk had changed, and the CIS Docker Benchmark, the external scan and the kernel’s socket table all reported an identical host. This is the honest shape of a file-editing fixer, and it is the reason the restart column exists.

Does the score move for somebody else's hardening?

Everything above is Hostveil fixing a host and independent instruments agreeing that something changed. It leaves the sharpest question open. A number that only responds to Hostveil’s own fixes is not a security score — it is a to-do list with a percentage on it.

So there is a second experiment, and it is the one this page was missing until 3.21. scripts/measure/control.sh hardens the same seeded host from the CIS Benchmarks and Lynis’s own suggestions — the sshd configuration, kernel parameters, file modes, the second root account, the Docker daemon — and Hostveil applies nothing at all. Every change is traceable to somebody else’s standard, and none of it was read off Hostveil’s fix registry. Then Hostveil scans, and the question is whether it noticed.

It moved, and it moved for the right reasons. Three axes rose and Hostveil had no hand in any of them: SSH, because a CIS drop-in turned off root login and password authentication; account hygiene, because the seeded second root account was deleted; file permissions, because the modes on /etc/shadow and the SSH host keys were tightened. Hostveil read the host afterwards and scored what it found.

Lynis agrees from the outside, and it agrees specifically: the two warnings it dropped are AUTH-9204, AUTH-9208, both of them the second root account — the same account Hostveil stopped reporting, found by a tool that has never heard of it.

What did not move is the other half of the answer. The firewall axis stays at 50 because control.sh enables no firewall, and the container axis stays at 0 because the findings that hold it down are the Manual ones the CIS Docker section does not touch — a socket mounted into Portainer, host networking, secrets in the environment. The five ports that stopped answering are visible to the external scanner and to docker-bench, and they do not lift that axis off the floor. A score that rose on every axis regardless of what was done to the host would be the more suspicious result.

Rollback fidelity

Not a security measurement — a claim about the recovery layer, and the one that has to hold before any of the rest is worth having. Hostveil asks an operator to let it edit files on a server, and “you can undo it” is the entire reason that is a reasonable thing to ask.

The one path that did not come back is /etc/shadow, and it is not a checkpoint that failed. The reviewed phase applies eighteen fixes hostveil itself marks not reversible — apt-get dist-upgrade among them, alongside a family of fixes that install rkhunter, fail2ban, auditd and five more hardening tools the host was missing. Installing a package leaves nothing to checkpoint; that is the fix doing exactly what it says, not the fix failing to do it. One of those installs adds a system account and rewrites /etc/shadow as an ordinary side effect, and rkhunter’s own dependency chain reinstalls the compiler toolchain — which is also why the compiler binaries carry no before-state: they did not exist when the before scan ran. Every one of the twenty-seven reversible changes came back byte for byte; the twenty-eighth was never reversible to begin with, and hostveil said so before it applied it.

The hashing is deliberately not done against the checkpoints. A checkpoint records what Hostveil believes it backed up, so comparing a restored file against it asks only whether Hostveil agrees with itself. The candidate paths come from the scan — every file a finding points at, plus every entry of /etc/sysctl.d, because a fix that creates a file is invisible in a before-state that only lists what exists — and they are hashed on the live filesystem before anything is applied. That is the state a rollback has to reproduce.

The restored column of the results table is the same claim measured a second way, by instrument rather than by hash — and it is the column that shows what a file rollback does not cover. Hostveil’s own score returns exactly, 29 to 34. The external scan does not: 7 ports answered before and 6 answer after.

That gap is not a rollback failure, and it is worth reading closely. Enabling a firewall is not a file edit and has no checkpoint, so ufw is still running when everything else has been put back. The two ports it is still holding shut are the ones no container publishes: 2375 and 6379, reported filtered rather than refused, which is a packet filter dropping them rather than nothing listening.

And the five that answer again answer through that firewall. Docker publishes a container port by writing its own rules ahead of ufw’s, so restoring the Compose files put those bindings back on 0.0.0.0 and the firewall never saw them. That is firewall.docker-bypass — a finding Hostveil reports and declines to fix, because the two remediations are unrelated and only the operator can choose between them — demonstrating itself, in an instrument that has never heard of it, on the way back down.

What this does not show

One host, one configuration. A seeded container, not a survey. It says what happened here; it does not predict what happens on yours.

Auto fixes only. fix --all applies exactly the fixes classified Auto. The Review and Manual findings — PermitRootLogin, the firewall, the Docker TCP socket, the second root account — were all left in place, and every one of them is a bigger change than anything measured above. This is a floor, not a ceiling.

Every domain ran. All thirteen, including CVEs and the AI-agent runtimes. Two of them, sysctl and systemd, used to score a silent 100 with nothing to find — not because the host was hardened, but because nothing in the seed gave either domain a question to ask: no operator-written systemd unit to grade, no sysctl override to catch. The seed now carries both — a hand-written backup unit with none of the sandboxing directives, and the two sysctl values every “gdb: Operation not permitted” guide tells you to relax — so the domain count moved from eleven, to twelve once the agent path was fixed, to thirteen now. The seed is in the repository and its paths are held against the checker by a test.

The scan comes from the Docker bridge, not from the internet. It reaches whatever is bound to 0.0.0.0 from another network namespace, which is the property being measured, but it does not cross a cloud provider’s own security groups. A port that answers here may still be unreachable from outside the machine — and a port that is refused here is refused for everyone.

Nothing here measures whether you get breached. Every instrument on this page, ours included, measures configuration. That is the honest limit of what a scanner reading files can claim.

Reproduce it

Both halves are in the repository: the script that builds the host, and the harness that measures it. Both edit the host they run on, irreversibly — point them at a container or a throwaway VM, never at your own machine. seed.sh refuses to run anywhere that does not look like one.

git clone https://github.com/seolcu/hostveil && cd hostveil
go build -o /usr/local/bin/hostveil ./cmd/hostveil

# Puts the host into the profile every figure above was taken on, and
# installs the three instruments. It permits root SSH login, adds a
# second UID 0 account and publishes databases on 0.0.0.0.
sudo scripts/measure/seed.sh

sudo scripts/measure/run.sh -c -p seeded /tmp/measurement.json

The seeding used to live outside the repository, which made these instructions incomplete in the way that matters: they told you to measure, and left you to guess the host. It also let the fixture drift away from the product without anything noticing — a copy of it wrote the OpenClaw config to ~/.config/openclaw/, which is not a path the agent checker scans, so the domain reported no runtime found on a host seeded to run two. Every path the seed writes is now held against the checker’s own table by a test.

The same seed scored lower than it used to, and that is the point. The previous committed run put this host at 40 before any fix; this one puts it at 29. The host did not get worse. Hostveil found more on it — 83 findings then, 140 now — and it stopped crediting itself for fixes that were written but not yet in force, which had been inflating the after column of every run before 3.20.1. A score that moves because the scorer got stricter is the one kind of regression worth publishing.

The run above was made repeatedly against separately seeded hosts on the same binary, and the figures that describe the host came out the same every time — the same scores, the same three CIS check IDs cleared at the restart and the same fourth at the reviewed phase, the same ports, the same files restored. What moved between runs was never the host: it was what was measuring it, and what had been seeded. A host without Trivy reported the CVE axis N/A and a correspondingly higher overall, because an excluded axis leaves the denominator rather than scoring zero. A host whose agent configs went to the wrong path did the same for that domain. Both are why the seeding is committed and pinned now, and why this run reports thirteen domains where an earlier one reported eleven, then twelve. Committed runs live in docs/measurements/, and every figure on this page is checked against the newest of them by a test — a stale number here is a failing build, not something a reader has to catch. Pass -c and the run also exits non-zero on the only two claims in its output that are promises rather than observations: that rollback restored every changed file exactly, and that Hostveil’s own score moved at all. Nothing else is asserted, because everything else on this page moves for reasons no change to Hostveil is responsible for.