Measured results
The September 11, 2026 rerun measured Hostveil alongside Lynis, the CIS Docker Benchmark, a TCP scan from the Docker bridge, and the kernel’s listening sockets. The tables include the changes that helped, the risks that remained, and the state rollback did not restore.
What this measurement establishes
Hostveil’s own score increasing after its own fixes only establishes consistency. Independent tools help check whether the host changed too. This is one seeded configuration on disposable VMs, not a security guarantee or a survey of production servers.
This final rerun supersedes September 10 for reporting. The fixes/rollback experiment and the control experiment each used a fresh VM, without manual pre-hardening. See the run record and limitations, the raw fixes/rollback JSON, and the raw control JSON. Earlier runs remain in the repository.
The measured host
The seed runs Nextcloud with PostgreSQL, Jellyfin with Redis, and Portainer with Watchtower. It permits root SSH login, disables the firewall and automatic updates, and seeds weak accounts, agent configurations and kernel settings. All 13 domains ran; 13 completed without a coverage gap.
| Operating system | Ubuntu 24.04.5 LTS |
| Architecture | x86_64 |
| Kernel | 6.8.0-134-generic |
| Hostveil version string | hostveil v3-dev |
| Measured at | 2026-09-11T04:52:54+00:00 |
| Source commit | 290b4d1ad429df94e95e6628fbb583eed800750d |
The binary reports v3-dev; the source commit above was recorded separately. These figures describe that build, not an arbitrary later release.
The instruments
| Instrument | Version / scope |
|---|---|
Hostveil scan --json | Configuration findings and score |
| Lynis | 3.0.9; 264 tests before, 270 after Review |
| docker-bench-security | 154869da6418089decf7e1ab0cfca0e1cdfc5c49; 44 image/container-runtime checks |
| Docker / Trivy | Docker version 29.8.0, build 88096ef / Version: 0.74.0 |
| External TCP scan | A connect scan from a container on the Docker bridge; answered, refused or filtered |
ss -ltn | Kernel sockets bound to wildcard/routable addresses, counted separately from reachability |
Five phases
| Phase | Action before measurement |
|---|---|
| before | Freshly seeded host |
| after | hostveil fix --all --yes; Auto fixes, before service restarts |
| restarted | Restart/reload services and run docker compose up -d for each stack |
| reviewed | hostveil fix --all --review --yes; then restart/reload again, including irreversible exec actions |
| restored | Roll back all checkpoints, then restart/reload; exec actions remain outside rollback |
File edits do not all take effect at once. File modes can change immediately, while a running container keeps its old port mapping until recreated. SSH changes that can cut off access require Review. Showing every phase keeps the effect of edits, restarts and accepted Review actions distinguishable.
The results
| Measurement | before | after | restarted | reviewed | restored |
|---|---|---|---|---|---|
| Hostveil score | 28 | 37 | 40 | 58 | 32 |
| Active findings | 140 | 97 | 83 | 52 | 121 |
| Lynis hardening index | 57 | 63 | 66 | 79 | 67 |
| Lynis warnings | 4 | 4 | 4 | 3 | 3 |
| Lynis suggestions | 37 | 33 | 33 | 22 | 31 |
| CIS Docker PASS | 17 | 17 | 19 | 20 | 16 |
| CIS Docker WARN | 15 | 15 | 13 | 12 | 16 |
| Wildcard/routable listening ports | 6 | 6 | 2 | 2 | 7 |
| Ports answering from the Docker bridge | 6 | 6 | 2 | 1 | 6 |
The Auto phase applied 55 fixes with no failures. Review applied another 40: 24 left checkpoints and 16 were marked [not reversible].
What changed, and what did not
After the first restart, the scan stopped reaching 5432, 6380, 8080, 8096. After Review, only 22 answered. The kernel still listed 22, 6379 as wildcard/routable listeners: Redis on 6379 remained bound while the scan recorded it as filtered. A binding and a successful TCP connection are different observations. Port 2375 was already refused before any fix.
CIS warnings 5.14, 5.26, 5.3 cleared at restart, and 5.11 (container memory limits) cleared after Review. Warning 5.32 also appeared, so four cleared warnings produced a net decrease of three. Check 5.3 is a measurement caveat: docker-bench accepts a non-empty SecurityOpt; no-new-privileges can satisfy that test without configuring SELinux.
Lynis improved at the Auto phase as well as after restarts and Review. Its warnings changed: AUTH-9283, PKGS-7392 cleared and KRNL-5830 appeared. AUTH-9204 and AUTH-9208 remained; both concern the seeded second UID-0 account, which Hostveil reports as accounts.uid0 and leaves for manual handling.
| Domain | before | reviewed |
|---|---|---|
| SSH | 10 | 100 |
| Containers | 0 | 2 |
| CVEs | 5 | 5 |
| AI agents | 0 | 13 |
SSH reaches full marks only after Review. The container axis remains low with risks such as mounted Docker sockets, host networking and secrets in environment variables. CVEs do not improve in this run; pulling a current image tag does not guarantee it is vulnerability-free. The agent axis improves but retains findings. These results describe the measured commit, not the current fix registry’s coverage.
Does the score respond to independent hardening?
A separate fresh VM ran scripts/measure/run.sh -C, which applies control.sh using CIS guidance and Lynis suggestions. Hostveil only scanned. This control used Ubuntu 24.04.4 LTS and was measured at 2026-09-11T05:07:49+00:00.
| Measurement | before | controlled |
|---|---|---|
| Hostveil score | 28 | 38 |
| Active findings | 140 | 126 |
| SSH axis | 10 | 59 |
| Accounts axis | 15 | 30 |
| File permissions axis | 25 | 50 |
| Kernel axis | 14 | 27 |
| Lynis hardening index | 57 | 62 |
| Lynis warnings | 4 | 2 |
| Lynis suggestions | 37 | 37 |
| CIS Docker PASS | 16 | 17 |
| CIS Docker WARN | 16 | 15 |
| Ports answering from the Docker bridge | 7 | 2 |
| Wildcard/routable listening ports | 7 | 7 |
SSH, accounts, file permissions and kernel scores improved without Hostveil applying fixes. Lynis cleared AUTH-9204, AUTH-9208, and CIS cleared 5.26. The firewall, container, CVE, agent and Docker daemon axes did not change.
The control scan reported fewer reachable ports even though ss listed the same seven wildcard/routable ports before and after. The run did not establish the cause. The observed drop in reachability must not be presented as proof that the control closed those bindings.
Rollback fidelity
| Paths watched | 60 |
| Paths changed | 28 |
| Paths restored exactly | 25 |
| Checkpoints rolled back | 79 |
| Fidelity (%) | 89 |
All checkpoint rollbacks completed without a reported failure, but the host did not return to its full original state. The paths that did not match their before-state hashes were /etc/issue, /etc/issue.net, /etc/shadow.
| Path | Recorded explanation |
|---|---|
/etc/shadow | ssh.emptypasswords locks an empty-password account with passwd -l, an exec action with no restore checkpoint. |
/etc/issue, /etc/issue.net | Hostveil’s own edits were rolled back, but the package installation triggered by updates.disabled also upgraded base-files and rewrote the banners. The September 10 investigation traced this through dpkg.log; September 11 reproduced the same residue without repeating that investigation. |
Before-state hashes come from the live host before fixes, not from Hostveil’s checkpoints. The run also records paths whose before-state was unobserved: /usr/bin/x86_64-linux-gnu-as, /usr/bin/x86_64-linux-gnu-as (0755), /usr/bin/x86_64-linux-gnu-ld.bfd, /usr/bin/x86_64-linux-gnu-ld.bfd (0755). They are not evidence of successful restoration.
After rollback, the score is 32, compared with 28 before. The scan reaches 22, 5432, 6380, 8080, 8096, 9000, while the kernel lists 22, 5432, 6379, 6380, 8080, 8096, 9000. Redis remains filtered and Portainer’s 9000 now answers. Equal port counts do not mean equal ports, and rolling back files does not undo package installs, account locks or firewall commands.
Limits of the result
This is a full Auto-plus-Review experiment, including irreversible actions. It is not the outcome of fix --all alone. The seeded host is intentionally weak, all domains ran, and the systemd and proxy axes already scored 100 before fixes. Domain availability does not prove that every rule was exercised.
The TCP scan originates on the Docker bridge, not on the public internet. Routing, filtering and timing can change its observations; it does not test cloud security groups or every possible network path. The control discrepancy remains unresolved. These instruments measure configuration and reachability, not breach probability.
The September 10 and 11 fixes/rollback runs reproduced the same scores, finding counts, auditor deltas and unrestored paths. That is repeatability on this fixture. Package repositories, images and vulnerability databases can change future runs; comparisons across Hostveil versions also include scoring and coverage changes.
Reproduce it
Use a disposable VM from scripts/measure/Vagrantfile. The scripts deliberately weaken and then modify that host. See the linked run record for the exact environment. Run the fixes experiment and the control on separate freshly seeded VMs.
# Inside the first disposable VM, using the recorded source commit:
sudo scripts/measure/seed.sh
sudo scripts/measure/run.sh -c -p cisc-w2026-seeded-v2 /tmp/fixes.json
# Inside a separate, freshly provisioned disposable VM:
sudo scripts/measure/seed.sh
sudo scripts/measure/run.sh -c -C -p cisc-w2026-control-v2 /tmp/control.jsonBoth recorded runs passed check.py. Its rollback check permits residue when Review includes actions marked not reversible; exit 0 does not mean 100% file restoration. The website’s figures are checked against the published raw run, with the control figures checked against their separately named file.