Seven Checks That Answered the Wrong Question

Two servers. Same PHP version string, same advisory, same pair of CVSS 9.2 remote code execution bugs in the interpreter. On the first server, apt upgrade was the whole job.

On the second, the identical command reported success and patched nothing.

What followed was four days of migration work, and the interesting part was not the CVE. It was that seven separate verification steps gave the wrong answer — most by passing while the thing they checked was broken, one by failing while the server was perfectly healthy. None of them was an exotic check. Each was the obvious one.

The CVE itself, and why php -v is the wrong way to check for it, is in the companion piece: PHP2Shell and WordPress: Why php -v Is the Wrong Check. This post is what happened after the patch turned out to be an infrastructure project.

Why one apt upgrade worked and the other did nothing

The second host ran a pre-packaged WordPress stack — the kind where the entire runtime lives under one vendor directory. The vendor’s own documentation is blunt: these stacks ship a bundled PHP, and even if you install PHP from system packages, the stack keeps using the bundled one.

So on that box, three signals:

• php -v after adding a PHP repository — green, new version.
• dpkg -l | grep php — zero packages. There never were any.
• The web server — still executing the vendor’s compiled binary, built over a year earlier.

Two of the three said “patched”. The one that mattered was the one nobody checks. The vendor’s remedy is not a package; it is migrating the instance. A five-minute patch had become a new server.

Never carry a fix between hosts because the symptom matches. Check how the runtime was installed before deciding what the fix even is.

The one I would put on a wall

Of the seven, this is the one worth the whole post.

The destination had a rule refusing to execute PHP inside the uploads directory — the standard defence against someone planting a file there. The test was textbook: drop a PHP file containing a unique marker into uploads, request it over HTTP, confirm a 403, confirm the marker never appears.

It returned 403. The marker did not appear. The gate reported PASS.

The gate was meaningless. A positive control — the same probe placed outside uploads, where it must execute — returned nothing at all. PHP was not running anywhere on that site.

The matcher was named for uploads but constrained only the file extension, never the path. It was returning 403 for every .php request on the server, index.php included. The site would have been dead from the second DNS pointed at it, and the security test would have gone on reporting PASS indefinitely, because a dead site and a protected directory produce identical output.

A test that only checks for the safe outcome cannot tell “protected” from “everything is broken”. Pair every negative security test with a positive control — a case that must succeed. It costs one extra request.

The validator that caused the outage it existed to prevent

The web server config validated cleanly. Valid configuration, exit 0. It then failed to start and took a co-tenant site down for ninety seconds.

The error was permission denied — opening its own access log.

The cause was the previous day. Validation had failed when run as an unprivileged user, so it was re-run as root. That root run created the log file, owned by root, mode 0600. The service runs as an unprivileged account and could not open the file that validating had just created for it.

Validation parses and adapts a config. It does not bind ports, open log files, or drop privileges. Everything that happens only at startup is invisible to it — and here, the act of validating manufactured the fault it was meant to catch.

The guard that aborted on a healthy server

The source schema used a newer collation family, and the provisioning script refused to run unless all three collations were present, looked up by name in the server’s catalog. On a server that supports all three perfectly well, the lookup found one, and the script stopped.

The database engine had introduced character-set-independent collation names two releases earlier. The catalog now reports the short form. The prefixed name is accepted and stored verbatim, but never appears as a row.

The guard checked a catalog string. What actually mattered was whether a table could be created using that collation — which is precisely what restoring the dump does anyway. Testing that instead takes four more lines and cannot go stale the next time someone renames something.

Two counts that disagreed, and neither was a bug

A list of CDN proxy ranges was fetched fresh rather than reused from a day-old copy — the right instinct. Counting it with wc -l returned 20 where the previous copy had 22, which produced a confident announcement that the list had changed.

It had not. The files carry no trailing newline, and wc -l counts newlines rather than lines, so it undercounts the final line of each file. Twenty-two both times.

Later, comparing the migrated database against the source, an options table showed 361 rows against a recorded 331 — exactly the shape of silent data loss during a migration. Running the same SQL on both sides returned 361 and 361. The original figure had come from a management tool that excludes transient rows.

These two are not the same error, and it is worth separating them. The first is a tool doing exactly what it documents while you assume it does something adjacent. The second is two different questions, one of which looked like a wrong answer to the other. When two counts disagree, confirm both sides were asked the same question before you believe either.

A permission test that ran outside the sandbox

Diagnosing the failed startup, the obvious check was whether the service account could write its log directory. test -w as that account said yes.

That test ran in a normal shell. The service runs under systemd confinement. Writability outside the sandbox tells you nothing about writability inside it. In this instance the real cause was simpler than the sandbox anyway — which is the point. A check that cannot fail for the right reason will cheerfully succeed for the wrong one.

The thing that was worse than the CVE

Partway through, an unrelated package install returned a non-zero exit code while obviously succeeding. That one confusing line unwound into this:

• The SSH daemon had two startup mechanisms enabled at once, each claiming port 22. One won at boot. The other had been failing with “address already in use” for close to six months.
• That failure broke the SSH package’s post-install step, leaving the package half-configured — for twenty-five days, provable from the package manager’s own log — which made every subsequent package operation exit non-zero regardless of outcome.
• The same failure disabled the tool that reports “you have services running outdated binaries”.
• The running SSH daemon was executing a binary from seven months earlier, already deleted from disk, kept alive only by the kernel holding its inode. Five security updates had been installed in that window. None had been loaded.

I want to state that last point mechanically rather than dramatically, because the mechanics are worse than any adjective. Nothing refused anything. The updates installed correctly. The files on disk were current. The process simply never restarted, so it kept running code that no longer existed anywhere on the filesystem — on a box where the firewall, the patch cadence and the monitoring all looked fine.

Every individual symptom was invisible or dismissible. A non-zero exit on an install that worked. A socket unit in a failed state nobody queries. A non-zero exit code on an install that “worked” is a thread worth pulling.

The honest part

Ninety seconds of downtime, on a site that had nothing to do with the migration, caused by the config change in the validator section above. Validation passed, startup did not, and the revert needed a second round trip to a remote host.

The fix is not “be more careful”. It is to make the risky step self-healing. The re-apply ran inside a single remote script that installed the new config, restarted, health-checked for eight seconds, and restored the backup automatically if the check failed.

Two things about that, and the second one matters more.

The deploy that caused the outage had no auto-revert. The revert was written afterwards, in response to the incident it would have prevented. And on both deploys since, the health check passed on the first second — so the revert branch has never executed. It is an untested recovery path.

I would be entitled, by the usual conventions of writing like this, to describe that as “now self-healing” and move on. But a recovery path that has only ever taken the happy path is exactly what the rest of this post is about. I have not seen it refuse anything. Until I have deliberately failed a health check and watched the backup come back, it is a plan, not a mechanism.

A post about misleading green checks that left out my own outage — or that quietly upgraded my untested fix into a working one — would be the thing it is criticising.

The through-line

Seven failures, one shape. Every one tested a proxy for the thing that actually mattered:

the proxy that was checked	the thing that mattered
the config parses	the service starts
the rule returns 403	the rule is scoped to what it names
the collation name is in the catalog	a table can use the collation
the CLI runtime loads an extension	the web runtime loads it
the interpreter’s version string	the package’s version
wc -l	lines, including one with no trailing newline
writable in a shell	writable inside the sandbox

None of these is exotic. Each is the obvious check, and each sits exactly one inferential step away from the real question. That step is where the confidence lives and where the error hides.

Two techniques caught the dangerous ones. Nothing else did.

Positive controls. Prove the probe can observe a failure before you trust it reporting success. The uploads rule is the entire argument: a security gate reporting PASS, indefinitely, on a site that was completely broken, with no other signal anywhere that anything was wrong.

Primary artefacts over reports about them. The package changelog instead of a vendor page. A created table instead of a catalog row. An HTTP response instead of a command-line version string. Each substitution costs one extra command, and each one caught something.

What the controls actually returned

Arguing for positive controls and then not running any would be its own punchline, so here is what they returned on the destination. Two of these are the primary-artefact substitution rather than a control, and I have marked which.

The data migrated intact, and the check could have told me if it hadn’t. 793 rows carry 4-byte characters — emoji and similar — which are the first thing to break under a wrong collation. Source and target both report 793 rows with an identical fingerprint, 9063c…7d39, computed as a hash over the content of every one of those rows rather than a count. The two sides report different database server versions, which is the cheap answer to “did you accidentally compare one host with itself”.

That is still only an absence of failure. So the detector was tested against a deliberately mangled copy — and the first attempt at that test passed for the wrong reason.

The mangled table was built by copying a real 4-byte row into a utf8mb3 table. Strict mode rejected the lossy insert outright, so the table was empty. The comparison then measured a real hash against an empty string, found them different, and printed CONTROL PASS. The fingerprint had never changed. The row did not exist.

I wrote that while writing a control, for a post about checks that pass for the wrong reason. It is the cheapest possible demonstration that this failure mode is not a beginner’s mistake — it is what happens when you check for a difference without first asserting that both things you are comparing exist.

The second attempt added that assertion, and relaxed strict mode so the conversion is lossy-but-successful, which is what a real mis-migration does on a non-strict server:

• intact row — 2,145 bytes, 2,122 characters, 1 question mark
• mangled row — 2,124 bytes, 2,122 characters, 8 question marks

Twenty-one bytes gone, seven emoji silently replaced, identical character count, different hashes. The detector can observe the failure. It did not observe it on the real data. Those are two separate facts and both had to be established.

The command line still disagrees with the web runtime, on the patched box. A live HTTP request reports sapi=fpm-fcgi ver=8.3.6 opcache=on. php -v on the same machine, at the same moment, reports PHP 8.3.6 (cli) — same version, and no opcode cache at all. Not a fault: the CLI disables it by default and registers it under a display name that a naive grep misses. The command line answers a different question and does not mention that it is doing so.

The changelog check works as advertised (primary artefact, not a control). On the destination, offline, one line returns both CVE identifiers, and the noble-security entry names the patch file directly — debian/patches/CVE-2026-109618_109619.patch, “Avoid state corruption via bailout” — on installed package 8.3.6-0ubuntu0.24.04.13. That is the check the companion post tells people to run, and it is the only local artefact that settles the backport question.

The SSH daemon is resolved (primary artefact, not a control). Before: the running process pointed at /usr/sbin/sshd (deleted) — a binary that existed only as an inode the kernel was still holding — on a build seven months old. After: a listener whose executable is a file that exists, on the current package, with the package state no longer half-configured and the duplicate socket mechanism disabled so boot is deterministic. The old deleted binary was written to disk first and a timer armed to restore access if the restart failed. It did not fire.

One leg is deliberately missing. The uploads rule — the lead of this post — was re-tested with both its legs after the matcher was fixed, and both behaved correctly. But that run was against a webroot containing only synthetic probes, not the real site, which was still landing at the time. I am not printing those numbers, because a positive control run against a scaffold proves the scaffold. The real figures go in when the real site has been tested, and if they come back worse than the scaffold did, that is the more interesting result and it goes in too.

If you take one thing

• Pair every negative security test with a positive control.
• Validation is not startup. Deploy behind an automatic revert, not behind care.
• On a backporting distribution, check package versions, never --version output.
• Never check a web runtime from the command line.
• Run a permission test inside the same sandbox as the service.
• When two counts disagree, confirm both sides were asked the same question.
• A non-zero exit code on an install that “worked” is a thread worth pulling.

The uncomfortable part is that every one of these servers passed its checks. What we need is not more checks but a habit of asking, once, what each one would look like if the thing underneath it were completely broken — and whether we would be able to tell.