When an image pipeline fails silently: a disk-quota postmortem

On a Sunday morning at the end of August I opened a property page on a Portland-area brokerage client's site and read the words "48 photos" above a grid of blank tiles. Nothing had thrown. No job had failed, no log line existed, no status code was out of place. This is the image pipeline postmortem for that morning: a silent failure that ran from a shared-hosting disk quota all the way to a visitor's screen without one layer in between raising its hand.

I am writing it in the shape a good engineering postmortem takes, because the shape is most of the value. Google's SRE book defines one as "a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring," and it is firm about tone: a blameless postmortem "assumes that everyone involved in an incident had good intentions and did the right thing with the information they had." When the team is one person, blameless is not a courtesy you extend to a colleague. It is the discipline that keeps you looking at the mechanism instead of at yourself, which is the only place a fix can come from.

The timeline

The site runs an IDX catalog off a RESO Web API feed from the regional MLS, roughly 76,500 listings once the full territory opened up. Photos are stored locally so they can be resized and served fast, which is the part that broke.

Why did nothing raise an alarm?

Because nothing failed, in the sense that any of the code was checking for. Five layers each did what they were written to do, and the composition of five correct-looking answers was a lie.

Start at the bottom. The account had reached its disk quota, so every write returned errno 122, which on Linux is EDQUOT, the user's quota of disk blocks exhausted. PHP's GD extension opened each destination file, wrote nothing into it, and reported success. That is documented behavior, not a surprise, and I had simply never read the line. From the manual page for imagewebp(), and word for word the same on imagejpeg():

Returns true on success or false on failure. Caution: However, if libgd fails to output the image, this function returns true.

So the encoder said yes. My cache layer then asked its usual question, which was whether a file existed at the rendition path and whether it was newer than the source. Both true. It treated the empty file as a finished rendition and sent it with Cache-Control: public, max-age=31536000, immutable, which is the correct header for a rendition and a catastrophic one for a blank. Browsers kept it. The edge kept it.

Meanwhile the queue that downloads photos recorded its own success honestly: rows were written, so the job returned "done, 0 added" and moved on. And the error log, on a host like this, is a file. On the disk that had no room. The one place a human might have looked was the one place that could not be written to.

Outward, every response was HTTP 200. That matters more than it sounds. Google's crawl budget documentation, last updated in July 2026, says the crawl capacity limit drops when a site "responds with server errors (5xx HTTP status codes) or rate-limiting signals (such as HTTP 429)". My server produced neither. It produced fast, cheap, confident 200s full of nothing, so there was no back pressure from anywhere.

None of this is exotic. Yuan and colleagues at Toronto studied 198 production failures across Cassandra, HBase, HDFS, MapReduce and Redis, and found that "almost all (92%) of the catastrophic system failures are the result of incorrect handling of non-fatal errors explicitly signaled in software", 35% of them in patterns as plain as an empty handler or a log line standing in for one. Mine was duller than that. I did check the return value. I just believed it.

Which link in the chain actually mattered?

Read the chain as five links: the quota is reached, the encoder writes nothing and says otherwise, the cache accepts the empty file as truth, the edge pins that truth for a year, and the job reports success because rows exist. Break any one link and the outage is a slow day rather than a blank website. The cheapest one to break, by a wide margin, is the third.

Two contributing factors are worth naming separately, because both are the kind of thing that only shows up in production. The first is that the quota is invisible from inside PHP: disk_free_space() reported terabytes free the entire time, because the limit is per hosting account and the volume underneath it is enormous. Anything that asks the filesystem how it is doing will get a cheerful and useless answer.

The second is who filled the disk. Over 40 GB of it was photo galleries, downloaded because something opened a listing page, and most of those somethings were not people. Imperva's Bad Bot Report 2026, published in April, put automated traffic at more than 53% of all web traffic in 2025 against 47% human. I had written a feature where one page view could commit the server to fetching several dozen files, and pointed it at the open internet. Building the catalog on a RESO Web API feed came with a hundred documented edge cases. This was not one of them.

Fix one: an empty file is never a cache hit

One condition, and it is the whole outage.

// before: a file at the cache path was a cache hit
if (!is_file($cache) || filemtime($cache) < filemtime($src)) { /* render */ }

// after: zero bytes means render again
if (!is_file($cache) || filesize($cache) < 1 || filemtime($cache) < filemtime($src)) { /* render */ }

The general form is worth more than the line: validate the artifact you produced, not the return value of the thing that produced it. A cache is a claim that something was already computed correctly, so every write into one is a small act of publishing, and nothing should get in unexamined, least of all from a library whose own documentation says it will lie to you.

The fallback matters too. With no rendition possible, the visitor now gets the unaltered original photograph instead of a hole. It is heavier than the preset, so it is served under a short cache window rather than pinned, and the next request gets a real rendition the moment one can be written. Degrade toward the expensive correct answer, never toward the cheap empty one.

Fix two: write to a temp name, then rename

The second change is older than the bug and would have prevented it anyway. Renditions are encoded to a temporary name and renamed into place only after they are checked, so the cache path holds a complete file or no file, and never a half of one.

$tmp = $cache . '.' . getmypid() . '.tmp';
$ok  = $webp ? @imagewebp($dst, $tmp, $qw) : @imagejpeg($dst, $tmp, $p['q']);
imagedestroy($dst);
clearstatcache(true, $tmp);
if (!$ok || !is_file($tmp) || filesize($tmp) < 1 || !@rename($tmp, $cache)) {
    @unlink($tmp);
    return false;
}
return true;

The guarantee comes from the operating system. The Linux manual page for rename(2) states that if the destination "already exists, it will be atomically replaced, so that there is no point at which another process attempting to access newpath will find it missing." Same directory, same filesystem, one instruction as far as any other process is concerned. Without it, a second request arriving mid-encode can read a partial file and, in the old code, cache it. The quota exposed that race, but the race was always there, and it would have surfaced eventually as a corrupt thumbnail nobody could reproduce.

The same discipline runs one layer up, in the code that downloads a photo from the feed: write the whole buffer or write nothing. A short write used to leave a zero-byte part-file behind and report a stored photo. There were 5,428 of those on disk when I went looking.

Fix three: every listing gets one photo before any listing gets forty

Nothing here was broken. I had simply never decided the order, and the order turned out to be the entire experience. A backfill of 64,000 covers does not finish on the day you start it: at roughly forty listings a scheduled tick it runs for weeks. Before the outage the queue served whoever had asked most recently, so one visitor on one sold listing from last winter could pull forty photographs while a thousand homes actually on the market still had no picture at all.

The rule now is simple to say and it is the right kind of simple: one photo for every listing before forty for any listing. Written out in English, the sort reads:

ORDER BY (on the market AND no local cover)  DESC,   -- 1. the blank tile a buyer meets today
         seq DESC,                                   -- 2. somebody has this page open
         (no local cover)                     DESC,   -- 3. sold homes, cover only
         id                                           -- 4. oldest first inside each group
LIMIT 40

Two details in there earned their place the hard way. The LIMIT 40 is a batch claim rather than a single job, because sorting this queue once per job, with several workers running, is what got flagged as heavy database use the night before. And every listing that has no local file yet falls back to the feed's own photo URL on its card, so a tile is never blank while it waits its turn. A queue that will be behind for two weeks needs a plan for what the site looks like while it is behind.

Fix four: a budget on the part that grows

Covers are bounded: one per listing, and I can multiply. Galleries are not, because they are triggered by page views and page views are supplied by the internet. So full sets now stop at a configurable ceiling, mls_media_full_budget_gb, set to 10. Past it, opening a listing page still guarantees that listing's cover and nothing more; the gallery is served from the feed's own URLs until there is room.

And because the filesystem lies about quota, the queue stops asking and simply tries, once per process:

const PROBE_BYTES = 2097152;   // a photo's worth: feed originals run to 1.5 MB

$probe = $dir . '/.probe-' . getmypid();
$n = @file_put_contents($probe, str_repeat('0', self::PROBE_BYTES));
@unlink($probe);
return self::$canWrite = ($n === self::PROBE_BYTES);

That constant started at 64 KB and was raised nine minutes later, in the review pass over my own fix, on the grounds that a probe should ask the question you actually care about. Sixty-four kilobytes of headroom does not mean a photograph will fit. Two megabytes does, because the originals coming off this feed run to about one and a half.

If the probe fails, the drain switches to recording photo URLs without storing files, puts the jobs back with a six-hour delay, and raises an alarm that writes an admin log line and sends one email, at most once every six hours for as long as the condition holds. That last part is the fix I care about most. Everything else makes the failure survivable; this is the part that makes it visible.

What generalizes from a postmortem like this?

Four things, none of them specific to photographs.

• Check the artifact, not the receipt. A function's return value is a claim about the past made by code that has already stopped paying attention. The file is the evidence. This is the same instinct behind verifying that a change is observed working rather than reported done, and behind an SEO agent that has to show its receipts: a summary is not evidence.
• A cache must never learn from a failure. Caches turn one bad answer into a permanent one, and an immutable header turns permanent into a year. Anything with a long time-to-live deserves a validity test at the moment it is written.
• Queue order is a product decision. Any backlog long enough to still be draining tomorrow is a design surface. Somebody should decide what the users see while it drains, and that somebody should not be the sort clause you wrote without thinking.
• Anything a stranger can trigger needs a ceiling. A page view that commits your server to dozens of downloads is a budget with no limit written on it, and the majority of your page views are now automated.

There is a fifth that is really a corollary: your alarm cannot live on the resource that ran out. A log file on a full disk is not monitoring, it is a hope. The health signal for this site is now a small JSON endpoint that an external watcher polls, so the thing doing the checking and the thing being checked no longer share a failure.

What is still owed

One loose end became its own small lesson. Working the outage, I saw the background tick had not advanced since the previous morning and concluded the host's scheduler had stopped. It had not. Measured properly that afternoon, by file access times and a timestamp the tick now writes for itself, the job was firing on the hour and the half hour exactly as configured. The work inside it was stuck behind a long pass that never finished. So the feed's share of the tick runs first now, and every run stamps itself, which makes the question answerable from a status page instead of by inference. I had spent an hour suspecting the wrong layer for want of a number.

And the studio's own engine, which shares this image code, has the guard on a work branch rather than in production. Nobody has been bitten there, because nobody has pointed 76,500 property galleries at it. That is a coincidence rather than a reason. Same instinct as measuring my own token bill rather than trusting the dashboard: the discovery was expensive and the fix is cheap, so pay the cheap part everywhere the code lives.

The one number I keep coming back to is 5,428. That is how many zero-byte files were sitting in the photo directory, each one the residue of a download that reported success. Nothing in the system could see them, and a two-minute sweep removed every one.