The Nginx Setting That Was Hiding Our Own 500 Errors

Bezmaske renders pages twice: once for people, once for crawlers, through a small Puppeteer service that sits behind Nginx. We had already built a fallback for when that service is unreachable — Nginx catches the failure and serves a plain, unrendered page instead of an error. It worked. We tested it. And it was still not enough.

A routine audit of the server logs turned up a handful of real HTTP 500 responses served straight to crawlers, weeks apart, on a page that was supposed to have a safety net. The fallback existed. It just never ran.

The failure mode our test missed

Our test scenario was the render service being down — connection refused, timeout. Nginx handles that correctly out of the box: a failed connection produces a 502 or 504 that it generated itself, and `error_page` catches those without any extra configuration.

What actually happened was different. The render service was up. It received the request, hit an internal error — the headless browser under memory pressure, in our case — and did the responsible thing: it caught the exception and replied with its own, deliberate `500`. That response is not a connection failure. It is a normal, well-formed HTTP response that happens to carry an error code, and Nginx treats it as exactly that: a normal response, to be relayed as-is.

location / {
    proxy_pass http://127.0.0.1:3003;

    # Without this, Nginx only intercepts errors it generates itself
    # (502/504 on connection failure). A 500 the backend sends on
    # purpose sails straight through to the visitor.
    proxy_intercept_errors on;

    error_page 500 502 503 504 = @fallback;
}

One line. It had been missing since the fallback was first written, because the difference between "Nginx generated this error" and "the app generated this error and Nginx is just carrying it" is not something you notice until you go looking for it specifically.

Why this is worth writing down

A fallback that only covers connection failures looks complete in every test you are likely to run, because connection failures are the easy way to simulate an outage. A backend that fails gracefully — catches its own exception, replies with a clean error instead of hanging — is invisible to that kind of test. It is also the more common failure in production, because graceful degradation is what you build the rest of the system to do.

A fallback you have not tested against a graceful failure has not really been tested against failure — only against the app disappearing entirely.