The Encoding Your Code Never Specified

We recently found a bug in our own pipeline that never actually broke anything in production. That’s an odd thing to write about, but the reason it never broke anything is the interesting part, and the reason it would have is why this sits in the Saya silo rather than somewhere else.

The short version: a line of Python was reading text files without telling Python what encoding to use. On our machine, that omission was harmless. On a different machine, it would have silently corrupted every em dash in every caption we publish — not with an error or a warning, just wrong characters, shipped.

The line that looked finished

Deep in our social scripting, a spec file gets read like this:

from pathlib import Path
import json

spec = json.loads(Path("caption-spec.json").read_text())

Path.read_text() is one of those methods that looks complete because it runs without complaint. It takes no required arguments, it returns a string, and the string looks right. What it doesn’t tell you is that when you don’t pass an encoding= argument, it doesn’t default to UTF-8 — it defaults to whatever the operating system’s locale says text should be decoded as. On macOS, where these scripts were written and always run, that locale is UTF-8. So read_text() did the right thing, every time, for as long as anyone tested it here.

That’s the whole trap. The code has an encoding. It was never “encoding vs. no encoding” — text on disk is always bytes, and bytes are always decoded according to some rule. The only question was whether the rule was written down in the code or left to whatever machine happened to run it. We’d left it to the machine, and our machine happened to agree with us.

What a different machine does with the same bytes

Windows’ default locale encoding is cp1252, not UTF-8. cp1252 is a single-byte encoding — every byte maps to exactly one character, with no concept of a multi-byte sequence. An em dash written in UTF-8 is three bytes. cp1252 doesn’t recognise that sequence as a unit; it decodes each byte on its own, as three separate characters, because that’s the only thing a single-byte encoding knows how to do. It doesn’t raise an exception — from cp1252’s point of view nothing is wrong, it produced three valid characters. They’re simply the wrong three characters.

We proved this rather than took it on faith: we read the same spec file both ways and compared. Decoded under the locale rule, one caption came back as 18 Google organic clicks — tota. Decoded as UTF-8, the same bytes read 18 Google organic clicks — total,. Notice the locale version isn’t just garbled at the dash — it’s truncated at a different point too, because once the byte count and the character count disagree, anything downstream that measures or slices the string by length is now working off the wrong number.

If we’d shipped that, the failure mode wouldn’t have been a crash. It would have been 86 captions going out with — sitting where an em dash belonged, and nothing in the pipeline objecting, because nothing asked the text whether it looked right — only whether it decoded without error. cp1252 always decodes without error. That’s what makes this class of bug dangerous: the code path that’s supposed to catch problems isn’t built to notice a decode that “succeeded” into the wrong content. This is the same shape of failure we wrote about in our post on a double-encoded em dash — that one was bad bytes that had already shipped; this one was a decode that would have manufactured bad bytes on the way out, on a machine we hadn’t tested on yet.

Why this lands on translated text first

An English-only pipeline can run for years without ever noticing this class of bug, and it’s worth being precise about why. Most of the characters in plain English prose are ASCII, and ASCII is a strict subset of both UTF-8 and cp1252 — the first 128 code points decode identically under either scheme. A sentence made almost entirely of unaccented Latin letters, digits, and basic punctuation will come out the same regardless of which locale reads it. The bug is there, but it has nothing to bite on.

Translated and localised text is a different case, because the whole point of translation is to stop being ASCII. Accented letters, curly quotes, em and en dashes, and non-Latin scripts entirely are exactly the characters that require more than one byte in UTF-8 — and exactly the characters a single-byte legacy encoding will mangle. The moment a pipeline starts handling copy in French, or Spanish with its accented vowels, or Japanese, the proportion of non-ASCII bytes in a typical string jumps from “rare” to “most of the interesting content.” A latent locale bug that was once a once-a-year fluke against English text becomes a bug that fires on nearly every string, every time, for anyone handling multilingual copy. If you’re writing systems that touch translated or localised content — as covered in our piece on what actually needs solving when you take a website’s content global — encoding correctness stops being a nice-to-have and becomes one of the first things worth checking.

The same reasoning applies to infrastructure, not just language. A script that reads files correctly on the developer’s laptop can behave differently the moment it runs inside a container, on a CI runner, or on a colleague’s machine with a different default locale — the kind of environment mismatch we’ve run into before with containerised infrastructure behaving differently. The code didn’t change. The environment did, and the code had quietly been depending on it all along.

The fix, and why it had to be broader than the one file

The fix itself is small: add encoding="utf-8" to the call, so the decode is explicit and no longer a function of whoever’s locale happens to be active.

spec = json.loads(Path("caption-spec.json").read_text(encoding="utf-8"))

The part worth being careful about is scope. This wasn’t a one-line patch to the file we happened to be working in. We went through every json.loads(...read_text()) call under scripts/social and added the encoding argument to each one, because the defect isn’t tied to a particular file — it’s tied to the pattern. Any spec file that carries prose, anywhere in that directory, was exposed to the same failure, whether or not it was the one in front of us that day. Fixing the instance we noticed and leaving the rest would have meant the next spec file with an em dash in it reintroduces the exact same bug, just somewhere we weren’t looking.

After the change, we re-measured everything the pipeline currently touches: four specs, 86 captions total, 0 warnings, 0 failures. Every caption, every dash, decoded and re-checked as the character it was meant to be.

The honest version of this story

We want to be direct about what this was and wasn’t. It wasn’t an outage. Nothing published wrong. No reader ever saw — on a live caption. This was caught in review, on the same machine that wrote the bug, before it ever reached a platform where the locale would have disagreed with us. The reason we’re writing it up isn’t that something broke — it’s that we could see, clearly, the exact conditions under which it would have: a Windows box, a CI runner with a non-UTF-8 default, any environment other than the one we happened to develop on. We didn’t need to wait for that environment to show up. The proof — reading the same bytes two ways and comparing the output character by character — was enough to show the failure was real and only a matter of which machine ran the script next.

That’s really the point of a pipeline that handles multilingual and localised text: the assumptions that are invisible in a monolingual English codebase are the first ones to matter once other languages are involved. If you’re maintaining a content or translation pipeline and haven’t specifically checked how your text is being decoded — not just where it’s stored, but every place it’s read back off disk — it’s worth a look before a different machine finds it for you. If it would help to have a second set of eyes on how your pipeline reads, decodes, and re-serialises text, get in touch with Saya and we’re glad to walk through it with you.

Saya provides translation and localisation for Australian businesses — what we do.