Mojibake in Production: When a Double-Encoded Character Reaches the Reader

Eleven scheduled social posts, across four brands, carried the same fault: a single em-dash, in the source text, had become a double-encoded byte sequence that rendered on screen as the three-character sequence readers of mangled text recognise instantly once they’ve seen it — where one clean em-dash belonged, an “a” with a hat on it, a euro sign, and a curly quote sat in its place instead.

That is what mojibake is, stated plainly rather than technically: a character that was encoded correctly once, then encoded again as though it hadn’t been, so what a reader sees is not the character but the debris of encoding it twice.

Why every gate let it through

We have gates. That’s the uncomfortable part of this story, and the reason it’s worth telling as a localisation problem rather than just a bug report. Every one of those eleven posts had already passed whatever checks stood between a source article and a scheduled social post. None of those checks caught this, for a specific and unglamorous reason: the corrupted string was valid UTF-8. It wasn’t malformed. It wasn’t a byte sequence a validator could flag as broken, because nothing about it was broken in the sense a validator checks for. It was simply the wrong characters, correctly encoded. A check built to catch invalid encoding has no opinion about text that is validly encoded and semantically wrong — those are different failure categories, and we only had tooling for one of them.

It’s the same distinction we’ve drawn when comparing machine translation against human review: a system can be technically correct by its own definition of correct and still miss the specific failure a reader actually experiences. A translation engine that produces grammatically valid output in the wrong register has the same shape of problem as a validator that accepts correctly-encoded bytes carrying the wrong characters — both are checking a real property, just not the one that determines whether the reader gets what they were supposed to get.

Where it actually came from

The source articles were clean. We checked that first, because it’s the easier thing to blame and it would have been a simpler story if it were true. It wasn’t. The em-dash was correct in the original article, in the correct encoding, right up until the point where our pipeline extracted a passage from that article to use as social copy. Somewhere in that copy-extraction step, a string that had already been correctly decoded got encoded a second time as though it were still raw bytes — the specific mechanical cause of mojibake, and the reason it shows up disproportionately in pipelines that move text between more than one processing stage. Every additional handoff is another chance for something to encode what was already encoded.

This is squarely a localisation-and-encoding problem rather than a generic bug, which is why it belongs in Saya’s silo rather than filed as a stray content-pipeline defect. Character encoding is the substrate multilingual publishing runs on, and it is exactly the kind of thing that looks solved — UTF-8 everywhere, no mystery — right up until a pipeline touches the same string twice and nobody is checking for that specific failure. We’ve written before about what website localisation actually requires, and encoding integrity through every processing hop is as much a part of that as the translation itself. A perfectly translated sentence that renders as mojibake has failed the reader in the same place a mistranslation would — it just failed for a mechanical reason instead of a linguistic one.

How it was found — and how it almost wasn’t

We ran a sweep of local files first, the way you’d expect, checking the article sources and the generated post drafts sitting in our own repositories for the corrupted byte sequence. That sweep came back with zero matches. Zero is a comforting number, and it was also wrong — not inaccurate, exactly, just answering a question we hadn’t meant to ask. The local files we searched did not contain the affected posts, because the affected posts existed only as scheduled items in the live social platform, generated from a copy-extraction step whose output never got written back to a file we had a local copy of. A clean sweep of the wrong corpus produces a clean result. It does not produce a clean answer.

We found the actual eleven on 19 September by reading the live scheduled data directly rather than trusting the local dump one more time — the only way to see the fault was to look at the artefact a reader would actually see, not a copy of it sitting somewhere upstream. That is a specific, narrow lesson and we’d rather state it narrowly than generalise it into something grander: when a pipeline has more than one place text could live, a check against one of those places tells you about that place, not about the pipeline. We had, without deciding to, been auditing the wrong copy of the truth.

What we changed, and what we didn’t overclaim

All eleven posts were repaired. We are not going to attach a broader statistic to that number, because we don’t have one that’s verified — we know these eleven, found in this specific sweep of live data, and we are not claiming to know the corruption rate across everything we’ve ever published, because that would require the same live-data check applied retroactively at a scale we haven’t done. What we can say is narrower and more useful operationally: the copy-extraction step is the specific point in the pipeline where a string gets handled a second time after it was already correctly decoded once, and that is now where an encoding check sits, checking a string against itself before and after the extraction step rather than assuming a validly-encoded string is automatically a correct one.

The broader habit this reinforced is one we’d also describe in infrastructure failures building the platform: the failures that actually reach a reader are rarely the ones a system was designed to prevent. They’re the ones nobody thought belonged in the category of things worth checking, until one of them sits in eleven scheduled posts looking, to every existing gate, like ordinary correctly-encoded text.

If your own pipeline moves text through more than one processing stage before it reaches a reader — translated, extracted, re-encoded, republished — it’s worth asking whether anything actually checks the string against itself at each hop, rather than just checking that it’s valid UTF-8. Saya’s team can talk it through.

Saya provides translation and localisation for Australian businesses — what we do.