Character encoding bugs that corrupt a dataset months later

The bug that doesn’t crash anything

Most scraper failures are loud. A selector breaks, a 403 shows up, a proxy pool dies, and you know within minutes. Encoding bugs are the opposite. They don’t throw exceptions. They write a slightly wrong string into a column and move on to the next row, week after week, until someone runs a query that depends on the string being right and gets nothing back.

We’ve had this happen on pipelines that had been running clean for months. Nobody touched the code. Nothing alerted. The rot was already in every row, just waiting for the right query to expose it.

Where a page’s real encoding actually lives

A page can declare its character encoding in three places, and they don’t always agree.

The HTTP response can send a Content-Type header with a charset parameter, like text/html; charset=utf-8. This is supposed to be authoritative, but plenty of servers omit it, or set it once at the CDN edge and never update it when the origin app changes.

The HTML document itself can declare encoding in a <meta charset="..."> tag, or the older <meta http-equiv="Content-Type" content="text/html; charset=..."> form. This is what the browser actually trusts when rendering, and it’s often more accurate than the HTTP header on older or poorly maintained sites, because whoever built the template set it once and it’s been correct ever since, even as the server config drifted.

A byte order mark (BOM) at the very start of the file can also signal encoding, mostly on UTF-16 or UTF-8-with-BOM files, which show up more in scraped CSV or XML feeds than HTML.

When these three disagree, a scraper that only checks one of them decodes the bytes wrong, silently.

What your scraper does when the signals disagree

Most HTTP libraries pick the header first. If the header is missing or wrong, behavior differs by library and version. Some fall back to a default (historically ISO-8859-1 for text/* content without an explicit charset, per the old HTTP spec), some run a statistical guesser like chardet or cchardet against the byte stream, and some just hand you raw bytes and expect you to decide.

ISO-8859-1 is the dangerous default, because it maps every byte value to something. It never fails to decode. Feed it UTF-8 bytes for an accented character, a curly quote, or an em dash, and it happily produces a valid but wrong string. No exception, no warning. A curly right single quote in UTF-8 is three bytes; decoded as ISO-8859-1, it becomes three separate garbage characters that get treated as perfectly normal text and stored as-is.

That’s mojibake, and it’s the most common form of scraped-text corruption. It doesn’t look like an error. It looks like a string with a few weird characters in it, and a human skimming a sample of ten rows might not even notice.

The double-encoding trap

The second-most-common failure is double-encoding, and it’s nastier because it can happen downstream of a correct decode. If you take a string that’s already correctly decoded to Unicode and encode it to UTF-8 bytes, then something later in the pipeline decodes those bytes as ISO-8859-1 or Windows-1252 and re-saves the result as a Python string, you get a specific, recognizable pattern: an accented character like “é” turns into “é”. If that string then gets encoded to UTF-8 again by a different stage that assumes it’s still clean text, you get a second layer of garbling on top of the first.

This tends to happen at pipeline boundaries: a scraper writes clean UTF-8 to a file, a second script reads that file with the wrong default encoding, transforms the text, and writes it back out. Each hop is individually reasonable-looking code. The corruption only exists because two hops disagree about what encoding the bytes in between represent.

Why it takes months to show up

None of this breaks anything at write time. The row gets inserted, the job finishes, the count matches expectations. The corruption sits there as data, not as a failure.

It surfaces later, and usually through one of a few specific triggers. A filter or LIKE query on a name or title that contains an accented character stops matching, because the stored bytes don’t match the bytes in the query anymore, even though both look “the same” on screen. A deduplication or join step that keys on a text field starts producing duplicates it shouldn’t, because two representations of what a human considers the same string decode to different byte sequences. NFC versus NFD normalization is a common source of this: some systems (notably certain macOS-originated content and some CMS export tools) store accented characters as a base letter plus a combining mark instead of a single precomposed character, and those two forms are visually identical but byte-for-byte different.

Or the trigger is external: a target site adds a section in a non-Latin script, or a new source gets added to the pipeline, and the heuristics that happened to work fine on mostly-Latin text fail hard on Cyrillic or CJK text, because a statistical charset guesser trained on the wrong sample size makes different mistakes at scale.

By the time any of this is noticed, the raw source bytes are usually long gone. If the pipeline only ever stored the decoded (and now wrongly decoded) string, there’s no way to recover the original text. Re-scraping the source might not even be possible if the page has changed or the source is gone.

The database layer adds its own trap

Encoding bugs aren’t only introduced during decoding. They can be introduced at the storage layer even when the decode was correct. MySQL’s utf8 character set, despite the name, is a 3-byte-max encoding and can’t store the full Unicode range, including a lot of CJK extension characters and all emoji, which are 4 bytes in UTF-8. A column defined as utf8 rather than utf8mb4 will silently truncate or reject those characters depending on the SQL mode, and in permissive modes it can insert a partial or replacement character instead of erroring.

This is a pure configuration bug, invisible until a specific character shows up in scraped content. It’s common on tables that were created years ago under an old default and never migrated, sitting quietly correct for every row that happens to only contain characters within the 3-byte range.

Building a pipeline that actually catches this

A few habits make this class of bug findable instead of invisible.

Keep raw response bytes, or at least a hash of them, alongside the decoded text for some retention window. If corruption is found later, having the original bytes is the only way to re-decode correctly instead of guessing.

Resolve encoding with an explicit priority chain rather than trusting a single signal: BOM first if present, then the HTTP header if it declares a charset, then the HTML meta tag, then a statistical guess as a last resort, and log which method was used per response. That log is what lets you go back later and find exactly which pages or which time range used the fallback guesser, instead of auditing the whole dataset.

Normalize text to a single Unicode normalization form (NFC is the common choice) at ingestion time, consistently, so that visually identical strings from different sources are also byte-identical.

Run a small canary check as part of routine pipeline monitoring: a handful of known strings containing accented Latin characters, curly punctuation, a CJK sample, and an emoji, decoded and compared against expected output. It’s cheap to run on a schedule and it catches a broken default encoding or a bad library upgrade before it’s written into a few hundred thousand rows.

None of this prevents every encoding bug. But it turns “we found garbage in the data and have no idea when it started or what the correct text was” into “we found garbage in the data, here’s the exact run it started in, and here are the raw bytes to fix it.”

If you’re building or auditing a scraping pipeline and want more from us on the infrastructure side of this work, take a look at what we cover on Data Research Tools.

Get new guides and videos first — join the Telegram channel.