Your cart is currently empty!
Parsing dates from pages in a dozen formats
The date on the page is rarely just a date
Pull the “published” or “updated” field off a hundred different sites and you will not get a hundred clean timestamps. You will get “March 3, 2026”, “03/03/2026”, “2026-03-03T14:22:00Z”, “3 days ago”, “Mar 3”, “il y a 3 jours”, a Unix epoch buried in a data attribute, and at least one site that puts the date in a title tooltip while showing “Yesterday” in the visible text. Every one of these represents the same underlying fact. None of them arrive in the same shape.
This is one of those problems that looks trivial until a pipeline actually depends on it. If you are deduping articles by publish date, sorting a feed chronologically, or filtering “show me everything from the last 7 days,” a wrong parse does not throw an error. It silently produces a wrong date, and that date flows downstream into a report or a decision. Date parsing in a scraper is a correctness problem that fails quietly. The formatting is the easy part.
Why one parser call does not survive contact with real pages
The instinct is to reach for a date library, pass it the scraped string, and move on. dateutil.parser.parse() in Python or Date.parse() in JS will happily accept most common formats out of the box, and for a single-source pipeline that can be enough. The trouble starts the moment you are pulling from more than a handful of domains.
A general-purpose parser has to guess when a string is ambiguous, and it guesses using its own default locale, not the locale the page was written in. Feed it a hundred strings from sites in different countries and some fraction will parse to a date that is technically valid but factually wrong, because the parser resolved an ambiguous format the wrong way. The failure mode is worse than a crash: the code runs fine, returns a datetime object, and that object is wrong. Nothing downstream flags it because nothing downstream knows the date was ambiguous in the first place.
The fix is a chain of attempts. One call will not get you there.
Building a fallback chain instead of a single call
A date parser that has to survive a dozen page layouts looks less like a function and more like an ordered list of attempts, each one more permissive than the last.
The first attempt checks structured data: <time datetime="...">, JSON-LD datePublished, or Open Graph article:published_time meta tags, before touching visible text. These are machine-readable by design and almost always ISO 8601, so when they exist, they’re the most reliable source on the page and skip the ambiguity problem entirely.
The second attempt runs a known-format regex. For a domain you scrape repeatedly, the date format on that domain does not change often, so a regex matched against a handful of formats you have actually seen from that site is faster and more predictable than a general parser, and it fails loudly (no match) instead of quietly (wrong match).
The third attempt falls through to a general library parser like dateutil or equivalent, only once the first two steps come up empty, and its output gets treated as lower confidence than theirs.
The fourth attempt handles relative dates: “3 days ago,” “yesterday,” “2h” need their own small parser that resolves against the scrape timestamp, not the current wall-clock time when you happen to reprocess the record later.
Each step that succeeds should tag the record with how it was parsed. That tag turns into an accuracy signal you can audit later, instead of a black box that either works or does not.
The DD/MM versus MM/DD problem
This is the single most common source of silent corruption in scraped date fields. 03/04/2026 is March 4th in the US and April 3rd in most of the rest of the world, and there is no way to tell which one it is from the string alone. A parser has to pick a default, and whatever it picks will be wrong for some fraction of your sources.
There are a few practical ways to reduce the damage, none of them a complete fix.
The page’s declared locale or lang attribute is a usable hint before falling back to a global default. The result is also worth bound-checking: if day-first parsing produces a date more than a day or two in the future, or absurdly old, retry as month-first, and vice versa, since neither number should be blindly trusted when the other reading produces something implausible. Unambiguous formats are worth preferring when they exist on the same page, many sites render both a human string and a machine datetime attribute, and if the attribute is there, trust it and stop guessing. Every ambiguous case you cannot resolve is worth logging rather than picking a default silently; a field marked “ambiguous, needs review” is more useful downstream than a field that is confidently wrong.
None of this eliminates the ambiguity. It just makes sure you know where it happened instead of finding out three reports later when someone asks why an article from April shows up in your March bucket.
Relative and vague dates need their own clock
“2 days ago” is not a date, it is a delta, and deltas need a reference point. The reference point is not “now” at read time, because if that record sits in a queue for six hours before processing, or gets reprocessed a week later during a backfill, “now” has moved. The correct reference is the timestamp your scraper recorded when it fetched the page. Store that alongside the raw scraped string, and resolve relative dates against it, not against the current system clock.
Vaguer strings like “yesterday,” “just now,” or a bare “Mar 3” with no year need explicit rules, not guesses. “Just now” typically means within the last few minutes and can usually be mapped to the fetch timestamp directly. A bare month-and-day with no year almost always means the current year, except right at a year boundary, where you are better off checking whether the resulting date is in the future and rolling it back a year if so.
Normalize to UTC, but keep the original
Sites publish in their own local timezone, sometimes with an explicit offset, often with none at all. Once you have a parsed date, convert it to UTC for storage and comparison. Do not try to infer a missing timezone from the site’s country, since that is one more unverifiable guess stacked on top of the ones you already made resolving the format.
Keep the raw scraped string alongside the parsed result. When a parse turns out wrong months later, and eventually one will, the raw string is the only way to figure out why. Storing only the parsed datetime throws away the evidence you’d need to debug it.
Cache the format per domain
Format guessing is the expensive part of this pipeline, and it is wasteful to redo it on every single page from a site you have already scraped a thousand times. Once you know example.com renders dates as Month D, YYYY in the visible text and has no structured markup, record that pattern against the domain. Future pages from that domain skip straight to the known regex and only fall back to general parsing if the known pattern fails to match, which itself is a useful signal that the site changed its template.
This also turns date parsing into something you can monitor. A sudden spike in fallback-parser usage or ambiguous-date flags from a domain that used to resolve cleanly on the first pass usually means the site redesigned its date markup, not that dates got harder to read. That is the kind of alert worth having before a downstream report quietly goes wrong.
What this actually buys you
None of this is exotic engineering. It is a fallback chain, a confidence tag per record, bound checks on ambiguous formats, a stored fetch timestamp for relative dates, and a per-domain cache. What it buys is a date field you can trust enough to sort, dedupe, and filter on without discovering months later that a chunk of your dataset has its months and days swapped. For a pipeline that runs across dozens of sources with no shared format convention, that is the actual bar, not “parses most strings.”
If you are building out the rest of the scraping stack around this, from proxy rotation to pipeline orchestration to the fingerprinting side of what sites are watching for, that is what we cover here.
Check out more breakdowns like this on the Data Research Tools home page.
Get new guides and videos first — join the Telegram channel.
Leave a Reply