Alerting when a scraper silently returns empty

The scraper that crashes is the good kind of broken. It throws a stack trace, the job fails, something pages you, and you fix it in twenty minutes. The scraper that quietly returns zero rows, exits with status 0, and writes nothing to your database is the one that costs you a week of missing data before anyone notices.

We run production scraping pipelines across a fleet of proxy infrastructure, and this failure mode is the one we’ve had to build the most tooling around. Not because it’s rare. Because it’s invisible by default, and every monitoring setup we’ve seen catch fire started with someone assuming “no errors” meant “it worked.”

Why empty results don’t look like failures

A scraper request can come back with a 200 status code, a valid HTML document, and zero usable data, all at once. Your HTTP client sees success. Your parser doesn’t throw because the page is well-formed, it just doesn’t contain what you expected. The job finishes on schedule. The exit code is clean. From the outside, everything about that run looks identical to a run that pulled 4,000 rows.

A few concrete ways this happens in real pipelines:

• A target site changes a class name or restructures its DOM, and your CSS selector or XPath now matches nothing instead of erroring.
• Content that used to render server-side moves behind client-side JavaScript, and a scraper built for static HTML gets a shell page with no data in it.
• A site serves a soft-block page to traffic it’s flagged as automated. This is often a 200-status page that looks plausible at a glance but contains a captcha prompt, a “please verify you’re human” message, or just a stripped-down template with none of the real content. Sites do this specifically because a hard block (403, 429) is easy for a scraper to detect and retry around. A soft block that still returns 200 is much harder to distinguish from a normal empty page unless you’re checking for it.
• Pagination logic breaks silently, a “next page” token changes format, and the scraper thinks it’s reached the end of results on page one.
• An authenticated session expires mid-run and the site redirects to a login page that still returns 200.
• A geo-restriction serves a regional landing page instead of the content you’re used to getting from that IP range.

None of these throw an exception. All of them produce an empty or near-empty result set from a job that reports success.

The monitoring gap: status codes aren’t enough

Most scraper monitoring starts and stops at “did the job run” and “did the job error.” That catches crashes, timeouts, and connection failures. It does nothing for the case above, because the job didn’t fail in any way your orchestrator understands.

The fix is to stop treating “ran without error” as the success signal and start treating “returned the expected shape and volume of data” as the success signal. That means monitoring the output of the scraper, not just its exit status.

What to actually check

Row count against a rolling baseline. Track how many records each scraping job produces per run and keep a trailing window, seven or fourteen days is usually enough to smooth out normal day-to-day variance. Alert when a run comes in meaningfully below that baseline, not just when it’s zero. A source that normally returns 2,000 rows and comes back with 40 is just as broken as one that returns nothing, and it’s the kind of partial silent failure that a “zero rows” check alone will miss entirely.

A hard zero-rows check, separate from the baseline check. Zero is unambiguous and should page faster and louder than a percentage drop. Don’t let it get buried in the same alert bucket as a 20% dip that might just be normal variance.

A canary selector or pinned record. Pick something on the target that should always exist, a specific product ID, a known listing, an element that’s present on every legitimate page render, and assert its presence before you trust the rest of the parse. If the canary is missing, you’re very likely looking at a changed layout or a block page, and you can fail the job loudly instead of writing an empty result set as if it were valid.

A content fingerprint for known bad pages. Soft-block pages, login redirects, and “content not available in your region” pages tend to be structurally distinct from the real thing, similar byte size, a captcha widget, different set of meta tags, a specific string in the body. If you’ve seen the block page before, hash it or grep for a distinctive string and check new responses against that signature before you assume the page is legitimate. This is defensive: you’re teaching your pipeline to recognize when a site has served it something other than real content, not trying to get past that response.

Byte length as a cheap first pass. Before you even parse anything, a response that’s a tenth the size of what you normally get from that endpoint is worth flagging. It’s a blunt signal but it’s nearly free to compute and it catches a lot of the cases above before you’ve spent any CPU on parsing.

Building the alert without drowning in noise

None of this is useful if it pages someone on every transient blip. Sites have real hiccups, proxies rotate and occasionally land on a slow exit node, a single run coming in 15% low isn’t a crisis.

A few things that have kept our alerting usable instead of ignored:

• Require the drop to be sustained across two or three consecutive runs before escalating, not just one. A single bad run gets logged and watched, not paged.
• Separate “zero rows” from “degraded volume” into different severities. Zero rows on a source that normally has data is worth waking someone up for. A 30% dip is worth a message in a channel someone checks that day.
• Route the alert to wherever your team actually looks, not just an internal log table nobody opens. A Slack or Telegram message with the job name, the row count, and the baseline it’s being compared against is enough context to triage without digging through logs first.
• Deduplicate. If a selector breaks and the job runs hourly, you don’t need twelve identical pages in a row. Bundle repeats of the same failure into one alert with a “still happening” timestamp update.
• Log the raw response alongside the alert, or at least a pointer to where it’s stored. When you get paged for zero rows, the first question is always “what did the page actually look like,” and if you didn’t save it, you’re re-running the scraper just to find out.

Diagnosing what kind of empty it is

Once you’re alerting on this reliably, the next useful step is distinguishing why a run came back empty, because the fix is different in each case. A changed selector needs a parser update. A soft-block page means the traffic pattern from that IP or session got flagged and needs a different approach to sourcing, whether that’s proxy rotation, request pacing, or reducing concurrency, not a code fix. An expired session needs re-authentication logic. A genuinely down site just needs a retry with backoff. Lumping all of these into one generic “scraper failed” alert means someone has to manually diagnose the cause every single time. A fingerprinted response, a canary check result, and a byte-length comparison, taken together, usually tell you which bucket you’re in before a human even opens the logs.

Sites actively defend against automated traffic, and that’s expected, not a bug in your pipeline. The goal of this kind of monitoring is to make sure that when something does break, you find out from an alert in the first hour instead of from a client asking why the dashboard hasn’t updated in a week. It won’t make a scraper immune to being blocked or to a target changing its layout.

If you’re building out scraping infrastructure and want more on the operational side, from proxy management to pipeline orchestration, you can find the rest of our writeups on the Data Research Tools home page.

Get new guides and videos first — join the Telegram channel.