
- Authors

- Name
- Nadim Tuhin
- @nadimtuhin
On this page
The worst scraper bug is the quiet one.
A site changes its layout. Your selector matches nothing. The scrape has failed, but the loop runs over zero items and the job exits 0. Nobody notices until someone asks why a month of data is empty.

Two small habits catch this.
1. Check against a number the page gives you
Pages often tell you how many items they have: a result count, numberOfItems in structured data, a "showing 1-20 of 340" label. Parse that too, and compare:
class Drift(Exception):
pass
def parse(h):
items = [{"name": n, "price": float(p)} for n, p in re.findall(
r'<span class="name">(.*?)</span><span class="price-tag">\$([\d.]+)</span>', h)]
if not items:
raise Drift("0 items parsed")
expected = int(re.search(r'"numberOfItems": (\d+)', h).group(1))
if len(items) != expected:
raise Drift(f"parsed {len(items)} items, page says {expected}")
return items
Zero results should fail the job. So should a count that doesn't match the page's own number.
2. Keep a saved copy of a good page
Save one real page as a fixture and run the parser on it in your tests. That tells you the parser works.
It can't tell you the live site is unchanged, though. So run the same check on every live fetch too.
What it printed
I tested both. I saved a good page, then changed the live layout the way a redesign would:
fixture parses: [{'name': 'Widget 1', 'price': 10.5}, {'name': 'Widget 2', 'price': 11.5}, {'name': 'Widget 3', 'price': 12.5}]
live page after redesign -> Drift: 0 items parsed
The fixture still parsed. The live page raised Drift straight away, with a message that names the problem:
Drift: 0 items parsed
No guessing, no empty table discovered a month later.
In practice: what to do with the error
- Let it fail the job, so your scheduler or monitor sees a non-zero exit.
- Don't write partial results over good ones.
- Log the URL and the count, and keep the bad HTML for debugging.
A scraper that stops is annoying. A scraper that's wrong and silent costs you the data you thought you had.
Frequently asked questions
- How do I know when a website layout change breaks my scraper?
- Make the scraper raise an error when it parses zero items or when the count differs from the number the page reports, such as numberOfItems in structured data. Without that check the loop runs over nothing and the job exits 0.
- What should a scraper do when parsing fails?
- Fail the job with a non-zero exit so your scheduler sees it, do not overwrite good data with partial results, and log the URL, the count and the bad HTML. A silent wrong scraper costs you data you thought you had.
- Are saved HTML fixtures enough to test a scraper?
- No. A fixture proves the parser still works on an old good page, but it cannot show that the live site is unchanged. Run the same count check on every live fetch as well.
Related posts
- Scraping: back off, honour Retry-After, and never retry a 404
A retry helper that backs off with jitter, reads Retry-After, and gives up fast on a 404. Run against a local server that fails on purpose.
- Scraping: look for the page's JSON before you write a CSS selector
Many pages ship their data as JSON inside the HTML. Reading that survives a redesign that breaks every class-name selector.
- Puppeteer in production: fallback selectors, real waits, clean exits
A Facebook class rename stopped 100+ accounts from posting. The fix: ARIA selectors, real waits, finally cleanup, a lock per account and screenshots on error.