Published on

Scraping: fail loudly when the page layout changes

Scraping: fail loudly when the page layout changes
Authors
On this page

The worst scraper bug is the quiet one.

A site changes its layout. Your selector matches nothing. The scrape has failed, but the loop runs over zero items and the job exits 0. Nobody notices until someone asks why a month of data is empty.

A server rack hums at night while a friendly robot shrugs over an empty basket next to a database.

Two small habits catch this.

1. Check against a number the page gives you

Pages often tell you how many items they have: a result count, numberOfItems in structured data, a "showing 1-20 of 340" label. Parse that too, and compare:

class Drift(Exception):
    pass

def parse(h):
    items = [{"name": n, "price": float(p)} for n, p in re.findall(
        r'<span class="name">(.*?)</span><span class="price-tag">\$([\d.]+)</span>', h)]
    if not items:
        raise Drift("0 items parsed")
    expected = int(re.search(r'"numberOfItems": (\d+)', h).group(1))
    if len(items) != expected:
        raise Drift(f"parsed {len(items)} items, page says {expected}")
    return items

Zero results should fail the job. So should a count that doesn't match the page's own number.

2. Keep a saved copy of a good page

Save one real page as a fixture and run the parser on it in your tests. That tells you the parser works.

It can't tell you the live site is unchanged, though. So run the same check on every live fetch too.

What it printed

I tested both. I saved a good page, then changed the live layout the way a redesign would:

fixture parses: [{'name': 'Widget 1', 'price': 10.5}, {'name': 'Widget 2', 'price': 11.5}, {'name': 'Widget 3', 'price': 12.5}]
live page after redesign -> Drift: 0 items parsed

The fixture still parsed. The live page raised Drift straight away, with a message that names the problem:

Drift: 0 items parsed

No guessing, no empty table discovered a month later.

In practice: what to do with the error

  • Let it fail the job, so your scheduler or monitor sees a non-zero exit.
  • Don't write partial results over good ones.
  • Log the URL and the count, and keep the bad HTML for debugging.

A scraper that stops is annoying. A scraper that's wrong and silent costs you the data you thought you had.

Frequently asked questions

How do I know when a website layout change breaks my scraper?
Make the scraper raise an error when it parses zero items or when the count differs from the number the page reports, such as numberOfItems in structured data. Without that check the loop runs over nothing and the job exits 0.
What should a scraper do when parsing fails?
Fail the job with a non-zero exit so your scheduler sees it, do not overwrite good data with partial results, and log the URL, the count and the bad HTML. A silent wrong scraper costs you data you thought you had.
Are saved HTML fixtures enough to test a scraper?
No. A fixture proves the parser still works on an old good page, but it cannot show that the live site is unchanged. Run the same count check on every live fetch as well.