Published on

Scraping: look for the JSON in the page before you write a CSS selector

scrapingweb-scrapingpythondata-extraction
Authors
On this page

Most scraping tutorials start with a CSS selector. That is the most fragile place to start.

Before you write one, open the page source and search for <script. A lot of pages carry the same data twice: once as rendered markup, and once as JSON for the page's own JavaScript. The usual places:

  • <script type="application/ld+json"> for structured data
  • <script id="__NEXT_DATA__" type="application/json"> on Next.js sites
  • a window.__STATE__ = {...} style assignment
  • an XHR call in the network tab that returns JSON directly

The JSON is the data model. The markup is a presentation of it, and presentation changes all the time.

A test you can run

I made a small local page with three items. Each item is in the markup with a price-tag class, and the same items sit in a __NEXT_DATA__ block. Then I wrote two extractors:

import re, json, requests

html = requests.get("http://127.0.0.1:8765/list", timeout=10).text

def by_css(h):
    return re.findall(r'<span class="price-tag">\$([\d.]+)</span>', h)

def by_json(h):
    m = re.search(r'<script id="__NEXT_DATA__" type="application/json">(.*?)</script>', h, re.S)
    return [x["price"] for x in json.loads(m.group(1))["props"]["items"]]

Both work on the first fetch:

before redesign  css: ['10.5', '11.5', '12.5'] json: [10.5, 11.5, 12.5]

Then I changed the page the way a redesign does. The class went from price-tag to a generated name. The data did not change. I fetched again:

after redesign   css: [] json: [10.5, 11.5, 12.5]

The selector returned nothing. The JSON extractor did not notice.

What to take from it

  • The JSON also gives you real types. The price came back as a number, not a string with a currency symbol to strip.
  • It is not a guarantee. The JSON shape can change too, so you still need to check the result (the next post is about that).
  • Look at the network tab before parsing HTML at all. If an endpoint returns the JSON, you can skip the HTML.

I still use selectors when there is no embedded data. I just stopped starting there.

Discuss on X