- Published on
Scraping: look for the page's JSON before you write a CSS selector

- Authors

- Name
- Nadim Tuhin
- @nadimtuhin
On this page
Most scraping tutorials start with a CSS selector. That's the most fragile place you could start.
Before you write one, open the page source and search for <script. A lot of pages carry their data twice: once as rendered markup for you, and once as JSON for the page's own JavaScript. Here's where I look:
<script type="application/ld+json">for structured data<script id="__NEXT_DATA__" type="application/json">on Next.js sites- a
window.__STATE__ = {...}style assignment - an XHR call in the network tab that returns JSON directly
The JSON is the data model. The markup is just one way of showing it, and that changes all the time.
A test you can run
I built a small local page with three items. Each item sits in the markup with a price-tag class, and the same items sit in a __NEXT_DATA__ block. Then I wrote two extractors:
import re, json, requests
html = requests.get("http://127.0.0.1:8765/list", timeout=10).text
def by_css(h):
return re.findall(r'<span class="price-tag">\$([\d.]+)</span>', h)
def by_json(h):
m = re.search(r'<script id="__NEXT_DATA__" type="application/json">(.*?)</script>', h, re.S)
return [x["price"] for x in json.loads(m.group(1))["props"]["items"]]
Both work on the first fetch:
before redesign css: ['10.5', '11.5', '12.5'] json: [10.5, 11.5, 12.5]
Then I redesigned the page
I changed the page the way a redesign does. The class went from price-tag to a generated name. The data didn't change at all. I fetched again:
after redesign css: [] json: [10.5, 11.5, 12.5]

The redesign broke the selector: 0 matches. The JSON extractor parsed all 3 records in under 2ms and never noticed the CSS change.
Takeaways
- You get real types. The price came back as a number, not a string with a currency symbol to strip.
- It's not a guarantee. The JSON shape can change too, so you still need to check the result (the next post is about that).
- Check the network tab first. If an endpoint already returns the JSON, you can skip the HTML entirely.
I still use selectors when a page has no embedded data. I just stopped starting there.
Frequently asked questions
- Where can I find the JSON data in a web page for scraping?
- Search the page source for script tags. Common places are application/ld+json structured data, a __NEXT_DATA__ block on Next.js sites, a window.__STATE__ assignment, and XHR calls in the network tab that return JSON directly.
- Why is parsing embedded JSON better than CSS selectors?
- The JSON is the data model and the markup is only its presentation, which changes often. In my test a redesign renamed the price-tag class and the CSS extractor returned nothing, while the JSON extractor still returned all three prices as numbers.
- Is embedded JSON a guaranteed stable source?
- No. The JSON shape can change too, so you still need to validate what you extract. I still use selectors when a page has no embedded data.
Related posts
- Scraping: fail loudly when the page layout changes
A scraper that parses nothing and exits 0 looks healthy while it fills your database with gaps. A count check and a saved fixture catch it.
- Scraping: back off, honour Retry-After, and never retry a 404
A retry helper that backs off with jitter, reads Retry-After, and gives up fast on a 404. Run against a local server that fails on purpose.
- How I track my money with Hermes: the flow, not the numbers
Bank SMS and email alerts flow through one parser, a dedupe hash and a Google Sheet. A morning cron job drafts the summary. Only the flow, no figures.