JSON Web Scraping: How to Find, Extract and Parse the JSON Behind Any Website

Developer guide · Last reviewed 28 September 2026 · 12 min read

JSON web scraping means collecting data from the JSON that websites already use, instead of parsing it out of HTML. Most modern sites load their products, listings and search results from JSON endpoints, and many embed structured JSON in the page itself. This guide shows where that JSON lives, how to find it with your browser’s developer tools, how to request and parse it in Python, and how to run JSON scraping reliably at scale with residential proxies.

Why JSON Web Scraping Beats HTML Parsing

A web page is a presentation layer. Somewhere behind it, the site has the same data in a structured form, and on most modern sites that form is JSON: JavaScript Object Notation, the lightweight text format defined in RFC 8259. When a page loads products after you scroll, filters results without reloading, or shows a price that appears a moment after the rest of the page, it is almost always fetching JSON.

• Cleaner data. Prices arrive as numbers, dates in standard formats, and IDs are included, with no text to clean from HTML.
• Less fragile. Redesigns change HTML classes all the time; data endpoints change far less often.
• Faster and lighter. One small JSON response can replace a full page with scripts, styles and images, which also cuts proxy bandwidth.
• More complete. Endpoints often return fields the page does not display, such as stock counts or product IDs.

Where JSON Hides on a Website

1. XHR and fetch requests

Single-page applications and infinite-scroll listings call JSON endpoints from the browser, for example a search or products endpoint with query parameters for page, filters and sorting.

2. Embedded state in the HTML

To render quickly, many sites send the initial data inside the HTML as a JSON object in a <script> tag, which the JavaScript framework then picks up. Search the page source for "price" or a product name and you will often find it.

3. JSON-LD structured data

Sites add JSON-LD blocks, <script type="application/ld+json">, to describe products, articles, events, organisations and reviews to search engines using schema.org vocabulary. Google’s documentation recommends JSON-LD as the format for structured data. For scrapers it is a gift: name, price, currency, availability and ratings in a documented structure.

4. Public APIs

Some sites offer documented APIs. If one exists for the data you need, use it: it is the most stable and clearly permitted route.

How to Find JSON Endpoints with DevTools

1. Open the page in Chrome or another browser, press F12 and select the Network tab.
2. Filter by Fetch/XHR and reload the page, then scroll, filter or click “load more” to trigger more requests.
3. Click each request and open its Preview or Response tab. Look for your products, prices or listings.
4. When you find it, check the Headers tab: the request URL, the method (GET or POST), query parameters and the request body.
5. Right-click the request and choose Copy as cURL to reproduce it exactly, then trim headers until you know which ones are needed.

Use the search box in the Network panel to search all responses for a value you can see on the page, such as a specific price. It finds the right request faster than clicking through dozens of them.

Before you write any code, test the request on its own. Paste the copied cURL command into a terminal, confirm it returns the same JSON, and then change one parameter at a time, such as the page number or category, to learn how the endpoint behaves. Ten minutes spent here usually saves hours of debugging later.

Request and Parse JSON in Python

import requests

PROXY = "http://USERNAME:PASSWORD@v2.proxyempire.io:5000"
proxies = {"http": PROXY, "https": PROXY}
headers = {"User-Agent": "Mozilla/5.0", "Accept": "application/json",
           "Accept-Language": "en-US"}

url = "https://www.example.com/api/products"
params = {"category": "laptops", "page": 1, "sort": "price_asc"}

r = requests.get(url, params=params, headers=headers, proxies=proxies, timeout=30)
r.raise_for_status()
data = r.json()                       # parsed into Python dicts and lists

for item in data.get("products", []):
    print(item.get("id"), item.get("name"), item.get("price"))

Check the response’s content type before calling .json(). If a site answers with an HTML error or CAPTCHA page, .json() raises an exception, which is better than silently storing bad data. Handle that case explicitly and log the URL.

Reading embedded state from the HTML

import json
from bs4 import BeautifulSoup

html = requests.get(page_url, headers=headers, proxies=proxies, timeout=30).text
soup = BeautifulSoup(html, "html.parser")
tag = soup.find("script", id="initial-state")      # the id or type differs per site
state = json.loads(tag.string)
print(state.keys())

Common pitfalls with JSON endpoints

• Required headers. Some endpoints expect an Accept header, a Referer or a custom header the page sends. If a request works in the browser but not in your script, add the browser’s headers back one at a time until you find the one that matters.
• Short-lived tokens. Endpoints that need a token issued to the page usually expect a fresh token per session. Load the page first in the same session to get it, and never reuse tokens that belong to someone else.
• Different data per country. The same endpoint can return other prices, currencies or products depending on the visitor’s IP. Record the country of every request next to the data.
• Silent schema changes. Sites rename or move fields without notice. Check that the fields you rely on exist in every response, and alert when they disappear.

How to Extract JSON-LD Structured Data

import json

def json_ld(soup):
    for tag in soup.find_all("script", type="application/ld+json"):
        try:
            block = json.loads(tag.string or "")
        except json.JSONDecodeError:
            continue
        for obj in (block if isinstance(block, list) else [block]):
            yield obj

for obj in json_ld(soup):
    if obj.get("@type") == "Product":
        offer = obj.get("offers", {})
        print(obj.get("name"), offer.get("price"), offer.get("priceCurrency"))

JSON-LD can be a single object, a list, or a graph under @graph, and offers can be one offer or a list. Write your extraction to handle each shape.

Pagination, Cursors and GraphQL

• Page numbers: increase page until the list comes back empty or you pass the total count the response reports.
• Offsets: add the page size to offset each time.
• Cursors: many APIs return a next token or URL; pass it to the next request and stop when it is missing.
• GraphQL: some sites use a single endpoint that accepts a POST with a query and variables. The GraphQL documentation describes how queries are sent over HTTP. Copy the query from DevTools and change only the variables, such as the cursor or category.

items, cursor = [], None
while True:
    params = {"limit": 50, **({"cursor": cursor} if cursor else {})}
    page = requests.get(url, params=params, headers=headers,
                        proxies=proxies, timeout=30).json()
    items += page["results"]
    cursor = page.get("next_cursor")
    if not cursor:
        break
print(len(items), "records")

Working with Nested JSON

Real responses are deeply nested: a product contains variants, each with prices by region and stock by warehouse. Three habits keep this manageable:

• Read defensively. Use .get() with defaults so a missing field does not crash the run, and count how often each field is missing.
• Use a query language such as JMESPath for complex selections, for example every variant price in one expression, instead of nested loops.
• Store raw and flat. Save the raw response as JSON Lines, one JSON object per line, and write a separate flat table with the fields you analyse. When you need a new field later, re-parse the raw file instead of scraping again.

Scaling JSON Scraping with Proxies

JSON endpoints are cheap to call, which is exactly why sites rate-limit them per IP. Hundreds of calls a minute from one server will soon get 429 “too many requests” answers or an HTML challenge page instead of JSON. Rotating residential proxies spread the calls over many household IPs, so each one sends only a few requests.

• Rotation: a new IP per request suits independent calls such as product IDs; a sticky session suits paginated sequences where the cursor or cookies are tied to one visitor.
• Location: prices, currencies, availability and catalogues in JSON responses often depend on the visitor’s country. Country, city and ZIP targeting return the local version.
• Bandwidth: JSON responses are small, so JSON scraping is one of the most cost-efficient ways to use pay-per-GB proxies.

For the Python side of proxies, including sessions, SOCKS5 and retries, see our Python web scraping guide. For pages that only produce their data inside a browser, see headless browsers.

Rules and Good Practice

• Public data only. Do not use endpoints behind logins you are not entitled to, and do not reuse other people’s tokens or session cookies.
• Official APIs first. If the site offers a documented API, use it and follow its limits.
• Moderate the rate. Limit concurrency per site and back off on 429 responses.
• Avoid personal data unless you have a lawful basis, and check each site’s terms.

ProxyEmpire proxies are for lawful data collection. Banking, payment and government websites are blocked by default on all products. See our web scraping basics for more on responsible scraping.

JSON Web Scraping FAQ

References

1. RFC 8259: The JSON Data Interchange FormatThe JSON standard
2. Google Search Central: Introduction to structured dataJSON-LD, schema.org and Google’s recommended format
3. JSON-LDJSON for Linked Data
4. Chrome DevTools: Network features referenceFiltering requests, previewing responses, copying as cURL
5. GraphQL: Serving over HTTPHow GraphQL queries are sent with GET and POST
6. JMESPathA query language for JSON
7. JSON LinesOne JSON value per line for storing records
8. Python: json moduleEncoding and decoding JSON