LLMs and Data Gathering: How AI Web Scraping, RAG and Agents Collect Web Data

AI data guide · Last reviewed 26 September 2026 · 13 min read

LLMs and data gathering meet in two directions. Data is gathered for language models, to train and fine-tune them. More and more, data is also gathered by language models: AI web scraping tools that read pages and return structured fields, retrieval systems that fetch fresh pages to answer a question, and agents that browse on a user’s behalf. This guide explains how each one works, shows a tested AI extraction example, and covers the part every approach shares, the fetch layer, where residential proxies decide whether the model sees the real page or a block page.

Two Directions of LLM Data Gathering

The phrase “LLMs and data gathering” covers four different jobs. They differ in volume, freshness and how they hit websites, and that changes what the collection layer has to do.

The first job has its own guide: LLM training data and where proxies fit. This article focuses on the other three, where the model is the one gathering.

What Is AI Web Scraping?

Classic scraping, covered in our web scraping 101 guide, uses selectors: “the price is the text of p.price_color“. It is fast and cheap, but every site needs its own selectors, and they break when the layout changes.

AI web scraping hands the page text to a language model with an instruction such as “return the product title, price and availability as JSON”. The model finds the values wherever they are on the page. One prompt can work across hundreds of differently built sites, and it can read things selectors can’t, such as the delivery terms buried in a paragraph or the overall sentiment of fifty reviews.

Many production systems mix the two. They use selectors for the high-volume sites they know well, and the model for the long tail and for pages where the selectors have just broken. A common pattern also uses the model once to write the selectors, then runs those cheaply at scale.

How AI Web Scraping Works

 URL list
    |
    v
 +--------------------+    via proxy     +-------------+
 |  HTTP client or    | ---------------> |   website   |
 |  headless browser  | <--------------- |             |
 +--------------------+     HTML         +-------------+
    |
    |  block page / CAPTCHA?  -->  retry on another IP, don't send to model
    v
 clean: drop scripts, nav, footer --> plain text (trimmed to a budget)
    |
    v
 +--------------------+
 |   LLM + schema     |  "return exactly these keys, null if absent"
 +--------------------+
    |
    v
 validate: JSON parses? keys present? values found in the page?
    |
    v
 store with URL, time and model version

1. Fetch the page with an ordinary HTTP client, or a headless browser if JavaScript builds it.
2. Check that you got the real page and not a challenge or error page.
3. Clean the HTML down to readable text. Stripping scripts, menus and footers cuts tokens and cost, and removes noise that confuses the model.
4. Prompt the model with the text and a strict output format: named fields, JSON only, null for anything the page doesn’t say.
5. Validate the answer: it parses, every field is present, and extracted values really appear in the source text.
6. Store the result with its URL, fetch time and the model version, so you can re-run or audit it later.

A Tested AI Extraction Example

This Python example fetches a product page, reduces it to text, asks a model for three fields, and rejects any string value that doesn’t appear in the page. call_llm is whatever client you use: it takes a prompt and returns the model’s text. We tested the pipeline on 26 September 2026 with a stand-in model that returned one real value and one invented one. The check kept the real title and price and set the invented availability to None.

import json
import os

import requests
from bs4 import BeautifulSoup

proxy = os.environ.get("PROXY_URL")          # e.g. http://user:pass@host:port
PROXIES = {"http": proxy, "https": proxy} if proxy else None
FIELDS = ["title", "price", "availability"]

PROMPT = """Extract these fields from the product page text below.
Return JSON only, with exactly these keys: {keys}.
Use null for any field the text does not state. Do not guess.

PAGE TEXT:
{text}"""


def page_text(url):
    html = requests.get(url, proxies=PROXIES, timeout=20).content
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(["script", "style", "nav", "header", "footer"]):
        tag.decompose()
    return " ".join(soup.get_text(" ").split())[:8000]


def extract(url, call_llm):
    text = page_text(url)
    data = json.loads(call_llm(PROMPT.format(keys=", ".join(FIELDS), text=text)))
    missing = set(FIELDS) - set(data)
    if missing:
        raise ValueError(f"model left out {missing}")
    for key, value in data.items():      # drop values that are not in the page
        if isinstance(value, str) and value not in text:
            data[key] = None
    return data

Things to add before you run it at scale:

• Structured output. Most model APIs can enforce a JSON schema. Use it, and keep the validation anyway.
• Normalise before checking. The substring check is deliberately strict. Prices written as “51.77” when the page says “£51.77” will be dropped, so normalise both sides or check numbers separately.
• Token budget. The example keeps the first 8,000 characters. For long pages, cut to the relevant section first, for example the main product block.
• Cache the fetch. Store the cleaned text, so a prompt change doesn’t mean downloading every page again.
• Record the model version with each row. A model update can change outputs silently.

RAG and Live Retrieval

Retrieval-augmented generation (RAG) was described by Lewis et al. in 2020 as combining a model’s “parametric” memory, what it learned in training, with a “non-parametric” memory it can look things up in, in their case a vector index of Wikipedia. Today the term covers any system that retrieves documents and passes them to the model as context before it answers.

Many RAG systems retrieve from a fixed internal index. Live web retrieval goes further and fetches pages at question time: search for the query, fetch the top results, cut them into passages, rank the passages, and give the best ones to the model. That changes the collection problem completely:

• Latency matters. A user is waiting. A slow or failed fetch means a worse answer, so you need fast exits and retries on a different IP.
• Many domains, few pages each. A question may touch ten sites you have never fetched before. Per-site selectors are useless here, which is why RAG leans on generic text extraction and on AI extraction.
• Location changes the answer. “Price of X”, “is Y in stock”, “best Z near me”: the correct page depends on the country the user is in. Fetching from the user’s country with geo-targeted residential proxies keeps the answer local.
• Freshness is the point. RAG exists largely because training data goes stale. Caching fetched pages for too long defeats the purpose.

AI Agents That Browse

An AI agent completes a task over many steps: search, open a result, click through, compare, maybe fill a form. Under the hood it drives a headless browser, usually Chromium, and the model decides the next action from what the page shows.

For the collection layer, an agent looks like a person browsing, and it needs what a person has:

• One IP for the whole task. Sites tie carts, searches and logins to the visitor’s address. Use sticky sessions. On ProxyEmpire a sticky IP has no fixed time limit and stays until you rotate it or the device behind it goes offline.
• A consistent fingerprint. The browser’s reported device, language and time zone should match the proxy’s location. Residential and mobile proxies can be targeted by OS fingerprint as well as country, city and ISP.
• Isolation. Each agent session gets its own browser context and its own exit, so one blocked session doesn’t affect the rest.

We cover this in depth in proxies for AI web agents at scale, why web automation runs on Chromium and designing a browser cluster.

The Fetch Layer Is Still the Hard Part

Models made extraction easier, but they did nothing for fetching. Every approach above still sends ordinary HTTP requests to websites, and websites still limit, localise and challenge those requests. There is also a new risk. A selector scraper that receives a CAPTCHA page returns an empty field, which is easy to spot. A model that receives a CAPTCHA page returns a fluent, confident summary of the CAPTCHA page, and that is much harder to spot.

So the fetch layer needs its own checks before anything reaches the model:

• Status and size. Treat 403, 429 and unusually small responses as failures, not content.
• Block-page detection. Look for challenge markers, and for pages whose title or text matches known interstitials.
• Retry on a fresh IP from the same country, with a backoff.
• Per-domain rate limits, so a burst of agent or RAG traffic doesn’t hammer one site.
• The right proxy type for the target. Residential proxies are the usual default for web scraping where sites check for bots, and static residential proxies give a fixed address for longer sessions.

Choosing Proxies for AI Data Gathering

On ProxyEmpire, rotating residential and mobile proxies can be targeted by country, region, city, ZIP, ISP or carrier, ASN and OS fingerprint at no extra charge. Static residential and datacenter proxies are targeted by country. The network has more than 30 million ethically sourced IPs and 99.9% uptime, with live figures on its public status page. Traffic runs over HTTP(S) or SOCKS5. Residential bandwidth costs $3.50/GB pay-as-you-go and drops to $1.50/GB on larger plans, datacenter starts at $0.35/GB, unused bandwidth rolls over, and the trial is $1.97.

Rules and Risks for AI Data Gathering

robots.txt and user-triggered fetches

robots.txt (RFC 9309) applies to automated crawling, and training or bulk extraction should honour it. User-triggered fetches are treated differently by some providers. OpenAI, for example, says its ChatGPT-User agent visits pages when users ask a question, isn’t used for automatic crawling, and that robots.txt rules may not apply to those user-initiated actions. If you build a similar feature, be transparent about it: use a distinct user agent, and don’t turn “on demand” into a disguised bulk crawl.

llms.txt

Some sites now publish an /llms.txt file, a proposal by Jeremy Howard for giving agents a clean, Markdown summary of a site and links to its key content. It is a proposal, not a standard, and not every site has one. Where it exists, it is often the cheapest and most welcome way for an agent to read that site.

Prompt injection from fetched pages

OWASP lists prompt injection as the top risk for LLM applications and describes “indirect” injection: instructions hidden in external content, such as a web page, that change the model’s behaviour when it reads them. Any system that feeds fetched pages to a model is exposed. Keep page text clearly separated from your instructions, never let the model act on instructions found in page content, limit what tools an agent can use, and validate outputs, as the example above does.

Personal data and terms

The model doesn’t change the law. Personal data you gather is still personal data, site terms still apply, and summarising copyrighted text at scale raises the same questions as copying it. This isn’t legal advice. Check the rules for your use case and jurisdiction.

Frequently Asked Questions

References

1. Lewis et al. — “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, arXiv:2005.11401. arxiv.org/abs/2005.11401
2. OpenAI — Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User). platform.openai.com/docs/bots
3. IETF — RFC 9309, “Robots Exclusion Protocol”. rfc-editor.org/rfc/rfc9309
4. OWASP GenAI Security Project — LLM01: Prompt Injection. genai.owasp.org/llmrisk/llm01-prompt-injection
5. llms.txt — “The /llms.txt file” proposal. llmstxt.org
6. Beautiful Soup — documentation. crummy.com/software/BeautifulSoup/bs4/doc
7. Requests — Advanced usage: proxies. requests.readthedocs.io/en/latest/user/advanced
8. Playwright for Python — documentation. playwright.dev/python/docs/intro