Most LLM training data starts as web pages. Someone has to fetch those pages, pull the text out, throw most of it away, and keep a clean, deduplicated, well-documented remainder. This guide explains where LLM training data comes from, how the collection pipeline works stage by stage, when a team needs its own crawl instead of an open dataset, where residential proxies fit in that crawl, and the opt-out and copyright rules a training crawl has to respect.
The short version
LLM training data is collected by crawling the web (or reusing a public crawl such as Common Crawl), extracting the text, filtering by language and quality, deduplicating, and removing personal data. Only a small share of the raw crawl survives. Proxies matter at one stage, the fetch, when a team crawls specific sites, languages or regions itself: they spread requests across IP addresses and let the crawler see each country’s version of a site. They never replace honouring robots.txt and AI opt-outs.
What LLM Training Data Is
Three stages, three kinds of dataA large language model learns in stages, and each stage uses different data:
- Pretraining teaches the model language and general knowledge by predicting the next token over a huge amount of text. This is where web data dominates. Open datasets built for pretraining are measured in trillions of tokens.
- Fine-tuning adapts a pretrained model to a task or a domain: support conversations, legal clauses, product descriptions in one market. The datasets are far smaller, but quality and relevance matter much more.
- Preference and evaluation data teach the model which answers people prefer and measure whether it improved. This is mostly written or rated by people, not scraped.
For scale, Hugging Face’s FineWeb dataset, one of the best-documented open pretraining sets, is 15 trillion tokens derived from 96 Common Crawl snapshots, according to its paper. Its authors note that the pretraining datasets behind leading open models such as Llama 3 and Mixtral are not public, and very little is known about how they were made. That is why FineWeb’s documented pipeline is a useful reference for everyone else.
Key takeaways
- Web text is the bulk of pretraining data. Fine-tuning needs smaller, targeted, higher-quality sets.
- Most of a raw crawl is thrown away. Filtering and deduplication are where the quality comes from.
- Start from public crawls and datasets. Crawl yourself only to fill gaps in freshness, language, region or domain depth.
- Proxies belong to the fetch stage of your own crawl. They help with scale and location, not with permission.
- Respect robots.txt and AI-specific opt-outs, and keep a record of where every document came from.
Where LLM Training Data Comes From
Reuse first, collect second| Source | What it is | Strengths | Limits |
|---|---|---|---|
| Public web crawls | Common Crawl’s open archive, collected regularly since 2008 and measured in petabytes | Free, huge, many snapshots | Raw and noisy. Coverage of a given site or language can be thin. |
| Curated open datasets | Filtered sets built from public crawls, such as FineWeb | Cleaned, deduplicated and documented | Fixed snapshot, mostly English, general-purpose |
| Your own crawl | Pages you fetch yourself from chosen sites | Fresh, targeted, any language or region | You carry the cost, the engineering and the compliance work |
| Licensed data | Archives bought or licensed from publishers and data owners | Clear rights, often high quality | Expensive, narrow |
| First-party data | Your own documents, tickets and product data | Highly relevant for fine-tuning | Needs consent and privacy review |
| Synthetic data | Text generated by other models | Cheap to scale, easy to shape | Can copy the source model’s mistakes, and licences may restrict it |
Most teams combine several. A typical pattern: pretrain on (or start from a model pretrained on) public web data, then fine-tune on a mix of licensed, first-party and freshly crawled domain data.
How the LLM Training Data Pipeline Works
From URL to clean tokensThe stages below follow the pipeline FineWeb documents on its dataset card. Other pipelines differ in the details, but the shape is the same.
seed URLs / sitemaps / public crawl index
|
v
+--------------------------+
| URL filter + frontier | drop blocklisted or unsafe domains
+--------------------------+
|
v
+--------------------------+ +------------------+
| robots.txt + opt-outs | --> | skip & record |
+--------------------------+ +------------------+
| allowed
v
+--------------------------+
| FETCH (proxy pool) | the only stage that touches the network
+--------------------------+
|
v
raw HTML + provenance (URL, time, status, hash)
|
v
text extraction -> language ID -> quality filters
|
v
deduplication -> PII removal -> tokenise -> dataset- URL filtering. Remove domains you never want, such as malicious or adult sites. FineWeb uses a blocklist plus word matching on URLs.
- Opt-out checks. Check robots.txt and AI-specific rules before fetching (more on these below).
- Fetching. Download the page and store the raw response with its URL, time, status code and a content hash. Common Crawl stores raw responses in WARC files. You can do the same.
- Text extraction. Strip menus, footers and markup to get the main text. FineWeb uses the Trafilatura library on raw HTML.
- Language identification. A classifier labels each document’s language. FineWeb’s English set keeps documents with a fastText English score of at least 0.65.
- Quality filtering. Heuristics remove boilerplate, lists of links, repeated lines and machine-generated junk. FineWeb applies Gopher and C4-style rules plus its own filters.
- Deduplication. The same text appears on thousands of URLs. FineWeb uses MinHash on 5-grams to find and drop near-duplicates within each crawl.
- PII handling. FineWeb replaces email addresses and public IP addresses. Stricter pipelines go further.
- Tokenisation and mixing. The text is tokenised and blended with other sources in chosen proportions.
Only one stage in the pipeline touches the network. Everything after the fetch decides what the model actually learns.
When You Need Your Own Crawl
Filling the gaps a public crawl leavesPublic crawls are broad but shallow in places. Teams run their own collection when they need:
If none of these apply, don’t crawl. Reusing an existing dataset is cheaper, faster and easier to document. For the collection basics, start with our web scraping guide for beginners.
Where Proxies Fit in Collecting Training Data
Scale and location at the fetch stageWhen your own crawler runs at volume, it runs into the same limits as any scraper, only harder:
- Per-IP rate limits. A crawler on a few cloud IPs quickly hits throttling, CAPTCHAs and 429 responses, even at polite per-site speeds, because sites count requests per address. Spreading requests across many IPs keeps each address’s footprint small.
- Location. Many sites serve different content by country: language, prices, local news, whole sections. A crawler that should learn Brazilian Portuguese e-commerce text needs to fetch from Brazil. Residential and mobile proxies with country and city targeting make that possible.
- Datacenter reputation. Cloud address ranges are easy to identify. Some sites serve them degraded pages or challenges, which corrupts a dataset quietly: you store a block page instead of the article.
- Reliability. A pool of exits means one blocked or slow address doesn’t stall the crawl.
What proxies are not for: getting round a site that has said no. If robots.txt disallows your crawler, or the site reserves its content from AI training, a different IP address doesn’t change that. Treat proxies as infrastructure for a crawl that is already allowed.
Proxies change where a request comes from. They don’t change whether you are allowed to make it.
Choosing Proxy Types for an LLM Data Crawl
Match the exit to the target| Type | Use it for | Trade-off | Targeting on ProxyEmpire |
|---|---|---|---|
| Datacenter | Tolerant sites, documentation, open repositories, bulk volume | Cheapest per GB. More likely to be challenged on strict sites. | Country |
| Rotating residential | Most commercial and regional sites, localised content | Higher cost per GB, much better acceptance | Country, region, city, ZIP, ISP, ASN, OS fingerprint |
| Rotating mobile | Mobile versions of sites, the strictest targets | Highest cost, highest trust | Country, region, city, ZIP, carrier, ASN, OS fingerprint |
| Static residential | Long sessions that must keep one IP | Fixed address, sold in selected countries | Country |
A sensible default is tiered: send tolerant domains through datacenter proxies, and route only the domains that challenge them through residential or mobile. Track the block rate per domain and move domains between tiers from the data, not from guesses.
Budgeting bandwidth. Don’t estimate from averages you read somewhere. Fetch a sample of 1,000 URLs from your real target list, record the transferred bytes, and multiply by the number of pages you plan to fetch. Include retries, and account for browsers downloading scripts and images if you render pages. Then compare tiers: ProxyEmpire’s network has more than 30 million ethically sourced IPs and 99.9% uptime (live figures on its public status page), residential is $3.50/GB pay-as-you-go down to $1.50/GB on larger plans, datacenter starts at $0.35/GB, and unused bandwidth rolls over to the next period. If the total runs to many terabytes, also price unlimited residential proxies, which bill by connection speed instead of per GB.
The same pool that feeds a training crawl also serves the other collection jobs a data team runs, such as a web scraping service, price monitoring or search-result tracking, so one account can cover them. Sessions can rotate on every request or stay sticky. A sticky IP has no fixed time limit and stays until you rotate it or the device behind it goes offline.
A Fetch Worker That Respects Opt-Outs
Check first, fetch second, record everythingThis minimal Python worker checks robots.txt for your crawler’s own user agent before each fetch, sends the request through a proxy if one is set, and writes one JSON line per URL with the provenance you will need later: time, status, content type and a SHA-256 hash of the body. We ran it against two practice sites on 26 September 2026.
import hashlib
import json
import os
from datetime import datetime, timezone
from urllib.parse import urlsplit
from urllib.robotparser import RobotFileParser
import requests
UA = "ExampleResearchBot/1.0 (+https://example.com/bot)"
proxy = os.environ.get("PROXY_URL") # e.g. http://user:pass@host:port
PROXIES = {"http": proxy, "https": proxy} if proxy else None
robots = {}
def allowed(url):
root = "{0.scheme}://{0.netloc}".format(urlsplit(url))
if root not in robots:
rp = RobotFileParser(root + "/robots.txt")
rp.read()
robots[root] = rp
return robots[root].can_fetch(UA, url)
def fetch(session, url):
if not allowed(url):
return {"url": url, "skipped": "robots.txt"}
r = session.get(url, headers={"User-Agent": UA}, proxies=PROXIES, timeout=30)
return {
"url": url,
"fetched_at": datetime.now(timezone.utc).isoformat(),
"status": r.status_code,
"content_type": r.headers.get("Content-Type", ""),
"sha256": hashlib.sha256(r.content).hexdigest(),
"html": r.text if r.ok else None,
}
with requests.Session() as s, open("raw.jsonl", "a", encoding="utf-8") as out:
for url in ["https://books.toscrape.com/", "https://quotes.toscrape.com/"]:
out.write(json.dumps(fetch(s, url)) + "\n")Notes for a production version:
- Name your crawler honestly and publish a page about it, so site owners can write robots.txt rules for it. Checking rules for an invented user agent that nobody knows about protects no one.
- Cache robots.txt, but not forever. RFC 9309 says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable.
- Keep the skipped records. A log of what you chose not to fetch, and why, is part of your provenance.
- Keep the hash. It lets you detect unchanged pages on a recrawl and cheaply drop exact duplicates before extraction.
- Rate-limit per domain, not per proxy. More IPs should mean less load per address, not more load per site.
Opt-Outs, Copyright and the Law
What a training crawl must respectrobots.txt
The Robots Exclusion Protocol, standardised as RFC 9309, lets site owners give rules to crawlers by user agent. The RFC is clear that the rules “are not a form of access authorization”. Honouring them is the baseline expectation for any crawler, and training crawls are watched more closely than most.
AI-specific user agents and tokens
Several AI companies publish separate controls for training:
- GPTBot. OpenAI’s documentation says GPTBot crawls content that may be used to train its foundation models, and disallowing GPTBot indicates a site’s content should not be used for training. OpenAI’s search crawler (OAI-SearchBot) is controlled separately.
- Google-Extended. Google describes it as a standalone product token that controls whether content Google crawls may be used to train future Gemini models and for grounding. It has no separate user agent string, and it doesn’t affect Google Search.
- CCBot. Common Crawl’s crawler identifies itself as CCBot and honours
User-agent: CCBotrules. Common Crawl also warns that other crawlers falsely identify themselves as CCBot.
If you build your own dataset, these signals tell you what site owners want, even though they are addressed to other crawlers. The cautious approach is to treat a disallow for GPTBot, Google-Extended or CCBot as a reservation against AI training in general and skip those sites.
The EU: text-and-data-mining opt-outs and the AI Act
Under Article 4 of the EU’s Copyright in the Digital Single Market Directive (2019/790), text and data mining of lawfully accessible content is allowed unless the rightholder has expressly reserved it “in an appropriate manner, such as machine-readable means” for content published online. The EU AI Act (Regulation 2024/1689), Article 53(1)(c), requires providers of general-purpose AI models to have a policy to comply with EU copyright law, in particular to identify and comply with those reservations, and point (d) requires a public summary of the content used for training. These general-purpose model obligations apply from 2 August 2025.
Personal data
Web pages contain names, emails, photos and profiles. Under laws such as the GDPR, public personal data is still personal data. Filter it out where you can, document why you keep what you keep, and avoid sources that are mostly about individuals.
This section is a summary, not legal advice. Rules differ by jurisdiction and are changing quickly, so take advice before you train on data you collected yourself.
A Practical Strategy for LLM Training Data
What experienced teams do- Start from what exists. Check open datasets and public crawls for your languages and domains before writing a crawler.
- Define the gap. Write down exactly which sites, languages, regions or time ranges are missing. That list is your crawl scope.
- Check permissions per source. robots.txt, AI tokens, terms and any licence, recorded per domain before the first fetch.
- Fetch with provenance. Raw responses, timestamps and hashes, kept separately from the cleaned text.
- Tier your proxies. Datacenter for tolerant sources, residential or mobile only where needed, country-targeted where content is local.
- Measure block pages. Detect challenge and error pages before extraction so they never reach the dataset.
- Filter and deduplicate early. Storage and tokenisation cost money. Duplicates also skew what the model learns.
- Document the dataset. Sources, dates, filters, exclusions and known gaps. You may need that summary for regulators, and you will certainly need it for your own debugging.
For the other half of the story, how LLMs gather web data at run time through retrieval and agents, see LLMs and data gathering. For broader AI collection work, see proxies for AI data collection and proxies for AI web agents.
Frequently Asked Questions
The short answersWhere does LLM training data come from?
Mostly from web text, usually through public crawls such as Common Crawl and datasets built from them, plus books, code, licensed archives, first-party documents and, increasingly, synthetic data. Fine-tuning adds smaller, targeted datasets, and preference data comes mostly from human ratings.
Do you need proxies to collect LLM training data?
Not if you use existing datasets or public crawls. You need them when you run your own crawl at volume, because sites limit requests per IP address and serve different content by country. Proxies spread the load and let you fetch local versions of sites. They don’t change what you are permitted to collect.
Which proxies are best for AI training data collection?
Use datacenter proxies for tolerant sources, because they are the cheapest per GB. Use rotating residential proxies for commercial and regional sites that challenge datacenter traffic, and mobile proxies for the strictest or mobile-only sources. Route by domain and move sources between tiers based on measured block rates.
How much of a web crawl ends up as training data?
A minority. Every pipeline discards duplicates, boilerplate, other languages, low-quality text and unsafe domains, and the exact share depends on the filters. Documented pipelines such as FineWeb show each filtering stage, which is the best way to understand the losses for your own data.
How do websites opt out of AI training?
Mainly through robots.txt rules for specific crawlers and tokens, such as GPTBot for OpenAI, Google-Extended for Gemini training and grounding, and CCBot for Common Crawl. In the EU, rightholders can also reserve text-and-data-mining rights in machine-readable form, and the AI Act requires general-purpose model providers to respect those reservations.
Is it legal to scrape websites for LLM training?
It depends on the jurisdiction, the source’s terms, whether rights were reserved, and whether personal data is involved. In the EU, text and data mining is allowed unless rights are expressly reserved, and model providers need a copyright policy. Other regions differ, and several questions are still before the courts. Take legal advice.
What is the difference between training data and RAG data?
Training data changes the model’s weights and is collected once or periodically in bulk. Retrieval-augmented generation (RAG) fetches documents at question time and passes them to the model as context, without retraining. RAG needs fresh, targeted fetches rather than bulk crawls. Our guide to LLMs and data gathering covers it.
References
Primary documentation- Penedo et al. — “The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale”, arXiv:2406.17557. arxiv.org/abs/2406.17557
- Hugging Face — FineWeb dataset card: data processing pipeline. huggingface.co/datasets/HuggingFaceFW/fineweb
- Common Crawl — Overview of the corpus. commoncrawl.org/overview
- Common Crawl — CCBot. commoncrawl.org/ccbot
- IETF — RFC 9309, “Robots Exclusion Protocol”. rfc-editor.org/rfc/rfc9309
- OpenAI — Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User). platform.openai.com/docs/bots
- Google — Google’s common crawlers, including Google-Extended. developers.google.com/search/docs/crawling-indexing/google-common-crawlers
- EU — Directive (EU) 2019/790 on copyright in the Digital Single Market, Article 4. eur-lex.europa.eu — Directive 2019/790
- EU — Regulation (EU) 2024/1689 (AI Act), Articles 53 and 113. eur-lex.europa.eu — Regulation 2024/1689
- Python — urllib.robotparser. docs.python.org/3/library/urllib.robotparser.html
Fetch the regions your dataset is missing
More than 30 million ethically sourced IPs and 99.9% uptime. Residential, mobile and datacenter proxies in one account, with country-to-ZIP, ISP and ASN targeting at no extra charge on residential and mobile, sticky or rotating sessions, and bandwidth that rolls over. 24/7 support from real people. Try ProxyEmpire for $1.97.
