Web Crawler vs Web Scraper: Key Differences, Tools and How They Work Together

Data collection guide · Last reviewed 1 October 2026 · 15 min read

Web crawler vs web scraper comes down to one question: are you looking for pages, or for data on pages? A web crawler discovers URLs by following links and reading sitemaps, and builds a list of what exists. A web scraper opens pages it already knows about and pulls out specific fields, such as a price, a title or a review score. Most real projects use both, one after the other. This guide explains how each one works, how robots.txt and sitemaps shape a crawl, the common tools, how to stay within a site’s rate limits, and where rotating residential proxies and sticky sessions fit. If you are new to the subject, start with our web scraping 101 guide.

What a web crawler does

A web crawler, also called a spider or a bot, is a program that discovers pages by following links. It starts from one or more seed URLs, downloads each page, finds the links in it, and adds the ones it hasn’t seen to a list of pages still to visit. Then it takes the next URL from that list and repeats. Google describes its own crawler the same way: Googlebot discovers new URLs mainly from links in pages it has already crawled.

The list of URLs waiting to be fetched is called the frontier, or the crawl queue. How the crawler orders that queue decides the shape of the crawl. A first-in, first-out queue visits pages level by level from the seed, which is breadth-first. A last-in, first-out queue follows one branch deep before coming back, which is depth-first. Scrapy’s documentation shows both: its default scheduler queues are last-in, first-out, and its DEPTH_PRIORITY setting switches toward breadth-first.

seed URLs
   |
   v
[ frontier ]  queue of URLs still to visit
   |
   v
[ fetcher ]   checks robots.txt, waits the
   |          per-host delay, downloads the page
   v
[ parser ]    pulls out every <a href> link
   |
   v
[ filter ]    normalises URLs, drops duplicates
   |          and anything out of scope
   |
   +------>  new URLs go back into the frontier

Every step in that loop has work in it. The filter has to normalise URLs so that /shoes, /shoes/ and /shoes?utm_source=mail are not treated as three different pages, and it has to keep the crawl in scope, for example one domain or one section of a site. The fetcher has to respect the site’s robots.txt and space out requests to the same host. A crawler that runs for weeks also needs to decide when to revisit pages it has already seen, because pages change.

What a crawler keeps from each page depends on its purpose. A search engine stores the page content for its index. An SEO audit tool records the status code, title, canonical tag and internal links of every URL. A crawler that only feeds a scraper may keep nothing but the URLs that match a pattern, such as product pages.

robots.txt and sitemaps: the rules of the crawl

Two files tell a crawler how a site would like to be crawled. The first is robots.txt, standardised in 2022 as the Robots Exclusion Protocol in RFC 9309. The second is the XML sitemap, described by the sitemaps.org protocol.

How robots.txt works

RFC 9309 says the file lives at the top level of a host, as /robots.txt, in UTF-8 plain text. It is made of groups. Each group names one or more crawlers in user-agent lines, followed by allow and disallow rules for URL paths. A crawler uses the group that matches its product token, matched without regard to case, and falls back to the * group if none matches. When an allow and a disallow rule both match a URL, the most specific one wins, meaning the rule with the most characters.

User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help

User-agent: ExampleBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

In this file, every crawler may fetch the site except the cart and the search pages, but /search/help is allowed, because Allow: /search/help is longer than Disallow: /search. A crawler that identifies as ExampleBot may fetch nothing.

The RFC also covers the edge cases a crawler has to handle:

• File not found. If robots.txt returns a 4xx status, such as 404, the crawler may access any resource on the server.
• Server error. If it returns a 5xx status or can’t be reached, the crawler must assume it may fetch nothing.
• Caching. Crawlers should not use a cached copy for more than 24 hours, unless the file is unreachable.
• Redirects and size. Crawlers should follow at least five consecutive redirects and must parse at least the first 500 kibibytes.

Two limits are worth knowing. RFC 9309 states that these rules are not a form of access authorization; they are instructions a well-behaved crawler follows, not a lock. And robots.txt controls crawling, not indexing: Google’s documentation notes that a URL disallowed for crawling may still appear in search results, without a snippet. Some crawlers also read a Crawl-delay line, but it isn’t part of RFC 9309, and Google says it doesn’t support it.

How sitemaps help discovery

A sitemap is an XML file that lists the URLs a site wants crawlers to find. The sitemaps.org protocol requires a urlset root with one url entry per page, each holding a loc with the full URL. Optional lastmod, changefreq and priority tags add hints. One sitemap can hold up to 50,000 URLs and 50 MB uncompressed; larger sites split their URLs across several sitemaps and list them in a sitemap index file. The Sitemap: line in robots.txt, shown in Figure 2, tells crawlers where to find it.

For a crawler, a sitemap is a shortcut. Instead of following links through category pages and pagination to reach every product, it can read the product URLs straight from the sitemap, and use lastmod to decide which pages changed since the last visit. The protocol itself says a sitemap doesn’t guarantee that search engines will include the URLs; it is a hint.

What a web scraper does

A web scraper is a program that extracts specific data from web pages. It doesn’t need to discover anything: you give it URLs, and it returns structured records. A scraper for an online shop might return one row per product with the name, price, currency, stock status and URL.

Every scraper runs the same four steps:

1. RequestSend an HTTP request for the page, the same request a browser sends, and receive the status code and the HTML or JSON.
2. ParseTurn the HTML into a tree of elements that code can search.
3. ExtractSelect the elements that hold the data with CSS selectors or XPath, and clean the values, for example turning “$1,299.00” into a number.
4. StoreWrite the records to a CSV file, JSON, a database or a spreadsheet, often with the time of collection.

The hard parts of scraping are different from the hard parts of crawling. A scraper has to cope with layout changes that break its selectors, prices shown in different formats, and pages where the data isn’t in the HTML at all because JavaScript builds it in the browser. For those pages a scraper either drives a real browser with a tool such as Playwright or Puppeteer, or finds the JSON the page loads in the background and requests that directly. Our headless browsers guide covers the browser route.

Scrapers also run on a schedule more often than crawlers do. A crawl to map a site may run once a week, while a scraper that tracks prices may revisit the same thousand product pages every hour. That pattern, many requests to the same pages, is what runs into rate limits, and it is where most of the proxy questions come from.

Web crawler vs web scraper: side-by-side comparison

Both programs send HTTP requests and read what comes back, so the web crawler vs web scraper difference is not in the plumbing. It is in what they are looking for and what they hand you at the end.

The line is not always sharp. A search engine crawler also extracts text from every page it visits, which is a kind of scraping. And a scraper that follows “next page” links through search results is doing a small crawl. The useful distinction is the goal: discovery or extraction.

How crawling and scraping work together

In practice, crawling and scraping are two stages of one pipeline. The crawl stage builds and refreshes the list of URLs worth reading. The scrape stage visits those URLs and extracts the data. Keeping them separate lets each one run at its own pace: the crawl can run weekly to catch new pages, while the scrape runs hourly on the pages you already know.

Example: a price comparison crawler

Say you compare prices for 20 online shops. Searchers call this a price comparison web crawler, but it is really a crawler and a scraper working together:

CRAWL (discovery)            SCRAPE (extraction)
-----------------            -------------------
sitemap.xml +                each product URL
category pages                     |
      |                            v
      v                      fetch the page
follow pagination                  |
links                              v
      |                      select name, price,
      v                      currency, stock
list of product  --------->        |
URLs                               v
                             CSV / JSON / database

The crawler reads each shop’s sitemap and category pages, follows pagination, and keeps only URLs that look like product pages. The scraper then fetches each product page, extracts the name, price, currency and stock status, and stores one record per product per visit. Matching products across shops, by barcode, model number or name, happens after that, on the stored data. Our price monitoring page covers the proxy side of this use case.

Frameworks such as Scrapy let you express both stages in one spider. Its CrawlSpider takes a list of rules, each built on a LinkExtractor: rules without a callback are followed to discover more pages, and rules with a callback send the matching pages to your extraction code.

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class ShopSpider(CrawlSpider):
    name = "shop"
    allowed_domains = ["shop.example.com"]
    start_urls = ["https://shop.example.com/"]
    custom_settings = {
        "ROBOTSTXT_OBEY": True,
        "AUTOTHROTTLE_ENABLED": True,
        "DOWNLOAD_DELAY": 1,
    }

    rules = (
        # crawl: follow category and pagination links
        Rule(LinkExtractor(allow=r"/category/")),
        # scrape: send product pages to parse_product
        Rule(LinkExtractor(allow=r"/product/"), callback="parse_product"),
    )

    def parse_product(self, response):
        yield {
            "url": response.url,
            "name": response.css("h1::text").get(),
            "price": response.css(".price::text").get(),
        }

The first rule has no callback, so Scrapy follows the links it matches. The second has a callback, so matching product pages go to parse_product, which yields one record per page. The selectors are placeholders: every shop needs its own. For shops with a good sitemap, Scrapy’s SitemapSpider does the discovery step from the sitemap instead, and can find the sitemap URL from robots.txt.

Common crawling and scraping tools

The tools below are listed by what they do, not ranked. Each description follows the project’s own documentation.

A common split is Requests plus Beautiful Soup for small scrapers of static pages, Scrapy when a project needs both crawling and scraping with scheduling, retries and throttling built in, Playwright or Puppeteer when pages need a real browser, and Nutch when the job is crawling and indexing a very large set of sites. They combine well: Scrapy can hand JavaScript-heavy pages to a browser, and Beautiful Soup can parse HTML that any of the others fetched. For a worked Python example, see our guide to web scraping for beginners.

Politeness and rate limits

A crawler or scraper that sends requests as fast as it can will slow a small site down, and most sites answer that with errors, CAPTCHAs or blocks. Politeness is the set of habits that keeps your job running and keeps the site healthy. It mostly comes down to how many requests you send to one host, and how fast.

• Limit concurrency per host. Cap the number of simultaneous requests to any one domain. Scrapy’s CONCURRENT_REQUESTS_PER_DOMAIN does this; its documentation shows a global default of 8, and projects created with startproject set it to 1.
• Add a delay. Wait between requests to the same host. Scrapy’s DOWNLOAD_DELAY is the minimum wait between two requests to the same domain, and its AutoThrottle extension adjusts the delay from measured response times, never letting a non-200 response speed things up.
• Respect 429 and Retry-After. RFC 6585 defines status 429, Too Many Requests, for rate limiting, and says the response may include a Retry-After header saying how long to wait. Back off when you see it, and retry later rather than straight away.
• Treat 5xx errors as a signal. A rising rate of 500 and 503 responses often means the server is struggling. Slow down.
• Don’t refetch what hasn’t changed. Conditional requests with If-Modified-Since or If-None-Match, defined in RFC 9110, let the server answer 304 Not Modified instead of sending the whole page again. Sitemap lastmod dates help decide what to revisit.
• Identify your crawler. Use a clear user-agent string, so a site’s operators know who is crawling and which robots.txt group applies to you.

For scale, Google’s own documentation says Googlebot shouldn’t access a site more than once every few seconds on average. Your job may need a different rate, but it is a useful reminder that even the largest crawler spreads its load. Politeness also helps you: a job that stays under a site’s limits gets fewer errors, fewer retries and cleaner data.

Where proxies fit

Sites limit how many requests one IP address can make in a period, and many show different prices, results and content by country or city. Both facts matter more to scrapers than to crawlers, because scrapers revisit the same pages often and usually need the version a local customer sees. Proxies help with three things.

Rotating or sticky?

Use rotating IPs for independent requests: product pages, search results, category pages, anything where each request stands on its own. Use a sticky session when requests depend on each other: signing in, paging through search results that keep state in a session cookie, adding items to a cart to see the final price, or any flow where the site ties a session to an IP. A crawl-then-scrape pipeline often uses both: sticky sessions for the stateful parts, rotation for the bulk page fetches.

Residential, mobile or datacenter?

Datacenter IPs are the cheapest option and work well on sites with light protection. Residential IPs come from real home connections and are the usual choice for sites that treat server traffic with suspicion, and for geo-targeted collection. Mobile IPs come from 4G and 5G carriers and suit mobile-first sites and apps. Many teams start a new target on datacenter, and move the pages that return errors or blocks to residential.

Setting a proxy is a single option in most tools. Scrapy reads it from the proxy key in a request’s meta, including a username and password, and Playwright takes a proxy setting with a server, username and password at launch or per browser context:

# Scrapy: per request, via HttpProxyMiddleware
yield scrapy.Request(
    url,
    meta={"proxy": "http://USERNAME:PASSWORD@v2.proxyempire.io:5000"},
)

# Playwright (Python): for the whole browser
browser = p.chromium.launch(proxy={
    "server": "http://v2.proxyempire.io:5000",
    "username": "USERNAME",
    "password": "PASSWORD",
})

Replace USERNAME and PASSWORD with the credentials from your ProxyEmpire dashboard. Proxies spread your requests across many IPs; the per-site pacing from the previous section still decides how fast each site is hit, and keeping it reasonable is what keeps success rates high.

Legal and terms considerations

Automated collection of public web data at scale is a normal part of business. Search engines, price comparison sites, market research firms, SEO platforms and brand protection teams all run crawlers and scrapers every day. What applies to a given project depends on the country, the data and how it is accessed. The questions that usually come up are:

• Site terms. Many sites set rules on automated access in their terms of service. How those terms apply varies by jurisdiction and by whether you have agreed to them, for example by creating an account.
• Personal data. Records that identify people, such as names or profiles, fall under data protection laws like the EU’s GDPR. Collect only what the project needs.
• Copyright and database rights. Written content, photos and curated databases can be protected, so what you do with collected content, analysing it or republishing it, matters.
• Access controls. Public pages are a different case from pages behind a login or a paywall.

robots.txt sits alongside these questions rather than answering them: RFC 9309 describes it as instructions for crawlers, not access authorization. Teams running large programmes usually have their own counsel review the data sources and the use.

Which one do you need?

The web crawler vs web scraper choice follows from the output you want: a list of pages, or data from pages.

• You need a list of every page on a site, for an SEO audit, a migration or broken-link checks: a crawler. Our SEO proxies page covers rank tracking and audits.
• You already know the URLs and need data from them, such as 500 product pages to check daily: a scraper.
• You need data from pages you haven’t found yet, such as every product in a shop’s catalogue: both, crawl then scrape.
• You need to know when new pages appear, such as new listings or new articles: a crawler that revisits category pages and sitemaps, feeding new URLs to a scraper.
• The data is on pages built by JavaScript: a scraper that drives a browser, or one that requests the page’s JSON directly.

Where ProxyEmpire fits

ProxyEmpire provides the proxies both stages run on. Rotating residential proxies come from 30M+ IPs in 179 countries and territories, and rotating mobile proxies from 4M+ 4G and 5G IPs in 174 countries, both with targeting by country, region, city, ZIP, ISP, ASN and OS fingerprint at no extra charge, and a choice of a new IP per request or sticky sessions. Rotating datacenter proxies start at $0.35/GB with country-level targeting, for the lighter targets. Every proxy works over HTTP, HTTPS and SOCKS5 with a username and password, through the v2.proxyempire.io gateway on port 5000 for residential and mobile, so it drops into Scrapy, Playwright, Puppeteer or any tool that accepts a proxy. See the web scraping proxies page for scraping-specific options.

Residential starts at $3.50/GB pay-as-you-go and falls to $1.50/GB, unused bandwidth rolls over, and the network runs at 99.9% uptime on the public status page, with 24/7 support from real people. Crawls that run around the clock can use unlimited residential proxies instead, billed by connection speed rather than per GB. The $1.97 trial includes 100 MB of residential and 50 MB of mobile traffic, enough to test a crawler and a scraper against your own targets.

Web crawler vs web scraper FAQ

References

1. IETF — RFC 9309, “Robots Exclusion Protocol”. rfc-editor.org/rfc/rfc9309
2. sitemaps.org — Sitemaps XML format. sitemaps.org/protocol.html
3. Google Search Central — Googlebot. developers.google.com/search/docs/crawling-indexing/googlebot
4. Google Search Central — How Google interprets the robots.txt specification. developers.google.com/search/docs/crawling-indexing/robots/robots_txt
5. Scrapy — Scrapy at a glance. docs.scrapy.org/en/latest/intro/overview.html
6. Scrapy — Spiders: CrawlSpider and SitemapSpider. docs.scrapy.org/en/latest/topics/spiders.html
7. Scrapy — Settings. docs.scrapy.org/en/latest/topics/settings.html
8. Scrapy — AutoThrottle extension. docs.scrapy.org/en/latest/topics/autothrottle.html
9. Scrapy — Downloader middleware: HttpProxyMiddleware, RobotsTxtMiddleware. docs.scrapy.org/en/latest/topics/downloader-middleware.html
10. Apache Nutch — project home. nutch.apache.org
11. Beautiful Soup — documentation. crummy.com/software/BeautifulSoup/bs4/doc
12. Playwright — Installation and overview. playwright.dev/docs/intro
13. Playwright — Network: HTTP proxy. playwright.dev/docs/network
14. Puppeteer — documentation. pptr.dev
15. IETF — RFC 6585, “Additional HTTP Status Codes”, 429 Too Many Requests. rfc-editor.org/rfc/rfc6585
16. IETF — RFC 9110, “HTTP Semantics”, conditional requests and Retry-After. rfc-editor.org/rfc/rfc9110