Designing Distributed Crawlers for Billion-Page Datasets: Architecture, Scheduling, and Fault Tolerance

Engineering guide · Last reviewed 8 October 2026 · 10 min read

A distributed crawler rarely fails because a single HTTP request is hard to execute. It fails because billions of requests create a distributed-systems problem. This guide walks through the architecture of a crawler built for billion-page datasets: the URL frontier, partitioning, backpressure, deduplication, fault tolerance, storage, the network layer, observability and freshness, with short Python and SQL examples for each idea.

From Scraper to Distributed Data Platform

Once a crawler grows beyond a small collection of workers, almost every component becomes stateful. URLs need to be scheduled, deduplicated, prioritized, retried, partitioned, stored and eventually revisited. Workers fail. Networks become inconsistent. Domains respond at different speeds. Data changes continuously.

At that point, “scraping” is no longer an appropriate architectural description. You are building a distributed data acquisition platform. If the distinction between the two is new, our comparison of web crawlers and web scrapers covers the basics.

URL Sources
     |
     v
URL Frontier
     |
     v
Scheduling Layer
     |
     v
Worker Cluster / Priority Queue
     |
     v
Fetching Layer
     |
     v
Raw Data Store + Metadata Store
     |
     v
Processing Pipeline
     |
     v
Search / AI / BI

The interesting engineering problem is not how to download a page. It is how to coordinate this entire system while maintaining throughput, correctness, freshness and predictable infrastructure costs.

Start With the URL Frontier, Not the Worker

A common architectural mistake is to start by optimizing the fetching workers. At billion-page scale, the URL frontier is often more important.

The frontier is the distributed system responsible for one decision: what should we fetch next? That decision can depend on priority, domain, freshness, crawl history, discovered links, business importance, previous failures and scheduling constraints.

The prototype version of a frontier is an in-memory queue:

from collections import deque

queue = deque()
queue.append("https://example.com/page")

url = queue.popleft()

This works for a prototype. It fails as soon as multiple workers, processes, machines and failure domains become involved. If the worker disappears, the queue disappears with it.

A production crawler therefore separates URL discovery from URL execution. A durable messaging system such as Kafka can provide the foundation for this architecture, but the important principle is broader:

Partitioning the Web

A billion URLs cannot realistically be treated as one queue. The system needs a partitioning strategy, and one common approach is partitioning by hostname or domain.

Partitioning provides two important properties:

• Horizontal scalability. Work can be spread across machines.
• Domain-level scheduling policies. The system can control how each site is crawled.

A crawler should not interpret “1,000 workers” as permission to send 1,000 simultaneous requests to the same website. Large-scale crawling requires politeness and rate control as first-class architectural concepts. A minimal per-host state might look like this:

from dataclasses import dataclass
from time import monotonic

@dataclass
class HostState:
    next_allowed: float = 0
    failures: int = 0

def can_schedule(host: HostState) -> bool:
    return monotonic() >= host.next_allowed

Backpressure Is What Keeps the System Alive

Suppose your crawler discovers 50,000 URLs per second but your processing infrastructure can only consume 30,000 URLs per second. The queue grows continuously, increasing memory, storage, latency and recovery time.

The answer is backpressure: a healthy distributed system allows downstream capacity to influence upstream production. Consumer lag can therefore become an operational control signal.

def adjust_workers(queue_depth, worker_count, max_workers):
    if queue_depth > 1_000_000:
        return min(worker_count + 10, max_workers)

    if queue_depth < 10_000:
        return max(worker_count - 5, 1)

    return worker_count

Scale according to system pressure, not according to a fixed worker count. The same principle applies to browser-based workloads, as covered in our guide to designing a browser cluster for 10,000 concurrent Chromium sessions.

Deduplication Becomes a Distributed Data Problem

At billion-page scale, URL duplication becomes extremely expensive. The same URL may appear:

• on multiple pages;
• through different discovery paths;
• with tracking parameters;
• in multiple sitemap files;
• across repeated crawl cycles.

A naive Python set cannot survive distributed execution. The state must be shared or partitioned. Bloom filters are particularly useful for large-scale probabilistic membership testing.

URL -> Hash Functions -> Bit Array
                           |-> probably seen
                           |-> definitely not seen

A Bloom filter does not store the entire URL. It stores a probabilistic representation, which dramatically reduces memory requirements at the cost of a controllable false-positive rate.

A production system can combine three components:

URL Canonicalization Is More Important Than It Looks

Consider three URLs:

https://example.com/product?id=123
https://example.com/product?id=123&utm_source=google
https://example.com/product/?id=123

Are these three resources or one? There is no universal answer. Canonicalization is therefore not simply string manipulation. It is part of the crawler’s data model.

from urllib.parse import urlsplit, urlunsplit

def normalize(url):
    parts = urlsplit(url)

    return urlunsplit((
        parts.scheme.lower(),
        parts.netloc.lower(),
        parts.path.rstrip("/") or "/",
        "",
        ""
    ))

Crawler correctness depends on understanding the difference between URL identity and resource identity.

Fault Tolerance: Assume Every Worker Will Fail

At sufficient scale, failures are normal operating conditions. A worker can disappear because of hardware failure, container termination, out-of-memory conditions, network errors, deployment or dependency failure.

The order of operations inside a worker is what makes that survivable:

Receive  ->  Process  ->  Persist Result  ->  Acknowledge

A worker should not acknowledge a job simply because it received it. If the worker fails between processing and acknowledgement, the system may process the job again.

Design for at-least-once execution

Instead of assuming that every job executes exactly once, make the operation idempotent. Retries should not corrupt the dataset.

INSERT INTO pages(url, content_hash)
VALUES (%s, %s)
ON CONFLICT (url)
DO UPDATE SET content_hash = EXCLUDED.content_hash

Storage Architecture Should Follow the Data Lifecycle

A billion-page crawler generates enormous quantities of information. Trying to put everything into one relational database is usually the wrong abstraction. Each crawl event produces several kinds of data, and each belongs somewhere different.

The Network Layer Is Part of the Architecture

At scale, networking becomes one of the largest variables in crawler performance. A worker may be healthy while the network path experiences latency variation, regional differences, connection failures, routing changes or bandwidth constraints.

For legitimate public-data collection, localization testing, market intelligence, price monitoring, SEO monitoring and similar workloads, ProxyEmpire can sit between the fetching layer and the public internet. Its infrastructure includes residential, mobile, static residential (ISP) and datacenter connectivity, with location targeting and common proxy protocols.

Crawler
   |
Network Abstraction
   |
Proxy Layer
   |
Internet

The crawler should not need to know how individual network endpoints are sourced or managed. It should request a connection profile appropriate for the workload. This keeps infrastructure replaceable and reduces coupling between application code and network topology.

Why Proxy Selection Should Be a Scheduling Decision

One architectural extension is to make network selection part of the scheduler. The fetch call states what the job needs:

fetch(
    url=url,
    region="US",
    network_type="residential"
)

And the job itself carries those requirements:

class CrawlJob:
    def __init__(self, url, region=None, network_type="default"):
        self.url = url
        self.region = region
        self.network_type = network_type

This design allows the same crawler to support different legitimate requirements without embedding proxy logic throughout the codebase. A global localization dataset may require region-specific access, while a high-volume dataset that is not location-sensitive may prioritize cost and latency.

Observability Is the Difference Between Scaling and Guessing

At billion-page scale, “the crawler is slow” is not an actionable diagnosis. Engineering teams need to know where the system is slow.

Instrumenting a request path takes only a few lines with a metrics client:

from prometheus_client import Counter, Histogram

requests = Counter("crawler_requests_total", "Total crawler requests")

latency = Histogram(
    "crawler_request_latency_seconds",
    "Crawler request latency"
)

def record_request(duration):
    requests.inc()
    latency.observe(duration)

Throughput alone is a poor success metric. A better KPI is useful, fresh, validated data per unit of infrastructure cost.

Freshness Changes the Architecture

A crawler that runs once is relatively simple. A crawler that maintains a dataset continuously is fundamentally different. Resources that change hourly, weekly and monthly should not receive identical recrawl schedules.

def next_crawl_interval(change_rate):
    if change_rate > 0.8:
        return 3600

    if change_rate > 0.3:
        return 21600

    return 86400

In this example the intervals are in seconds, so the three tiers map to the following schedule:

A mature crawler can assign recrawl intervals based on historical change frequency. This transforms the crawler from a downloader into a continuous data freshness engine.

The Real Scaling Problem Is Coordination

It is tempting to think that billion-page crawling is primarily a compute problem. Compute is comparatively easy to scale horizontally. The difficult problems are coordination, state, scheduling, correctness and failure recovery.

Adding another hundred workers is easy. Ensuring those workers do not duplicate work, overload the same host, lose jobs, corrupt state or overwhelm downstream storage is considerably harder.

Control Plane
  |-- Scheduler
  `-- Policy
        |
   Message Bus
        |
   Worker Fleet
        |
 Processing Layer
        |
Object Storage + Analytics

That is no longer a scraper. It is a distributed data platform.

Designing for the Next Billion Pages

The architecture of a large-scale crawler should be based on how the system behaves when every component is under pressure simultaneously. Each design choice in this guide answers one of those pressures.

ProxyEmpire can occupy that network abstraction layer for organizations collecting public data, performing global market research, monitoring prices, validating localized experiences or building distributed data infrastructure.

Conclusion: A Crawler at Scale Is a Distributed System

A billion-page crawler is not a faster scraper. It is a distributed system with the same fundamental problems found in large-scale backend infrastructure: state management, scheduling, partitioning, queues, fault tolerance, observability, storage and resource optimization.

The HTTP request is merely the smallest unit of work. The real engineering challenge is coordinating millions or billions of those units while maintaining correctness and freshness.

For CTOs building AI data pipelines, competitive intelligence platforms, search infrastructure, market research systems or large-scale public-data platforms, this distinction matters. Once the crawler becomes infrastructure, it should be engineered like infrastructure.

Frequently Asked Questions