429 Too Many Requests in Web Scraping: Rate Limits, Retry-After and Backoff (2026)

Developer guide · Last reviewed 20 October 2026 · 15 min read

A 429 Too Many Requests response is the server telling your client that it is sending requests faster than it is allowed to. For scrapers and API clients, the HTTP 429 error is less a bug than a control signal: it says you are being limited, sometimes says for how long, and hints at whether a retry can succeed. This guide covers what 429 means in the specs, how it differs from 403, 503 and 407, how to read Retry-After and rate-limit headers, and how to make Python requests, aiohttp and Scrapy clients slow down correctly.

What 429 Too Many Requests Means

The 429 status code was added to HTTP by RFC 6585 in 2012. Its definition is one sentence: the user has sent too many requests in a given amount of time, which the RFC calls rate limiting. The response should explain the condition and may include a Retry-After header saying how long to wait.

Two details matter to scraper developers. The RFC explicitly does not define how the server identifies the user or counts requests: it may count per resource, per site or across servers, and identify you by IP, credentials or a cookie. And 429 responses must not be stored by caches, so each one is a fresh decision about your recent traffic.

A 429 can come from the application, an API gateway, a CDN or WAF rule, or occasionally a proxy you route through. A typical API-style response:

HTTP/1.1 429 Too Many Requests
Date: Tue, 20 Oct 2026 09:14:05 GMT
Retry-After: 30
Content-Type: application/json
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1792487675

{"error": "rate_limited", "message": "Too many requests, retry in 30 seconds"}

That response has everything a client needs. Many HTML sites send a bare 429 page with no headers at all, which is why your client needs a fallback strategy.

429 vs 403, 503 and 407

Lumping every non-200 response into “blocked” leads to the wrong fix. These four codes mean very different things:

403 is a refusal, not a pacing signal; the same request usually gets another 403. Some bot-management systems answer automated traffic with 403, so a wave of them is a reason to stop and investigate. Our explainer on how DataDome detects automated traffic covers that side.

503 is about the server’s condition. RFC 9110 lets the server send Retry-After with it, which is why many retry libraries treat 429 and 503 alike. Some CDNs also use 503 for block pages, so check the body.

407 comes from your proxy: it rejected or never received your proxy credentials, and no backoff fixes it. See our guides to proxy error codes and fixing 407 Proxy Authentication Required.

Retry-After and Rate-Limit Headers

Retry-After: two formats

Retry-After is defined in RFC 9110, section 10.2.3. Its value is either an HTTP-date or a number of seconds, which the spec calls delay-seconds:

• Retry-After: 120 means wait two minutes after receiving the response. The value is a non-negative decimal integer.
• Retry-After: Wed, 21 Oct 2026 07:28:00 GMT means do not retry before that moment.

The date form is expressed in the server’s clock, so if your clock is off, a naive date - now() gives the wrong wait. Measuring against the response’s own Date header removes the skew. Malformed values exist in the wild, so parse defensively and fall back to your own backoff.

Vendor rate-limit headers

Many APIs also advertise the quota, usually with headers in the style of X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset. These are conventions, not a standard: names vary, one API’s reset is a Unix timestamp and another’s is seconds remaining, and windows range from a minute to a day. Read the provider’s documentation. When present, these headers let you slow down before a 429 by spreading the remaining requests over the time until the reset.

The IETF RateLimit fields are still a draft

The IETF HTTPAPI working group is standardizing this in draft-ietf-httpapi-ratelimit-headers. As of October 2026 it is an active working-group Internet-Draft (revision -11, May 2026), not an RFC. It defines two structured fields, RateLimit-Policy for the quota policy and RateLimit for what is available now, for example RateLimit: "default";r=50;t=30 (50 units left, window resets in 30 seconds). Earlier revisions used different field names and the syntax may change again, so support it opportunistically. The draft itself tells clients not to expect the fields on every response and to ignore malformed values.

What the Limit Is Keyed On

A rate limit is a counter attached to a key, and the key decides which countermeasures work. Common keys:

Behind the key is an algorithm: a fixed window, a sliding window, or a token bucket that refills steadily and allows short bursts. The token bucket is a good mental default because it explains why a short burst succeeds and a sustained one fails.

The Client-Side Playbook

A client that handles 429 well combines four mechanisms:

1. Honor Retry-After. The server’s number beats any guess. Cap it, and when it exceeds the cap, park the URL and reschedule it instead of holding a worker.
2. Exponential backoff with jitter. Without a header, double the wait ceiling on each attempt and pick a random wait below it. This “full jitter” variant was popularized by the AWS Architecture Blog post Exponential Backoff And Jitter. Without jitter, workers that failed together retry together.
3. A per-host limiter. Retries only react to failures. A token bucket per host sets the pace up front and gives every worker a shared place to pause when a 429 arrives.
4. Bounded concurrency. Cap in-flight requests per host and in total, so retries never pile up into a burst.

response
  |
  +-- 2xx / 3xx ------------------> success: nudge host rate up
  |
  +-- 429 ---+-- Retry-After? --yes--> wait = header value
  |          |                    no--> wait = backoff with jitter
  |          +-- wait > max? ----yes--> park URL, reschedule later
  |          +-- pause the whole host for `wait`, halve its rate
  |
  +-- 500/502/503/504 / timeout -> backoff this request only
  |
  +-- 401 / 403 / 404 / 407 ----> do not retry: fix, skip or investigate

These standard-library helpers are reused by the later examples:

import random
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime


def parse_retry_after(value, date_header=None):
    """Return the Retry-After delay in seconds, or None if absent/invalid.

    Handles both forms from RFC 9110: delay-seconds ("120") and
    HTTP-date ("Wed, 21 Oct 2026 07:28:00 GMT").
    """
    if not value:
        return None
    value = value.strip()
    if value.isascii() and value.isdigit():
        return float(value)
    try:
        when = parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return None  # malformed: fall back to your own backoff
    if when.tzinfo is None:
        when = when.replace(tzinfo=timezone.utc)
    # Measure against the server's clock when it sent a Date header,
    # so local clock skew does not shorten or stretch the wait.
    now = datetime.now(timezone.utc)
    if date_header:
        try:
            server_now = parsedate_to_datetime(date_header)
            if server_now.tzinfo is not None:
                now = server_now
        except (TypeError, ValueError):
            pass
    return max(0.0, (when - now).total_seconds())


def full_jitter_backoff(attempt, base=1.0, cap=60.0):
    """Exponential backoff with full jitter: uniform(0, min(cap, base * 2**attempt))."""
    return random.uniform(0, min(cap, base * (2 ** attempt)))

Catching both exceptions is deliberate: Python 3.10+ raises ValueError for an unparseable date, 3.9 raises TypeError.

Python requests: urllib3 Retry

requests does not retry on its own, but its HTTPAdapter accepts a urllib3 Retry, which understands 429 and Retry-After. Parameter names match the current urllib3 2.x documentation.

import os

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

PROXY_USER = os.environ.get("PROXY_USER")   # copy from Connection Builder
PROXY_PASS = os.environ.get("PROXY_PASS")

retry = Retry(
    total=6,
    connect=3,
    read=2,
    status=5,
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET", "HEAD"],     # never blindly replay POSTs
    backoff_factor=1.0,                  # 0s, 2s, 4s, 8s ... between retries
    backoff_max=60,
    backoff_jitter=1.0,                  # + uniform(0, 1) seconds
    respect_retry_after_header=True,     # the default, stated for clarity
    retry_after_max=120,                 # urllib3 2.6.3+; older versions lack it
    raise_on_status=False,               # hand back the last 429 instead of raising
)

session = requests.Session()
adapter = HTTPAdapter(max_retries=retry, pool_maxsize=10)
session.mount("http://", adapter)
session.mount("https://", adapter)

if PROXY_USER:
    proxy = f"http://{PROXY_USER}:{PROXY_PASS}@v2.proxyempire.io:5000"
    session.proxies = {"http": proxy, "https": proxy}

resp = session.get("https://example.com/catalog?page=1", timeout=(10, 30))
history = resp.raw.retries.history if resp.raw.retries else ()
print(resp.status_code, "after", len(history), "retries")
if resp.status_code == 429:
    print("still throttled; Retry-After =", resp.headers.get("Retry-After"))

Details worth knowing:

• Retry-After wins. With respect_retry_after_header=True (the default), urllib3 sleeps for the header value, in either form, on 413, 429 and 503, and uses backoff only when the header is missing.
• The first retry is immediate. Backoff is zero for the first retry, then backoff_factor * 2 ** (n - 1) capped at backoff_max. A 429 without a header is retried at once.
• Long waits block the thread. urllib3 calls time.sleep(). Since 2.6.3 it caps Retry-After at six hours by default; retry_after_max lowers that. For crawlers, a short cap plus rescheduling beats a sleeping thread.
• Only idempotent methods retry. The default allowed_methods excludes POST. Keep it that way unless replay is safe.
• No pacing. Retry reacts to failures; pair it with a limiter in your own loop.

For a broader look at proxies with requests, httpx, Scrapy and Playwright, see Python web scraping with residential proxies.

aiohttp: A Per-Host Limiter That Honors Retry-After

Hundreds of coroutines can hit one host in the same second, and if each handles its own 429, the others keep firing while it sleeps. Make the limit a shared object per host instead: a token bucket every worker draws from, with a not_before timestamp a 429 can push forward for everyone.

The limiter also adapts: success raises the rate slightly, a 429 halves it. This additive-increase, multiplicative-decrease (AIMD) pattern converges on what a host tolerates without knowing the limit.

import asyncio
import os
import time
from collections import Counter, defaultdict
from urllib.parse import urlsplit

import aiohttp

from backoff import full_jitter_backoff, parse_retry_after  # Example 2

RETRYABLE = {429, 500, 502, 503, 504}
MAX_RETRY_AFTER = 300  # longer than this: park the URL, do not hold a worker


class HostLimiter:
    """Token bucket for one host, with AIMD rate control and a shared pause."""

    def __init__(self, rate=2.0, burst=4, min_rate=0.2, max_rate=10.0):
        self.rate, self.burst = rate, burst
        self.min_rate, self.max_rate = min_rate, max_rate
        self.tokens = float(burst)
        self.updated = time.monotonic()
        self.not_before = 0.0
        self.lock = asyncio.Lock()

    async def acquire(self):
        async with self.lock:          # waiters queue up behind the lock
            while True:
                now = time.monotonic()
                if now < self.not_before:          # honoring Retry-After
                    await asyncio.sleep(self.not_before - now)
                    continue
                elapsed = now - self.updated
                self.tokens = min(self.burst, self.tokens + elapsed * self.rate)
                self.updated = now
                if self.tokens >= 1:
                    self.tokens -= 1
                    return
                await asyncio.sleep((1 - self.tokens) / self.rate)

    def on_success(self):
        self.rate = min(self.max_rate, self.rate + 0.05)    # additive increase

    def on_throttle(self, wait):
        self.rate = max(self.min_rate, self.rate / 2)       # multiplicative decrease
        self.not_before = max(self.not_before, time.monotonic() + wait)


limiters = defaultdict(HostLimiter)    # one bucket per target host
stats = Counter()


class GiveUp(Exception):
    pass


async def fetch(session, url, proxy=None, proxy_auth=None, max_attempts=6):
    limiter = limiters[urlsplit(url).hostname]
    for attempt in range(max_attempts):
        await limiter.acquire()
        wait, status = None, None
        try:
            async with session.get(url, proxy=proxy, proxy_auth=proxy_auth) as resp:
                status = resp.status
                stats[status] += 1
                if status < 400:
                    limiter.on_success()
                    return await resp.text()
                if status not in RETRYABLE:
                    raise GiveUp(f"{status} {url}")      # 403/404/407: retrying won't help
                wait = parse_retry_after(
                    resp.headers.get("Retry-After"), resp.headers.get("Date")
                )
        except (aiohttp.ClientConnectionError, asyncio.TimeoutError):
            stats["network_error"] += 1
        if wait is not None and wait > MAX_RETRY_AFTER:
            raise GiveUp(f"Retry-After {wait:.0f}s for {url}; reschedule it later")
        if wait is None:
            wait = full_jitter_backoff(attempt, base=1.0, cap=60.0)
        if status == 429:
            limiter.on_throttle(wait)     # every worker on this host pauses
        else:
            await asyncio.sleep(wait)     # only this request backs off
    raise GiveUp(f"retries exhausted for {url}")


async def worker(session, queue, results):
    proxy = "http://v2.proxyempire.io:5000" if os.environ.get("PROXY_USER") else None
    auth = (
        aiohttp.BasicAuth(os.environ["PROXY_USER"], os.environ["PROXY_PASS"])
        if proxy else None
    )
    while True:
        url = await queue.get()
        try:
            results[url] = await fetch(session, url, proxy, auth)
        except GiveUp as exc:
            results[url] = exc
        finally:
            queue.task_done()


async def main(urls, concurrency=8):
    queue = asyncio.Queue()
    for url in urls:
        queue.put_nowait(url)
    results = {}
    timeout = aiohttp.ClientTimeout(total=45, connect=10)
    connector = aiohttp.TCPConnector(limit=concurrency, limit_per_host=concurrency)
    async with aiohttp.ClientSession(timeout=timeout, connector=connector) as session:
        workers = [
            asyncio.create_task(worker(session, queue, results))
            for _ in range(concurrency)
        ]
        await queue.join()
        for w in workers:
            w.cancel()
        await asyncio.gather(*workers, return_exceptions=True)
    print(dict(stats))
    return results


if __name__ == "__main__":
    asyncio.run(main([f"https://example.com/catalog?page={i}" for i in range(1, 21)]))

• A 429 pauses the host. on_throttle() moves not_before forward for every worker on that host; a 503 or timeout only delays its own request.
• Concurrency is bounded twice. The worker count caps in-flight requests, TCPConnector(limit=..., limit_per_host=...) caps sockets, and the bucket caps the rate.
• Long waits leave the worker. A Retry-After above MAX_RETRY_AFTER raises GiveUp, so a scheduler can requeue the URL for later.

The starting values are placeholders; tune them per target. The same limiter works around httpx.AsyncClient, which takes the proxy URL through its proxy argument in current versions. For aiohttp proxy configuration in depth, see our aiohttp proxy guide.

Scrapy: AutoThrottle, RETRY_HTTP_CODES and Retry-After

AutoThrottle adjusts the download delay per slot (by default, per hostname) from response latency, and the built-in RetryMiddleware already retries 429: the default RETRY_HTTP_CODES is [500, 502, 503, 504, 522, 524, 408, 429], with RETRY_TIMES = 2.

# settings.py
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 2.0          # initial per-slot delay, seconds
AUTOTHROTTLE_MAX_DELAY = 120.0          # ceiling when a site is slow or throttling
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0   # average parallel requests per site
AUTOTHROTTLE_DEBUG = False              # True logs every delay adjustment

CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 4      # hard cap; AutoThrottle works below it
DOWNLOAD_DELAY = 1.0                    # with AutoThrottle on, the minimum delay

RETRY_ENABLED = True
RETRY_TIMES = 5
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524, 408]
RETRY_PRIORITY_ADJUST = -1              # retries go behind fresh requests

# A host with a published limit gets its own slot settings.
DOWNLOAD_SLOTS = {
    "api.example.com": {"concurrency": 1, "delay": 2.0},
}

DOWNLOADER_MIDDLEWARES = {
    # Higher number = sees responses before RetryMiddleware (550).
    "myproject.middlewares.RetryAfterMiddleware": 560,
}
RETRY_AFTER_MAX_WAIT = 300              # custom setting, read by Example 6

Two limitations matter. The built-in RetryMiddleware does not read Retry-After (checked against the Scrapy 2.19 source); the retry goes out at the slot’s normal pace. And AutoThrottle reacts to latency and never lowers the delay on a non-200 response, so a fast 429 page does not speed you up, but it does not slow you down much either. With AutoThrottle on, a DOWNLOAD_SLOTS delay is only a starting value; its concurrency stays a hard cap.

A small downloader middleware closes the gap by raising the slot delay when a 429 or 503 carries Retry-After:

from backoff import parse_retry_after  # Example 2


class RetryAfterMiddleware:
    """Turn Retry-After on 429/503 into a slower pace for that download slot."""

    def __init__(self, crawler):
        self.crawler = crawler
        # Custom setting (not built into Scrapy): longest Retry-After we wait for.
        self.max_wait = crawler.settings.getfloat("RETRY_AFTER_MAX_WAIT", 300.0)

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler)

    def process_response(self, request, response, spider=None):
        if response.status not in (429, 503):
            return response
        raw = response.headers.get(b"Retry-After")
        date = response.headers.get(b"Date")
        wait = parse_retry_after(
            raw.decode("latin-1") if raw else None,
            date.decode("latin-1") if date else None,
        )
        stats = self.crawler.stats
        stats.inc_value(f"retry_after/status_{response.status}")
        if wait is None:
            return response               # RetryMiddleware retries with its own timing
        if wait > self.max_wait:
            stats.inc_value("retry_after/too_long")
            request.meta["dont_retry"] = True    # give up; reschedule outside Scrapy
            return response
        slot = self.crawler.engine.downloader.slots.get(request.meta.get("download_slot"))
        if slot is not None:
            # Slot delays are randomized (+/-50% by default); scale up so the
            # shortest randomized gap is still at least `wait`.
            jitter = getattr(slot, "jitter", 0.5)
            scale = 1 / (1 - jitter) if jitter < 1 else 2.0
            slot.delay = max(slot.delay, wait * scale)
        return response

• At order 560 it sees each response before RetryMiddleware (550). It returns the response unchanged, so the retry still happens, at the slower pace.
• It uses crawler.engine.downloader.slots, the internal structure AutoThrottle also uses. We ran it on Scrapy 2.13 and checked it against the 2.19 source; re-test after upgrades. The getattr covers versions without slot.jitter.
• AutoThrottle eases the delay back down as 200 responses arrive, so the crawl recovers gradually.
• spider=None keeps it compatible with older Scrapy, which passes the spider, and newer releases, which deprecate it.

To route Scrapy through a proxy, set request.meta["proxy"] to the proxy URL (for example http://USERNAME:[email protected]:5000); the built-in HttpProxyMiddleware handles the credentials.

Status Codes: What to Do With Each

Retry policy is mostly classification. A reasonable default, to adjust per target:

Cap retries per URL as well as per request, and log why each URL was abandoned.

When Rotating Proxies Help, and When They Don’t

Where rotation helps

When a public site applies a per-IP limit, a crawler on one server IP concentrates all its traffic on one counter. Spreading requests across a pool of exit IPs distributes the load the way ordinary visitors are distributed. Rotation also helps when one IP range has a poor reputation, such as a datacenter range a site treats more strictly than residential traffic.

Rotation is not a substitute for pacing. Keep the per-host limiter global across all exits, sized to what the site can reasonably serve, and keep honoring Retry-After: the same total load through a thousand IPs still lands on one server. Respect robots.txt, the site’s terms and its capacity, and prefer an official API where one exists. Our piece on designing distributed crawlers covers per-host politeness at scale.

Where rotation does not help, or is inappropriate

• Account and API-key limits. The counter follows the credential. Spreading one workload across many accounts to multiply a quota usually violates the provider’s terms.
• Published API quotas. Documented limits are part of the contract. Cache, batch where supported, use conditional requests for unchanged data, and ask for a higher tier if you need more.
• Session and cookie limits. Changing IP mid-session can break the session, and the limit still follows the cookie.
• Concurrency limits. Lower your in-flight requests.

Sticky sessions for stateful flows

Multi-step flows, such as paginating search results tied to a session cookie or working inside an area you are authorized to access, expect one visitor on one IP. Use a sticky session for the flow and rotate between flows, not within them. If a sticky IP starts getting 429s, slow that session down rather than switching: a fresh IP inherits the same behavior. See sticky vs rotating proxies for the trade-offs.

Setting It Up With ProxyEmpire

ProxyEmpire serves every proxy type through one gateway, v2.proxyempire.io on port 5000, over HTTP, HTTPS or SOCKS5. Authentication is username and password; targeting and session settings live in the username, which you generate (and can test) under Proxy Manager → Connection Builder. Copy it from there rather than assembling it by hand.

The rotation mode matters most for 429 handling:

• New IP on every request: stateless fetches of public pages behind per-IP limits. Keep your per-host limiter global.
• Same IP (sticky session): stateful flows. The username carries a session ID (sid), and you can create your own 8-digit IDs, one per worker or flow. See the sticky sessions documentation.
• Custom rotation time, up to 2 hours: a stable IP for a while, but not indefinitely.

Test the credentials first, so a 407 is not mistaken for something else:

curl -x http://USERNAME:[email protected]:5000 https://api.ipify.org

For sticky workers, export many sessions at once as a host:port:user:pass list and pin each worker to one line; the per-host limiter keeps the combined rate polite:

from urllib.parse import quote


def load_sessions(path):
    """Read host:port:user:pass lines exported from the dashboard."""
    sessions = []
    with open(path) as fh:
        for line in fh:
            line = line.strip()
            if not line:
                continue
            host, port, user, password = line.split(":", 3)
            sessions.append(
                f"http://{quote(user, safe='')}:{quote(password, safe='')}@{host}:{port}"
            )
    return sessions


sessions = load_sessions("proxyempire_sessions.txt")


def proxy_for_worker(worker_id):
    # Same worker, same session, same exit IP for the life of its flow.
    return sessions[worker_id % len(sessions)]

Rotating residential proxies (30M+ IPs with country, region, city, ZIP, ISP and ASN targeting) fit most public-web collection. Rotating mobile proxies use real 4G carrier IPs with carrier targeting; datacenter and static residential (ISP) proxies suit cases where speed or a fixed IP matters more. With Playwright, use the http:// gateway URL: its Chromium rejects username and password authentication on SOCKS5 proxies.

Monitoring Your 429 Rate

A crawler that retries quietly can look healthy while it spends most of its time waiting. Track, per host and per proxy pool:

• 429 ratio: 429s divided by all responses over a rolling window. The headline number.
• Retry-After values: a jump from seconds to hours means you moved from a burst limit to a longer quota.
• Retries per success and time spent throttled: the real cost of an aggressive pace.
• Effective rate: where the AIMD limiter settles. A falling equilibrium is an early warning.
• Abandoned URLs by reason: retries exhausted, Retry-After too long, or non-retryable status.

Pick a 429 budget per target (1% of responses is an illustrative starting point, not a standard) and alert when a host exceeds it for several minutes. If it stays high, a circuit breaker that pauses the host for a cool-down is cheaper than burning retries. A spike across every host at once usually means a change on your side, such as a deploy that raised concurrency. In Scrapy, the retry_after/* counters from Example 6 and the built-in downloader/response_status_count/429 stat cover this.

Frequently Asked Questions