Scrapy Proxy Middleware: Rotating Residential Proxies and scrapy-playwright (2026)

Developer guide · Last reviewed 20 October 2026 · 17 min read

A Scrapy proxy setup can be one line: put a proxy URL in request.meta["proxy"] and Scrapy routes the request through it. A real crawl through rotating residential proxies needs more: which requests share an exit IP, what happens on a 403 or 429, how hard each domain is hit, and how scrapy-playwright pages get a proxy at all, since they ignore that meta key. This guide covers Scrapy 2.19 and scrapy-playwright 0.0.48: how Scrapy applies proxies, a proxy middleware that assigns ProxyEmpire sessions, retry and AutoThrottle settings, browser and per-context proxies, and how to check your exit IP.

The short version

For plain Scrapy requests, set meta["proxy"] to http://USER:[email protected]:5000, and let a small downloader middleware at priority 740 pick the username: a rotating one, or a sticky one with its own session ID per domain or cookiejar. HttpProxyMiddleware (priority 750) turns the credentials into a Proxy-Authorization header, and for HTTPS targets that header goes in the CONNECT request. Retry 429 (and, with care, 403) on a new session and throttle per domain. scrapy-playwright requests take their proxy from PLAYWRIGHT_LAUNCH_OPTIONS or a browser context, never from meta["proxy"], and need an http:// URL because Chromium rejects SOCKS5 authentication.

Start a $1.97 trial See residential proxies

Versions and placeholders. Everything here was checked against Scrapy 2.19.0 and scrapy-playwright 0.0.48. Both need Python 3.10 or newer. Concurrency, delay and retry numbers are starting points to tune. example.com URLs and CSS selectors are placeholders, and ProxyEmpire usernames are always copied from the dashboard, never typed out here.

Key takeaways

meta["proxy"] is the only switch. HttpProxyMiddleware (750) reads it, strips the credentials and sends them as Proxy-Authorization.
Your own proxy middleware needs a priority below 750 so it runs first, and above RetryMiddleware (550) so it sees 403/429 responses before a retry is scheduled. 740 works.
Rotating vs sticky is a username choice. Build sticky usernames from a template you copy from the Connection Builder, with your own 8-digit sid.
Never retry 407: it means the proxy credentials are wrong. Add 403 to RETRY_HTTP_CODES only together with a new session and a low retry count.
Download slots, and so CONCURRENT_REQUESTS_PER_DOMAIN and AutoThrottle, are per target domain. More exit IPs do not make a site more willing to take load.
scrapy-playwright ignores meta["proxy"]. Use PLAYWRIGHT_LAUNCH_OPTIONS, PLAYWRIGHT_CONTEXTS or playwright_context_kwargs, always with an http:// server.
start_requests() was removed in 2.19 (deprecated in 2.13). Use start().

How Scrapy Applies a Proxy

One meta key, one middleware, two download paths

Check the proxy outside Scrapy first. If this command doesn’t print a residential IP, no spider setting will fix it:

Example 1 — check the gateway with curl (shell)

curl -x http://USERNAME:[email protected]:5000 https://api.ipify.org

Copy the username and password from Proxy Manager → Connection Builder in the dashboard, which can also test the string. The curl proxy guide covers the flags.

Inside Scrapy, a proxy is just a value in request.meta["proxy"]. The built-in HttpProxyMiddleware, enabled by default at priority 750, processes it on every request:

It parses the URL. If it contains user:password@, it percent-decodes both, Base64-encodes them (encoding set by HTTPPROXY_AUTH_ENCODING, default latin-1) and sets a Proxy-Authorization: Basic … header.
It rewrites meta["proxy"] to the same URL without the credentials.
If a request’s proxy changes, for example on a retry, the header is replaced or removed, so credentials meant for one proxy aren’t sent to another.

The download handler then decides how to talk to the proxy. With the default HTTP/1.1 handler, a plain http:// target is sent to the proxy as a normal proxied request. An https:// target goes through a tunnel: Scrapy sends CONNECT host:443 to the proxy with the Proxy-Authorization header, then starts TLS with the target inside the tunnel, and strips the header from the tunneled request. The target never sees your proxy credentials.

Figure 1 — where the proxy is applied (plain Scrapy request)

spider yields Request(url)
        |
        v
downloader middlewares, process_request in ascending order
  100  RobotsTxtMiddleware
  550  RetryMiddleware
  700  CookiesMiddleware
  740  ProxyEmpireMiddleware   -> sets meta["proxy"] = http://user:pass@gateway
  750  HttpProxyMiddleware     -> Proxy-Authorization header, credentials removed from meta
        |
        v
HTTP/1.1 download handler
  http://  target -> GET http://target/... sent to the proxy
  https:// target -> CONNECT target:443 (+ Proxy-Authorization), then TLS to target
        |
        v
v2.proxyempire.io:5000 -> residential exit IP -> target site

The smallest working spider sets the meta key itself. Note the quote(..., safe=""): if a password contains @, : or /, an unencoded URL breaks parsing, and Scrapy decodes the percent-encoding back before building the header.

Example 2 — one proxied request through meta[“proxy”] (Python)

import os
from urllib.parse import quote

import scrapy

USER = quote(os.environ["PROXYEMPIRE_USER"], safe="")
PASSWORD = quote(os.environ["PROXYEMPIRE_PASSWORD"], safe="")
PROXY = f"http://{USER}:{PASSWORD}@v2.proxyempire.io:5000"


class OneProxySpider(scrapy.Spider):
    name = "one_proxy"

    async def start(self):
        yield scrapy.Request(
            "https://api.ipify.org?format=json",
            meta={"proxy": PROXY},
        )

    def parse(self, response):
        yield {"exit_ip": response.json()["ip"]}

Environment variables

HttpProxyMiddleware also follows the standard library’s proxy environment variables, read through urllib.request.getproxies(). Both lowercase and uppercase names work (http_proxy/HTTP_PROXY, https_proxy/HTTPS_PROXY, no_proxy/NO_PROXY), and lowercase wins if both are set. Three details matter in practice:

The variables are read once, when the middleware is created at crawler start. Changing them mid-crawl does nothing.
They are keyed by the target’s scheme: HTTPS_PROXY covers https:// URLs, HTTP_PROXY covers http:// URLs. Set both.
A request with meta["proxy"] set, even to None, ignores the environment and NO_PROXY. meta["proxy"] = None is how you send a single request direct.

Example 3 — route a whole crawl through environment variables (shell)

# Read once, when HttpProxyMiddleware starts; applies to requests
# that don't set meta["proxy"] themselves.
export HTTP_PROXY="http://USERNAME:[email protected]:5000"
export HTTPS_PROXY="$HTTP_PROXY"
export NO_PROXY="localhost,127.0.0.1"
scrapy crawl myspider

That’s fine for a quick run on one rotating username, but it can’t give different requests different sessions.

Which proxy schemes work

Use http:// for the gateway. The default HTTP/1.1 handler has no SOCKS support, and an https:// proxy URL with an https:// target raises NotImplementedError. Scrapy’s httpx-based handler does support socks5://, but you don’t need it: an HTTP proxy tunnels HTTPS end to end with CONNECT. The SOCKS vs HTTP proxy comparison explains the difference.

Rotating vs Sticky Sessions in a Crawl

The username decides the IP behavior

ProxyEmpire puts targeting and session behavior in the proxy username. In the Connection Builder you pick the location and one of three rotation modes: a new IP on every request, the same IP (a sticky session), or a custom rotation time of up to two hours. A sticky username contains a session ID, the sid; every request that sends the same sid keeps the same exit IP, and a new sid gets a new one. You can make up your own 8-digit session IDs, which lets the crawler create sessions on demand. The sticky sessions documentation shows where the sid sits.

The rest of this guide doesn’t hard-code a username format. It uses two values you copy from the dashboard:

A rotating username, used as-is.
A sticky username template: copy a sticky username and replace its session ID value with the literal text {sid}. The code fills in a fresh 8-digit ID for each session.

Example 4 — credentials from the Connection Builder, kept out of the code (shell)

# Copied from Proxy Manager -> Connection Builder
export PROXYEMPIRE_PASSWORD='your-proxy-password'
export PROXYEMPIRE_ROTATING_USER='paste-the-rotating-username'
# Sticky username with its session ID value replaced by the text {sid}
export PROXYEMPIRE_STICKY_USER_TEMPLATE='paste-the-sticky-username-with-{sid}'

Which mode fits depends on what the requests have in common. The general trade-offs are covered in sticky vs rotating proxies; in Scrapy terms it comes down to three patterns:

Pattern	Use it when	Scrapy key
Rotating per request	Independent pages: product URLs, search results, public listings. Nothing links one request to the next	None needed
Sticky per domain	A site ties cookies, CSRF tokens or region selection to the IP, or behaves oddly when the IP changes mid-crawl	Target hostname
Sticky per cookiejar	Several parallel sessions on one site, each with its own cookies and its own consistent IP	`meta["cookiejar"]`

A sticky session that starts collecting 403s or 429s should be replaced; the middleware below does that automatically. For pool-level strategies, see how proxy pools and rotation strategies work.

A ProxyEmpire Session Middleware

Rotating or sticky, chosen in settings

First, a small credential helper shared by the middleware and the Playwright settings:

Example 5 — myproject/proxyempire.py, the credential helper (Python)

"""myproject/proxyempire.py: build ProxyEmpire gateway credentials.

Usernames come from the Connection Builder (Proxy Manager -> Connection
Builder). Copy a rotating username as-is. For sticky sessions, copy a sticky
username and replace the session ID value inside it with the literal text
{sid}, so this module can generate new sessions.
"""

import os
import secrets
from urllib.parse import quote

GATEWAY_HOST = os.environ.get("PROXYEMPIRE_HOST", "v2.proxyempire.io")
GATEWAY_PORT = int(os.environ.get("PROXYEMPIRE_PORT", "5000"))


def new_sid():
    """Return a random 8-digit session ID."""
    return str(10_000_000 + secrets.randbelow(90_000_000))


def username(sid=None):
    """Rotating username when sid is None, otherwise a sticky one."""
    if sid is None:
        return os.environ["PROXYEMPIRE_ROTATING_USER"]
    template = os.environ["PROXYEMPIRE_STICKY_USER_TEMPLATE"]
    if "{sid}" not in template:
        raise ValueError("PROXYEMPIRE_STICKY_USER_TEMPLATE requires "
                         "a {sid} placeholder")
    return template.replace("{sid}", sid)


def password():
    return os.environ["PROXYEMPIRE_PASSWORD"]


def scrapy_proxy_url(sid=None):
    """Proxy URL for request.meta["proxy"]; credentials are percent-encoded."""
    user = quote(username(sid), safe="")
    pwd = quote(password(), safe="")
    return f"http://{user}:{pwd}@{GATEWAY_HOST}:{GATEWAY_PORT}"


def playwright_proxy(sid=None):
    """Proxy dict for Playwright launch options or browser contexts.

    Always http://: Chromium rejects username/password auth on SOCKS5.
    """
    return {
        "server": f"http://{GATEWAY_HOST}:{GATEWAY_PORT}",
        "username": username(sid),
        "password": password(),
    }

The middleware skips browser requests (see the scrapy-playwright section), respects a proxy the spider set explicitly, and in sticky mode maps each session key to a sid. On a status in PROXYEMPIRE_NEW_SESSION_CODES or a download exception, it drops that session, so the next request with that key, including the retry, gets a new sid and a new IP.

Example 6 — myproject/middlewares.py, rotating or sticky sessions (Python)

"""myproject/middlewares.py: assign ProxyEmpire sessions to Scrapy requests."""

from urllib.parse import urlparse

from scrapy.exceptions import NotConfigured

from myproject import proxyempire


class ProxyEmpireMiddleware:
    """Set request.meta["proxy"] for every plain (non-browser) request.

    PROXYEMPIRE_MODE = "rotating": every request uses the rotating username,
    so the gateway can hand out a new exit IP each time.
    PROXYEMPIRE_MODE = "sticky": one session ID per key, where the key is the
    target hostname (PROXYEMPIRE_STICKY_KEY = "domain") or the cookiejar
    meta value (PROXYEMPIRE_STICKY_KEY = "cookiejar").
    """

    def __init__(self, settings, stats):
        if not settings.getbool("PROXYEMPIRE_ENABLED", True):
            raise NotConfigured
        self.mode = settings.get("PROXYEMPIRE_MODE", "rotating")
        self.key_by = settings.get("PROXYEMPIRE_STICKY_KEY", "domain")
        self.new_session_codes = {
            int(code)
            for code in settings.getlist("PROXYEMPIRE_NEW_SESSION_CODES", [403, 429])
        }
        if self.mode not in {"rotating", "sticky"}:
            raise NotConfigured(f"Unknown PROXYEMPIRE_MODE: {self.mode!r}")
        self.stats = stats
        self.sessions = {}  # session key -> sid

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings, crawler.stats)

    def _session_key(self, request):
        if self.key_by == "cookiejar":
            return ("cookiejar", request.meta.get("cookiejar"))
        return ("domain", urlparse(request.url).hostname)

    def process_request(self, request):
        if request.meta.get("playwright"):
            # Browser requests take their proxy from the Playwright context.
            # meta["proxy"] = None also stops HttpProxyMiddleware from adding
            # a Proxy-Authorization header from HTTP(S)_PROXY env vars.
            request.meta.setdefault("proxy", None)
            return None
        if "proxy" in request.meta and "pe_mode" not in request.meta:
            return None  # the spider chose a proxy (or None) explicitly

        if self.mode == "rotating":
            request.meta["proxy"] = proxyempire.scrapy_proxy_url()
        else:
            key = self._session_key(request)
            sid = self.sessions.get(key)
            if sid is None:
                sid = self.sessions[key] = proxyempire.new_sid()
                self.stats.inc_value("proxyempire/sessions_created")
            request.meta["pe_key"] = key
            request.meta["pe_sid"] = sid
            request.meta["proxy"] = proxyempire.scrapy_proxy_url(sid)
        request.meta["pe_mode"] = self.mode
        return None

    def _drop_session(self, request, reason):
        key = request.meta.get("pe_key")
        if key is not None and self.sessions.get(key) == request.meta.get("pe_sid"):
            del self.sessions[key]
            self.stats.inc_value(f"proxyempire/sessions_dropped/{reason}")

    def process_response(self, request, response):
        if response.status in self.new_session_codes:
            self._drop_session(request, response.status)
        return response

    def process_exception(self, request, exception):
        self._drop_session(request, type(exception).__name__)
        return None

A few design points are easy to miss:

Retries are re-assigned. RetryMiddleware retries with a copy of the request, meta included. The pe_mode marker tells the middleware it set this proxy itself, so it builds a new proxy URL for the copy instead of treating the old one as an explicit choice.
Only the current session is dropped. Several in-flight requests can share a sid. The sessions.get(key) == sid check stops a late failure from discarding a session that was already replaced.
Method signatures. Since Scrapy 2.14, custom downloader middleware methods that require a spider argument are deprecated, so these take only the request (and response or exception). On older Scrapy versions, add a spider parameter to each process_* method.

Where to put it in DOWNLOADER_MIDDLEWARES

Scrapy merges your DOWNLOADER_MIDDLEWARES with the built-in defaults and sorts them by number. process_request runs from low to high, and process_response and process_exception run from high to low. Two constraints follow:

Middleware (default priority)	Why it constrains yours
HttpProxyMiddleware (750)	Yours must run before it in `process_request`, so it needs a lower number. Above 750, the meta key is set after the credentials were already processed
RetryMiddleware (550)	When it decides to retry, it returns a new request and the response chain stops there. Yours needs a higher number to see the 403/429 first and drop the session
CookiesMiddleware (700)	Not a constraint, but useful context: per-cookiejar sessions line up with the jars it keeps

Anything from 551 to 749 works; 740 keeps it next to the middleware it feeds.

403, 407 and 429: Retries and Throttling

Retry what a new IP can fix, slow down for the rest

These three codes need different handling. The proxy error codes guide covers the wider set.

407 Proxy Authentication Required comes from the proxy: the credentials are wrong or badly encoded. Retrying can’t succeed, so keep it out of RETRY_HTTP_CODES. For http:// targets you get a 407 response. For https:// targets the proxy refuses the CONNECT, and Scrapy raises TunnelError with 'status': 407 in the message. TunnelError is in the default RETRY_EXCEPTIONS, so expect a couple of pointless retries before it fails.
429 Too Many Requests is the site asking you to slow down. It’s in the default RETRY_HTTP_CODES. A new IP may get the retry through, but the real fix is a lower request rate. The built-in RetryMiddleware doesn’t read the Retry-After header.
403 Forbidden is ambiguous: this exit IP may be blocked, which a new session fixes, or the content isn’t open to automated access, which nothing should “fix”. It isn’t retried by default; if you add it, pair it with a new session and a low RETRY_TIMES.

Example 7 — settings.py: middleware, retries, AutoThrottle, robots.txt (Python)

# myproject/settings.py (proxy-related parts)

ROBOTSTXT_OBEY = True

DOWNLOADER_MIDDLEWARES = {
    # Must be lower than HttpProxyMiddleware (750) so meta["proxy"] is set
    # first, and higher than RetryMiddleware (550) so it sees 403/429
    # responses before a retry is scheduled.
    "myproject.middlewares.ProxyEmpireMiddleware": 740,
}

PROXYEMPIRE_MODE = "sticky"            # or "rotating"
PROXYEMPIRE_STICKY_KEY = "domain"      # or "cookiejar"
PROXYEMPIRE_NEW_SESSION_CODES = [403, 429]

# Retries: 429 is already in the default list; 403 is not.
RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429, 403]

# Politeness: concurrency is counted per target domain, not per exit IP.
CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 4
DOWNLOAD_DELAY = 0.5
DOWNLOAD_TIMEOUT = 60

AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 2.0
AUTOTHROTTLE_MAX_DELAY = 60.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0
AUTOTHROTTLE_DEBUG = False

RETRY_TIMES counts retries in addition to the first attempt, so 3 means up to four downloads per URL. You can override it for a single request with meta["max_retry_times"], or switch retries off with meta["dont_retry"]. Retried requests get a lower priority by default (RETRY_PRIORITY_ADJUST = -1), so they wait behind fresh work.

AutoThrottle and per-domain concurrency

Scrapy schedules downloads in slots, keyed by default on the target hostname; the proxy plays no part. Through one exit IP or a thousand, CONCURRENT_REQUESTS_PER_DOMAIN caps in-flight requests to a domain and DOWNLOAD_DELAY spaces them. That is correct: the site carries the same load either way.

AutoThrottle adjusts each slot’s delay from measured latency. It aims for AUTOTHROTTLE_TARGET_CONCURRENCY parallel requests per site, starts at AUTOTHROTTLE_START_DELAY, never goes above AUTOTHROTTLE_MAX_DELAY or below DOWNLOAD_DELAY, and doesn’t let non-200 responses lower the delay. Latency through a residential route is part of what it measures, and CONCURRENT_REQUESTS_PER_DOMAIN remains a hard cap. AUTOTHROTTLE_DEBUG = True shows the adjustments.

Rotating proxies change where requests come from. They don’t change how many a site has to serve.

Per-Request Overrides

Cookiejars, pinned sessions and direct requests

The middleware covers the default, and individual requests can still opt out. Three common cases:

Example 8 — sticky per cookiejar, a pinned session, and a direct request (Python)

import scrapy

from myproject import proxyempire


class AccountAreaSpider(scrapy.Spider):
    name = "overrides"

    async def start(self):
        # 1. Sticky per cookiejar: each jar keeps its own cookies and exit IP
        #    (needs PROXYEMPIRE_STICKY_KEY = "cookiejar").
        for jar in range(3):
            yield scrapy.Request(
                "https://example.com/region-settings",
                meta={"cookiejar": jar},
                dont_filter=True,
            )

        # 2. Pin one request to a specific session you created yourself.
        yield scrapy.Request(
            "https://example.com/basket",
            meta={"proxy": proxyempire.scrapy_proxy_url("48151623")},
        )

        # 3. Skip the proxy entirely, e.g. for your own internal API.
        yield scrapy.Request(
            "https://internal.example.net/health",
            meta={"proxy": None, "dont_retry": True},
        )

    def parse(self, response):
        jar = response.meta.get("cookiejar")
        if jar is not None:
            yield response.follow(
                "/prices",
                callback=self.parse_prices,
                meta={"cookiejar": jar},  # cookiejar is not sticky: pass it on
            )

    def parse_prices(self, response):
        yield {"url": response.url, "jar": response.meta["cookiejar"]}

Two caveats. The cookiejar meta key isn’t carried over automatically, so pass it on in every follow-up request. And with PROXYEMPIRE_STICKY_KEY = "cookiejar", requests without a jar all share the None key, and with it one session.

scrapy-playwright Setup and Launch-Level Proxy

Browser requests take a different path

scrapy-playwright is a Scrapy download handler that fetches a request with a real browser when the request’s meta contains "playwright": True. Other requests still use Scrapy’s normal HTTP handler, so you render only pages that need JavaScript. For background, see Headless Browsers 101.

Example 9 — install scrapy-playwright and a browser (shell)

python3 -m venv .venv && . .venv/bin/activate
pip install scrapy scrapy-playwright
playwright install chromium

Then register the handler and the asyncio reactor, which has been the TWISTED_REACTOR default since Scrapy 2.13.

Example 10 — scrapy-playwright settings with a launch-level proxy (Python)

# myproject/settings.py (scrapy-playwright parts)
from myproject import proxyempire

DOWNLOAD_HANDLERS = {
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    # Add "http" only if you must render plain-http:// pages in the browser.
}
# The default since Scrapy 2.13; set it explicitly anyway.
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

PLAYWRIGHT_BROWSER_TYPE = "chromium"
PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": True,
    # Launch-level proxy: every context that doesn't set its own uses it.
    "proxy": proxyempire.playwright_proxy(),  # rotating username, http://
}
PLAYWRIGHT_MAX_CONTEXTS = 8
PLAYWRIGHT_MAX_PAGES_PER_CONTEXT = 2
PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT = 45_000  # milliseconds


def should_abort_request(request):
    return request.resource_type in {"image", "media", "font"}


PLAYWRIGHT_ABORT_REQUEST = should_abort_request

This is why the middleware skips browser requests. The scrapy-playwright docs list it as a known issue: meta["proxy"] is not supported; the browser uses its launch or context proxy. Setting the key anyway is worse than useless. HttpProxyMiddleware would still add a Proxy-Authorization header to the Scrapy request, and scrapy-playwright’s default header handling copies Scrapy’s headers onto the browser’s navigation request, so your proxy credentials could be sent to the target site. The middleware also sets meta["proxy"] = None on browser requests, which stops an environment-variable proxy from adding the header.

Use http:// for the browser proxy. Playwright’s proxy option takes server, username, password and bypass. Chromium refuses username/password authentication on SOCKS5 proxies (“Browser does not support socks5 proxy authentication”), so point server at http://v2.proxyempire.io:5000. HTTPS sites still work: the browser tunnels them through the HTTP proxy with CONNECT.

A launch-level proxy applies to every context without its own. One page load opens many connections, and with a rotating username they can leave from different IPs; where that matters, give each context a sticky session. Aborting images, media and fonts with PLAYWRIGHT_ABORT_REQUEST cuts traffic per page.

Per-Context Proxies for Browser Sessions

One browser context, one sticky identity

A Playwright browser context is an isolated profile with its own cookies, storage and cache, and it can have its own proxy. scrapy-playwright gives you two ways to set one: PLAYWRIGHT_CONTEXTS for contexts created at startup, and the playwright_context_kwargs meta key for contexts created while crawling. A request picks its context by name with playwright_context. If no context with that name exists yet, it’s created from the request’s playwright_context_kwargs. If it already exists, the kwargs are ignored.

Example 11 — startup contexts and contexts created on the fly (Python)

# Option A: named sticky contexts created at startup
PLAYWRIGHT_CONTEXTS = {
    "default": {"proxy": proxyempire.playwright_proxy()},  # rotating
    "us-session-a": {
        "proxy": proxyempire.playwright_proxy("31415926"),
        "locale": "en-US",
    },
}

# Option B: create a context while crawling, from a spider callback
yield scrapy.Request(
    url,
    meta={
        "playwright": True,
        "playwright_context": "session-27182818",
        "playwright_context_kwargs": {
            "proxy": proxyempire.playwright_proxy("27182818"),
        },
    },
)

One context per sid gives each browser identity a consistent exit IP and its own cookies. Keep memory in check:

PLAYWRIGHT_MAX_CONTEXTS caps concurrent contexts. If you don’t close contexts you no longer need, the crawl blocks once the cap is reached. Close them from callbacks and errbacks.
PLAYWRIGHT_MAX_PAGES_PER_CONTEXT defaults to CONCURRENT_REQUESTS, which is far too many pages for one identity. Two to four is a saner start.
Persistent contexts (user_data_dir) are launched separately and don’t inherit PLAYWRIGHT_LAUNCH_OPTIONS, so give them their own proxy.

Past one machine, see the browser cluster design guide.

Mixing Plain and Browser Requests in One Spider

Render only what needs rendering

The common shape: listing pages over plain Scrapy, detail pages rendered in a few browser identities. Below, four identities each get a context with its own sticky sid; a 403 or 429 retires the identity and retries the URL with a new one.

Example 12 — a hybrid spider with rotating browser identities (Python)

import scrapy
from scrapy.downloadermiddlewares.retry import get_retry_request
from scrapy_playwright.page import PageMethod

from myproject import proxyempire

BLOCK_CODES = [403, 429]


class HybridSpider(scrapy.Spider):
    """Plain HTTP for listings, a browser only for pages that need JavaScript."""

    name = "hybrid"
    identities = 4  # browser identities = sticky sessions = contexts

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.sids = [proxyempire.new_sid() for _ in range(self.identities)]

    def browser_meta(self, slot):
        sid = self.sids[slot]
        return {
            "playwright": True,
            "playwright_include_page": True,
            "playwright_context": f"identity-{slot}-{sid}",
            "playwright_context_kwargs": {
                "proxy": proxyempire.playwright_proxy(sid),
                "locale": "en-US",
            },
            "playwright_page_methods": [
                PageMethod("wait_for_load_state", "networkidle"),
            ],
            "handle_httpstatus_list": BLOCK_CODES,  # let the callback see them
            "dont_retry": True,  # retried below, with a fresh identity
            "identity": (slot, sid),
        }

    async def start(self):
        # No "playwright" key: Scrapy's HTTP handler + ProxyEmpireMiddleware.
        yield scrapy.Request(
            "https://example.com/catalog?page=1",
            callback=self.parse_listing,
        )

    def parse_listing(self, response):
        # Placeholder selectors: inspect your own target and replace them.
        links = response.css("a.product-link::attr(href)").getall()
        for n, href in enumerate(links):
            yield response.follow(
                href,
                callback=self.parse_product,
                errback=self.on_browser_error,
                meta=self.browser_meta(n % self.identities),
            )
        next_page = response.css("a[rel=next]::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse_listing)

    async def parse_product(self, response):
        page = response.meta["playwright_page"]
        slot, sid = response.meta["identity"]

        if response.status in BLOCK_CODES:
            await page.close()
            await page.context.close()  # retire this identity
            if self.sids[slot] == sid:
                self.sids[slot] = proxyempire.new_sid()
            meta = {k: v for k, v in response.meta.items() if k != "playwright_page"}
            meta.update(self.browser_meta(slot))
            retry = get_retry_request(
                response.request.replace(meta=meta),
                spider=self,
                reason=f"HTTP {response.status}",
            )
            if retry is not None:
                yield retry
            return

        title = await page.title()
        await page.close()
        yield {"url": response.url, "title": title, "identity": slot}

    async def on_browser_error(self, failure):
        page = failure.request.meta.get("playwright_page")
        if page is not None and not page.is_closed():
            await page.close()
        self.logger.warning("Browser request failed: %r", failure.value)

How it fits together:

Two retry paths. Plain requests use RetryMiddleware and the session middleware. Browser requests set dont_retry and handle_httpstatus_list, so the callback sees a 403/429, closes the page and its context, and calls get_retry_request(). That helper still enforces RETRY_TIMES and keeps the retry stats.
Why not let RetryMiddleware retry browser pages? With playwright_include_page, the page object is stored in the request meta, and scrapy-playwright reuses a page passed in meta. A plain retry would load the URL again in the same page, the same context and the same IP.
Closing a context fails its other open pages, which then reach on_browser_error. Another reason to keep pages per context low.
Async callbacks. Callbacks may be async generators; start() must be one. Scrapy 2.19 removed start_requests(), so older examples that only define it no longer run.

For more patterns that combine requests, httpx and Playwright in Python, see Python web scraping with residential proxies.

Testing the Exit IP

Trust the IP the target sees, not the config

Before a long crawl, run a spider that does nothing but ask api.ipify.org which IP it sees, through both download paths:

Example 13 — compare exit IPs for plain and browser requests (Python)

import json

import scrapy

from myproject import proxyempire

IPIFY = "https://api.ipify.org?format=json"


class ExitIpSpider(scrapy.Spider):
    name = "exit_ip"

    async def start(self):
        for i in range(3):
            yield scrapy.Request(IPIFY, dont_filter=True, cb_kwargs={"via": "scrapy"})
        for i in range(3):
            sid = proxyempire.new_sid()
            yield scrapy.Request(
                IPIFY,
                dont_filter=True,
                cb_kwargs={"via": f"browser sid={sid}"},
                meta={
                    "playwright": True,
                    "playwright_context": f"check-{sid}",
                    "playwright_context_kwargs": {
                        "proxy": proxyempire.playwright_proxy(sid),
                    },
                },
            )

    def parse(self, response, via):
        # Chromium shows JSON inside a <pre> element; plain Scrapy gets raw JSON.
        raw = response.css("pre::text").get() or response.text
        yield {
            "via": via,
            "exit_ip": json.loads(raw)["ip"],
            "sid": response.meta.get("pe_sid"),
        }

Example 14 — run it and read the results (shell)

scrapy crawl exit_ip -O exit_ip.jsonl
cat exit_ip.jsonl

What to expect:

Sticky per domain: the three plain rows share one sid and one IP, because they hit the same host.
Rotating mode: the plain rows show sid: null and will usually show different IPs.
Browser rows: one IP per sid, set by each context’s proxy.

A row with your server’s own IP means that path isn’t proxied; for browser rows, usually because the proxy went into meta["proxy"].

Settings Reference

Defaults verified against Scrapy 2.19 and scrapy-playwright 0.0.48

Setting	Default	Starting point with proxies
ROBOTSTXT_OBEY	`False` (new projects: `True`)	`True`
HTTPPROXY_ENABLED	`True`	Leave on
HTTPPROXY_AUTH_ENCODING	`"latin-1"`	Leave unless credentials are non-ASCII
CONCURRENT_REQUESTS	16	16–64, limited by plan and memory
CONCURRENT_REQUESTS_PER_DOMAIN	8 (new projects: 1)	1–4 per site
DOWNLOAD_DELAY	0 (new projects: 1)	0.5–1 s floor
DOWNLOAD_TIMEOUT	180 s	30–60 s
AUTOTHROTTLE_ENABLED	`False`	`True`
AUTOTHROTTLE_START_DELAY	5.0	1–5
AUTOTHROTTLE_MAX_DELAY	60.0	60
AUTOTHROTTLE_TARGET_CONCURRENCY	1.0	1.0–2.0
RETRY_TIMES	2	2–3
RETRY_HTTP_CODES	500, 502, 503, 504, 522, 524, 408, 429	Default, plus 403 only with session rotation
TWISTED_REACTOR	Asyncio reactor (since 2.13)	Required by scrapy-playwright
PLAYWRIGHT_MAX_CONTEXTS	None (no limit)	4–8 per process
PLAYWRIGHT_MAX_PAGES_PER_CONTEXT	Value of `CONCURRENT_REQUESTS`	1–4
PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT	None (Playwright’s 30,000 ms)	30,000–60,000 ms

“New projects” are the values that scrapy startproject writes into settings.py, which override the framework defaults.

Troubleshooting

Symptom, cause, fix

Symptom	Likely cause	Fix
407 on http:// URLs, or `TunnelError` with status 407 on https://	Wrong or mis-encoded proxy credentials	Test with curl; copy again from the Connection Builder; `quote(..., safe="")` both parts
`TunnelError` without a status, or connect timeouts	Gateway unreachable: firewall, wrong host or port	Check `v2.proxyempire.io:5000` is reachable from the server
`NotImplementedError`: HTTPS proxies for HTTPS destinations	`https://` proxy URL with the default handler	Use an `http://` proxy URL
socks5:// proxy fails in a plain spider	The default handler has no SOCKS support	Use `http://`
“Browser does not support socks5 proxy authentication”	SOCKS5 with credentials in Playwright/Chromium	Use `http://` as the `server`
Browser pages show your own IP	Proxy set in `meta["proxy"]`, which scrapy-playwright ignores	Set it in launch options or the context
Logins or carts break between requests	Rotating mode, so the IP changes every request	Sticky per cookiejar or per context
Same IP for everything in rotating mode	A sticky username was copied as the rotating one	Re-copy with “new IP on every request”
Waves of 429s	Too much concurrency per domain	Lower `CONCURRENT_REQUESTS_PER_DOMAIN`, enable AutoThrottle
403 persists after new sessions	The content is not open to automated access	Stop retrying; check the site’s terms or an official API
Browser crawl hangs	Unclosed pages or contexts with `PLAYWRIGHT_MAX_CONTEXTS` set	Close pages in callbacks and contexts in errbacks
Spider starts and finishes with no requests	Only `start_requests()` defined, on Scrapy 2.19+	Define `async def start()`

Crawling Responsibly at Scale

More IPs, same manners

Residential proxies change where a crawl comes from, not what it’s allowed to do:

Obey robots.txt. Keep ROBOTSTXT_OBEY = True. The framework default is False, but new projects turn it on. RobotsTxtMiddleware fetches robots.txt through the same middleware chain, so it uses your proxy too.
Rate-limit per site with the per-domain settings above, whatever the size of the IP pool.
Prefer official APIs and data feeds where they exist, and collect only public data you have a legitimate reason to process.
Identify yourself through USER_AGENT where the use case allows it, and stop when a site clearly says no.

When one process isn’t enough, proxies become one layer of a system with a shared frontier and per-domain politeness across workers; the distributed crawler architecture guide picks up from there. To weigh crawling against targeted scraping first, see web crawler vs web scraper. If you’d rather not run the request layer yourself, the ProxyEmpire Scraping API is the managed alternative.

Frequently Asked Questions

Scrapy proxies, in short

How do I use a proxy in Scrapy?

Set request.meta["proxy"] to a URL such as http://USER:[email protected]:5000. The built-in HttpProxyMiddleware (priority 750) turns the credentials into a Proxy-Authorization header and, for HTTPS targets, sends it in the CONNECT request to the proxy. For a whole crawl through a single proxy, the HTTP_PROXY and HTTPS_PROXY environment variables also work.

What priority should a custom Scrapy proxy middleware have?

Lower than 750, so it sets meta["proxy"] before HttpProxyMiddleware runs. If it reacts to responses, also higher than RetryMiddleware’s 550, because process_response runs from high to low and the retry ends the chain. 740 satisfies both.

How do I rotate proxies in Scrapy?

With a gateway like ProxyEmpire’s, you don’t rotate a list of IPs yourself. Use a rotating username to get a new IP on every request, or a sticky username with a session ID (sid) to keep one IP, and change the sid when you want a new one. A downloader middleware can assign sessions per request, per domain or per cookiejar, and replace a session after a 403 or 429.

Does scrapy-playwright use meta[“proxy”]?

No. The scrapy-playwright docs list it as a known issue. Set the proxy in PLAYWRIGHT_LAUNCH_OPTIONS for the whole browser, in PLAYWRIGHT_CONTEXTS for named contexts, or in the playwright_context_kwargs meta key for contexts created while crawling.

Can I use a SOCKS5 proxy with Scrapy or scrapy-playwright?

Not with the defaults. Scrapy’s default HTTP/1.1 download handler doesn’t support SOCKS proxies, and Chromium refuses SOCKS5 proxies that need a username and password. Use the http:// gateway URL. HTTPS sites still work through CONNECT tunnels.

Should I add 403 and 407 to RETRY_HTTP_CODES?

Never 407: it means the proxy rejected your credentials, and a retry sends the same ones again. 403 can make sense if each retry gets a new proxy session and RETRY_TIMES stays low. If 403s continue across fresh sessions, the content isn’t open to automated access, and you should stop.

Do rotating proxies let me raise CONCURRENT_REQUESTS_PER_DOMAIN?

They shouldn’t. Scrapy’s download slots are per target domain, not per exit IP, and the site serves the same load whichever IPs it comes from. Keep per-domain concurrency modest and let AutoThrottle adjust delays from measured latency.

Run your Scrapy crawls through ProxyEmpire

One gateway for rotating or sticky residential, mobile and datacenter IPs over HTTP, HTTPS and SOCKS5, with country, region, city, ZIP, ISP and ASN targeting. Copy your username from the Connection Builder and plug it into the middleware above.

Start a $1.97 trial See pricing

ProxyEmpire proxy types and prices

Proxies for data collection at scale.

Residential, mobile, ISP and datacenter proxies for scraping, ad verification, SEO monitoring and multi-account work. Choose from 30M+ ethically sourced residential IPs and 4M+ 4G/5G mobile IPs in 170+ countries, target by country, city, ZIP, ISP or carrier, and connect over HTTP(S) or SOCKS5 with 99.9% uptime. Try residential and mobile proxies for $1.97.

🏘️ Rotating Residential Proxies

30M+ Premium Residential IPs
170+ Countries
Every residential IP in our network corresponds to an actual desktop device with a precise geographical location. Our residential proxies are fast and reliable, with 99.9% uptime, and work for a wide range of use cases. You can use country, region, city, ZIP, ISP, ASN and OS fingerprint targeting for our rotating residential proxies, at no extra charge.

See our Rotating Residential Proxies

📍 Static Residential Proxies

24 Countries
Buy a dedicated static residential IP from one of the 24 countries that we offer proxies in. Keep the same IP for a month or longer, while benefiting from their fast speed and stability.

See our Static Residential Proxies

📳 Rotating Mobile Proxies

4M+ Premium Mobile IPs
170+ Countries
Access millions of clean mobile IPs with precise targeting by country, region, city, ZIP, mobile carrier, ASN and OS fingerprint, at no extra charge. Get far fewer IP blocks and CAPTCHAs with our 4G and 5G proxies.

See our Mobile Proxies

📱 Dedicated Mobile Proxies

4 Countries
50+ Locations
Get your own dedicated mobile proxy in one of our supported locations, with unlimited bandwidth and unlimited IP changes on demand. A great choice when you need a small number of mobile IPs and a lot of proxy bandwidth.

See our 4G & 5G Proxies

🌐 Rotating Datacenter Proxies

197,000+ IPs Premium IPs
62 Countries
On a budget and need to do some simple scraping tasks? Our datacenter proxies are the perfect fit! Get started with as little as $2

See our Datacenter Proxies

♾️ Unlimited Residential Proxies

1M+ Daily Residential IPs
85+ Countries
Our unlimited residential proxies bill by connection speed, not gigabytes: 100 to 1000 Mbit with unlimited bandwidth, on daily, weekly or monthly plans. Ask us for a 1-hour trial at 50 Mbit before you pick a plan.

See our Unlimited Residential Proxies

30M+ rotating IPs

See all locations

99.9% uptime.

Try for $1.97 only!

24/7 Dedicated Support.

Fair Pricing.