Crawlee Tutorial: Web Scraping with Proxies in Node.js and Python (2026)

Developer guide · Last reviewed 20 October 2026 · 15 min read

Crawlee is Apify’s open-source crawling and scraping library, available for Node.js/TypeScript and for Python. It ships what every serious crawler needs: a request queue, retries, a session pool, proxy rotation, autoscaled concurrency and result storage. This Crawlee tutorial shows how those parts fit together, then wires them to a real proxy gateway: rotating IPs for stateless pages, sticky sessions where one Crawlee session maps to one proxy session, tiered proxies from datacenter to mobile, and block handling that doesn’t burn bandwidth.

The short version

Create a ProxyConfiguration and pass it to any Crawlee crawler. Use proxyUrls (proxy_urls in Python) with one rotating gateway URL for stateless pages. For stateful flows, use newUrlFunction (new_url_function) so each Crawlee session gets its own sticky proxy session ID and keeps its cookies and IP together. Use tieredProxyUrls (tiered_proxy_urls) to start on cheap IPs and escalate only for domains that block. Always use the http:// gateway URL with browser crawlers, cap concurrency with maxConcurrency and maxRequestsPerMinute, and let the session pool retire identities that get 401, 403 or 429.

Versions used here. JavaScript examples target Crawlee 3.18, the current stable npm release (Node.js 16+). A v4 release candidate exists under the rc tag and needs Node.js 22.13+; check its changelog before upgrading. Python examples target Crawlee for Python 1.11 (Python 3.10+). The example target, books.toscrape.com, is a public sandbox built for scraping practice.

Key takeaways

  • ProxyConfiguration works the same way in every crawler class, HTTP or browser, in both languages.
  • A Crawlee session should map to exactly one proxy session. newUrlFunction is the cleanest way to do that with a sticky gateway.
  • Tiered proxies let Crawlee use the cheapest pool that works, per domain, and step back down when blocking stops.
  • Use the http:// gateway with Playwright. Chromium rejects username and password authentication on SOCKS5 proxies.
  • 401, 403 and 429 retire the session by default. Treat a 200 with the wrong content as a block too.
  • Pair maxConcurrency with a per-minute request budget.

What Crawlee Is, and When It Fits

A crawling framework, not just a parser

Crawlee is an open-source library from Apify for crawling and scraping websites, licensed under Apache 2.0. There are two implementations: Crawlee for JavaScript/TypeScript on npm, and Crawlee for Python on PyPI. They share concepts and most option names, translated to each language’s conventions, so maxRequestRetries in Node.js is max_request_retries in Python.

The value is in what you don’t have to write: URL frontier, deduplication, retries, proxy rotation, cookie handling, concurrency control and output. Crawlee ships all of them behind one crawler interface, and switching from HTTP fetching to a headless browser means changing the crawler class, not rewriting the job. If the distinction between crawling and extraction is new to you, our explainer on web crawlers vs web scrapers covers it.

It fits jobs that span many pages, such as catalogs, listings or scheduled monitoring runs. For a handful of fixed URLs, a plain HTTP client is simpler.

Crawler classes

Every crawler extends a basic crawler that owns the queue, retries, sessions and autoscaling. The subclasses differ in how they fetch and parse a page.

CrawlerLanguageUse it when
CheerioCrawlerNode.jsServer-rendered HTML; fast, no browser
HttpCrawlerBothYou want the raw body, such as JSON endpoints
BeautifulSoupCrawlerPythonServer-rendered HTML with a BeautifulSoup API
ParselCrawlerPythonServer-rendered HTML with CSS and XPath selectors
PlaywrightCrawlerBothContent needs JavaScript or interaction
AdaptivePlaywrightCrawlerBothA site mixes static and rendered pages; it learns which need a browser

Node.js also has JSDOMCrawler and PuppeteerCrawler. The adaptive crawler is marked experimental in the JavaScript API.

How Crawlee Works: Queue, Crawler, Handlers, Dataset

Where the proxy plugs in

A Crawlee run is a loop around a request queue seeded with start URLs. The crawler takes a request, picks a session from the session pool, asks the ProxyConfiguration for a proxy URL for that session, fetches the page, and calls your handler. Your handler extracts data into a dataset and adds new URLs to the queue with enqueueLinks(). Meanwhile the autoscaled pool decides how many requests run at once.

Figure 1 — Crawlee’s moving parts and where the proxy sits
 start URLs / enqueueLinks()
            |
            v
 +----------------------+       +---------------------------+
 |    Request queue     |       |  AutoscaledPool           |
 |  (dedupes by URL,    |<------|  scales concurrency from   |
 |   persists to disk)  |       |  CPU, memory, event loop  |
 +----------------------+       +---------------------------+
            |
            v
 +----------------------+       +---------------------------+
 |  Crawler             |------>|  Session pool             |
 |  Cheerio / HTTP /    |       |  session = cookies +      |
 |  Playwright / ...    |       |  error score + proxy URL  |
 +----------------------+       +---------------------------+
            |                                |
            |   ProxyConfiguration.newUrl(session.id)
            v                                v
 +----------------------+       +---------------------------+
 |  Your handlers       |       |  Proxy gateway            |
 |  router / labels     |       |  v2.proxyempire.io:5000   |
 +----------------------+       +---------------------------+
            |
            v
   Dataset (rows)  +  Key-value store (files, state)
Request queueHolds pending URLs, dedupes them by unique key and stores state under ./storage by default.
Session poolA set of identities. Each session carries cookies, an error score and a usage count, and is retired when it is blocked, too old or overused.
Autoscaled poolRaises or lowers concurrency between your minimum and maximum based on CPU, memory and event-loop load.
StorageDatasets for rows of results, key-value stores for files and state. Both write to ./storage locally.

The proxy decision happens per session, not per crawler. If a session should look like one consistent visitor, its proxy URL must keep the same exit IP for as long as the session lives.

Install Crawlee and Test the Proxy

Verify the gateway before writing a crawler

Test the credentials first. If this curl command prints an IP address, the proxy side is working, and anything that fails later is in the crawler. In the ProxyEmpire dashboard, open Proxy Manager, then Connection Builder, pick the product, location and rotation mode, and copy or test the proxy string there.

Example 1 — check the gateway from the shell (bash)
curl -x http://USERNAME:[email protected]:5000 https://api.ipify.org

For more proxy flags, see our guide to using curl with a proxy. Then install Crawlee. The Node.js CLI scaffolds a project from a template; manual installation adds Crawlee and Playwright to an existing project. Playwright isn’t bundled with Crawlee, so install it and its browser yourself.

Example 2 — install Crawlee for Node.js (bash)
# Option A: scaffold a project from a template
npx crawlee create my-crawler
cd my-crawler && npm start

# Option B: add Crawlee to an existing project
npm install crawlee playwright
npx playwright install chromium
Example 3 — install the pip package or use the CLI (bash)
# Crawlee for Python needs Python 3.10 or newer
python -m pip install 'crawlee[beautifulsoup,playwright]'
playwright install chromium

# Or scaffold a project with the CLI (uses uv)
uvx 'crawlee[cli]' create my-crawler

Optional features live in extras: beautifulsoup and playwright cover this guide, parsel and adaptive-crawler add the others, and all installs everything.

Node.js: CheerioCrawler with ProxyConfiguration

A rotating gateway in one option

Start with the simplest useful setup: a CheerioCrawler that pages through a catalog, with every request going out through the ProxyEmpire gateway. Choose “new IP on every request” in the Connection Builder, then keep the username and password in environment variables rather than in code.

Example 4 — CheerioCrawler through a rotating gateway (TypeScript)
import { CheerioCrawler, ProxyConfiguration } from 'crawlee';

// Copy the username and password from Proxy Manager -> Connection Builder
// (rotation mode: new IP on every request).
const user = encodeURIComponent(process.env.PROXYEMPIRE_USERNAME ?? '');
const pass = encodeURIComponent(process.env.PROXYEMPIRE_PASSWORD ?? '');

const proxyConfiguration = new ProxyConfiguration({
    proxyUrls: [`http://${user}:${pass}@v2.proxyempire.io:5000`],
});

const crawler = new CheerioCrawler({
    proxyConfiguration,
    maxConcurrency: 10,          // hard ceiling for parallel requests
    maxRequestsPerMinute: 120,   // politeness budget for the whole crawler
    maxRequestRetries: 3,
    maxRequestsPerCrawl: 100,    // safety net while developing
    respectRobotsTxtFile: true,

    async requestHandler({ request, $, enqueueLinks, pushData, proxyInfo, log }) {
        const books = $('article.product_pod')
            .map((_, el) => ({
                title: $(el).find('h3 a').attr('title'),
                price: $(el).find('.price_color').text().trim(),
            }))
            .get();

        log.info(`${request.loadedUrl}: ${books.length} books via ${proxyInfo?.hostname}`);
        await pushData(books.map((book) => ({ ...book, page: request.loadedUrl })));

        // Follow pagination; enqueueLinks() deduplicates requests.
        // The request queue prevents repeat URLs.
        await enqueueLinks({ selector: 'li.next a' });
    },

    failedRequestHandler({ request, log }, error) {
        log.error(`Gave up on ${request.url}: ${error.message}`);
    },
});

await crawler.run(['https://books.toscrape.com/']);
await crawler.exportData('./books.csv');

encodeURIComponent protects credentials that contain URL-special characters. proxyInfo tells you which proxy served the request, the first thing to log when debugging. pushData() writes to the default dataset under ./storage/datasets/default, and exportData() writes that dataset out as CSV or JSON at the end.

With one rotating URL, every request gets a new exit IP while Crawlee’s sessions still hold cookies. That suits stateless pages; when a site ties cookies to the IP that received them, use the sticky setup below.

Use http://, not socks5://, with browser crawlers. ProxyEmpire supports HTTP, HTTPS and SOCKS5, but Chromium rejects username and password authentication on SOCKS5 proxies (“Browser does not support socks5 proxy authentication”). Crawlee’s Node.js browser pool hands authenticated proxies to the browser through a small local forwarding proxy, and Crawlee for Python passes the credentials to Playwright’s context proxy settings. The http:// gateway URL works the same way in both, and it also works for the HTTP crawlers. Our comparison of SOCKS vs HTTP proxies covers the protocol differences.

Sticky Sessions: One Crawlee Session, One Proxy Session

Keep cookies and IP together

A Crawlee session is the crawler’s idea of one visitor: a cookie jar, an error score and a usage counter. A sticky proxy session is the provider’s idea of one visitor: the same exit IP for a period of time. When the two drift apart, a session’s cookies arrive from a new IP on every request, or two sessions share one IP.

The fix is a one-to-one mapping. ProxyEmpire’s Same IP (sticky) mode puts a session ID, sid, into the proxy username, and you can create your own 8-digit session IDs. Give every Crawlee session its own 8-digit sid and the session lifecycle becomes your IP lifecycle: when the pool retires a session after a 403, the next one gets a new sid and a new IP.

Figure 2 — the mapping you want
Crawlee session A  -->  sid 1xxxxxxx  -->  exit IP A   (cookies A)
Crawlee session B  -->  sid 2xxxxxxx  -->  exit IP B   (cookies B)
Crawlee session C  -->  sid 3xxxxxxx  -->  exit IP C   (cookies C)

session B retired (403)  -->  session D  -->  new sid  -->  new exit IP

Build the username from a template

Don’t hand-write the username format. In the Connection Builder, select Same IP rotation with your location settings and copy the generated username. Replace its session ID with the literal token {sid} and store the result in PROXYEMPIRE_STICKY_TEMPLATE; leave everything else as generated. The sticky sessions documentation shows where the session ID sits.

newUrlFunction receives the Crawlee session ID and returns a proxy URL. Crawlee may call it for every request, so cache the sid per session.

Example 5 — PlaywrightCrawler with one sticky proxy session per Crawlee session (TypeScript)
import { randomInt } from 'node:crypto';
import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';

// A sticky ("Same IP") username copied from the Connection Builder, with its
// session ID replaced by the literal token {sid}. Do not hand-build this string.
const TEMPLATE = process.env.PROXYEMPIRE_STICKY_TEMPLATE ?? '';
const PASSWORD = encodeURIComponent(process.env.PROXYEMPIRE_PASSWORD ?? '');

// One Crawlee session id -> one 8-digit ProxyEmpire session id.
const sidBySession = new Map<string, string>();

function proxyUrlFor(sessionId: string | number): string {
    const key = String(sessionId);
    let sid = sidBySession.get(key);
    if (!sid) {
        sid = String(randomInt(10_000_000, 100_000_000)); // always 8 digits
        sidBySession.set(key, sid);
    }
    const username = encodeURIComponent(TEMPLATE.replace('{sid}', sid));
    return `http://${username}:${PASSWORD}@v2.proxyempire.io:5000`;
}

const crawler = new PlaywrightCrawler({
    proxyConfiguration: new ProxyConfiguration({
        newUrlFunction: (sessionId) => proxyUrlFor(sessionId),
    }),
    useSessionPool: true,            // default, shown for clarity
    persistCookiesPerSession: true,  // cookies stay with their session (and IP)
    sessionPoolOptions: {
        maxPoolSize: 10,             // at most 10 identities in rotation
        sessionOptions: {
            maxUsageCount: 30,       // retire an identity after 30 requests
            maxAgeSecs: 4 * 60,      // example: below a 5m sticky TTL
        },
    },
    maxConcurrency: 5,
    navigationTimeoutSecs: 45,

    async requestHandler({ page, request, session, proxyInfo, pushData, log }) {
        // Resolve the sticky sid from the Crawlee session ID.
        // Each session remains paired with its cached proxy ID.
        const sid = proxyInfo?.sessionId ? sidBySession.get(proxyInfo.sessionId) : undefined;
        await page.waitForSelector('article.product_pod');
        const titles = await page
            .locator('article.product_pod h3 a')
            .evaluateAll((links) => links.map((a) => a.getAttribute('title')));

        log.info(`${request.url} | crawlee session=${session?.id} | proxy sid=${sid}`);
        await pushData({ url: request.url, titles, session: session?.id, sid });
    },
});

await crawler.run(
    Array.from({ length: 5 }, (_, i) => `https://books.toscrape.com/catalogue/page-${i + 1}.html`),
);

maxPoolSize caps how many identities, and so sticky IPs, exist at once. maxAgeSecs should be shorter than the rotation time you chose in the Connection Builder (up to 2 hours), so Crawlee retires the session before the IP changes underneath it. Crawlee’s own defaults are a 3,000-second maximum age, 50 uses per session and a pool of 1,000 sessions.

In the Node.js browser crawler, retiring a session also retires the browser that was launched for it, so the old cookies and fingerprint go with it. For the same pattern in Puppeteer without Crawlee, see our guide to Puppeteer proxies with authentication. For a broader comparison of the two rotation styles, read sticky vs rotating proxies.

No template? Use an exported proxy list

The Connection Builder can also export many sticky sessions at once in host:port:user:pass format. Convert each line to a URL and pass the list as proxyUrls. Crawlee pairs each session ID with one URL from the list the first time it sees it, assigning them round-robin, and keeps that pairing for as long as the session lives.

Example 6 — sticky sessions from an exported list (TypeScript)
import { readFileSync } from 'node:fs';
import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';

// proxies.txt: sticky sessions exported from the Connection Builder,
// one per line, in host:port:user:pass format.
const proxyUrls = readFileSync('proxies.txt', 'utf8')
    .split('\n')
    .map((line) => line.trim())
    .filter(Boolean)
    .map((line) => {
        const [host, port, user, pass] = line.split(':');
        return `http://${encodeURIComponent(user)}:` +
            `${encodeURIComponent(pass)}@${host}:${port}`;
    });

const crawler = new PlaywrightCrawler({
    proxyConfiguration: new ProxyConfiguration({ proxyUrls }),
    sessionPoolOptions: { maxPoolSize: proxyUrls.length },
    // Keep Crawlee sessions at or below exported sticky IPs.
    async requestHandler({ request, session, log }) {
        log.info(`${request.url} handled by ${session?.id}`);
    },
});

Keep maxPoolSize at or below the number of lines. If there are more sessions than URLs, the round-robin assignment puts two sessions on the same sticky IP. Our article on proxy pools and rotation strategies covers sizing in more depth.

ProxyEmpire rotation modeCrawlee setupGood for
New IP on every requestproxyUrls with one gateway URLStateless listings and product pages
Same IP (sticky)newUrlFunction mapping session to sidCookie-bound flows, multi-step navigation
Custom rotation timeSticky, with maxAgeSecs below the rotation timeLonger browsing sessions
Exported proxy listproxyUrls with one URL per lineA fixed set of identities

Tiered Proxies: Datacenter, Residential, Mobile

Pay for the expensive IPs only where you need them

Some sites serve datacenter IPs without complaint, some need residential, a few only behave for mobile carrier IPs. tieredProxyUrls lets Crawlee work that out per domain. It starts on tier 0. When a request to a domain fails and is retried, Crawlee records an error against the tier it used for that domain and moves that domain up a tier. Each error penalizes the tier for roughly the next ten requests to the domain, after which Crawlee tries the lower tier again and stays there if it works.

Example 7 — escalate from datacenter to residential to mobile (TypeScript)
import { CheerioCrawler, ProxyConfiguration } from 'crawlee';

// Each URL is a full http:// proxy URL built from the Connection Builder
// for that product, in rotating (new IP per request) mode.
const proxyConfiguration = new ProxyConfiguration({
    tieredProxyUrls: [
        [process.env.PROXYEMPIRE_DATACENTER_URL!],   // tier 0: DC
        // Cheapest route; tried first.
        [process.env.PROXYEMPIRE_RESIDENTIAL_URL!],  // tier 1
        // Used when the datacenter tier is blocked.
        [process.env.PROXYEMPIRE_MOBILE_URL!],       // tier 2: last resort
    ],
});

const crawler = new CheerioCrawler({
    proxyConfiguration,
    async requestHandler({ request, proxyInfo, log }) {
        log.info(`${request.url} served through tier ${proxyInfo?.proxyTier}`);
    },
});

Build each tier’s URL in the Connection Builder for the relevant product, datacenter, residential or mobile, in rotating mode. You can also put null in a tier to mean “no proxy”, which is useful as a free tier 0 for your own sites or for targets that don’t rate-limit.

Three caveats come from how the feature is implemented. Tiers only work when the configuration is used by a crawler, because the tier choice depends on the request; calling newUrl() directly won’t escalate. In Node.js, URLs within a tier rotate round-robin per request instead of staying paired to sessions (Crawlee for Python pins a session to the first URL it received), so use rotating usernames for tier URLs, not sticky ones. And tieredProxyUrls, proxyUrls and newUrlFunction can’t be combined in one configuration. If you need sticky sessions and escalation together, implement both inside your own newUrlFunction. For the cost and detection trade-offs between the pools, see datacenter vs residential proxies.

Crawlee for Python with Proxies

The same model, async and typed

Crawlee for Python is built on asyncio with the same building blocks: ProxyConfiguration (proxy_urls, tiered_proxy_urls or new_url_function), a session pool, autoscaling and local storage. Handlers are registered on the router with decorators, and HTTP crawlers fetch with ImpitHttpClient by default.

BeautifulSoupCrawler through a rotating gateway

Example 8 — BeautifulSoupCrawler with ProxyConfiguration (Python)
import asyncio
import os
from urllib.parse import quote

from crawlee import ConcurrencySettings
from crawlee.crawlers import (
    BasicCrawlingContext,
    BeautifulSoupCrawler,
    BeautifulSoupCrawlingContext,
)
from crawlee.proxy_configuration import ProxyConfiguration

USER = quote(os.environ['PROXYEMPIRE_USERNAME'], safe='')
PASSWORD = quote(os.environ['PROXYEMPIRE_PASSWORD'], safe='')
PROXY_URL = f'http://{USER}:{PASSWORD}@v2.proxyempire.io:5000'


async def main() -> None:
    crawler = BeautifulSoupCrawler(
        proxy_configuration=ProxyConfiguration(proxy_urls=[PROXY_URL]),
        concurrency_settings=ConcurrencySettings(
            max_concurrency=10,
            desired_concurrency=5,  # must not exceed max_concurrency
            max_tasks_per_minute=120,
        ),
        max_request_retries=3,
        max_requests_per_crawl=100,
        respect_robots_txt_file=True,
    )

    @crawler.router.default_handler
    async def handle_listing(context: BeautifulSoupCrawlingContext) -> None:
        for book in context.soup.select('article.product_pod'):
            link = book.select_one('h3 a')
            price = book.select_one('.price_color')
            await context.push_data({
                'title': link.get('title') if link else None,
                'price': price.get_text(strip=True) if price else None,
                'page': context.request.url,
            })
        await context.enqueue_links(selector='li.next a')

    @crawler.failed_request_handler
    async def on_failed(context: BasicCrawlingContext, error: Exception) -> None:
        context.log.error(f'Gave up on {context.request.url}: {error}')

    await crawler.run(['https://books.toscrape.com/'])
    await crawler.export_data('books.csv')


if __name__ == '__main__':
    asyncio.run(main())

One Python-specific trap: ConcurrencySettings has a desired_concurrency default of 10 and raises a ValueError if that is higher than max_concurrency. If you set max_concurrency below 10, set desired_concurrency as well. max_tasks_per_minute is the Python name for the per-minute budget that Node.js calls maxRequestsPerMinute.

PlaywrightCrawler with sticky sessions

The Python version of the session-to-sid mapping looks almost the same. new_url_function receives the session ID and the request, and can return a string or an awaitable.

Example 9 — PlaywrightCrawler with one sticky proxy session per Crawlee session (Python)
import asyncio
import os
import secrets
from datetime import timedelta
from urllib.parse import quote

from crawlee import ConcurrencySettings, Request
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
from crawlee.errors import SessionError
from crawlee.proxy_configuration import ProxyConfiguration
from crawlee.sessions import SessionPool

# Sticky username from the Connection Builder: replace the
# session ID with the literal token {sid}.
TEMPLATE = os.environ['PROXYEMPIRE_STICKY_TEMPLATE']
PASSWORD = quote(os.environ['PROXYEMPIRE_PASSWORD'], safe='')

_sid_by_session: dict[str, str] = {}


def sticky_proxy_url(session_id: str | None = None,
                     request: Request | None = None) -> str:
    key = session_id or 'no-session'
    if key not in _sid_by_session:
        _sid_by_session[key] = str(
            10_000_000 + secrets.randbelow(90_000_000))  # 8 digits
    username = quote(TEMPLATE.replace('{sid}', _sid_by_session[key]), safe='')
    return f'http://{username}:{PASSWORD}@v2.proxyempire.io:5000'


async def main() -> None:
    crawler = PlaywrightCrawler(
        proxy_configuration=ProxyConfiguration(new_url_function=sticky_proxy_url),
        # Without this, pages in one browser share a context and its proxy.
        use_incognito_pages=True,
        session_pool=SessionPool(
            max_pool_size=10,
            create_session_settings={
                'max_usage_count': 30,
                'max_age': timedelta(minutes=4),
            },
        ),
        concurrency_settings=ConcurrencySettings(max_concurrency=5, desired_concurrency=2),
        max_session_rotations=5,
        max_requests_per_crawl=50,
        headless=True,
    )

    @crawler.router.default_handler
    async def handler(context: PlaywrightCrawlingContext) -> None:
        cards = context.page.locator('article.product_pod')
        if await cards.count() == 0:
            raise SessionError(f'Expected listing markup missing on {context.request.url}')

        titles = await cards.locator('h3 a').evaluate_all(
            'links => links.map(a => a.getAttribute("title"))'
        )
        session_id = context.session.id if context.session else None
        await context.push_data({
            'url': context.request.url,
            'titles': titles,
            'session': session_id,
            'sid': _sid_by_session.get(session_id or ''),
        })

    await crawler.run([
        f'https://books.toscrape.com/catalogue/page-{n}.html' for n in range(1, 6)
    ])


if __name__ == '__main__':
    asyncio.run(main())

Note use_incognito_pages=True. In Crawlee for Python, pages in one browser share a single browser context by default, and that context is created with the proxy of the first page that opened it. With incognito pages, each page gets its own context with the proxy for its own session, which is what the sticky mapping needs. The session settings are passed through create_session_settings, with max_age as a timedelta.

Tiered proxies use the same shape as in Node.js:

Example 10 — tiered proxies in Python (fragment)
proxy_configuration = ProxyConfiguration(
    tiered_proxy_urls=[
        [os.environ['PROXYEMPIRE_DATACENTER_URL']],
        [os.environ['PROXYEMPIRE_RESIDENTIAL_URL']],
        [os.environ['PROXYEMPIRE_MOBILE_URL']],
    ],
)

If you are weighing Crawlee against other Python stacks, our guide to Python web scraping with residential proxies shows the same proxy wired into requests, httpx, Scrapy and Playwright.

Handling Blocked Responses and Retries

Rotate identity, or stop asking

Crawlee has two retry budgets, and knowing which one an error spends explains most retry behavior.

maxRequestRetries (default 3)Ordinary failures: timeouts, server errors and exceptions thrown from your handler.
maxSessionRotations (default 10)Errors of type SessionError, including proxy connection errors: the request is retried with a different session and does not use up the ordinary retry budget.
Blocked status codes401, 403 and 429 retire the session by default. Change the list with blockedStatusCodes in the session pool options.
Soft blocksA 200 that contains an interstitial or an empty shell. Crawlee can’t know; your handler should check for expected content and throw SessionError.

The languages differ at the edges. In Python, a blocked status code raises a SessionError, and other 4xx errors are final, not retried. In Node.js, a blocked status code retires the session and the retry counts against maxRequestRetries. Either way the next attempt uses a fresh session and, with the sticky mapping above, a fresh IP.

Example 11 — soft-block detection, retry logging and a failed-requests dataset (TypeScript)
import { CheerioCrawler, Dataset, ProxyConfiguration, SessionError } from 'crawlee';

const failed = await Dataset.open('failed-requests');

const crawler = new CheerioCrawler({
    proxyConfiguration: new ProxyConfiguration({
        proxyUrls: [process.env.PROXYEMPIRE_URL!],
    }),
    maxRequestRetries: 3,      // timeouts, 5xx and handler errors
    // Ordinary retry budget.
    maxSessionRotations: 5,    // retry with new identities
    // Counted separately from normal request retries.
    sessionPoolOptions: {
        blockedStatusCodes: [401, 403, 429],  // defaults
        // These status codes retire the session.
    },

    async requestHandler({ $, request, pushData }) {
        // A 200 can still be an interstitial, a consent wall or an empty shell.
        if ($('article.product_pod').length === 0) {
            // Crawlee retires the current session and retries with another one.
            throw new SessionError(`Expected listing markup missing on ${request.url}`);
        }
        await pushData({ url: request.url, count: $('article.product_pod').length });
    },

    // Runs before each retry.
    async errorHandler({ request, log }, error) {
        log.warning(`Retry ${request.retryCount} for ${request.url}: ${error.message}`);
    },

    // Runs once retries are exhausted. Keep these for a later re-run.
    async failedRequestHandler({ request }, error) {
        await failed.pushData({
            url: request.url,
            retries: request.retryCount,
            errors: request.errorMessages,
            last: error.message,
        });
    },
});

await crawler.run(['https://books.toscrape.com/']);

errorHandler runs before every retry, the right place for logging and metrics. failedRequestHandler runs once retries are exhausted; a separate dataset gives you a clean list to re-run. In Python, the same hooks are the @crawler.error_handler and @crawler.failed_request_handler decorators. For what each status code usually means on a proxied request, keep our proxy error codes reference handy.

A retry should change something: the session, the IP, the timing. Retrying the same request from the same identity just repeats the failure.

Keep block handling proportionate. A rising 429 rate means the target wants fewer requests: lower the per-minute budget rather than adding IPs. A CAPTCHA, a login wall or a terms-of-service restriction is a signal to stop and check whether you have permission or an official API. Crawlee and a proxy network are for collecting public data reliably, not for getting past access controls.

Concurrency Limits and Politeness

Autoscaling decides how much your machine can take, not the target

Crawlee’s autoscaled pool starts at a desired concurrency and adjusts between minConcurrency and maxConcurrency based on the host’s CPU, memory and event-loop load. That protects your machine. It knows nothing about the target’s capacity or your proxy budget, so set those limits explicitly.

ControlNode.jsPython
Parallel requests ceilingmaxConcurrency (default 200)ConcurrencySettings(max_concurrency=...) (default 100)
Requests per minutemaxRequestsPerMinutemax_tasks_per_minute
Delay per domainsameDomainDelaySecsUse the per-minute budget
Stop after N pagesmaxRequestsPerCrawlmax_requests_per_crawl
robots.txtrespectRobotsTxtFilerespect_robots_txt_file
Identities in rotationsessionPoolOptions.maxPoolSizeSessionPool(max_pool_size=...)

Start with a low maxConcurrency and a per-minute budget you could defend to the site owner, measure success rate and latency, then raise one limit at a time. Browser crawlers need far lower ceilings than HTTP crawlers because each page is a real Chromium tab. When one machine is no longer enough, the problems change; our write-ups on designing a browser cluster for 10,000 concurrent Chromium sessions and distributed crawlers for billion-page datasets cover that stage.

Concurrency and IP count are separate. Ten parallel requests through a rotating gateway can leave from ten IPs; through one sticky session they all leave from one. With sticky sessions, load per IP is roughly concurrency divided by pool size, so a small pool with high concurrency concentrates traffic on a few IPs.

Crawlee vs Scrapy vs Plain Playwright

Pick by the work you’d otherwise write yourself
CapabilityCrawleeScrapyPlaywright alone
LanguagesJS/TS and PythonPythonJS/TS, Python, Java, .NET
HTTP and browserSame interface, swap the classHTTP; browsers via plugins such as scrapy-playwrightBrowser only
URL queue and dedupeBuilt inBuilt in (scheduler)You build it
Proxy rotationProxyConfiguration, incl. tiersPer request via meta['proxy']; rotation via middlewarePer browser or context; rotation is yours
Session to proxy bindingBuilt in (session pool)Custom middlewareYou build it
Concurrency controlAutoscaled by system loadSettings plus AutoThrottleYou build it
Result storageDatasets, key-value storesItem pipelines, feed exportsYou build it

Choose Scrapy for a Python-only team with mostly server-rendered targets and a need for its middleware ecosystem. Choose plain Playwright for short scripted flows, such as a screenshot job, where a framework is overhead; our headless browsers primer covers that ground. Choose Crawlee when a crawl mixes static and JavaScript-heavy pages, or when you want sessions and proxies managed together in Node.js or Python.

Common Crawlee Proxy Mistakes

Most of them are mismatched lifecycles
  1. Sticky username with a static sid for every session. All sessions share one IP and their cookies collide. Generate one sid per Crawlee session.
  2. newUrlFunction that returns a new sid on every call. It may run per request, so the IP rotates mid-session. Cache by session ID.
  3. Session lifetime longer than the proxy rotation time. The IP changes under a live session. Keep maxAgeSecs below the rotation time you chose.
  4. socks5:// URLs with Playwright. Chromium rejects SOCKS5 authentication. Use the http:// gateway.
  5. Python Playwright with shared contexts. Without use_incognito_pages=True, pages in a browser reuse the first context’s proxy.

Frequently Asked Questions

Crawlee and proxies, in short
What is Crawlee?

Apify’s open-source crawling and scraping library for Node.js/TypeScript and Python. It bundles a request queue, retries, a session pool, proxy rotation, autoscaled concurrency and storage, with HTTP crawlers such as CheerioCrawler and BeautifulSoupCrawler and browser crawlers such as PlaywrightCrawler.

How do I use a proxy with Crawlee?

Create a ProxyConfiguration with proxyUrls (proxy_urls in Python) and pass it to the crawler as proxyConfiguration (proxy_configuration). Every request then goes through those proxies, and the handler’s proxyInfo shows which one was used.

How do Crawlee sessions relate to sticky proxy sessions?

A Crawlee session is one visitor identity with its own cookies, and it should map to exactly one sticky proxy session. In newUrlFunction, generate one 8-digit sid per Crawlee session and build the username from a template copied from the Connection Builder. A retired session’s replacement gets a new sid and IP.

What are tiered proxies in Crawlee?

tieredProxyUrls (tiered_proxy_urls) is a list of proxy tiers, such as datacenter, residential and mobile. Crawlee starts each domain on the lowest tier, moves up when requests to that domain fail and are retried, and periodically tries lower tiers again. It only works when the configuration is used inside a crawler.

Can I use a SOCKS5 proxy with Crawlee’s PlaywrightCrawler?

Use the http:// gateway URL instead. Chromium rejects username and password authentication on SOCKS5 proxies, and HTTP proxies tunnel HTTPS sites without problems.

What Python version does Crawlee for Python need?

Crawlee for Python 1.11 requires Python 3.10 or newer. Install it with pip install 'crawlee[beautifulsoup,playwright]' or scaffold a project with uvx 'crawlee[cli]' create my-crawler.

Run your Crawlee crawlers through ProxyEmpire

One gateway for rotating or sticky residential, mobile and datacenter IPs, with country, region, city, ZIP, ISP and ASN targeting, ready for ProxyConfiguration in Node.js and Python.

ProxyEmpire proxy types and prices

ProxyEmpire

Proxies for data collection at scale.

Residential, mobile, ISP and datacenter proxies for scraping, ad verification, SEO monitoring and multi-account work. Choose from 30M+ ethically sourced residential IPs and 4M+ 4G/5G mobile IPs in 170+ countries, target by country, city, ZIP, ISP or carrier, and connect over HTTP(S) or SOCKS5 with 99.9% uptime. Try residential and mobile proxies for $1.97.

🏘️ Rotating Residential Proxies
  • 30M+ Premium Residential IPs
  •  170+ Countries
    Every residential IP in our network corresponds to an actual desktop device with a precise geographical location. Our residential proxies are fast and reliable, with 99.9% uptime, and work for a wide range of use cases. You can use country, region, city, ZIP, ISP, ASN and OS fingerprint targeting for our rotating residential proxies, at no extra charge.

See our Rotating Residential Proxies

📍 Static Residential Proxies
  • 24 Countries
    Buy a dedicated static residential IP from one of the 24 countries that we offer proxies in. Keep the same IP for a month or longer, while benefiting from their fast speed and stability.

See our Static Residential Proxies

📳 Rotating Mobile Proxies
  • 4M+ Premium Mobile IPs
  •  170+ Countries
    Access millions of clean mobile IPs with precise targeting by country, region, city, ZIP, mobile carrier, ASN and OS fingerprint, at no extra charge. Get far fewer IP blocks and CAPTCHAs with our 4G and 5G proxies.

See our Mobile Proxies

📱 Dedicated Mobile Proxies
  • 4 Countries
  • 50+ Locations
    Get your own dedicated mobile proxy in one of our supported locations, with unlimited bandwidth and unlimited IP changes on demand. A great choice when you need a small number of mobile IPs and a lot of proxy bandwidth.

See our 4G & 5G Proxies

🌐 Rotating Datacenter Proxies
  • 197,000+ IPs Premium IPs
  •  62 Countries
    On a budget and need to do some simple scraping tasks? Our datacenter proxies are the perfect fit! Get started with as little as $2

See our Datacenter Proxies

♾️ Unlimited Residential Proxies
  • 1M+ Daily Residential IPs
  • 85+ Countries
    Our unlimited residential proxies bill by connection speed, not gigabytes: 100 to 1000 Mbit with unlimited bandwidth, on daily, weekly or monthly plans. Ask us for a 1-hour trial at 50 Mbit before you pick a plan.

See our Unlimited Residential Proxies

proxy locations

30M+ rotating IPs

99.9% uptime - high speed

99.9% uptime.

dedicated support team

24/7 Dedicated Support.

fair price

Fair Pricing.

🏠 Residential Proxies Rotating / Static / Unlimited
📱 Mobile Proxies Rotating and Dedicated
🖥️ Datacenter Proxies Rotating
🌍 IP Pool 30M+ residential + 4M+ mobile IPs
📶 Uptime 99.9% · Live status
💳 Payment Card · PayPal · Crypto · Bank transfer
💬 Support 24/7 live chat · [email protected]