Playwright Scraping at Scale: Designing a Browser Cluster for 10,000 Concurrent Chromium Sessions

Engineering guide · Last reviewed 27 September 2026 · 18 min read

Running a few Playwright jobs on one machine is easy. Playwright scraping at scale, with 10,000 browser sessions open at the same moment, is a distributed-systems problem: memory, scheduling, network identity, failure recovery and observability all have to be designed. This guide is the hands-on companion to our explainer on why modern web automation runs on Chromium. It walks through a reference architecture for a browser cluster, with the Python code, Kubernetes manifests and proxy policy you need, and shows where large browser fleets usually fail.

Modern websites depend on client-side rendering, JavaScript APIs, service workers and application state that a plain HTTP client cannot reproduce. That is why data teams run real browsers for search datasets, e-commerce monitoring, ad verification, AI data pipelines and SEO intelligence. Opening one page is not the hard part. The hard part is running thousands of isolated sessions while memory stays predictable, network identity holds, failed jobs recover and cost stays under control. At that point the fleet is a platform, not a script.

What “10,000 Concurrent Sessions” Actually Means

“10,000 browsers” is usually said loosely. A production design has to separate three units, because they cost very different amounts:

For most high-density Playwright scraping deployments the practical scheduling unit is the browser context. Launching a new Chromium process for every task adds startup overhead and wastes memory. A controlled pool of processes that hands out short-lived contexts gives far better density and still keeps each session isolated. Playwright’s documentation describes a new context as one that “won’t share cookies/cache with other browser contexts”, which is exactly the isolation a multi-tenant scraper needs.

That definition still hides a lot of processes. Chromium’s multi-process design means one browser can run several renderer, GPU, network-service and utility processes, and a page with cross-site frames can use more than one renderer. Our Chromium architecture explainer covers the process model in detail. For capacity planning, the point is simpler: measure the whole process tree, not just the browser process.

Capacity Planning Before Architecture

There is no universal “contexts per server” number. A lightweight product page and a heavy single-page application can differ several times over in memory, CPU, connection count and execution time. Benchmark representative targets before you pick node sizes, and do it for each class of target you plan to run.

Use a measurable capacity formula

usable_memory    = node_memory x (1 - safety_reserve)

context_capacity = usable_memory / p95_memory_per_context

node_capacity    = min(
    context_capacity,
    cpu_capacity,
    file_descriptor_capacity,
    network_capacity,
)

Keep a safety reserve of roughly 15–25% so the node can absorb short memory spikes, process recycling, operating-system activity and uneven workloads. Base the per-context figure on p95 or p99 observations, never on the average. Averages hide the heavy pages that cause out-of-memory kills.

A worked example with hypothetical benchmark results: suppose light targets measure 180 MB per context at p95, including that context’s share of renderer and shared memory. A node with 64 GiB and a 20% reserve has about 52,400 MiB usable, which fits roughly 290 contexts by memory. If CPU saturates at 250 contexts on the same benchmark, the node capacity is 250. The limit that bites first sets the number.

An illustrative topology for 10,000 contexts

This is a topology example, not a purchasing recommendation. Your benchmark may support fewer or more contexts per process. Whatever the number, cap contexts per Chromium process so that one leak or renderer crash cannot take out a large share of the fleet. Small pods help for the same reason: losing a pod with 50 contexts is a retry, while losing a node-sized worker with 250 is an incident.

A Reference Architecture for Playwright Scraping at Scale

A scalable browser cluster separates task control, browser execution, network identity, results and analytics. Each layer can then scale and fail on its own.

Client / API
    |
    v
Durable task queue  <----  Scheduler + tenant quotas
    |
    +--> Light worker pool
    |       +--> Browser supervisor (process pool)
    |       +--> Fixed context slots
    |       +--> Proxy session manager
    |
    +--> Heavy worker pool
            +--> Browser supervisor (process pool)
            +--> Fixed context slots
            +--> Proxy session manager

Workers --> Result store / metadata DB
Workers --> Event stream --> Metrics and analytics
Proxy session manager --> ProxyEmpire --> Target websites

The API never assigns a task to a specific browser. It enqueues a durable task that carries the URL, workflow name, deadline, geographic requirement, session policy and tenant identity. Workers claim tasks when they have a free slot. That one rule is what lets you add, drain and lose workers without the API noticing.

Separate queues for light, medium and heavy targets stop a handful of difficult sites from eating the whole cluster, and let you size each worker pool differently. The scheduler can also enforce per-customer concurrency and bandwidth limits, which matters as soon as more than one team shares the cluster.

Not every task needs a browser. Static pages, server-rendered HTML and public JSON endpoints are cheaper through a plain HTTP client or a scraping API. Route those before they reach the browser queue, and keep Chromium for targets that really need JavaScript execution. If you are deciding which targets belong where, our headless browsers primer is a good starting point.

Building the Browser Worker

The browser worker is the core execution unit. It turns a queued task into a browser context, a proxy session, a result and an acknowledgement. It should expose a fixed number of context slots instead of creating as many asynchronous tasks as the queue will hand it.

A browser supervisor with fixed slots and draining

import asyncio
import time
from dataclasses import dataclass, field

from playwright.async_api import Browser, Playwright


@dataclass
class ChromiumProcess:
    browser: Browser
    started: float = field(default_factory=time.monotonic)
    active: int = 0
    completed: int = 0
    draining: bool = False


class BrowserSupervisor:
    """A fixed pool of Chromium processes, each with a fixed number of context slots."""

    def __init__(self, pw: Playwright, processes: int, contexts_per_process: int,
                 max_age_s: int = 3600, max_tasks: int = 2000):
        self.pw = pw
        self.processes = processes
        self.cap = contexts_per_process
        self.max_age_s = max_age_s
        self.max_tasks = max_tasks
        self.pool: list[ChromiumProcess] = []
        self.slots = asyncio.Semaphore(processes * contexts_per_process)
        self.lock = asyncio.Lock()

    async def _launch(self) -> ChromiumProcess:
        return ChromiumProcess(await self.pw.chromium.launch(headless=True))

    async def start(self) -> None:
        for _ in range(self.processes):
            self.pool.append(await self._launch())

    async def acquire(self) -> ChromiumProcess:
        await self.slots.acquire()              # backpressure: wait for a free slot
        async with self.lock:
            ready = [p for p in self.pool
                     if not p.draining and p.browser.is_connected() and p.active < self.cap]
            if not ready:                       # every process is draining or dead
                self.slots.release()
                raise RuntimeError("no healthy Chromium process available")
            proc = min(ready, key=lambda p: p.active)
            proc.active += 1
            return proc

    async def release(self, proc: ChromiumProcess) -> None:
        retire = False
        async with self.lock:
            proc.active -= 1
            proc.completed += 1
            worn_out = (time.monotonic() - proc.started > self.max_age_s
                        or proc.completed >= self.max_tasks
                        or not proc.browser.is_connected())
            if worn_out and not proc.draining:
                proc.draining = True            # stop handing out its slots
                self.pool.append(await self._launch())
            if proc.draining and proc.active == 0:
                self.pool.remove(proc)
                retire = True
        if retire:
            await proc.browser.close()
        self.slots.release()

The property that matters is backpressure: when every slot is busy, the worker waits instead of opening more pages and sockets. The supervisor also drains a process once it is too old, has served too many tasks or has disconnected. It launches a replacement first, stops handing out the old process’s slots, and closes it only when its last context has finished. A production version adds a background health loop that replaces crashed processes straight away, plus metrics on every state change.

Always close the context

async def run_task(sup: BrowserSupervisor, task: Task) -> dict:
    proc = await sup.acquire()
    context = None
    try:
        context = await proc.browser.new_context(
            proxy=proxy_for(task),              # one proxy session per context
            locale=task.locale,                 # keep locale and timezone coherent
            timezone_id=task.timezone,          # with the exit country
        )
        context.set_default_timeout(15_000)
        page = await context.new_page()
        await page.goto(task.url, wait_until="domcontentloaded", timeout=30_000)
        await page.locator(task.ready_selector).wait_for()
        return await extract(page)
    finally:
        if context is not None:
            await context.close()
        await sup.release(proc)

Closing the whole context is safer than trying to clear cookies, storage, service workers and open pages one by one, and it stops state leaking between customers or tasks. Playwright’s API reference recommends explicitly closing contexts you create with new_context() before closing the browser, so artifacts such as HAR files and videos are flushed.

Two details in the example are deliberate. The readiness check waits for a workload-specific locator rather than networkidle, because pages with analytics, polling or WebSockets may never go idle. And locale and timezone are set per context, so the browser’s language and clock agree with the country the proxy exits in.

Proxy and Network Identity Orchestration

At browser scale, network identity belongs in task scheduling. Websites can weigh IP reputation, location consistency, session history, TLS behaviour, request timing and browser state together. A perfectly good browser session routed through an unsuitable or unstable IP can still fail, and at 10,000 sessions that failure shows up as a fleet-wide success-rate drop rather than one broken script.

Assign the proxy per context

Playwright accepts a proxy option on browser.new_context(), with a server, username and password, so each context can carry its own route while sharing one Chromium process. The scheduler turns task requirements into a proxy policy. Credentials live in one place instead of being scattered across thousands of scripts.

import os

PROXY_SERVER = "http://v2.proxyempire.io:5000"


def proxy_for(task: Task) -> dict:
    # Stateful flows reuse one session for the whole flow; independent
    # pages get a fresh session per task, which means a fresh IP per task.
    session_key = task.flow_id if task.stateful else task.task_id
    return {
        "server": PROXY_SERVER,
        # ProxyEmpire encodes targeting (country, city, ISP...) and the session
        # in the username. Build it from the format shown in your dashboard's
        # Proxy Manager rather than hard-coding a string here.
        "username": build_username(country=task.country, city=task.city,
                                   session=session_key),
        "password": os.environ["PROXY_PASSWORD"],
    }

ProxyEmpire residential proxies can sit between the browser workers and the target sites. The pool has 30M+ ethically sourced residential IPs, plus 4M+ mobile IPs for targets that trust carrier networks. Targeting by country, region, city, ZIP, ISP (carrier on mobile), ASN and OS fingerprint is included at no extra charge. The same sessions can be sticky or rotating. For targets that accept them, rotating datacenter proxies are the cheaper route, with country-level targeting.

Rotate between tasks, not inside a page

A browser page load is not one request. It is dozens or hundreds: HTML, scripts, API calls, fonts and images. If the proxy rotates the IP on every request, one page load arrives from many addresses, which looks nothing like a real visitor and can break cookies set mid-load. For browser work, give every context its own sticky session. To rotate, start the next task with a new session key.

A ProxyEmpire sticky session keeps its IP until you rotate it or the residential device behind it goes offline. Design for the second case: when a sticky session drops mid-flow, restart that flow from its last checkpoint with a new session instead of carrying on from a different IP.

The proxy session manager should track connection failures, connect latency, bandwidth, per-target success rate and session age. Retries must be classified: a browser crash, a proxy connection failure, a target timeout and an application-level block each need a different response. Our handbook on proxies for AI web agents at scale goes deeper on routing policy when the browser is driven by an agent rather than a fixed script.

Reliable Task Delivery and Recovery

If a worker removes a job from the queue before the browser finishes, a crash loses that job silently. Use a queue or stream with acknowledgements, visibility timeouts or leases, and dead-letter handling:

1. A worker claims a task and takes a temporary lease on it.
2. The task stays recoverable while the browser runs.
3. The worker stores the result and task metadata durably.
4. Only then does it acknowledge the task.
5. If the worker disappears, the lease expires and another worker reclaims the task.

Redis Streams, RabbitMQ, NATS JetStream, Kafka and managed cloud queues can all do this when configured properly. Pick the one your team already operates well. The product matters less than the rule: acknowledge only after durable success. With Redis Streams, consumer groups keep unacknowledged entries in a pending list, and XAUTOCLAIM transfers entries that have been idle longer than a minimum time to another consumer.

import redis.asyncio as redis

STREAM, GROUP, DEAD = "browser:tasks:light", "workers", "browser:tasks:dead"
LEASE_MS = 180_000          # longer than the longest task deadline


async def handle(r: redis.Redis, sup: BrowserSupervisor, msg_id: str, fields: dict) -> None:
    task = Task.from_fields(fields)
    try:
        result = await asyncio.wait_for(run_task(sup, task), timeout=task.deadline_s)
    except PermanentError as exc:           # e.g. extraction bug, 404
        await r.xadd(DEAD, {**fields, "error": exc.code})
        await r.xack(STREAM, GROUP, msg_id)
        return
    # Any other exception propagates: no ack, so the entry stays pending
    # and another worker reclaims it after LEASE_MS.
    await store_result(task, result)        # durable write first...
    await r.xack(STREAM, GROUP, msg_id)     # ...then acknowledge


async def reclaim(r: redis.Redis, worker_id: str) -> list:
    # Take over tasks whose previous owner stopped acknowledging them.
    _next, claimed, _deleted = await r.xautoclaim(
        STREAM, GROUP, worker_id, min_idle_time=LEASE_MS, start_id="0-0", count=10)
    return claimed

Set the lease longer than the longest task deadline, or a slow task will be reclaimed while it is still running and processed twice. Cap retries too: the pending-entries list records how many times each entry has been delivered, so an entry that keeps crashing workers can be moved to the dead-letter stream instead of looping forever. And make result writes idempotent, keyed on the task ID, because at-least-once delivery means a duplicate will eventually happen.

Make retries intentional

Networking and Operating-System Limits

A large browser fleet opens huge numbers of sockets, DNS lookups, TLS handshakes, pipes, temporary files and proxy connections. CPU and RAM can look fine while workers fail because file descriptors, ephemeral ports, NAT state or connection tracking have run out.

• File descriptors: set process and system limits from measured connection counts, and check the effective limit inside the container, not on the host.
• Ephemeral ports: Linux picks outgoing ports from ip_local_port_range, which the kernel documentation gives as 32768–60999 by default. Spread egress across nodes or source addresses and watch TIME_WAIT before you reach for aggressive tuning.
• Connection tracking and NAT: cluster egress through a NAT gateway shares its port and state table across every pod behind it. Monitor it as a capacity limit.
• DNS: use reliable resolvers, cache where appropriate, and track lookup latency. A slow resolver shows up as slow page loads everywhere.
• Connection reuse: let Chromium and the proxy layer reuse healthy connections where the target allows it.
• Bandwidth: model average and burst traffic, including images, video, fonts and failed retries.

Tune after you observe. Raising every limit to an extreme value hides leaks and widens the blast radius of a runaway worker. Set per-node budgets, alert on them, and run load tests that reproduce your real proxy and target mix.

Running Playwright in Docker and Kubernetes

Containers give you repeatable browser versions and process isolation. Kubernetes, or any other orchestrator, gives you scheduling, rollout control and worker replacement. Neither fixes an inefficient worker. They just reproduce it faster.

Shared memory matters

Chromium leans heavily on shared memory, and Docker’s default /dev/shm is small. Playwright’s Docker documentation recommends --ipc=host when using Chromium, because without it “Chromium can run out of memory and crash”. It also recommends --init to avoid zombie processes. In Kubernetes, the equivalent is a memory-backed emptyDir mounted at /dev/shm. Treat --disable-dev-shm-usage as a fallback, because it moves that traffic to disk instead of fixing the capacity problem.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: browser-workers-light
spec:
  replicas: 200
  selector:
    matchLabels: {app: browser-workers-light}
  template:
    metadata:
      labels: {app: browser-workers-light}
    spec:
      terminationGracePeriodSeconds: 240     # time to drain active contexts
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
      volumes:
      - name: dshm
        emptyDir: {medium: Memory, sizeLimit: 1Gi}
      containers:
      - name: worker
        image: registry.example.com/browser-worker:1.0.0   # pin the version
        env:
        - {name: PROCESSES, value: "2"}
        - {name: CONTEXTS_PER_PROCESS, value: "25"}
        volumeMounts:
        - {name: dshm, mountPath: /dev/shm}
        resources:
          requests: {cpu: "3", memory: 12Gi}
          limits:   {cpu: "3", memory: 12Gi}

A few choices here are deliberate. Memory requests equal limits, so the scheduler never packs more browsers onto a node than it can hold. The Kubernetes documentation notes that a memory-backed emptyDir counts against the container’s memory limit, so the 1 GiB of /dev/shm comes out of the 12 GiB, not on top of it. The grace period gives the supervisor time to drain in-flight contexts when a pod is replaced. And the image is pinned, so a browser upgrade is a deliberate rollout you can roll back, not a surprise.

Autoscaling From Queue Pressure, Not CPU Alone

CPU is a useful guardrail but an incomplete demand signal. A browser job can sit waiting on the network while the queue behind it grows. Scale on queue depth and the age of the oldest waiting task, with memory pressure and worker startup time as constraints.

A practical target is ready tasks per available context slot. Scale out when sustained demand exceeds what current workers can clear within your service-level objective. Scale in slowly, and only by draining, so you never kill a pod with 50 half-finished sessions.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: browser-workers-light
spec:
  scaleTargetRef:
    name: browser-workers-light
  minReplicaCount: 20
  maxReplicaCount: 200
  triggers:
  - type: redis-streams
    metadata:
      address: redis.queue.svc:6379
      stream: browser:tasks:light
      consumerGroup: workers
      lagCount: "50"             # about one pod's worth of slots
      activationLagCount: "10"

KEDA’s Redis Streams scaler can scale on pending entries, stream length or consumer-group lag. Setting the per-replica target to one pod’s slot count, 50 in this example, makes the scaler reason in the same unit as the cluster. Account for startup time as well: a new pod has to be scheduled, pull the image and launch Chromium before it takes work, so keep a warm minimum for predictable peaks.

Browser Lifecycle and Failure Isolation

Long-running Chromium processes accumulate memory fragmentation, caches, abandoned renderer state and site-specific leaks. Don’t wait for a process to become critically unhealthy. Recycle it on a predictable policy.

Draining means the supervisor stops giving new contexts to a process, waits for active contexts to finish or time out, closes the process, launches a replacement and returns the slots to the pool. Routine recycling then never kills a healthy task.

proxy session    --> one context retries with a new session
browser context  --> one task retries
Chromium process --> its 25 contexts retry elsewhere
worker pod       --> its 50 leased tasks are reclaimed
node             --> its 5 pods reschedule; leases expire

Keep every failure domain small. A failed proxy session should never force a browser restart. A Chromium crash should affect only its own contexts. A node failure should affect only that node’s leased tasks.

Observability: Measure the Whole Execution Path

Structured logs should carry the task ID, tenant ID, worker, Chromium process, proxy session, target domain, attempt number, stage, duration and a normalised error class. With those fields you can answer the questions that matter in an incident: did the failure happen before or after launch, was it the worker, the route or the target, and did a new Chromium version raise the crash rate?

Skip verbose per-step logs for every successful page. High-cardinality noise can end up costing more than the browsers. Capture screenshots, HAR files and traces for failures and a small sample of successes, and redact them: browser artifacts can contain credentials and personal data.

Cost Control Through Workload Classification

• Use a direct HTTP client whenever a browser is not required.
• Block images, media, fonts and third-party scripts with context.route() when doing so does not change the data you collect.
• Split light and heavy targets into separate queues and size their workers differently.
• Reuse Chromium processes, but replace the browser context after every isolated task.
• Apply retention rules to screenshots, HTML snapshots, videos and traces.
• Track compute cost and proxy bandwidth per tenant, target and workflow.

Proxy bandwidth is easy to underestimate because a browser downloads everything a page asks for. The arithmetic is simple, so run it with your own measurements. If an average task moves 1.5 MB through the proxy, a million tasks move about 1.5 TB. If blocking images and fonts brings that to 0.6 MB without changing the extracted data, the same run moves about 600 GB. Retries count too, so a lower block rate is also a lower bandwidth bill.

ProxyEmpire residential bandwidth starts at $3.50/GB pay-as-you-go and falls to $1.50/GB at volume, and unused bandwidth rolls over. Rotating datacenter starts from $0.35/GB for targets that accept datacenter IPs. See the pricing table for the full tiers.

Security: Treat Every Page as Untrusted

Browser workers execute JavaScript written by the sites they visit. Run them with container isolation, resource limits and a non-root user. Restrict their access to internal networks, and block cloud metadata endpoints such as 169.254.169.254 at the network-policy level so a compromised page cannot read instance credentials. Keep Chromium and Playwright patched, and keep the sandbox on unless the environment needs it off and gives you equivalent isolation.

Keep proxy credentials and target-site logins in a secret store, inject them per task, and never write them into logs, traces or screenshots. For multi-tenant clusters, isolate tenants with quotas and separate queues, and put untrusted workloads on separate node pools.

Scale also brings responsibility toward the sites you visit. Set per-target rate limits in the scheduler, respect each site’s terms, and collect only data you are allowed to use.

Production Checklist and Final Architecture

1. Define “session” precisely and set process, context and page limits.
2. Benchmark representative targets and size from p95 resource usage.
3. Use durable queues with acknowledgements, leases and dead-letter handling.
4. Expose a fixed number of context slots per worker to create backpressure.
5. Assign proxy identity and session policy at the scheduler or context layer.
6. Recycle Chromium processes by graceful draining.
7. Autoscale from queue demand and memory pressure, not CPU alone.
8. Measure success, latency, bandwidth, crashes, retries and proxy health end to end.
9. Isolate tenants with quotas, separate queues and per-customer cost attribution.
10. Test node loss, browser loss, proxy failure and queue recovery before launch.

API --> Durable queue + tenant quotas
             |
             +--> Scheduler / autoscaler (queue lag)
             |
             +--> Browser worker pools (light / heavy)
                    +--> Browser supervisor
                    +--> Fixed context slots
                    +--> Lifecycle and health manager
                    +--> Proxy session manager --> ProxyEmpire --> Targets

Workers --> Results / metadata (idempotent writes)
Workers --> Event stream --> Monitoring / analytics

A 10,000-session browser cluster is not mainly a Playwright problem. It is a capacity-planning, scheduling, networking and reliability problem. The fleets that scale well don’t treat each browser as a precious long-lived instance. They keep controlled process pools, hand out short-lived contexts, keep task and proxy state outside the browser, and replace unhealthy parts automatically.

When browser state, geographic targeting and network identity are orchestrated together, session behaviour can be tuned per workflow instead of being hard-coded into thousands of scripts. ProxyEmpire runs that network layer with 99.9% uptime, which you can check on the public status page, and 24/7 support from real people.

Frequently Asked Questions

Technical References

1. Playwright Documentation — “Browser” API, including new_context() and its proxy option. playwright.dev/python/docs/api/class-browser
2. Playwright Documentation — “Isolation” and browser contexts. playwright.dev/python/docs/browser-contexts
3. Playwright Documentation — “Network”: HTTP proxy and request routing. playwright.dev/python/docs/network
4. Playwright Documentation — “Docker”, including --ipc=host, --init and the seccomp profile. playwright.dev/python/docs/docker
5. Chromium Project — “Multi-process Architecture.” chromium.org/developers/design-documents/multi-process-architecture
6. Kubernetes Documentation — “Volumes”, emptyDir with medium: Memory. kubernetes.io/docs/concepts/storage/volumes
7. KEDA Documentation — “Redis Streams” scaler. keda.sh/docs/latest/scalers/redis-streams
8. Redis Documentation — XAUTOCLAIM. redis.io/docs/latest/commands/xautoclaim
9. Linux Kernel Documentation — IP sysctl, ip_local_port_range. kernel.org/doc/html/latest/networking/ip-sysctl.html