Crawl4AI vs Firecrawl: Self-Hosted, LLM-Ready Web Crawling with Proxies (2026)

Developer guide · Last reviewed 20 October 2026 · 15 min read

Crawl4AI vs Firecrawl is the comparison teams run into when they want web pages turned into clean Markdown or JSON for an LLM, on infrastructure they control. Both are open source and render JavaScript with Playwright, but Firecrawl is an API service you deploy with Docker Compose, while Crawl4AI is a Python library you import. This guide explains what “LLM-ready” output means, how each tool is put together, how to self-host Firecrawl and drive the Firecrawl API from Python, how to run Crawl4AI with proxy_config, and how to route both through ProxyEmpire with the right session type for a crawl.

What “LLM-Ready” Output Actually Means

Raw HTML is a poor input for a language model: most of the bytes are navigation, scripts, banners and layout markup that burn context and confuse chunking. “LLM-ready” is shorthand for three things.

Both tools produce all three. Firecrawl returns markdown, html, rawHtml, links and a json format from one scrape call. Crawl4AI returns a result whose markdown carries both raw_markdown and a filtered fit_markdown, plus extracted_content when you attach an extraction strategy. For the bigger picture of how retrieval and agents consume this data, see our overview of LLMs and data gathering.

How Firecrawl and Crawl4AI Are Built

Firecrawl: an API stack

Firecrawl is a TypeScript API service. According to the repository’s SELF_HOST.md, the root Docker Compose file runs the Firecrawl API and workers, a Playwright service, Redis, RabbitMQ, a PostgreSQL-backed queue (NuQ) and FoundationDB services for an optional alternative queue backend. Only the API is published, on port 3002. Workers take jobs from the queue and fetch each page with plain HTTP or through the Playwright service.

Crawl4AI: a library you embed

Crawl4AI is a Python package (it needs Python 3.10 or newer). You create an AsyncWebCrawler, which manages Playwright browsers, and pass a CrawlerRunConfig to each arun() or arun_many() call. Markdown generation, filtering, extraction, deep crawling and proxies are pluggable strategies on that config; concurrency and rate limits live in a dispatcher. An optional Docker server (port 11235) wraps the library in a REST API with MCP endpoints.

FIRECRAWL (self-hosted)                 CRAWL4AI (library)

client / SDK / MCP                      your Python process
      |                                       |
      v                                       v
API :3002 --> queue (Redis, RabbitMQ,   AsyncWebCrawler
      |        NuQ Postgres)                  |
      v                                       +-- CrawlerRunConfig
workers                                       |     proxy_config / rotation
      |                                       |     markdown + extraction
      +-- fetch engine ---+                   v
      +-- playwright -----+               Playwright (Chromium)
                          |                   |
           PROXY_SERVER (one per stack)   ProxyConfig (per request)
                          |                   |
                          v                   v
                 v2.proxyempire.io:5000  v2.proxyempire.io:5000
                          |                   |
                          v                   v
                     target site         target site

In self-hosted Firecrawl, the fetch engine and the Playwright service both read the same PROXY_SERVER variable, so every request uses the same proxy settings. In Crawl4AI, each run config can carry a different proxy, an ordered list of proxies, or a rotation strategy.

Crawl4AI vs Firecrawl: Comparison Table

This table compares the open-source, self-hosted versions. Both projects also sell hosted services with extra features, which are left out here.

On licensing: the AGPL’s network clause matters when you modify the software and let others use it over a network, which is different from running it unmodified for your own pipelines. Crawl4AI’s README calls attribution “recommended”, while its LICENSE file states an attribution requirement. If you build a product on either, have counsel read the LICENSE files.

Firecrawl Self-Host with Docker Compose

Firecrawl’s self-hosting guide is deliberately conservative: pin a release, start with API authentication off on a trusted network, keep the PostgreSQL queue, add AI providers and extra engines only when you need them. You need Git, Docker Engine or Docker Desktop, Docker Compose v2 and a free port 3002. Firecrawl does not publish a minimum host size.

Proxy settings go in the root .env file next to docker-compose.yaml. The Compose file passes PROXY_SERVER, PROXY_USERNAME and PROXY_PASSWORD to both the API workers and the Playwright service. (The same variables appear in apps/api/.env.example under “Proxy Settings for Playwright”, but the docs say that file is for API development and should not be used as a Compose file.)

git clone https://github.com/firecrawl/firecrawl.git
cd firecrawl
git checkout v2.11.162

cat > .env <<'EOF'
USE_DB_AUTHENTICATION=false
POSTGRES_USER=postgres
POSTGRES_PASSWORD=replace-with-at-least-32-random-characters
POSTGRES_DB=postgres

# ProxyEmpire gateway. Copy the username from
# Proxy Manager -> Connection Builder in the dashboard.
PROXY_SERVER=http://v2.proxyempire.io:5000
PROXY_USERNAME=YOUR_PROXYEMPIRE_USERNAME
PROXY_PASSWORD=YOUR_PROXYEMPIRE_PASSWORD

# Optional: skip media requests in Playwright to save bandwidth
BLOCK_MEDIA=true
EOF

docker compose up --build -d
docker compose ps --all

The readiness endpoint only checks that the API answers, not Redis, the queue, Playwright or outbound network access. The real check is a scrape:

curl --fail --silent --show-error --max-time 5 \
  http://localhost:3002/v0/health/readiness

curl --fail-with-body --silent --show-error --max-time 75 \
  -X POST http://localhost:3002/v2/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://example.com", "formats": ["markdown"], "timeout": 60000}'

A response with "success": true and a markdown field means the API, a scraping engine, the proxy and outbound access all work. If it fails while readiness passes, check docker compose logs --tail=200 api playwright-service; our guide to proxy error codes explains what a 407 or a 502 from the proxy layer means.

What self-hosted Firecrawl does and does not include

Before the API leaves a trusted network, add an authentication design, TLS at a reverse proxy and network policy. The default API is unauthenticated, and the Compose file defines no persistent volumes for PostgreSQL, Redis or RabbitMQ.

The Firecrawl API from Python

The official Python SDK installs with pip install firecrawl and exposes a Firecrawl client. Its api_url argument defaults to the cloud, so set it to your deployment. With USE_DB_AUTHENTICATION=false no API key is needed, and the SDK sends an Authorization header only when you give it a key.

import os

from firecrawl import Firecrawl

firecrawl = Firecrawl(
    api_key=os.getenv("FIRECRAWL_API_KEY"),  # None is fine with auth disabled
    api_url=os.getenv("FIRECRAWL_API_URL", "http://localhost:3002"),
)

# 1. Clean Markdown of the main content
doc = firecrawl.scrape(
    "https://example.com",
    formats=["markdown", "links"],
    only_main_content=True,
    timeout=60000,  # milliseconds
)
print(doc.metadata.status_code, doc.metadata.source_url)
print(doc.markdown[:500])

# 2. Structured JSON (needs OPENAI_API_KEY / OPENAI_BASE_URL or
#    OLLAMA_BASE_URL plus MODEL_NAME set in the Firecrawl .env)
product_schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "price": {"type": "string"},
        "currency": {"type": "string"},
        "in_stock": {"type": "boolean"},
    },
    "required": ["name", "price"],
}

doc = firecrawl.scrape(
    "https://shop.example.com/products/123",
    formats=[{
        "type": "json",
        "prompt": "Extract the main product on this page.",
        "schema": product_schema,
    }],
)
print(doc.json)

For whole sites, map() lists URLs quickly (from sitemaps and links) without fetching content, and crawl() starts a crawl job and polls until it finishes. Both take the limits you should always set: a page cap, path filters, a depth limit, a delay and a concurrency ceiling.

site_map = firecrawl.map("https://docs.example.com", limit=500)
print(len(site_map.links), "URLs discovered")
for link in site_map.links[:10]:
    print(link.url, "-", link.title)

job = firecrawl.crawl(
    "https://docs.example.com/guides/",
    limit=200,                    # hard page cap
    include_paths=["guides/.*"],  # regex on the URL path
    max_discovery_depth=3,
    delay=1,                      # seconds between scrapes (max 60)
    max_concurrency=1,            # delay forces concurrency to one
    formats=["markdown"],
    only_main_content=True,
    poll_interval=5,
)

print(job.status, job.completed, "/", job.total)
for page in job.data:
    source = page.metadata.source_url if page.metadata else None
    print(source, len(page.markdown or ""))

The same operations are plain HTTP calls (POST /v2/crawl returns a job ID, GET /v2/crawl/{id} returns status and pages), so services in any language can share one deployment. That shared API is the main reason to self-host Firecrawl rather than embed a library.

Crawl4AI with proxy_config

Install the library and its browser once with pip install -U crawl4ai and then crawl4ai-setup. The Crawl4AI proxy documentation recommends setting proxies per request on CrawlerRunConfig.proxy_config. The older proxy argument on BrowserConfig is deprecated. ProxyConfig takes server, username and password, and ProxyConfig.from_string() also parses host:port:user:pass lines.

import asyncio
import os

from crawl4ai import (
    AsyncWebCrawler,
    BrowserConfig,
    CacheMode,
    CrawlerRunConfig,
    DefaultMarkdownGenerator,
    PruningContentFilter,
    ProxyConfig,
)

proxy = ProxyConfig(
    server="http://v2.proxyempire.io:5000",
    username=os.environ["PROXYEMPIRE_USERNAME"],  # from the Connection Builder
    password=os.environ["PROXYEMPIRE_PASSWORD"],
)

run_config = CrawlerRunConfig(
    proxy_config=proxy,
    check_robots_txt=True,          # off by default in Crawl4AI
    cache_mode=CacheMode.BYPASS,
    page_timeout=60000,             # milliseconds
    markdown_generator=DefaultMarkdownGenerator(
        content_filter=PruningContentFilter(threshold=0.48, threshold_type="fixed"),
    ),
)


async def main():
    async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
        result = await crawler.arun("https://example.com", config=run_config)
        if not result.success:
            print("failed:", result.status_code, result.error_message)
            return
        print(result.markdown.raw_markdown[:300])   # everything converted
        print(result.markdown.fit_markdown[:300])   # boilerplate pruned


asyncio.run(main())

raw_markdown is the whole page converted. fit_markdown is what remains after the content filter drops low-value blocks, and it is usually what you want to embed. BM25ContentFilter (relevance to a query) and LLMContentFilter are alternatives to pruning.

Structured extraction with an LLM

For repeating layouts, JsonCssExtractionStrategy extracts fields with CSS selectors and no model at all, which is cheaper and deterministic. When layouts vary, LLMExtractionStrategy sends the page (by default as Markdown) to any provider LiteLLM supports, including a local Ollama model, and asks for JSON matching your schema.

import asyncio
import json
import os
from typing import Optional

from pydantic import BaseModel, Field
from crawl4ai import (
    AsyncWebCrawler,
    CacheMode,
    CrawlerRunConfig,
    DefaultMarkdownGenerator,
    LLMConfig,
    LLMExtractionStrategy,
    ProxyConfig,
    PruningContentFilter,
)


class Article(BaseModel):
    title: str = Field(..., description="Headline of the article")
    author: Optional[str] = Field(None, description="Byline, if present")
    published: Optional[str] = Field(
        None, description="Publication date as shown")
    summary: str = Field(..., description="Two-sentence neutral summary")


llm_strategy = LLMExtractionStrategy(
    llm_config=LLMConfig(
        provider=os.getenv("LLM_PROVIDER", "ollama/llama3.3"),
        api_token=os.getenv("LLM_API_TOKEN", "no-token"),
    ),
    schema=Article.model_json_schema(),
    extraction_type="schema",
    input_format="fit_markdown",
    instruction=("Extract the article described by the schema. "
                 "Do not invent fields."),
)

run_config = CrawlerRunConfig(
    proxy_config=ProxyConfig.from_string(os.environ["PROXYEMPIRE_PROXY"]),
    markdown_generator=DefaultMarkdownGenerator(
        content_filter=PruningContentFilter(),  # fit_markdown
        # The LLM reads the filtered Markdown output.
    ),
    extraction_strategy=llm_strategy,
    check_robots_txt=True,
    cache_mode=CacheMode.BYPASS,
)


async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://blog.example.com/post", config=run_config)
        if result.success:
            print(json.loads(result.extracted_content))


asyncio.run(main())

Here PROXYEMPIRE_PROXY holds one line in v2.proxyempire.io:5000:USERNAME:PASSWORD form, which from_string() turns into an http:// server plus credentials. The pruning filter matters here: input_format="fit_markdown" sends the filtered Markdown to the model, which cuts tokens and keeps navigation text out of the extraction.

The Crawl4AI Docker server and proxies

The Docker server treats request bodies as untrusted: they may not set proxy_config, rotation strategies, proxy sessions, JavaScript or LLM strategies, and the server answers such requests with a 400. The operator sets an upstream proxy instead, through CRAWL4AI_UPSTREAM_PROXY (or the standard HTTPS_PROXY/HTTP_PROXY). The server’s own egress proxy resolves and validates each target IP, then asks the upstream to CONNECT to that pinned IP. Only plain http:// upstreams are supported.

export CRAWL4AI_API_TOKEN="$(openssl rand -hex 32)"

docker run -d -p 127.0.0.1:11235:11235 --name crawl4ai --shm-size=1g \
  -e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN" \
  -e CRAWL4AI_UPSTREAM_PROXY="http://USERNAME:[email protected]:5000" \
  unclecode/crawl4ai:0.9.5

curl -s http://localhost:11235/md \
  -H "Authorization: Bearer $CRAWL4AI_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

URL-encode any reserved characters in the username or password. Because the server resolves DNS itself and tunnels to an IP, test one request through your proxy before committing to this setup. Crawl4AI’s docs say proxies that refuse CONNECT to an IP are not supported yet.

Plugging ProxyEmpire into Both

ProxyEmpire uses one gateway for residential, mobile and datacenter traffic: host v2.proxyempire.io, port 5000, with HTTP, HTTPS and SOCKS5 support. Authentication is username and password. Targeting (country, region, city, ISP or ASN; carrier for mobile) and the session type are encoded in the username, which you copy from Proxy Manager → Connection Builder in the dashboard. Test it from the crawler host first:

curl -x http://USERNAME:[email protected]:5000 https://api.ipify.org

Use the http:// scheme in all three. Both tools drive Chromium through Playwright, and Chromium rejects SOCKS5 proxies that need a username and password (Playwright reports “Browser does not support socks5 proxy authentication”). (Crawl4AI’s docs suggest SOCKS5 in places; that only works without authentication.) HTTP proxies tunnel HTTPS with CONNECT, so nothing is lost. Our SOCKS vs HTTP proxy comparison covers the protocol details.

For public pages that load fine from datacenter IPs, datacenter proxies are the economical default. Switch to rotating residential proxies when you need country or city targeting, or when content differs by consumer network. Use mobile proxies for carrier-specific views. In Crawl4AI, an ordered list such as proxy_config=[datacenter, residential] tries the second proxy only when it detects that the first attempt was blocked, and result.crawl_stats records which one answered. Keep the list short: a site that keeps refusing you is stating a policy.

Sticky vs Rotating Sessions for Crawls

ProxyEmpire offers three rotation modes: the same IP (a sticky session, with a session ID sid in the username), a new IP on every request, or a custom rotation time of up to two hours. Which one fits depends on what a “unit of work” is in your crawl. The general trade-offs are covered in sticky vs rotating proxies. For LLM crawls they usually come down to this:

Firecrawl: choose one mode per deployment

Self-hosted Firecrawl reads a single PROXY_USERNAME, so the rotation mode encoded in it applies to every job. A sticky username would push all traffic through one IP, so one block becomes an outage. For a shared instance, use a rotating username or a custom rotation time of a few minutes. If one workload needs sticky behavior, give it a second small deployment with its own .env, or move it to Crawl4AI.

Crawl4AI: a sticky session per site

Crawl4AI can do this inside one process. Export a batch of sticky sessions from the ProxyEmpire dashboard (proxy lists come out as host:port:user:pass, each line carrying its own sid). Load them into a RoundRobinProxyStrategy, then give each site a proxy_session_id. Every request that shares that ID reuses the same proxy, and therefore the same sticky exit, until proxy_session_ttl expires.

import asyncio
import os
from urllib.parse import urlparse

from crawl4ai import (
    AsyncWebCrawler,
    CacheMode,
    CrawlerRunConfig,
    MemoryAdaptiveDispatcher,
    ProxyConfig,
    RateLimiter,
    RoundRobinProxyStrategy,
)

# Sessions exported from the dashboard, one host:port:user:pass per line
sticky_proxies = [
    ProxyConfig.from_string(line)
    for line in os.environ["PROXYEMPIRE_STICKY_LIST"].splitlines()
    if line.strip()
]
strategy = RoundRobinProxyStrategy(sticky_proxies)


def config_for(site: str) -> CrawlerRunConfig:
    return CrawlerRunConfig(
        proxy_rotation_strategy=strategy,
        proxy_session_id=site,          # all pages of this site share one exit
        proxy_session_ttl=900,          # seconds; then a new proxy is assigned
        proxy_session_auto_release=True,
        check_robots_txt=True,
        cache_mode=CacheMode.BYPASS,
    )


dispatcher = MemoryAdaptiveDispatcher(
    max_session_permit=8,               # concurrent browser pages
    rate_limiter=RateLimiter(
        base_delay=(1.0, 3.0),          # random delay per domain, in seconds
        max_delay=60.0,
        max_retries=2,                  # backs off on 429 and 503
    ),
)


async def crawl_site(crawler, urls):
    site = urlparse(urls[0]).hostname
    results = await crawler.arun_many(urls, config=config_for(site), dispatcher=dispatcher)
    for r in results:
        if not r.success:
            print("failed:", r.url, r.status_code, r.error_message)
            continue
        print(r.status_code, r.url, len(r.markdown.raw_markdown))


async def main():
    jobs = {
        "docs.example.com": [f"https://docs.example.com/page/{i}" for i in range(1, 21)],
        "shop.example.org": [f"https://shop.example.org/list?page={i}" for i in range(1, 11)],
    }
    async with AsyncWebCrawler() as crawler:
        for urls in jobs.values():
            await crawl_site(crawler, urls)


asyncio.run(main())

To rotate instead, drop proxy_session_id, or skip the list and use one rotating ProxyEmpire username. If you build your own pool logic, our guide to proxy pools covers rotation strategies and failure handling.

Firecrawl MCP and the Crawl4AI MCP Server

Both projects can be exposed to agents and IDE assistants over the Model Context Protocol, so a model can call scraping and crawling as tools.

Firecrawl MCP is a separate MIT-licensed package, firecrawl-mcp, developed in the firecrawl/firecrawl-mcp-server repository. Its tools include firecrawl_scrape, firecrawl_map, firecrawl_crawl and firecrawl_search. FIRECRAWL_API_URL points it at a self-hosted API (the API key is then optional if auth is off), so the proxy in Firecrawl’s .env covers agent traffic too. The local MCP docs pin an exact package version; do the same.

{
  "mcpServers": {
    "firecrawl": {
      "command": "npx",
      "args": ["-y", "firecrawl-mcp"],
      "env": {
        "FIRECRAWL_API_URL": "http://localhost:3002"
      }
    }
  }
}

Crawl4AI’s MCP support is built into its Docker server: Server-Sent Events at /mcp/sse, WebSocket at /mcp/ws, and tool schemas at /mcp/schema. The tools include md, html, screenshot, pdf, execute_js, crawl and ask. All endpoints except /health require the bearer token, so pass it when you register the server:

claude mcp add --transport sse c4ai http://localhost:11235/mcp/sse \
  --header "Authorization: Bearer $CRAWL4AI_API_TOKEN"
claude mcp list

Keep both MCP servers on localhost or a private network: an agent with a crawler tool can generate a lot of traffic quickly. If you are building agents that need a full browser instead, see our walkthrough of an AI browser agent with Playwright.

Responsible Crawling: robots.txt, Rate Limits and Data

A proxy changes the network path, not what you are allowed to collect. A few practices keep LLM crawls defensible:

• robots.txt. Firecrawl crawls honor robots.txt disallow rules by default and report blocked URLs with an explanation. The current code does not apply a robots Crawl-delay automatically, so set delay yourself. Crawl4AI checks only when you set check_robots_txt=True, and then returns a result with “Access denied by robots.txt”. Turn it on.
• Rate limits. In Firecrawl, set delay (seconds, up to 60) and max_concurrency on crawls, and size CRAWL_CONCURRENT_REQUESTS and the worker settings in .env. In Crawl4AI, use a RateLimiter on the dispatcher. It keeps per-domain delays and backs off on 429 and 503 by default.
• Bounded crawls. Always set a page limit, path filters and a depth limit (limit, include_paths, max_discovery_depth in Firecrawl; max_pages, max_depth and a FilterChain on Crawl4AI’s deep-crawl strategies).
• Official APIs and terms. If a site offers an API or a data feed for the content you need, use it. Read the terms for the pages you collect.
• Personal data and caching. Avoid personal data you don’t need, set retention rules early, and cache unchanged pages (Firecrawl’s max_age, Crawl4AI’s CacheMode) instead of re-fetching them.

At larger scale, scheduling, deduplication and politeness move out of the crawler and into a frontier service. Our article on distributed crawlers for billion-page datasets covers that architecture, and how LLM training data is collected covers the dataset side.

When to Choose Which (and Is Crawl4AI a Firecrawl Alternative?)

Crawl4AI is the open-source Firecrawl alternative most teams evaluate, but the two only partly overlap. The choice is mostly about who calls the crawler and how much control each call needs.

Many teams run both: Firecrawl as the shared API for agents, Crawl4AI inside Python jobs with unusual session or extraction logic, and one ProxyEmpire gateway underneath. If you run browser fleets yourself, the capacity planning in our browser cluster design guide applies to Firecrawl’s Playwright service and to Crawl4AI workers alike.

Frequently Asked Questions

Crawl4AI attribution: this article discusses software developed by UncleCode as part of the Crawl4AI project.