Firecrawl Scrape Client (collectors/firecrawl_client.py)

Thin async wrapper around Firecrawl Cloud API for JavaScript-rendered web scraping with bot evasion and proxy handling.

# Source: https://github.com/polarsource/polar | License: Apache-2.0 | Copyright You must retain, in the Source form of any Derivative Works
# Provided via justclaw.it.com; keep this notice with the code.
"""Thin wrapper around the Firecrawl Cloud `/scrape` endpoint.

Firecrawl renders JavaScript, evades bot/Cloudflare blocks, and uses proxies
server-side. It backs the JS-render path of the organization-review website
collector; the browser egress is Firecrawl's network rather than ours.
"""

import functools
from dataclasses import dataclass
from typing import TYPE_CHECKING

import structlog

from polar.config import settings

if TYPE_CHECKING:
    from firecrawl import AsyncFirecrawl

log = structlog.get_logger(__name__)

# Firecrawl expresses both of these in milliseconds. `wait_for` gi

... (truncated -- full source via MCP)