Python package · Scrapy transport

TLS profiles and proxy policy for Scrapy

A small, pip-installable extension for Scrapy projects that need browser TLS profiles, consistent request headers, and domain-aware proxy fallback. The components are opt-in and can be configured independently.

Source repository Installation and usage

Capabilities

Browser TLS profiles

CurlDownloadHandler adds opt-in Chrome, Firefox, Safari, and other curl_cffi impersonation profiles to Scrapy.

Coherent headers

BrowserFingerprintMiddleware keeps user-agent, referer, and connection behavior consistent.

Adaptive proxy fallback

AdaptiveProxyMiddleware starts direct, escalates on configurable refusal codes, and remembers per-domain decisions.

Configuration example

pip install "hexdigest-scrapy-fingerprint[curl]"

DOWNLOAD_HANDLERS = {
    "http": "hexdigest_scrapy_fingerprint.CurlDownloadHandler",
    "https": "hexdigest_scrapy_fingerprint.CurlDownloadHandler",
}
FINGERPRINT_ENABLED = True

Use it only with permission, respect robots.txt and terms, and apply conservative concurrency and rate limits.

Scope

The package contains transport and middleware code extracted from HexDigest. Marketplace adapters, extraction logic, storage schema, and proprietary datasets are not included.