HexDigestStable ownership
SHA-256 assignment keeps a URL or canonical work key on one shard across restarts and machines.
Python package · crawler coordination
Deterministic work ownership for Python crawlers
A small library for stable shard assignment, SQLite leases, retries, and worker launch coordination. It can be used with an existing Scrapy or Python crawler without adopting the rest of the HexDigest stack.
Capabilities
Lease-aware SQLite state
SQLiteWorkStore atomically claims pending work, recovers expired leases, and records retries and failures.
Safe worker launches
hexdigest-shard run starts workers without a shell, assigns shard and proxy environment, and preserves readable logs.
Minimal example
pip install hexdigest-crawl-sharding
from hexdigest_crawl_sharding import ShardSpec
spec = ShardSpec(index=2, count=8)
if spec.owns("https://example.com/products/123"):
fetch_product("https://example.com/products/123")The package also includes a compatibility helper for legacy row-id or sitemap-position assignment when migrating an existing crawl.
Scope
The package contains shard assignment, lease state, retry bookkeeping, and worker launch helpers extracted from HexDigest. Product extractors, marketplace adapters, and proprietary data remain separate.