Release 0.4.17 — Scrapy 2.16 Crawl Fix
Released: 2026-05-22 Chart:
oci://ghcr.io/cnoe-io/charts/ai-platform-engineering:0.4.17Previous release: 0.4.16
Highlights
0.4.17 is a targeted fix for RAG web ingestion. Scrapy 2.16.0 added validation that raised AttributeError whenever a spider defined a singular start_url attribute without a populated start_urls, which broke every crawl mode — single, recursive, and sitemap — resulting in zero pages crawled for any web ingestion request. This release renames the conflicting attribute, adds the new async spider entrypoint, and pins a stable Twisted to resolve a Scrapy 2.16 TLS bug.
