A developer has scraped planning portals from 241 councils across England, Scotland and Wales, pulling 2.6 million decision records into a public site, planninglens.co.uk. The data, technically open, sits behind more than 400 distinct web interfaces, many built on dated ASP.NET or protected by AWS WAF. He built three custom scrapers—plain requests, Playwright for browser emulation, and curl_cffi for TLS fingerprinting—to bypass the assorted blocks.

Most councils use Idox software, but each installation varies, causing issues like TLS fingerprint checks and aggressive rate limits that can ban an IP within minutes. Liverpool’s portal, for example, triggers a JavaScript challenge behind AWS WAF; the Playwright scraper solves it once, then hits a daily limit, leaving only 60 k historic decisions available.

Nationally the approval rate sits at 88%, but it fluctuates dramatically by ward. Delay figures are worse: among 119 English and Welsh councils, 36.5% of home‑extension applications missed the statutory eight‑week deadline in 2025, up from 27.9% in 2019. Guildford leads the lag, with two‑thirds of cases taking an average 13.3 weeks. The site now offers a free postcode checker and paid PDF reports priced at £19 and £79, though no customers have signed up yet.