Weighing 97 grams with a stiff carbon-fiber plate in the sole, Adidas’ new shoes helped Sabastian Sawe beat the two-hour mark in the London Marathon, a time barrier once considered insurmountable.
I've been scraping 241 UK council planning portals – 2.6M decisions so far
UK planning data is technically public. In practice it's locked behind 400+ different council portals, some still running bespoke ASP.NET that looks like it dates from 2004, some behind AWS WAF, all with subtly different schemas. I've spent four months scraping them. I'm now at 241 councils and 2.6 million decisions across England, Scotland and Wales.
The scraping problem
Most UK councils run one of a handful of portal systems, Idox being the most common. In theory this makes things easy. In practice every council has configured theirs differently, some block non-browser requests via TLS fingerprinting, some have rate limits that will get you banned inside 10 minutes, and a handful are running the aforementioned bespoke ASP.NET.
I ended up writing several scrapers: a standard requests-based one, a Playwright-based one for councils that block anything that doesn't look like a real browser, and a curl_cffi one for TLS fingerprinting. Some councils I still can't get. Liverpool's portal sits behind AWS WAF with a JavaScript challenge. I have a working Playwright-based scraper that solves the challenge once and reuses cookies, but the WAF rate-limits the IP after about 10 requests and then blocks me for a day. So I have 60k Liverpool decisions from an old scrape and no easy way to add more.
What I found
The approval rate stuff is what most people come for. Nationally it's around 88%, but it varies wildly by ward within a council, not just between councils.
The more interesting finding came from the time-to-decision data. Across 119 English and Welsh councils, 36.5% of home extension applications missed the statutory 8-week target in 2025, up from 27.9% in 2019. Guildford is the worst at scale: 66% of decisions over target, averaging 13.3 weeks.
What it is now
A postcode checker (free) and paid PDF reports (£19/£79). Zero paying customers so far, which is fine. I've been heads down on data quality and coverage.
Site is planninglens.co.uk if you want to poke around. AMA on the scraping side – that's where the interesting problems are.
Comments URL: https://news.ycombinator.com/item?id=47931470
Points: 21
# Comments: 18
Article URL: https://gtfobins.org/
Comments URL: https://news.ycombinator.com/item?id=47931035
Points: 88
# Comments: 32
Golden Child is launching with two "five-star" products: a fresh frozen meal system and, more intriguingly, a "drizzle." It has also raised $37 million in funding.
Find the latest Best Buy promo codes and offers, including 10% back in rewards for new cardmembers and free 2-day shipping with My Best Buy Plus, here at WIRED.
Save 30% with a OnePlus coupon this month, plus save up to 10% on earbuds, phones, and more.
Students can get a Hulu plan for $1.99 per month. Get more details on this and other great deals below.
Get 20% off your next website, 10% off with exclusive Squarespace discount code, 50% off plans, and more top coupons from WIRED.
Article URL: https://purplesyringa.moe/blog/wasm-is-not-quite-a-stack-machine/
Comments URL: https://news.ycombinator.com/item?id=47930493
Points: 8
# Comments: 1
On Monday, the courtroom battle between Elon Musk and Sam Altman over alleged broken promises at OpenAI started, as usual, with jury selection. The only tricky part? A lot of the prospective jurors already have an opinion about Elon Musk, and it's not a good one. The Verge reporter Elizabeth Lopatto, who was there at […]