19 points by datafreak_ 3 hours ago | 3 comments
pixel_popping 5 minutes ago
addedlovely 1 hour ago
Nice one.

I've been doing the same bit wider scope, for the whole Crux list, pruned to apex domains, and looking for CMS signals - how's your throughput?

I'm not doing any headless browser stuff, or many requests, so hyper optimised for speed.

I do grab robots.txt - didn't really see much in llms.txt or humans.txt in the wild, does yours?

addedlovely 1 hour ago
Ohh Cloudflare verified bot status, interesting I'll check that out.

I'm seeing about 6.6% block rate, but that does climb over time.

datafreak_ 1 hour ago
[dead]
idid 1 hour ago
One extension, beyond stack: market category/domain/application - or any combo that tells me what the product does.

Fab project otherwise!

datafreak_ 1 hour ago
[flagged]