Built in Camden. By one person. So far.
WhoCrawls is run by Zaki Qadus, a Camden-based software developer who got tired of telling clients to make sure AI crawlers could read their site without ever checking if they could read his own.
The wedge
There are at least five free llms.txt generators on GitHub. They all do the same job: paste a domain, get a markdown file. None of them tell you whether the bots actually showed up after you installed the file. That gap is the product.
WhoCrawls logs every bot hit on the sites you control. GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and a dozen others. Per-page counts, last-seen timestamps, verified vs unverified source IPs. The free generator stays free. The paid dashboard is the one you'll keep open in a tab.
What we collect (the short version)
Bot user-agents. That's it. No IP addresses for humans, no cookies, no fingerprinting, no analytics SDK. Our database schema is audited in CI on every commit and the build fails if anyone adds a column matching *ip*, *cookie*, *session*, *human_*, or uses Postgres inet / cidr types. The audit script is in the public repo.
The full version lives at whocrawls.com/privacy.
The Camden tech agency
WhoCrawls is the first product in a small portfolio out of a Camden office space. If your team needs hands-on AI SEO work that goes beyond the dashboard, the agency does retained engagements. Drop a line to hello@whocrawls.com with the word "agency" in the subject.
Stack
- Next.js 15 + React 19 + Tailwind CSS 4 on Vercel (marketing + dashboard).
- Cloudflare Workers + KV (snippet ingestion + hosted llms.txt endpoint).
- Neon Postgres (account data + crawler hit logs, range-partitioned monthly).
- Polar (Merchant of Record: £15 Lite + £9/mo Pro with 7-day trial).
- Resend (transactional + weekly digest).
What's next
A mobile companion app for Pro customers is plausible for v2 once the web product is paying its own rent. Push when a new bot first hits your domain. Daily / weekly digest cards on your phone. Until then, all surface area lives at whocrawls.com.
Contact
hello@whocrawls.com
Camden, London