CRAWL
Challenge
AI crawlers never respect the rules...
Live service, no source handout. Just a URL.
Approach
"Rules" plus "crawler" points straight at robots.txt, the file sites use to tell well-behaved crawlers which paths not to index. It's not access control, it's a polite request, and the description is making fun of the fact that plenty of crawlers (and every attacker) just ignore it. Fetching it turns up a disallowed path:
User-agent: * Disallow: /super_secret/
A path being listed in Disallow doesn't block anyone from requesting it directly, it just asks search engines nicely not to list it. So the obvious next step is to just go look at what's actually in there.