Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

this reminds: webcrawlers are supposed to include an info url with their user agent header

ex: Googlebot/2.1 (+http://www.google.com/bot.html)



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: