Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Browsers and other programs can use the User-Agent[1] header to send along a bit of information about themselves with each request.

This and other information is then used to filter out various types of visitor.

In this case, requests claiming to be a Google Search crawler will receive a boring page with lots of text that it can index and use as search results.

Most browsers' devtools let you change your user-agent string, and a listing of the ones used by Google crawlers is publicly available. Not saying that you should, but you could check this out for yourself... entirely at your own risk of course :)

https://en.wikipedia.org/wiki/User_agent

https://developers.google.com/search/docs/advanced/crawling/...



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: