Getting rid of the sneaky search engines
To my big surprise, I saw the domain already listed on Google when it’s barely been a month I bought it and it was never used before, go figure!
Some people would do anything to get their domain directly listed on Google without having to go by the sandbox, I know. But what if, because it is now listed and still in development, the search bots fall on a 404 or other kind of errors? You’re right, you start getting penalized.
Now, since the domain is in Google and I better be cautious not to start getting penalized for it, I just created the magic robots.txt file to disallow all crawling starting from the root directory. It’s so damn obvious, but who knows:
User-agent: *
Disallow: *
I could have also made some content pages with the keywords I want the domain to rank for but I don’t have time for that now, maybe when I put up the splash page?