Update robots.txt to improve crawler access#323

rafaelfranca merged 2 commits intorails:mainrails/rails-contributors:mainfrom

fgo:fgo-patch-1fgo/rails-contributors:fgo-patch-1Copy head branch name to clipboard