Robots.txt respect should be ON by default #614

Robots.txt is ignored by default, as far as I can see:
https://github.com/gocolly/colly/blob/master/colly.go#L348-L354

This is disrespectful to site owners. There is no good reason this should be ignored by default:

• If scraping a site you control, then you already know whether you want a scraper to ignore the robots.txt directives, and can set the flag accordingly.
• If scraping someone else's site, then you should actively have to choose to ignore their request not to capture material. People specifically go to the effort of putting directives into the robots.txt file (which a well-understood standard) for good reasons, and there is an expectation that legitimate bot usage will respect the directives.

We've just had one of your software users fall into our spamtrap, so their activity is now banned across every site on our server. I doubt that's what they actually wanted, simply because they failed to respect our robots.txt file, probably inadvertently.