cubicweb-seo #2909429 Rate limiting for certain search bots [in-progress]

Add a crawling rate limiting for certain bots. For example

User-Agent: AhrefsBot
Crawl-Delay: 2

source : https://ahrefs.com/robot/index.php

Others : http://www.majestic12.co.uk/projects/dsearch/mj12bot.php

If this is not a general idea. Make it possible to configure by instance.

prioritynormal
typeenhancement
done in0.3.0
load1.000
load left1.000
closed by<not specified>
patch[robotstxt] Tie a rule to a user_agent. [pending-review]