Our web servers were recently getting hit pretty hard by web crawler bots that simply ignore your Robots.