Our web servers were recently getting hit pretty hard by web crawler bots that simply ignore your Robots.txt file. We were getting hit especially hard by a bot called MJ12bot, which is supposed to be for a distributed search engine which they say lives in an alpha state at search.majestic12.co.uk (at the time of writing that is down). It seems they get people to run their SETI@Home like software that uses your power, CPU and bandwidth to help them build what they say is a distributed search engi…