The traffic of our football news syndicating website (Kick News) has been steadily growing a lot since it launched. When we redeveloped it a couple of years ago, we used an in-process cache, by creating an IQueryable extension method that uses an md5 hash of the underlying SQL query as the key. This worked reasonably well, but has it’s obvious problems, such as the caches needing to be refilled when the app pool recycles or when the server is restarted. On our busy site, this means we had to wait until the caches are full before we serve any requests or it would overload our database server, which is unacceptable. Before the site gets any busier we’re going to move to an out-of-process cache and the are 3 main options we’ve considered are Redis, Memcached and Windows Server AppFabric (there are other options however, such as Coherence). There is already quite a lot of info on Redis vs Memcached, so I won’t go into that here, but I was curious as to how AppFabric performed against Redis. 

This is an extremely limited and unscientific test, however all I was looking for was an indication and couldn’t find any data elsewhere. I used Dusan Majkic’s Windows port of Redis and the ServiceStack.Redis C# Client Library, so results might be different if I used the native version of Redis, but not enough for me to care about in this little test.

Setting up AppFabric on my test machine was a breeze and Scott Hanselman has an easy to follow guide here. To run the test, I created an ASP.Net MVC 3 project and used a System.Diagnostics.Stopwatch() to time the execution of putting and retrieving 1,000 4 byte strings and then the same with a byte array of a 115KB image. I ran the test 6 times, resetting everything in between runs and then averaged the results. There was a some variation in each run (the 4 byte string tests where all roughly the same), but I think this gives me a pretty good idea of what I wanted to know.

Set 1,000 4 byte strings:
AppFabric: 918 ms
Redis: 240 ms

Get 1,000 4 byte strings:
AppFabric: 587 ms
Redis: 137 ms

Set 1,000 115 KB byte arrays:
AppFabric: 11,195 ms
Redis: 15,812 ms

Get 1,000 115 KB byte arrays:
AppFabric: 11,976 ms
Redis: 7,163 ms

I forgot to mark them, but 1 is set and 2 is get. The conclusion I draw from this is that Redis is as fast as everyone says it is (I don’t believe AppFabric is any kind of slouch). I’m not sure why Redis ended up slower in the larger byte array set test, but there was one run when it was quite a lot slower, which would have dragged it’s average down a bit. To answer the question properly, we’d need a much better benchmarking test, but I think it’s safe to say that overall Redis is quite a lot faster overall than AppFabric, however AppFabric is very easy to setup and cluster. I think we’re going to go with Redis on a couple of dedicated Debian VMs over Memcached or AppFabric, but I think all would be a massive improvement for us over our existing in-process caching. 

I needed to setup a maintenance plan for a client to backup their databases and remove backups older than 4 weeks. They run SQL Server 2005, so I created the maintenance plan with a Back Up Database Task and a Maintenance Cleanup Task to remove the old backups, which is straightforward enough. When I tried to test it however I got the error: “Unable to start execution of step 1 (reason: line(1): Syntax error). The step failed.” 

I had a quick look at the version of the server and it was 9.0.1399 (SQL Server 2005 RTM!!!). So I downloaded SP4 and went to install it, however when it gave me the options for the components to update, it wouldn’t let me check Database Services and when I selected it, the details box said “This update requires language ENU. The language for product instance MSSQLSERVER is . Download the update for language .”

Now, I know this particular server hosts a lot of Sage 200 databases as well as the database for my app. The guy that setup Sage 200 changed the language to be British English because for some reason Sage don’t use ISO date formatting (yyyyMMdd) and instead use UK style dates (ddMMyyyy). 

The fix for this was to change the language values for SQL Server in the registry.

Stop the SQL Server services via services.msc, fire up regedit.exe and write down the current decimal value for the Language value in the following two keys. 

HKLM\Software\Microsoft\Microsoft SQL Server\MSSQL.1\setup

HKLM\Software\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\CurrentVersion

Now change them both to 1033 decimal (which is US English) and re-run SP4 setup, which should now allow you to upgrade the Database Services component. Once you’re upgraded return the registry values to their original settings.

I actually ran into another problem after this (Error Code: 0x80070534 (1332) Windows Error Text: No mapping between account names and security IDs was done.), which I had to dig around in the SP4 install log for, but it turned out quite easy to fix by deleting some registry keys and letting SP4 setup recreate them.

Our web servers were recently getting hit pretty hard by web crawler bots that simply ignore your Robots.txt file. We were getting hit especially hard by a bot called MJ12bot, which is supposed to be for a distributed search engine which they say lives in an alpha state at search.majestic12.co.uk (at the time of writing that is down). It seems they get people to run their SETI@Home like software that uses your power, CPU and bandwidth to help them build what they say is a distributed search engine. What they definitely are doing with the data at present is running a SEO business, that tells you how many back-links you have, etc. via what seems to be the huge index that they have built via people donating their computing resources. 

We use Windows 2008 R2, so the solution for us was to block these crawlers via their user agent (easily done with Apache as well). To do so, open your site in IIS 7 and select the URL Rewrite module (you may have to add this via the Add/Remove Windows Features tool). Select Add Rule (in the top right) and then select ‘Blank rule’. Set you’re new rule up like the image below shows…

And add a condition like the one below…

Our regex string is “MJ12bot|soso|baidu|youdao|NaverBot|Yeti|ichiro|moget|sogou|Speedy”, however you may want to include Yandex in that as well. I’ve just removed them as they emailed us asking us to allow them to index our site and said that they obey Robots.txt rules, such as crawl-rate and disallow. 

***UPDATE***

Steve from Majestic12 has posted a detailed response regarding MJ12bot below.