Before we go any further, here’s the sample app used for this article. Even if you don’t like my writing, you should play with it. Yeah, you! optimistic-concurrency.zip

One problem space that fits very well with functional patterns is in developing apps that have to use optimistic concurrency to maintain data consistency at scale. Here at Hive7 we build PvP games. In such games, multiple people and background processes are often affecting the same entity at the same time. We can’t use coarse grained locks or high isolation levels in MS-SQL, or the whole game would come to a halt. Here’s a common scenario in a game like Knighthood:

Multiple rival lords are attacking my Kingdom at once trying to steal my most prized vassal, my wife! My wall is staffed with a heavy defense, and my hospital has a strong set of medics healing my kingdom over time. But to keep a handle on the attack I also have to continuously spend gold to heal my defensive army.

In this common use case there are a number of subtleties. First, multiple people are attacking me at once. That means they’re doing damage to my defenses in real time, and at the same time. My hospital is healing my vassals over time. This occurs in a background process once every few minutes. And I’m triggering an instant heal to my defensive vassals using my gold supply. My Marketplace is also generating gold for me over time in another background process. To top it all off, this is happening across a cluster of application servers that are certain to be processing multiple requests simultaneously. Phew!

So what does all that mean? Well, basically, there are a lot of possibilities for change conflicts. And we have to deal with those conflicts to both keep a consistent data model and perform well.

There are a a number of potential strategies for managing these change conflicts in the persistent store – a few beefy Microsoft SQL Server databases in our case. We chose to go with optimistic concurrency and an abort on conflict transaction strategy. That basically means when we write data to the database we make sure we are always writing the most recent version of a row. If an application attempts to write an old version of the row, the data access layer throws an exception and aborts the transaction. Knighthood uses NHibernate so the validation is done for us automatically using a simple version number on the row. The basic algorithm is:

1. Read data and serialize into objects (done by NHibernate)
2. Modify objects in code
3. Tell NHibernate to persist the changes, which does the following
   1. Increments the version number
   2. Finds all the changes and batches up insert/update calls
   3. Uses the version number in the WHERE clause of updates like: “UPDATE Table SET Col1=’blah’ WHERE Version=36″
   4. Checks the rows modified reported by SQL server and throws an exception if it’s an unexpected number

As you can imagine, this fails regularly in a high concurrency scenario, but it succeeds orders of magnitude more often than not. It’s also pretty standard for any web app nowadays.

The only problem is, to preserve consistency, an exception is thrown and the transaction is aborted when change conflicts occur. That means whatever request the application or user issued fails. We could show the user a friendly error message, but that would be a frustrating experience. Nobody likes seeing errors for non-obvious reasons. And in the case of headless software running in the background the error would just be in a log somewhere. If it’s something important that needs to happen, then we have to make sure it gets done! So us imperative programmers devise a retry scheme and write a loop with an exception trap around our code. Maybe you get clever and create a class that does this which raises an event any time you need to execute your retry-able code. But, this gets pretty cumbersome. Enter functional programming!

We have a little class named DataActions that is used to simplify and consolidate this retry process and make it painless to use. I’m going to use LINQ to SQL as the example here. Here’s some usage code:

DataActions.ExecuteOptimisticSubmitChanges<GameDataContext>(
    dc =>
    {
        var playerToMod = dc.Players.Where(p => p.ID == playerId).Single();
        SetRandomGold(playerToMod);
    });

As you can see it’s really straight forward. Notice all the goodness going on there. We don’t have to instantiate our own DataContext, manually submit the changes, or worry at all about transactions. It’s all handled by the wrapper. And, you just have to provide some code to execute once the DataContext has been instantiated.

The ExecuteOptimisticSubmitChanges helper method itself is pretty simple as well:

public static void
ExecuteOptimisticSubmitChanges<TDataContext>(Action<TDataContext> action)
    where TDataContext : DataContext, new()
{
    Retry(() =>
        {
            using (var ts = new TransactionScope())
            {
                using (var dc = new TDataContext())
                {
                    action(dc);
                    dc.SubmitChanges();
                    ts.Complete();
                }
            }
        });
}

And, finally, we have the Retry method:

public static void Retry(Action a)
{
    const int retries = 5;
    for (int i = 0; i < retries; i++)
    {
        try
        {
            a();
            break;
        }
        catch
        {
            if (i == retries - 1) throw;

            //exponential/random retry back-off.
            var rand = new Random(Guid.NewGuid().GetHashCode());
            int nextTry = rand.Next(
              (int)Math.Pow(i, 2), (int)Math.Pow(i + 1, 2) + 1);

            Thread.Sleep(nextTry);
        }
    }
}

When you string all this together you get pseudo-stacks that look like:

MyCode
  ExecuteOptimisticSubmitChanges
    Retry
      ExecuteOptimisticSubmitChanges
        MyCode

So, why should you care? The calling code is really easy to read, and you get a number of other benefits with this code. In addition to handling exceptions caused by concurrency errors, you also get retries on deadlocks, and more common Sql Connection errors.

I put together a little sample application you can play with. It uses these helpers and has a SQL Database with it. The sample simulates really high concurrency and you can watch it deal gracefully with deadlocks. Then you can change line 29 of Program.cs and execute the same concurrent code without retries enabled. It ouputs the number of failed transactions and a bunch of other interesting stuff to the console. Here’s some example output:

...

Retrying after iteration 0 in 1ms
Retrying after iteration 0 in 0ms
Thread finished with 0 failures. Concurrency at 3
Retrying after iteration 1 in 3ms
Retrying after iteration 1 in 4ms
Thread finished with 0 failures. Concurrency at 2
Retrying after iteration 2 in 5ms
Thread finished with 0 failures. Concurrency at 1
Retrying after iteration 3 in 15ms
Thread finished with 0 failures. Concurrency at 0

0 total failures and 7 total retries.
All done. Hit enter to exit.

And the same test run with retries disabled:

...

Starting worker. Concurrency at 8
Thread finished with 0 failures. Concurrency at 7
Thread finished with 0 failures. Concurrency at 6
Thread finished with 1 failures. Concurrency at 5
Thread finished with 1 failures. Concurrency at 4
Thread finished with 1 failures. Concurrency at 2
Thread finished with 2 failures. Concurrency at 3
Thread finished with 0 failures. Concurrency at 1
Thread finished with 2 failures. Concurrency at 0

7 total failures and 0 total retries.
All done. Hit enter to exit.

Here’s the download link again: optimistic-concurrency.zip

Let me know if you have any questions.

At Hive7 we make web based social games for platforms like Facebook and Myspace. We’re a small startup, but producing a successful game on these platforms means we’re writing code to deal with millions of monthly users, and thousands of simultaneous users pounding away at our games. Because our games are web based they’re basically written like you’d write any other web application. They’re stateless, with multiple RDBMS back end servers for most of the data storage. Game state is pretty small so we don’t really store that much data per user. We don’t have Google sized problems to solve or anything. Our main problem is with speed.

When you’re surfing the web you want it to be fast but can live with a page taking a few second to load here and there. When you’re playing a game, on the other hand, you want instant gratification. A full second is just way too long to wait to see the results of your action. Your character’s life might be on the line!

To accomplish this speed in our games we currently buy high end commodity hardware for our database servers, and have a huge cluster of memcached that we tap into. It works. But, properly implementing caching is complex. And those DB servers are big 3U power hungry monsters! Here’s a typical disk configuration of one of our DB servers:

Each of those drives are 15k RPM 72 GB SAS (or whatever the fastest is at the time of build). And the RAID controllers are very high end with loads of cache. And here’s the kicker! We can only use about 25% of the capacity of these arrays before the database write load gets too high and performance starts to suffer. They cost us about $10k a piece. Sure, there are much more complex architectures we could use to gain performance. Or we could spend a few hundred grand and pick up a good SAN of some sort. Or we could drop some coin for Samsung SSD’s. But, those options are a bit out of the price we want to spend for our hardware, not to mention the necessary rack space and power requirements.

Enter the ioDrive. With read/write speeds that are very close to the 24 SSD monster that Samsung recently touted, at a way lower price, I have a hard time imagining choosing the 24 drive option. Maybe if you had massive storage requirements, but for pure performance you can’t beat the ioDrive price/performance ratio right now. I don’t remember if I’m allowed to comment on pricing, but you can contact a sales rep at Fusion-io for more info.

Last month we picked up one of these bad boys for testing. In summary, “WOW!” I spent a few hours this week putting the ioDrive through the ringer and comparing it to a couple different disk configurations in our datacenter. My main goal was to see if this is a viable option to help us consolidate databases and/or speed up existing servers.

The Configuration

ioDrive System (my workstation)

• Windows Server 2008 x64 Standard Edition
• 4 CPU Cores
• 6 GB Ram
• 80 GB ioDrive
• Log and Data files on same drive

Fast Disk System

• Windows Server 2008 x64 Standard Edition
• 8 CPU Cores
• 8 GB Ram
• 16 15k RPM 72 GB SAS Drives (visualization above)
• Log and Data files on different arrays

Big and Slow Disk System

• Windows Server 2008 x64 Standard Edition
• 4 CPU Cores
• 8 GB Ram
• 12 7200 RPM 500 GB SATA Drives
• Log and Data files on different arrays

Test Configuration

For this test I used SQLIOSim with two five minute test runs. We were really only interested in simulating database workloads. If you want a more comprehensive set of tests check out Tom’s Hardware. I should also mention that this was obviously not a test of equals. Both disk based systems have a clear RAM advantage and the fast disk system has a clear CPU advantage. The hardware chipsets and CPU’s are also slightly different, but they’re the same generation of Intel chips. In any case, when you see the results you’ll see how this had a negligible effect. We’re talking orders of magnitude differences in performance here…

I ran two different configurations through SQLIOSim. One was the “Default” configuration that ships with the tool. It represents a pretty typical load on a SQL Server disk system for a general use SQL server. The other was one I created called “Write Heavy Memory Constrained”. The write heavy one was designed to simulate the usage in a typical game, where, due to caching, we have way more writes than reads to a database. Also, the write heavy one is much more parallel. It uses 100 simulated simultaneous random access users where the default one has only 8. And, with the write heavy one there is no chance the entire data set can be cached in memory. It puts a serious strain on the disk subsystem.

I took the output from SQLIOSim and imported it into Excel to do some analysis. I was primarily concerned with two metrics: IO Duration and IO Operation count. These two things tell me all I need to know. First, how long does it take the device to perform IO on average, and how many can it get done in the given time period.

Test Results

Write Heavy Memory Constrained Workload

Wow, 100x faster IO’s on average!

Over 20x less time spent doing IO operations!

And over 3x more operations performed. This would have been way higher, but the ioDrive system was CPU constrained, taking 100% CPU. Looks like we’ll be loading up at least 8 cores in any database servers we build with these cards!

Default Workload

40x faster on average in this workload! Looks like the bulk operations and larger IO’s present in this workload narrowed the gap a bit.

This time, a little under 30x less time spent doing IO operations!

Only 1.5x more total operations this round. This time we weren’t CPU constrained, and I didn’t take the time to dig in to the “why” on this one. Based on the raw data I would guess this is caused by IO blocking a lot more often for ioDrive than the fast RAID system. This probably has to do with the caching system in the RAID cards under this mixed write workload. You’ll notice if you look at the raw report, that the ioDrive has no read or write cache at the device level. It doesn’t really need it.

In case you want to see the raw data or the SQLIOSim configuration files, you can download the package here:ioDrive Test Results

Conclusion

Wow! ioDrive is going to be scary fast in a database server, especially when it comes to tiny random write IO’s, parallelism, and memory constraints. I think we’ll be seeing a lot of new interesting software development and system architectures due to this type of technology. The industry is changing. You no longer need either tons of cache (or cash) or tons of RAM to get great performance out of your data store. We’re talking 100x better performance than our fast commodity arrays. I think it’s safe to say we’ll be using these devices in production in the near future. Since this device is currently plugged into my workstation, maybe I’ll post another review about how it’s improving my development productivity so you can convince your boss to buy you one. : )

Pretty Loaded

Onesize Reel 2008

And just to keep posting more about stuff not web game related, check out this incredibly talented group in the uk that has done consistently amazing work in motion graphics.  Mainframe

Seriously though, its always good to look outside your domain for inspiration and ideation.  The most impactful innovation happens when translating ideas from one domain to another.

When a game can create moments powerful enough for people to feel nostalgia, I get a warm tingly feeling that we’ve done something good. Gaming can bring people together online and be more than just killing boredom.

But, let’s get real. That stuff doesn’t matter.

Why do you write code? Well, chances are someone pays you to do it. Of course the best programmers also love what they do and do it for fun. But at the end of the day it’s your profession. Maybe you’re (un)lucky enough to be doing it for yourself in your own startup.

In the early years of a startup only one thing should matter to a programmer: shipping your product to meet your customers’ needs. Everything else we do is simply a result of this, the humblest of goals. Without the product people want you have no revenue. Without the revenue you have no company. Without the company, well, you get the idea.

Startups are widely considered the purest form of a company. You exist to meet a perceived need with a pretty small scope. There aren’t layers of management or TPS reports to get in the way of getting things done. The only barrier to getting something done is yourself. No excuses. It is in this environment where an engineer’s need for perfection must be replaced with a hacker’s passion to get things done, and get them done fast. Nothing else matters.

Maintainability isn’t a factor. Best practices don’t matter. Design patterns don’t matter. All that matters is getting things done. Don’t worry about scalability until you have to. Instantiate that object. Who cares about the factory. Skip the interface, and create a static class. Some day if you need the interface come back and re-factor your code. With the power of the IDE these days re-factoring is a lot less scary than it used to be.

This may sound short sighted, and it is. In fact, that’s the point. Who knows if the company will even exist in a year to have anything to maintain. Projects change. You have to adapt. You will never know how your code will be used 5 years from now. Stop thinking about it. 5 years ago did you think you’d be integrating your [random business application] with this Facebook thing? I bet you’ve thought about it now. Not to mention, 5 years from now it’s likely the entire programming paradigm will have changed. Were they thinking about AJAX when they designed ASP.NET? How about 3D graphics in desktop applications when the window message pump was developed? Such is life in our fast paced world. No amount of overly designed or perfectly formatted code will change it.

If you find yourself maintaining this horribly designed, hacked together legacy code from the early days of a company be thankful and bask in its glory. Without that spaghetti nightmare you wouldn’t have that job. It was that short sighted thinking that was able to get something done and create a profitable product/company.

Of course, I’m not advocating you just toss all your code in a button’s click event or anything that silly. Be smart, organize things well, but don’t waste time overly designing code to be flexible. If you have to spend more than a couple hours sketching out your design, it’s probably too complicated. Write some code. Re-factor it if you need to. You don’t need a proper RESTful architecture, or a perfect DDD. Your application isn’t going to change from Microsoft SQL to MySQL some day.

Alright, I’ll admit it. If you’re building enterprise server products, or work on a large team, or are building framework products for developers to use, then ignore everything I’ve said. Of course, then I’d question why you’re a startup in that position in the first place….

So I urge you, especially if you’re in a startup, to put down put down the abstract factory and get something done.

Disney’s pixar will be releasing a Car’s flavored mmo next year.

Cartoon networks is releasing fusion fall this month.

Fusion Fall

I did get to come up with a couple title character designs that I still enjoy looking back on. There was only a couple of days to concept the two characters, the result being wholesome Abbey and her mischievous litte brother. Oh and a couple cute and cuddly sidekicks of course. I’m such a sucker for those clean vector lines ala Illustrator.

Abbey

Touchgrind Trailer - Illusion Labs from Streetlab on Vimeo.

We just recently launched our second game called Mobties.  Its a mob flavored social strategy game based off of knighthood.  While knighthood was developed in a more startup fashion, we built on the success of knighthood and fleshed out a more cohesive visual direction this time around.  Right now its in limited beta, meaning we are limiting access to the game while we work out the bugs.

Check it out here.