Google App Engine for indie developers

As promised, here's the second part of my post about web hosting. I thought I would highlight the technology that we use for the main Wolfire site, Google App Engine, and explain why we chose it.

Before I explain what App Engine is exactly, let me talk about what I look for in a web host.

- Scalability and reliability
It is critical for your website to be online. If your site has been featured on Slashdot, Digg, etc. that is not an excuse for downtime -- that is more reason than ever for your site to be up! Your site's speed and stability should be independent of its load or you have built a glass ceiling for yourself.

- Speed and latency
Your site should be fast -- this is pretty obvious. Both Amazon and Google actually have run experiments that quantify how responsiveness affects their sales or ad clicks respectively. According to Google's experience, "half a second delay caused a 20% drop in traffic."

- Time and money
In my opinion, the most important feature, is that it should be easy to develop for. Fast prototyping is a mantra in the indie community and I think it extends into web development as well. Furthermore, price is a very sensitive issue. Of course, we want the top of the line, but at the same time, almost by definition, most indies just don't have much cash to invest in our own data center.

What is Google App Engine?

To summarize what App Engine is, it lets you create web applications that can be deployed on Google's ridiculous server infrastructure relatively easily. You are given a neat set of APIs to use, for instance, Google's BigTable database system, a memcached compatible API, an email API, and a number of other things.

The idea is that if you code your web app in a certain way within Google's best practices, you can relax and your web app will scale pretty much indefinitely, with great speed, and near perfect uptime.

Here's how it works. Basically, when you write your site using App Engine, Google has tricked you into making a self contained, independent piece of code that nicely interacts with other parts of itself through rigid APIs. Your website is therefore not defined in terms of servers, databases, or hardware, it is defined in terms of nice little Python processes.

When someone types in your URL in their browser, there is no defined machine for it to go to. It is routed to Google, and Google can run your application on any number of machines to handle the request in an optimal manner. If you just were featured on Digg, you can bet that your code is running on a ton of machines and your site will be as responsive as ever.

The beauty is that you don't have to think about that. Google handles all of that for you.

How well does it work?

From our experience, App Engine scales quite well. We've been linked to by a number of sites: Slashdot, Ars Technica, Digg (via Ars Technica), Reddit, and App Engine didn't break a sweat.

The biggest test came when Lugaru was part of the MacHeist Giving Tree. When the Giving Tree was unveiled, we experienced a sudden spike that generated about 30 visits / second. While many of the other participating apps featured were immediately destroyed (even the Giving Tree itself at some points), Google kept up without breaking a sweat.

How much does it cost?

App Engine is a shining example of "software as a service". You pay for exactly what you use, and in fact, the first chunk is free (i.e. the first 5 million page views / month, more or less). Needless to say, at Wolfire, we are not even close to exceeding our free quota and when we do, we are not going to be too concerned with forking over the 10 cents per gigabyte and other nickels and dimes Google charges based on your precise usage. Click here for the full breakdown.

Disadvantages

A common criticism of App Engine is that you become locked into the platform. This is mostly because Google's BigTable database is unlike many other database systems, so if you make a complicated, database heavy app, it might be non-trivial to port it to another system, since you wrote it the "Google way". The Wolfire website is simple enough that I could easily port it to another system in less than a day, but if you are working on a highly complicated web application, I can see how this might be a concern.

Also, App Engine is very young. You will probably run into minor bugs every once in a while. Lucky for you, Wolfire and a number of other early adopters have been putting App Engine through its paces and have found a number of bugs for you. Google has actually offered to come to my Wolfire office (aka my bedroom) and observe my App Engine usage to help improve it. We'll see if they are still comfortable once they learn that fact.

What do you guys use for hosting? Would you guys like to hear more about App Engine or Amazon's web services? I have barely scratched the surface.