Adam Leach

In Making a mobile connection I describe how after just a few seconds of inactivity your mobile phone demotes the radio link to your carrier network. It typically takes 1-2 seconds to re-establish the radio link to full bandwidth capacity. This is a huge delay!

A few days ago I was discussing desktop vs. mobile page load times with some web performance wonks. These times were gathered from real users via the W3C Nav Timing API. We started chatting about why the mobile times were worse – slower connection speeds, less cache space, etc. – and it hit me that taking 2 seconds to re-establish the radio link might account for much of what makes mobile sites slower, especially in RUM (Real User Monitoring) vs. synthetic testing. And I wondered:

After some initial testing it looks like the answers are:

I started by creating a Nav Timing test page that shows the values from Nav Timing. If you load the page you’ll see something like this. (Please look at page source to see how I calculate these conceptual time values.)

total time = 239 ms
dns = 119 ms
connect = 16 ms
ttfb = 61 ms
HTML = 0 ms
frontend = 42 ms

NOTE: Nav timing is available in Android 4. I’m not aware of any other mobile platform that has it, so you’ll need an Android 4 device to run these tests. You should close all/most currently running apps on your mobile device as they might be keeping the radio link alive in the background. On Android 4 this is done under Settings | Apps | Running. I had to stop Google Services.

You can determine if the radio link promotion delay occurs based on whether any of the times are greater than 2 seconds. Here’s a key:

no 2 second times

If all of the times are less than 2 seconds then the radio link was already active. You can create this result by loading the page multiple times in quick succession. All the times should be pretty fast because you have a radio link, the DNS resolution is cached, and you have a persistent connection to the web server.

dns > 2 seconds

If you wait 10-20 seconds (and closed all background apps) the radio link gets demoted. At this point clicking on one of the buttons to open the test page on another domain will force a DNS lookup. Normally the DNS lookup should take a few hundred milliseconds, but if the radio link needs to be promoted the DNS time jumps to 2000+ milliseconds. This page is hosted on three different domains. If you use all three pages thus caching all three DNS resolutions, the only way I know of to clear the DNS cache is to power cycle the phone.

connect > 2 seconds

If you allow the radio link to be demoted by waiting 10-20 seconds and reload the page (or click the button for the same page) you might see the connect time is greater than 2 seconds. This happens when the DNS is cached but there’s no persistent connection to the server. This is harder to reproduce – it depends on the browser’s policy for closing persistent connections.

ttfb > 2 seconds

If the radio link is demoted, the DNS is cached, and there’s a persistent connection to the server you’ll see the time-to-first-byte (ttfb) is greater than 2 seconds. This is what happens most frequently when you load the same page multiple times with a 10-20 second gap in-between.

It’s important that developers focusing on performance be aware of the impact of radio link promotion on nav timing for mobile traffic so you don’t waste time solving the wrong problem: If you’re gathering RUM data via nav timing and see slow DNS times, you might think about investing in your DNS infrastructure – even though those slow DNS times might be caused by radio link promotion. Similarly, if you see long connection times it might not make sense to investigate how your servers manage persistent connections. And slow time-to-first-byte values may or may not indicate a backend app layer performance problem.

My website doesn’t generate enough mobile traffic to verify this theory, but I believe that websites with enough mobile nav timing data will see bimodal distributions of their timing data for dns, connection, and ttfb where the modes are ~2 seconds apart. If anyone has enough data (you know who you are) please take a look and comment below. It might be possible to develop heuristics that help us determine when radio link delays are having an impact. I’d love to get some stats on the percentage of page views that incur this delay.

Unlike Zend Framework 1, the view layer in Zend Framework 2 separates the variables assigned to each view model. This means that when you are in the layout view script, you don't automatically have access to variables that were assigned the the action's view model and vice versa.

Accessing action variables in the layout

Consider this controller code:

class IndexController extends ActionController
{
    public function indexAction()
    {
        return array('myvar' => 'test');
    }
}

If you are in the layout.phtml, then to retrieve this value you do:

layout.phtml:

<?php
$children = $this->viewModel()->getCurrent()->getChildren();
$child = $children[0];
?>
<!-- some HTML -->
<?php echo $this->escape($child->myvar);?>

If you really want to make sure you collect the correct child view model, then you could iterate over $children and look for the child that has the correct captureTo name set. For the action's view model, this defaults to content:

layout.phtml:

<?php
$children = $this->viewModel()->getCurrent()->getChildren();
foreach($children as $child) {
    if ($child->captureTo() == 'content') {
        break;
    }
}
?>
<!-- some HTML -->
<?php echo $this->escape($child->myvar);?>

Accessing layout variables in the action view

If you have assigned a variable to the layout's view model in, say, an event listener within Module.php:

Module.php:

    public function onBootstrap($e)
    {
        $application = $e->getParam('application');
        $viewModel = $application->getMvcEvent()->getViewModel();
        $viewModel->some_config_var = '12345';
    }

This is how you access some_config_var in the action view:

view/index/index.html:

<?php echo $this->escape($this->layout()->some_config_var); ?>

Another, more long winded way is to use the getRoot() method on the viewModel view helper:

view/index/index.html:

<?php
$layoutViewModel = $this->viewModel()->getRoot(); 
?>
<!-- Some HTML -->
<?php echo $this->escape($layoutViewModel->some_config_var); ?>

Setting configuration variables into the view

It therefore follows that if you need to set a variable that could be accessed from any view script, it's easiest to set it into the layout's view model and then access it via the layout() view script. This is handy for view layer config variables that you want to store in your config files, such as the Google search API key.

Application/config/module.config.php:

<?php
return array(
    'layout' => array(
        'google_search_api_key' => '1234567890',
    ),

Application/Module.php:

    public function onBootstrap($e)
    {
        $application = $e->getParam('application');
        $config = $e->getParam('config');

        $viewModel = $application->getMvcEvent()->getViewModel();
        $viewModel->config = $config->layout;
    }

view/search/index.html:

<?php echo $this->layout()->config->google_search_api_key; ?>

This post tells a story.

A long time ago, I set out to write my own blog platform. Yes, WordPress is a fine blogging platform, as is Serendipity (aka "s9y", and my previous platform). And yes, I know about Habari. And, for those of you skimming ahead, yes, I'm quite aware of Jekyll, thank you anyways.

Why write something of my own? Well, of course, there's the fact that I'm a developer, and have control issues. Then there's also the fact that a blog is both a simple enough domain to allow easily experimenting with new technology and paradigms, while simultaneously providing a complex enough domain to expose non-trivial issues.

When I started this project, it was a technology-centered endeavor; I wanted to play with document databases such as CouchDB and MongoDB, and with caching technologies like memcached and redis.

Not long after I started, I also realized it was a great playground for me to prototype ideas for ZF2; in fact, the original DI and MVC prototypes lived as branches of my blog. (My repository is still named "zf2sandbox" to this day, though it technically houses just my site.)

Over time, I had a few realizations. First, my actual blog was suffering. I wasn't taking the time to perform security updates, nor even normal upgrades, and was so far behind as to make the process non-trivial, particularly as I had a custom theme, and because I was proxying to my blog via a ZF app in order to facilitate a cohesive site look-and-feel. I needed to either sink time into upgrading, or finish my blog.

My second realization, however, was the more important one: I wanted a platform where I could write how I want to write. I am a keyboard-centric developer and computer user, and while I love the web, I hate typing in its forms. Additionally, my posts often take longer than a typical browser session -- which leaves me either losing my work in a GUI admin, or having to write first in my editor of choice, and then cut-and-paste it to the web forms. Finally, I want versions I can easily browse with standard diffing tools.

When it came down to it, my blog content is basically static. Occasionally, I'll update a post, but it's rare. Comments are really the only dynamic aspect of the blog... and what I had with s9y was not cutting it, as I was getting more spam than I could keep up with. New commenting platforms such as Livefyre and Disqus provide more features than most blogging platforms I know, and provide another side benefit: because they are javascript-based, you can simply drop in a small amount of markup into your post once -- meaning your pages can be fully static!

Add these thoughts to the rise of static blogging platforms such as the aforementioned Jekyll, and I had a kernel of an idea: take the work I'd done already, and create a static blog generator.

I've been following the products of WonderNetwork for a while as they do some interesting stuff with servers around the world. I particularly like Wonder VPN as a drop dead simple and reliable VPN is very handy for any mobile user who wants some security when using a wireless network in Starbucks!

Recently they have been working on a new product called Natural Load Testing which is intended to make load testing your web application very simple. It's a very friendly service which is immediately apparent on the home page where it says "not logged in, no greeting. how sad". It's a little touch that appeals to me.

Natural Load Testing is currently in private beta and the WonderNetwork guys invited me into the beta and have been generous enough to spend a small part of their marketing budget to enable me to be able to publish this article. They've even managed to cope with my list of complaints and still talk to me! In this tutorial I will walk through how to use the Natural Load Testing product (as is today) to get some results. Interpreting the results and improving your website and server infrastructure is your job though :)

Getting Started

Natural Load Testing (shortened to NLT in this article!) consists of 4 main areas where you interact with it:

• Creating tests
• Configuring test suites
• Running test suites
• Reviewing results

Upon log in, the home page consists of a set of friendly buttons:

We start at the beginning: creating tests.

Creating tests

When load testing you need to configure your testing tool with the URLs of the pages and resources the you want to be tested. Natural Load Testing makes this job trivial by cleverly leveraging their proxy technology. By configuring your browser to use the NLT proxy, it will record every page you visit and provide you with a list of URLs which you can then use to create tests.

Obviously, as load testing involves hitting the server with a lot of requests, by the time NLT is out of beta, you will need to authorise each domain that you want to use. This is done in the "Manage Authorised Domains" section.

There is a helpful page on the NLT site that explains how to do this. Don't forget to turn off the proxy after recording. The WonderNetwork people have also remembered that you may want to record HTTPS traffic for load testing, and have provided a root certificate that you'll need to install. Again, there's help available on how to install this.

Once you have recorded some URLs, you can then click on the "Create" button and you are presented with a list of the most recent URLs you have recorded.

To create the test, you select the URLs that you require to be included (the checkbox in the header selects all within the group, which is very handy!). You then choose a name that will help you remember what this test does and create the test.

In order to create some interesting load tests on our site, I created a good few tests which I could then group into test suites.

Configuring test suites

Your tests are grouped into suites that are then run either sequentially or as a set of random loads. To create a suite, you click on the "Configure" button which then shows you the list of the suites that you have already created.

In this screenshot you can see two test suites that I've created. The second test "ZF2 Tutorial only" is the simplest suite possible as it contains a single page load within a single step. The top suite shows a more complex suite which hints at the complexity of the suites that you can create.

A suite is composed of one or more steps which are run in sequence. Each step can have multiple tests. If there are multiple test in the same step, then NLT will randomly pick one test for each run.
In this suite I have four tests in step 1 which are a set of pages within the BRI website. Step 2 is the contact page and so this test suite is measuring visiting one of 4 pages within the site and then choosing to go to the contact page.

Creation of a new test suite is done via drag and drop:

You simply drag your green tests over to the list of steps on the right hand side. It's all quite easy. Rather weirdly, you can't reuse the same test in multiple steps, so if you're testing a cycle, then you need to duplicate the same test so that you can place it into two different steps. I haven't found a way to edit the steps within a test suite once created either, so make sure you get it right!

Once you have added tests to steps, you can then tell NLT to send data whilst load testing. This is useful for filling in forms or logging into the website, for instance. I haven't used this section as I haven't yet tested any pages with forms or requiring login.

Rather useful fully, you can change the domain that a test suite uses. This enables you to duplicate a test suite and then change the domain to test domain that's running on another server or using a different configuration which makes side-by-side testing a little easier.

Calibration

Once we have created a test suite, the next step is to calibrate it. NLT will not allow you to run a test suite before calibration, so don't forget this bit! Calibration is done from the test suites list page where there is a drop down box of operations you can do on each suite:

Simply pick Calibrate and press Go. NLT will then perform a single run over your suite and present the standard results page. Ideally, I would like to see this page look different from the standard results page as it's a calibration run, not a standard test and so I was slightly confused at this point. Also, whilst the test is running, you simply see one of those "spinning gifs" to let you know that something is happening. I ran into a bug in this section which resulted in the calibration run failing, but I was never notified on the page that an error had occurred.

The calibration results on my ZF2 tutorial page suite look like this:

The intent of calibration is to provide a base-line to the NLT of the performance of the suite when there is no load. i.e. the idea is that this single run has provided an indication of the optimal conditions and we are expecting to be within 15% of this performance when testing under load.

Running test suites

Now that we have calibrated test suites, we can run some load tests and see what happens. In NLT parlance, we run test suites by pressing the Play button at which point the terminology changes to from play to execute. Simply select your test suite and press the Execute button. You are then presented with a form in order to set the run parameters:

There are three parameters you need to set:

We set these numbers up to create the required load testing profile. I've been using 100 users at 250ms spin-up for 2000 total executions as this takes around 30 to 60 seconds to complete a run and seems a reasonable profile for my blog given its traffic levels.

Upon pressing execute, NLT will spin up a number of worker processes on its servers and then display a graph that updates every few seconds showing you what happening:

As you can see in the screenshot above, you get this information:

• A bar showing number of active requests (blue) on top of number of requests initiated.
• A line chart of median response time.

The x-axis is in seconds. Hence the red bar chart shows the number of requests initiated during each second and the blue bar shows the number of requests that were still active at the end of the second. Hence a request that completed within a second is counted in the red bar, but not in the blue. Ideally, therefore we would like to see smaller blue bars than red bars and we definitely don't want to see blue bars getting bigger.

The green line shows us the average time it takes to server all requests in that second. Ideally, we want this number to be lower. Clearly, if there are any errors serving a request (e.g. an nginx bad gateway error), the rather fast failure time is not counted as it would bring the average down!

We can also view the data as a table:

Obviously, this is right at the start of the run, and so all look good. Clicking on a given run id will provide us with more information. This is a run that's later in the test:

As you can see, the server was struggling now! The red background means that the request took more than 15% longer than the calibration run and the difference from calibration is shown in brackets in the Response Time column. In this particular run, the main HTML took 760ms to deliver which as 411ms longer than the calibration run. We're also struggling to serve images in a sensible time. This is clearly not ideal!

Looking at what's going on on the server whilst running a load test is also instructive. A good introduction to the server tools that are useful is 16 Linux server monitoring commands you really need to know by Steven Vaughan-Nichols.

Reviewing results

Once you have a few runs under your belt, the Review page becomes useful. In this page we can see a list of all our previous runs and most importantly, we can edit the title of each run and give it a useful name. It would be nice to be able to store additional notes about each run though.

Adding the title makes it much easier to remember why a given run was performed and I've found it useful.

Using Natural Load Testing to improve performance

The obvious first target for testing NLT, was my blog at akrabat.com. This is a simple WordPress blog which recently moved to a new server. I haven't been particularly worried about performance and don't really expect to ever be slash-dotted (or is it Cal-dotted, nowadays!) any time soon.

However, as I've done nothing to tune the system, this is a good time to see how it behaved. The first run produced these results:

This isn't good! The only good sign was that none of the 2000 runs actually resulted in an error. A median response time around 1.5 seconds didn't sound good and the maximum run time was 13 seconds!

Something had to be done!

A little bit of investigating showed that neither APC or WP Super Cache was installed on the server. So I turned them on.

The effect of APC can be seen here:

We now have an average response time of much less than 500ms which seems much saner. There were a number of very long requests of over 1 second though.

To see if I could improve this further, I then installed and enabled WP Super Cache in PHP mode and re-ran Natural Load Testing. The results were:

Note that you need to be careful when comparing graphs as the axes are automatically scaled. Looking at the results with WP Super Cache was enabled, I can see that the response time display of the graph is much smoother. The tabular data backs this up as it shows that the vast majority of requests were taking less than 200ms with significantly fewer runs taking much longer. Also, the curve of the number of requests initiated vs request active at the end of each second is much smoother, which indicates that with WP Super Cache enabled, the server should be able to hold its own over a longer time period much more easily.

I tested this hypothesis by doing 5000 runs rather than the 2000 shown above. The result was:

Again, the median response time is nicely under 150ms for most of the time, but we have more variance in the response time, with a couple of seconds where the median was significantly over 250ms.

On the whole, my investigations with NLT have made my blog much more responsive and more likely to be able to handle more traffic than before I started this process.

Other thoughts

Having walked though what Natural Load Testing does, I also need to point out that when using it, you can tell that this is a product still in beta! Incidentally, the WonderNetwork people have responded to my reports quickly, even if half the time it was to let me know that my idea or complaint was already on their list!

Nearly all the issues I have with NLT at the moment are related to usability, which I'm sure will be ironed out over time. My "favourite" annoyance as noted above is on the create test suite page where the Save button is above the section where you set up the suite's steps. If you accidentally press Save after entering a title, then you find you can't actually add any tests to the steps! There's also no menu bar or easy way to get from one section to another. You very quickly learn to click on the NLT logo at the top which takes you to the home page.

It would also be good if NLT would allow me to store "execution profiles" so that I don't have to keep typing in that I want 100 users for 2000 total runs.

I would also like to see more aggregate results. In particular, I'd like to have stats on the response time for say the 50th and 95th percentile. I'd also like to graph these against multiple runs along with the run title so I can see how the site's performance changed over time. This then leads to the idea that it would be nice if I could schedule a load test once a month at 3am local time!

Conclusion

Natural Load Testing is the first product I've used that makes me actually want to do load testing. It makes it easy to run a specific load test and repeat exactly the same test as frequently as you need to. Tweaking the server to see the effect that any given change has becomes an interesting task and your websites can only benefit as you reduce and remove the bottle necks that you find.

On the whole, I'm quite impressed with Natural Load Testing and can see that it could turn into a very useful tool in the web developer's toolbox.

As I mentioned at the top, NLT is currently in private beta. If you want to get an invite, head over to the sign up form and fill in your details. You also have to read the thank you for signing up message!

Read more »

We hung out with the DreamHost team for the first time at HostingCon in August 2011. They threw a great party, had awesome t-shirts, and exuded the kind of excitement and passion for hosting that CloudFlare has for making websites faster and more secure. We immediately knew we wanted to partner with them.
 
Fast forward nine months to today and we are happy to announce that DreamHost is now an official Certified Hosting Provider. Beginning today they're offering CloudFlare to all their customers with a one-click-simple integration. Prior to this partnership, we had thousands of DreamHost customers who signed up for CloudFlare directly through our site. Now, every DreamHost customer has simple, easy access to CloudFlare with a click of a button and without having to mess with their DNS. Other bells and whistles like mod_cloudflare are now included in all the default DreamHost configurations, so even existing CloudFlare users on the DreamHost network will benefit.

CloudFlare Plus

DreamHost is the latest CloudFlare Certified Hosting Partner, a program that makes CloudFlare one-click simple for any hosts to provide to their customers. We're trying a new experiment and allowingDreamHost to offer a special plan they've dubbed CloudFlare Plus. We worked with them to create this custom CloudFlare plan with the features they thought would be the most interesting for their customers and a price point lower than our current Pro product. For those folks for whom CloudFlare Pro is a bit more than they need, DreamHost now offers another option with some of our most popular paid features.

Just Sayin': CloudFlare ≈ Voltron

I do have to say that DreamHost will also always hold a special place in my heart for what must be the most fun press release I've ever seen, a full copy of which is below. If you only read one part, read the quote near the end that I bolded which is certifiably awesome.

FOR IMMEDIATE RELEASE

DreamHost Partners With CloudFlare

CloudFlare to Provide Free Site Performance Optimization and Security Services for all DreamHost customers

LOS ANGELES, California—April 5th, 2012—DreamHost, a global full-service web hosting company, has today announced a partnership with leading Internet web performance and security company, CloudFlare. DreamHost customers now have immediate access to CloudFlare's robust infrastructure at little or no cost as a standard feature of their hosting plans.

Shared web hosting customers, for many years, have rarely been able to take advantage of the benefits provided by Content Delivery Networks as a result of either the cost or complexity involved. CloudFlare has removed both barriers to entry by distilling the entire setup and configuration process down to a single checkbox and removing the cost entirely. CloudFlare brings the performance and security tools previously available only to Internet giants to anyone with a website.

Hundreds of nodes around the globe power CloudFlare's network ensuring that websites load quickly and consistently, regardless of where in the world users happen to be. CloudFlare's Anycast technology works with static and dynamic sites, routing users to the node on their network for the fastest performance — all without breaking a sweat!

CloudFlare's “Always Online” technology ensures that sites taking advantage of the CloudFlare platform will remain online, continuing to serve cached content, even if the hosting servers on which they are housed become temporarily unreachable. If the hosting industry had a holy grail, it might look a little something like CloudFlare.

In addition to CloudFlare's free offering, DreamHost has also worked with the CloudFlare team to create “CloudFlare Plus,” a bundling of CloudFlare's most popular features available exclusively to DreamHost customers. CloudFlare Plus is an optional paid upgrade, weighing in at $9.95 per month, and adds automatic image optimization and support for Secure Socket Layer (SSL) connections. Provisioning of either option has been integrated within the DreamHost customer control panel.

“When we first met with the CloudFlare team at HostingCon 2011 we had no idea if what they were telling us was true,” said Kathy Brahm, DreamHost's Vice President of Customer Experience and Partnerships. “You know how typical sales people can be — schmoozy, smiley, 'Let-me-buy-you-dinnery', handsy … all the while making outrageous claims about their product. CloudFlare's team has been the complete opposite of those — and a true pleasure to deal with. We've spent the past few weeks putting CloudFlare through its paces and running some tests of our own. Some of us nearly fainted when the first speed tests came back. One guy cried. Me? I buried my feelings deep inside so I don't have to deal with them. It's just what I do.”

“From days of long ago, from uncharted regions of the web, comes a legend; the legend of CloudFlare, Defender of the Interwebs: a mighty robot, loved by good, feared by evil,” explains Matthew Prince, co-founder and CEO of CloudFlare, doing his best Peter Cullen impersonation. “As CloudFlare's legend grew, peace settled across the network. From Los Angeles, an Interweb Alliance was led by DreamHost. Together with other good hosts of the network, DreamHost helped maintain peace throughout the Interwebs, until a new horrible menace threatened. A closer relationship with CloudFlare was needed. This is the story of the super force of web explorers, specially trained by DreamHost, to more tightly integrate CloudFlare, Defender of the Interwebs!”

CloudFlare's free offering and the DreamHost-exclusive “CloudFlare Plus” are now available to all DreamHost customers.

Permalink | Leave a comment  »

From tax preparation to safe social networks, Amazon RDS brings new and innovative applications to the cloud

Empowering innovation is at the heart of everything we do at Amazon Web Services (AWS). I often get to meet, discuss, and learn from innovators how they are using AWS to deliver transformative applications to their users, customers and partners. Often we think about innovation as doing 'new things' or based on revolutionary new technologies such as DynamoDB, but it is more important to ensure that one can also innovate based on existing paradigms. One of the services that is very successful in driving innovation at our customers in this context is Amazon RDS, the Relational Database Service. Amazon RDS removes the headaches of running a relational database service reliably at scale, allowing Amazon RDS customers to focus on innovation for their customers.

Recently I had great conversations with Troy Otillio, Senior Development Manager at Intuit and Jack Murgia, Senior DevOps Engineer at Edmodo. Troy and his team have added a contextual social offering to the popular TurboTax and Intuit applications. Jack and his engineers have created a safe social app for teachers and students. These innovators use Amazon RDS in conjunction with other Amazon Web Services to build, scale and operate their applications. Below is my dialog with them. Read on.

Note: If you want to see how Amazon RDS can enable your creative agenda, sign up for the 60 day free trial.

Troy, Jack, Tell me a little bit about your app. What's unique and innovative about your service?

Troy: Live Community Platform is Intuit's flagship Contextual Social offering – Live Community makes it easy to find answers when and where you need them. This is a unique and innovative platform.

• Intelligent Social network - Facilitate topical Q&A conversations among employees, customers and our most valued super contributors.
• Large Seasonal Peaks – Our largest community supports TurboTax where the peak traffic during February or April is often 100's of times greater than a quiet day in June. Live Community Experience is deeply integrated into the tax experience, so we built a highly responsive and reliable web experience.
• Read-your-mind contextual integration – Our core innovation and underlying secret sauce involves selecting the most relevant content for a given page if not given user – to provide the right answer at the right time to our users.

Jack: Edmodo is the safe social network for education used by a network of over 6 million teachers and students worldwide that allows teachers to create and maintain their classroom communities. Some unique and innovative characteristics of Edmodo are:

• Edmodo is as easy to use as other social network sites, but secure - the teacher has the same control over access, content and behavior in Edmodo as he/she does in the classroom.
• Students gain experience they need for the modern workplace, learning how to work responsibly and effectively in a collaborative, project-based manner on a social website.
• Teachers can use Edmodo to share educational content, manage projects and assignments, handle notifications, and conduct quizzes and events.
• Teachers can interact with their colleagues in professional learning networks.
• Schools and districts can claim unique Edmodo web addresses for added communication and customization.

How are your users adopting and responding to your service?

Troy: We moved our service from internal servers to AWS. Our 25+ million strong TurboTax and Intuit user community grows every year and Live Community is an integral component of the overall product experience. Moving to AWS has enabled us with operational agility to deliver more value to those customers without having to worry about scale and infrastructure maintenance. We now have more time to focus on innovation while being confident that when demand increases we can easily add more capacity.

Jack:Since our launch in late 2008, we've grown to over 6 million teachers and students globally primarily through word of mouth of teachers who have shared Edmodo with each other. In addition to using Edmodo to engage students in classroom activities, teachers all over the world build profile pages on Edmodo, which they use to discover and share content, meet and stay in contact with other educators, and best practices and top resources.

Jack, how did this idea come about? How did you choose a SQL approach to solve this problem?

Jack: After years of seeing teachers struggle to share the web with their classroom, Edmodo founders Nic Borg and Jeff O'Hara knew there was a need for a highly scalable, secure social network targeted at K-12. SQL was the right choice because it was an established and proven technology for use in similar environments, and the massive knowledge base that exists around it.

And how about you Troy? Why did you choose a SQL approach to build your social community app?

Troy: The initial architecture was based on MySQL– we've continued with use of SQL but are now leveraging RDS. Of course, with as much textual data as we have we are leveraging Lucene/SOLR (a NoSQL solution) for Search and Semantic processing. More recently we've expanded our platform to include additional forms of user interaction observation in support of our real-time analytics – here we've begun to leverage NoSQL technologies like Redis. Going forward we'll continue to employ a hybrid approach using RDS for the necessary transactional computation and services like DynamoDB for high performance and scalability for structured data.

What did you find unique about RDS? What has been your experience so far?

Troy: We love RDS – it's reduced our operational workload by a noticeable factor but even more exciting is the benefits around fast recovery enabled by the Multi-Availability Zone capability. My team often brags about the one-click creation of read replicas, ability to upsize or downsize the database without downtime and automatic back-up. However, the shining moment occurred just last month – during peak load there was a hardware failure on the Server powering a RDS Master Database – RDS automatically failed over to the alternate zone within minutes and our customers experience was fully functional shortly thereafter. The best part was that the entire process was what I call Òhands freeÓ and took near zero development effort. With self-hosted databases we would have invested considerable engineering effort to implement, test and retest failover – to achieve fast recovery with RDS we simply changed our configuration. And when the actual production event took place the recovery required no manual intervention – the response from our CTO after hearing what happened: "that's cool".

We encountered a few situations that required help from the Amazon team – for example, we didn't know that the I/O capacity of the Server is governed by the size of storage and size of the server. When we first attempted to load our production database it took 28 hours – after a few days of attempts to reduce the load time through well-known optimizations (mostly documented on the RDS website) we were stuck at 8 hours. We consulted directly with Amazon and learned that the storage and the DB Server size affected I/O throughput – after altering our size we dropped our load time to 1 hour which was within expectations relative to native database.

Jack: Based on our experience during this period of phenomenal growth while our team productivity is stretched to the max, we see that:

• RDS is a huge time-saver
• RDS provides peace of mind about our data

Anything that saves time and simplifies processes for employees of a young startup has a positive affect that CAN NOT be overstated. The peace of mind part needs no explanation. Nobody on our team regrets moving to RDS MySQL - quite the opposite; we all agree we don't want to think about where we would have been without RDS. We have been able to meet our goal of architecting our application for 0% "maintenance downtime"

Out of the box, RDS' CloudWatch data and graphs speed up the troubleshooting process.

• Complete certainty in a DB environment is VERY unique- we never worry that:
• our DB parameters are identical across replicas, and changes propagate at a time of my choosing
• the recoverability of data

This is great to hear. We are glad we RDS meets your needs. Now, what's next on your innovation agenda?

Jack: We want to deliver an even more performance and rock solid experience for our global user base of teachers, students, administrators and parents. We will be:

• Building "incident managers" which utilize the cloud watch data and AWS APIs to automatically replace servers and/or re-deploy when problems arise.
• Building "incident creators" - servers which test our ability to maintain peak performance. 

We believe that by leveraging the services that Amazon provides to the fullest we can continue to scale our exceptional user experience so that Edmodo can be the platform for classrooms around the world on devices of all shapes and sizes.

Troy: At Intuit, we can go further to leverage the benefits of elasticity and further improve our resiliency. We are investing in use of CloudFormation coupled with Chef – the result will enable us to lower costs and further reduce risk.

Prior to AWS we had several tiers that we now think can be delegated to AWS services – this should free up our team to focus on our domain problems. For example, we are intending to replace our EC2/Memcache tier with ElastiCache, our batch processing with Simple Workflow and Web servers with CloudFront.

With our newfound agility we can launch new services quickly and there are a few on our plate in the near term. In some cases we are refactoring our system into smaller, discrete services, while in other cases we are creating wholesale new services Our core problem domain consists of extracting greater value out of textual and behavioral data which means that use of EMR and even the newly released Workflow should enable us to focus more on the domain and less on the system engineering.

Troy, Jack, Thank you both very much for sharing your unique experience. I look forward to hearing your progress.

Jack: Thank you. This has been a great dialog.

Troy: Thank you Werner. We appreciate the opportunity.

As I noted before, it's a great pleasure to talk to these innovators and how AWS helps their journey. If you have never used RDS before, you can sign up for a 60 day free trial What innovation will you bring to market? How will it change the world? We won't know until you try and build something.

Amazon ElastiCache makes it easy for you to deploy, scale, and run a cloud-based in-memory cache that is protocol-compliant with Memcached. ElastiCache improves the performance of web applications and reduces the load on your databases by retrieving data from a fast, managed, Memcached-compatible, in-memory caching system, instead of relying entirely on disk-based storage. It can significantly improve throughput for read-heavy or compute-intensive workloads including Social Networking, Mobile and Social Gaming, E-Commerce Sites, Media Sites, and Recommendation Engines.

In order to make ElastiCache an even better value, we are adding a full suite of Reserved Cache Node options -- Light, Medium, and Heavy with both 1 and 3 year terms. See the ElastiCache pricing for additional information. Reserved Cache Nodes can provide savings of up to 70% compared to On-Demand pricing. More information, including pricing, is available on our new Reserved Cache Nodespage.

You can easily migrate from Memcached to ElastiCache using our How Do I Migrate FAQ as a guide; you can also use the ElastiCache CloudFormation template to launch a cache cluster.

Finally, you'll also find useful information in the recorded version of our "Turbo-charge Your Apps Using Amazon ElastiCache" webinar:

-- Jeff;

What a crazy few weeks. Since my crazy, fuck you to homophobia coming out post almost a month ago, I’ve seem to have unintentionally gone into full overdrive with the social issues in technology culture posts: discussing how sexism in technology conferences insults everybody by making women and gay men invisible, and by portraying straight men as stupid, misogynistic idiots who think only with their dicks. Then there was the Brendan Eich post. Which wasn’t really about Brendan Eich so much as about whether it’s legitimate to call Eich out and the tired old trope about how the oppressed become the oppressors by, err, talking about homophobia on Twitter. Then to round it off, a post on the reaction to Ryan Funduk’s post about drinking in tech culture.

I wanted to share a little reading list: Derailing for Fun and Profit by Peter Aronoff, which deals with the ever so tiresome response “oh, but they have a right to their opinion”. Which is really a big old red herring. Sure, people have the right to an opinion. You have the right in the strict legal sense to believe that Queen Elizabeth II is actually a shape-shifting lizard from the Draco constellation. Only I also have the right to consider that absolutely fucking crazy and to think that you are off your rocker.

Chris Heilmann has written a post discussing whether Twitter is a good place to have these kinds of arguments, and includes the excellent TEDx video from Jay Smooth on racism.

Chris is obviously right about the potential for grandstanding and sloganeering on platforms like Twitter. But, I think he goes too far. DISREGARD THAT, I’m an idiot. Chris wasn’t saying what I think he was saying. Apologies.

I think that with a few obvious limits,¹ honesty is a better policy than hushing things up in order to give outsiders the view that the tech community is free of disagreement. However painful talking about things like sexism and homophobia and the social issues around geek culture can be, it’s better to have the conversation than keep quiet.

Finally, read Natalie Reed’s Hipster Misogyny. Because the reaction to the Boston API Hackathon thing was so clearly hipster misogyny. I didn’t cover it in my post as that wasn’t what the post was about. Remember what they said in their non-apology apology? “While we thought this was a fun, harmless comment poking fun at the fact that hack-a-thons are typically male-dominated, others were offended.”

Yeah, LOL GUYZ WHY SO SERIOUS ON THE SEXISM SHIT? That about sums it up.

This week I finally got time to do some coding on the HTTP Archive. Coincidentally (ironically?) I needed to focus on performance. Hah! This turned out to be a good story with a few takeaways – info about the HTTP Archive, some MySQL optimizations, and a lesson learned about dynamic script loaders.

Setting the stage

The HTTP Archive started in November 2010 by analyzing 10K URLs and storing their information (subresource URLs, HTTP headers, sizes, etc.) in a MySQL database. We do these runs twice each month. In November 2011 we began increasing the number of URLs to 25K, 50K, 75K, and finally hit 100K this month. Our goal is to hit 1M URLs by the end of 2012.

The MySQL schema in use today is by-and-large the same one I wrote in a few hours back in November 2010. I didn’t spend much time on it – I’ve created numerous databases like this and was able to quickly get something that got the job done and was fast. I knew it wouldn’t scale as the size of the archive and number of URLs grew, but I left that for another day.

That day had arrived.

DB schema

The website was feeling slow. I figured I had reached that curve in the hockey stick where my year-old schema that worked on two orders of magnitude less data was showing its warts. I saw plenty of slow queries in the log. I occasionally did some profiling and was easily able to identify queries that took 500 ms or more; some even took 10+ seconds. I’ve built big databases before and had some tricks up my sleeve so I sat down today to pinpoint the long poles in the tent and cut them down.

The first was pretty simple. The urls table has over 1M URLs. The only index was based on the URL string – a blob. It took 500-1000 ms to do a lookup. The main place this happens is looking up the URL’s rank, for example, in the last crawl Whole Foods was ranked 5,872 (according to Alexa). This is a fairly non-critical piece of information, so slowing down the page 500-1000 ms wasn’t acceptable. Plus this seems like a simple lookup ripe for optimizing.

When I described this problem to my Velocity co-chair, John Allspaw, he suggested creating a hash for the URL that would be faster to index. I understood the concept but had never done this before. I didn’t find any obvious pointers out there on “the Web” so I rolled my own. I started with md5(), but that produced a fairly long string that was alphanumeric (hex):

select md5("http://www.wholefoodsmarket.com/");
=> 0a0936fe5c690a3b468a6895efaaff83

I didn’t think it would be that much faster to index off the md5() hex string (although I didn’t test this). Assuming that md5() strings are evenly distributed, I settled on taking a substring:

select substring(md5("http://www.wholefoodsmarket.com/"), 1, 4);
=> 0a09

This was still hex and I thought an int would be a faster index (but again, I didn’t test this). So I added a call to conv() to convert the hex to an int:

select conv(substring(md5("http://www.wholefoodsmarket.com/"), 1, 4), 16, 10);
=> 2569

I was pretty happy. This maps URLs across 64K hashes. I’m assuming they’re evenly distributed. This conversion is only done a few times per page so the overhead is low. If you have a better solution please comment below, but overall I thought this would work – and it did! Those 500+ ms queries went down to < 1 ms. Yay!

But the page was still slow. Darn!

Duh – it’s the frontend

This and a few other MySQL changes shaved a good 2-3 seconds of the page load time but the page still felt slow. The biggest problem was rendering – I could tell the page arrived quickly but something was blocking the rendering. This is more familiar performance territory for me so I gleefully rolled up my sleeves and pulled out my WPO toolbox.

The page being optimized is viewsite.php. I used WebPagetest to capture a waterfall chart and screenshots for Chrome 18, Firefox 11, IE 8, and IE 9. The blocking behavior and rendering times were not what I consider high performance. (Click on the waterfall chart to go to the detailed WebPagetest results.)

Chrome 18:

Firefox 11:

Internet Explorer 8:

Internet Explorer 9:

These waterfall charts looked really wrong to me. The start render times (green vertical line) were all too high: Chrome 1.2 seconds, Firefox 2.6 seconds, IE8 1.6 seconds, and IE9 2.4 seconds. Also, too many resources were downloading and potentially blocking start render. This page has a lot of content, but most of the scripts are loaded asynchronously and so shouldn’t block rendering. Something was defeating that optimization.

Docwrite blocks

I immediately honed in on jquery.min.js because it was often in the critical path or appeared to push out the start render time. I saw in the code that it was being loaded using Google Libraries API. Here’s the code that was being used to load jquery.min.js:

<script src="http://www.google.com/jsapi"></script>
<script>
google.load("jquery", "1.5.1");
</script>

I’ve looked at (and built) numerous async script loaders and know there are a lot of details to get right, so I dug into the jsapi script to see what was happening. I saw the typical createElement-insertBefore pattern popularized by the Google Analytics async snippet. But upon walking through the code I discovered that jquery.min.js was being loaded by this line:

m.write('<script src="http://www.stevesouders.com/blog'+b+'" type="text/javascript"></script>'):

The jsapi script was using document.write to load jquery.min.js. While it’s true that document.write has some asynchronous benefits, it’s more limited than the createElement-insertBefore pattern. Serendipitously, I was just talking with someone a few weeks ago about deprecating the jsapi script because it introduces an extra HTTP request, and instead recommend that people just load the script directly. So that’s what I did.

We don’t need no stinkin’ script loader

In my case I knew that jquery.min.js could be loaded async, so I replaced the google.load code with this:

var sNew = document.createElement("script");
sNew.async = true;
sNew.src="http://ajax.googleapis.com/ajax/libs/jquery/1.5.1/jquery.min.js";
var s0 = document.getElementsByTagName('script')[0];
s0.parentNode.insertBefore(sNew, s0);

This made the start render times and waterfall charts look much better:

Chrome 18:

Firefox 11:

Internet Explorer 8:

Internet Explorer 9:

There was better parallelization of downloads and the start render times improved. Chrome went from 1.2 to 0.9 seconds. Firefox went from 2.6 to 1.3 seconds. IE8 went from 1.6 to 1.1 seconds. IE9 went from 2.4 to 1.0 seconds.

This was a fun day spent making the HTTP Archive faster. Even though I consider myself a seasoned veteran when it comes to web performance, I still found a handful of takeaways including some oldies that still ring true:

• Even for web pages that have significant backend delays, don’t forget to focus on the frontend. After all, that is the Performance Golden Rule.
• Be careful using script loaders. They have to handle diverse script loading scenarios across a large number of browsers. If you know what you want it might be better to just do it yourself.
• Be careful using JavaScript libraries. In this case jquery.min.js is only being used for the drop down About menu. That’s 84K (~30K compressed) of JavaScript for a fairly simple behavior.

If you’re curious about why document.write results in worse performance for dynamic script loading, I’ll dig into that in tomorrow’s blog post. Hasta mañana.

In yesterday’s blog post, Making the HTTP Archive faster, one of the biggest speedups came from not using a script loader. It turns out that script loader was using document.write to load scripts dynamically. I wrote about the document.write technique in Loading Script Without Blocking back in April 2009, as well as in Even Faster Web Sites (chapter 4). It looks something like this:

document.write('<script src="http://www.stevesouders.com/blog' + src + '" type="text/javascript"></script>'):

The problem with document.write for script loading is:

• Every DOM element below the inserted script is blocked from rendering until the script is done downloading (example).
• It blocks other dynamic scripts (example). One exception is if multiple scripts are inserted using document.write within the same SCRIPT block (example).

Because the script loader was using document.write, the page I was optimizing rendered late and other async scripts in the page took longer to download. I removed the script loader and instead wrote my own code to load the script asynchronously following the createElement-insertBefore pattern popularized by the Google Analytics async snippet:

var sNew = document.createElement("script");
sNew.async = true;
sNew.src="http://ajax.googleapis.com/ajax/libs/jquery/1.5.1/jquery.min.js";
var s0 = document.getElementsByTagName('script')[0];
s0.parentNode.insertBefore(sNew, s0);

Why does using document.write to dynamically insert scripts produce these bad performance effects?

It’s really not surprising if we walk through it step-by-step: We know that loading scripts using normal SCRIPT SRC= markup blocks rendering for all subsequent DOM elements. And we know that document.write is evaluated immediately before script execution releases control and the page resumes being parsed. Therefore, the document.write technique inserts a script using normal SCRIPT SRC= which blocks the rest of the page from rendering.

On the other hand, scripts inserted using the createElement-insertBefore technique do not block rendering. In fact, if document.write generated a createElement-insertBefore snippet then rendering would also not be blocked.

At the bottom of my Loading Script Without Blocking blog post is a decision tree to help developers choose which async technique to use under different scenarios. If you look closely you’ll notice that document.write is never recommended. A lot of things change on the Web, but that advice was true in 2009 and is still true today.

Today Amazon Web Services is introducing Amazon CloudSearch, a new web service that brings the power of the Amazon.com’s search technology to every developer. Amazon CloudSearch provides a fully-featured search engine that is easy to manage and scale. It offers full-text search with features like faceting and user-defined rank functions. And like most AWS services, Amazon CloudSearch scales automatically as your data and traffic grow, making it an easy choice for applications small to large. With Amazon CloudSearch, developers just create a Search Domain, upload data, and start querying.

Why Search?

Search is an essential part of many of today's cloud-centric applications. While in our daily lives we are mostly familiar with the search functionality offered by web search, there are in fact many more cases where search is a fundamental component of an application. Search is a much broader technology than just the indexing of large collections of web pages. Many organizations have large collections of documents, structured and unstructured, that can benefit from a specialized search service. With the rise of the App developer culture there is an increasing number of consumer data sources that cannot be simply queried with a web search engine. Using specialized ranking functions these apps can give their customers a highly specialized search experience.

And increasingly, search is applied to data that, though called a "document" for the purposes of search, is really just a record in a database or an object in a NoSQL system. On the query side, we are used to seeing search results as users, but search results are increasingly being used at the core of complex distributed systems where the results are consumed by machines, not people.

With these applications in mind, our customers have told us that a cloud-based managed search service is high on their wish lists. Their main motivation is that existing search technologies, both commercial and open source, have proven to be hard to manage and complex to configure.

Amazon CloudSearch will have democratization effect as it offers features that have been out of reach for many customers. With Amazon CloudSearch, a powerful search engine is now in the hands of every developer, at our familiar low prices, using a pay-as-you-go model. It will allow developers to improve functionality of their products, at lower costs with almost zero administration. It is very simple to get started; customers can create a Search Domain, upload their documents, and can immediately start querying.

How it Works

Developers set up a Search Domain -- a set of resources in AWS that will serve as the home for one collection of data. Developers then access their domain through two HTTP-based endpoints: a document upload endpoint and a query endpoint. As developers send documents to the upload endpoint they are quickly incorporated into the searchable index and become searchable.

Developers can upload data either through the AWS console, from the command-line tools, or by sending their own HTTP POST requests to the upload endpoint.

There are three features that make it easy to configure and customize the search results to meet exactly the needs of the application.

Filtering: Conceptually, this is using a match in a document field to restrict the match set. For example, if documents have a "color" field, you can filter the matches for the color "red".

Ranking: Search has at least two major phases: matching and ranking. The query specifies which documents match, generating a match set. After that, scores are computed (or direct sort criterion is applied) for each of the matching documents to rank them best to worst. Amazon CloudSearch provides the ability to have customized ranking functions to fine tune the search results.

Faceting: Faceting allows you to categorize your search results into refinements on which the user can further search. For example, a user might search for ‘umbrellas’, and facets allow you to group the results by price, such as $0-$10, $10-$20, $20-$40, etc. Amazon CloudSearch also allows for result counts to be included in facets, so that each refinement has a count of the number of documents in that group. The example could then be: $0-$10 (4 items), $10-$20 (123 items), $20-$40 (57 items), etc.

For more information on the different configuration possibilities visit the Amazon CloudSearch detail page.

Automatic Scaling

Amazon CloudSearch is itself built on AWS, which enables it to handle scale.

Amazon CloudSearch supports both horizontal and vertical scaling. The main search index is kept in memory to ensure that requests can be served at very high rates. As developers add data, CloudSearch increases either the size of your underlying node or it increases the number of nodes in the cluster. To handle growing request rates, the service autoscales the number of instances handling queries.

Amazon CloudSearch is based on more than a decade of developing high quality search technologies for Amazon.com. It has been developed by A9, the Amazon.com subsidiary that focuses on search technologies. The technology that is used at all the different places where you can search on Amazon.com is also at the core of at Amazon CloudSearch.

Summary

With the launch of Amazon CloudSearch the Amazon Web Services remove yet another pain point for developers. Almost every application these days needs some form of search and as such every developer has to spend significant time implementing it. With Amazon CloudSearch developers can now simply focus on their application and leave the management of search to the cloud.

For more information see the Amazon CloudSearch detail pages, the Amazon CloudSearch Developer Guide and the posting on the AWS developer blog.

You can sign up for the Introduction To Amazon CloudSearch webinar on May 10.

Today Amazon Web Services launched AWS Marketplace, an online store that makes it easy for you to find, buy, and immediately start using software and services that run on the AWS Cloud. You can use AWS Marketplace’s 1-Click deployment to quickly launch pre-configured software on your own Amazon EC2 instances and pay only for what you use, by the hour or month. AWS handles billing and payments, and software charges appear on your AWS bill.

Marketplace has software listings from well-known vendors including 10gen, CA, Canonical, Couchbase, Check Point Software, IBM, Microsoft, SAP, Zend, and others, as well as many widely used open source offerings including Wordpress, Drupal, and MediaWiki.

AWS Marketplace brings the same simple and trusted online shopping experience that customers enjoy on Amazon.com to software built for the AWS platform, streamlining the process of doing research and purchasing software. It features a wide selection of development and business software, including software infrastructure, developer tools, and business applications. Product prices are clearly stated and appear on the same bill as your other AWS services.

AWS Marketplace also simplifies many of the challenges software companies face, such as acquiring customers, developing distribution channels, and billing for their software.

Why shop here?

The way businesses are buying applications is changing. There is a new generation of leaders that have very different expectations about how they can select the products and tools they need to be successful. Last week I met with a CIO for a discussion about how her IT department can use AWS to help make their business units be more agile and move faster. One of the stumbling blocks she mentioned was how to select the best software running on AWS, in a way that was completely in line with the “Cloud Experience”: no software to install, no sales cycle, no procurement delays, and a selection of licensing models to choose from. She jokingly asked for an “Amazon 1-Click” experience for software. I am sure she will be a very happy CIO today.

AWS Marketplace features a wide selection of commercial and free IT and business software. AWS Marketplace enables you to compare options, read reviews, and quickly find the software you want.

We wanted to shrink the time between finding what you want and getting it up and running. Once you find software you like, you can deploy that software to your own EC2 instance with 1-Click -- like the CIO suggested -- or using popular management tools like the AWS Console.

In addition, for most products, software prices are clearly posted on the website so you can purchase software immediately, with the payment instrument you already have on file with Amazon Web Services. Software charges appear on the same monthly bill as your AWS infrastructure charges.

Why sell here?

The Amazon Web Services have helped to create great ecosystem of ISVs that are selling software and services to other customers running in the cloud. It has had a true democratization effect: no longer does the dominant vendor in a market automatically get chosen. I have many IT decision makers ask me who are the young and exciting companies they should be paying attention to. Who are the companies that have a native cloud product, who are the ones that have innovative new licensing models, who are the young and hungry companies that break with the old style of enterprise software vending and are truly customer-centric. At the same time the up-and-coming companies often ask me how we can help them get in front of more customers such that they can compete in an open and honest way. And they also often ask whether we can help them with what Amazon.com does so well for its sellers: handle billing and charging.

AWS Marketplace includes both large, well known companies as well as exciting up and coming companies. If you’re a software provider with an offering that runs on the AWS cloud, you can gain new customers, enable usage-based billing without much additional work, and ensure that customers have a fast and easy deployment experience with their software.

AWS Marketplace helps software and SaaS providers find new customers by exposing their products to some of the hundreds of thousands of AWS customers, ranging from individual software developers to large enterprises.

Additionally, if you are interested in adding hourly billing to your software, AWS Marketplace can help. Simply upload an Amazon Machine Image to AWS and provide the hourly cost. Billing is managed by AWS Marketplace, relieving sellers of the responsibility of metering usage, managing customer accounts, and processing payments, leaving software developers more time to focus on building great software.

Summary

At Amazon we have a long experience with buyers and sellers in a marketplace. We know that something great happens when you solve problems for both the people selling things and those buying things – the market becomes more and more vibrant. We know that for buyers it is important to have very convenient ways of discovering and buying products. For sellers it is important to get their products in front of as many relevant customers as possible and make the sales process as painless as possible.

But more important that anything else for both parties is trust: easy to understand product information, high quality, relevant reviews by other customers, that the seller is reputable and has a history of delivery, and that the buyer will only be charged for his exact usage. For the seller it removes the burden of having to manage customers, measuring their usage and collecting payments for it.

The AWS Marketplace is a great step forward in making easier to buy and deploy software. It also makes it dead simple for ISVs for add hourly billing to their offerings and get their software in from of hundreds of thousands of active AWS customers.

For more information see the announcement at the AWS website, the "Introducing AWS Marketplace" video, the posting on the AWS blog and off course visit the AWS Marketplace for a test drive. Happy shopping!

Over the past several years I’ve spent much of my time traveling around the world speaking about distributed systems. From building infinitely scalable data stores, architectures for high performance computing, to the challenges imposed by the CAP theorem, there are wonderful, complex, fascinating problems to be solved in the area of distributed computing. During my travels I’ve met thousands of brilliant engineers who are leveraging the cloud to deliver exciting new products and revolutionize IT as we know it. One thing that’s become obvious to me is that there are innovative, inspiring developers in every corner of the planet from Australia to Iceland and from Israel to Peru.

And that leads me to another distributed problem – finding good engineers to help AWS build the next generation of cloud computing services. We’ve got a big vision and to realize it we need to find qualified engineers to join us on our journey. A quick look at the AWS career web sites reveals that we are hiring hundreds of people around the world.

Click here for our current job openings in the U.S.

Click here for our current job openings in Europe, Asia, and South Africa

Distributed problems call for innovative solutions. So next month we will be taking a distributed approach to finding engineers who want to join AWS. On May 17th and 18th we will be traveling to Houston, Minneapolis, and Nashville to interview candidates who want to join the AWS team. If you live in or near one of those cities and are interested in a meeting with us about careers in AWS check out this page. You can also simply email your resume to aws-recruiting@amazon.com