Several years ago, some suspected cyber criminals on the Internet wrote a family of malware dubbed DNSChanger. About a year ago, law enforcement tracked down the suspected cyber criminals behind this malware, arrested them, and took over the servers they were using to redirect customers to rogue sites.

As a result of a court order, the Internet Systems Consortium (ISC) under the direction of the FBI, has continued to run the DNS servers used by the malware for the last year. However, the court order will soon expire and those servers are scheduled to be shut down on July 9, 2012. When that happens, hundreds of thousands of Internet users whose systems are still infected and/or affected could lose access to the web, email, and anything else that depends on DNS. This is the story of how two Internet infrastructure startups — CloudFlare and OpenDNS — are playing a small part to help solve the problem.

A Bit of DNS Background

Up front, in order to understand this story, you need to understand there are two types of DNS servers: recursive and authoritative. Everyone who uses the Internet needs a recursive DNS server. Your ISP usually provides these types of services or you can use a provider like OpenDNS, Google, DNSAdvantage, other public resolvers, or even run a server yourself to handle your recursive DNS queries.

On the other hand, every domain needs at least one authoritative DNS server. Authoritative servers are where a particular domain's records are hosted and published. Many domain registrars provide authoritative DNS servers, or you can use a service like CloudFlare and we provide authoritative DNS. When an Internet user types a Universal Resource Identifier (URI) aka Universal Resource Locator (URL) into their browser, clicks on a link, or sends an email, their computer queries their recursive DNS provider. If the recursive DNS provider has the answer cached then it responds. If it doesn't have the answer cached, or if the answer it has is stale, then the recursive DNS server queries the authoritative DNS server.

As mentioned above, OpenDNS provides recursive DNS. Their customers are web surfers and they provide a terrific service that helps speed up Internet browsing and protect people on the web from malware. CloudFlare provides authoritative DNS. Our customers are websites and we make those sites faster and protect sites from attacks directed at them. While we're often asked if OpenDNS and CloudFlare are competitive, in reality both services are complementary just using different parts of DNS (recursive and authoritative) to achieve a similar mission: a faster, safer, better Internet.

How Suspected Cyber Criminals Use DNS to Do Bad Things

The DNSChanger malware family was designed to change the recursive DNS server that Internet users’ computers queries. Instead of directing DNS queries at the recursive server you or your ISP configured, the malware modified computer settings to route queries to recursive DNS servers controlled by the suspected cyber criminals.

The job of DNS is to translate a domain name such as dcwg.org, which humans prefer, into an IP address, like 108.162.205.64, which servers and routers can use. If you are a cyber criminal and you can gain control over someone’s recursive DNS then you can direct traffic to certain sites to a fake version of the site. Once DNSChanger had web surfers querying rogue recursive DNS servers, all requests for legitimate websites could be directed to a fake website. For example, even if you typed your bank's domain name into your browser, if the suspected cyber criminals control recursive DNS then they can send you to a malicious site and steal your information.

Over the years DNSChanger operated unchecked, more than a million computers and home routers had their DNS configurations modified. Thankfully, law enforcement was able to track down the suspected cyber criminals behind the malware, arrest them, and seize control of the rogue recursive DNS servers. Unfortunately, hundreds of thousands of computers are still using the formerly rogue recursive DNS servers. On July 9, 2012 the court order directing ISC to operate the servers expires and those servers are scheduled to be shut down. On that date, all systems which still have their DNS settings modified by DNSChanger will effectively be cut off from the Internet.

Getting the Word Out

The DNSChanger Working Group (DCWG), a loosely affiliated organization comprised of some of the world’s largest and most competent ISPs, search engines vendors, software vendors, security companies, and others, has been working to get the word out about the problem and reduce the impact of the shutdown of the DNSChanger recursive servers. The DCWG launched a website (dcwg.org) to provide information about the malware, let people test whether they are infected, and provide recommendations on how to fix their systems. CloudFlare first became involved when the folks at dcwg.org reached out to us because their site was under heavy load after attention from major media outlets. CloudFlare helped keep the dcwg.org website online under the load caused by media attention over the last 10 days. We offloaded more than 95% of the traffic to the site, ensuring the site ran fast and stable even when it was being featured on the front page of cnn.com.

Unfortunately, one of the challenges in trying to address situations like DNSChanger is that you only know to go to the dcwg.org website if you already know about it. What you needed was something akin to an emergency broadcast system that would inform people who were infected that they had a problem as they surfed the web. In the process of working with the DCWG, we realized we might be able to help.

Some of our engineers created an app named Visitor DNSChanger Detector App. Any website on CloudFlare can enable the app with a single click from our apps marketplace. The app installs a small bit of Javascript on the page that tests visitors to see if they're infected. If the tests do not detect anything, nothing happens. If the tests indicate that the DNSChanger recursive servers are being used, then a banner is displayed across the top of the page and visitors are directed to instructions on how to clean up the infection (more on that in a second).

More than 470 million people pass through CloudFlare's network on a monthly basis. Our data suggest that more than half of the people infected with DNSChanger would visit at least one site on CloudFlare per month. The power of the Visitor DNSChanger Detector App is that as CloudFlare publishers enable it then there is an increasing likelihood that people who are infected will get information about their infection before they are no longer able to use the Internet on July 9, 2012.

While we've made it extremely easy for publishers on CloudFlare's network to help get the word out, we didn't want to restrict participation to only those sites using our service. We therefore decided to release the code for the checks publicly and as open source so anyone who can install a few lines of Javascript on their web pages will be able to install it on their own sites to inform their potentially infected users. You can access the code from the following GitHub Repo. We're hopeful that sites both large and small will take the time to install the code in order to help inform their visitors who may be infected.

What Should People Notified of This Infection Do?

While CloudFlare is able to assist with informing web surfers they have an infection, we aren’t particularly well situated to actually fix the problem. After all, it isn’t our customers that are directly impacted, but rather the customers of our customers. Many of the folks infected can get help from their ISPs, but for some this might not be an option. CloudFlare reached out to David Ulevitch, the CEO of OpenDNS and he saw this as a great opportunity to further OpenDNS's mission of helping build a better Internet. We added OpenDNS as a resource for publishers to display to their customers when the Javascript detects the use of the DNSChanger recursive servers.

The Power of the DNS

This incident illustrates to me the importance and power of the DNS system that underpins the Internet. The suspected cyber criminals were able to modify DNS settings to steal advertising revenue and perform other illegal activities. CloudFlare uses authoritative DNS in order to provision powerful tools to make sites faster and even help create a sort of emergency warning system for the Internet. OpenDNS provides high performance recursive DNS caching services for their customers. Combined, we hope to help the DCWG get the word out so the hundreds of thousands of Internet users still impacted by the DNSChanger malware will be able to take steps to ensure they’ll be able to use the Internet on July 10, 2012 and beyond.

Permalink | Leave a comment  »

Today Amazon Web Services launched AWS Marketplace, an online store that makes it easy for you to find, buy, and immediately start using software and services that run on the AWS Cloud. You can use AWS Marketplace’s 1-Click deployment to quickly launch pre-configured software on your own Amazon EC2 instances and pay only for what you use, by the hour or month. AWS handles billing and payments, and software charges appear on your AWS bill.

Marketplace has software listings from well-known vendors including 10gen, CA, Canonical, Couchbase, Check Point Software, IBM, Microsoft, SAP, Zend, and others, as well as many widely used open source offerings including Wordpress, Drupal, and MediaWiki.

AWS Marketplace brings the same simple and trusted online shopping experience that customers enjoy on Amazon.com to software built for the AWS platform, streamlining the process of doing research and purchasing software. It features a wide selection of development and business software, including software infrastructure, developer tools, and business applications. Product prices are clearly stated and appear on the same bill as your other AWS services.

AWS Marketplace also simplifies many of the challenges software companies face, such as acquiring customers, developing distribution channels, and billing for their software.

Why shop here?

The way businesses are buying applications is changing. There is a new generation of leaders that have very different expectations about how they can select the products and tools they need to be successful. Last week I met with a CIO for a discussion about how her IT department can use AWS to help make their business units be more agile and move faster. One of the stumbling blocks she mentioned was how to select the best software running on AWS, in a way that was completely in line with the “Cloud Experience”: no software to install, no sales cycle, no procurement delays, and a selection of licensing models to choose from. She jokingly asked for an “Amazon 1-Click” experience for software. I am sure she will be a very happy CIO today.

AWS Marketplace features a wide selection of commercial and free IT and business software. AWS Marketplace enables you to compare options, read reviews, and quickly find the software you want.

We wanted to shrink the time between finding what you want and getting it up and running. Once you find software you like, you can deploy that software to your own EC2 instance with 1-Click -- like the CIO suggested -- or using popular management tools like the AWS Console.

In addition, for most products, software prices are clearly posted on the website so you can purchase software immediately, with the payment instrument you already have on file with Amazon Web Services. Software charges appear on the same monthly bill as your AWS infrastructure charges.

Why sell here?

The Amazon Web Services have helped to create great ecosystem of ISVs that are selling software and services to other customers running in the cloud. It has had a true democratization effect: no longer does the dominant vendor in a market automatically get chosen. I have many IT decision makers ask me who are the young and exciting companies they should be paying attention to. Who are the companies that have a native cloud product, who are the ones that have innovative new licensing models, who are the young and hungry companies that break with the old style of enterprise software vending and are truly customer-centric. At the same time the up-and-coming companies often ask me how we can help them get in front of more customers such that they can compete in an open and honest way. And they also often ask whether we can help them with what Amazon.com does so well for its sellers: handle billing and charging.

AWS Marketplace includes both large, well known companies as well as exciting up and coming companies. If you’re a software provider with an offering that runs on the AWS cloud, you can gain new customers, enable usage-based billing without much additional work, and ensure that customers have a fast and easy deployment experience with their software.

AWS Marketplace helps software and SaaS providers find new customers by exposing their products to some of the hundreds of thousands of AWS customers, ranging from individual software developers to large enterprises.

Additionally, if you are interested in adding hourly billing to your software, AWS Marketplace can help. Simply upload an Amazon Machine Image to AWS and provide the hourly cost. Billing is managed by AWS Marketplace, relieving sellers of the responsibility of metering usage, managing customer accounts, and processing payments, leaving software developers more time to focus on building great software.

Summary

At Amazon we have a long experience with buyers and sellers in a marketplace. We know that something great happens when you solve problems for both the people selling things and those buying things – the market becomes more and more vibrant. We know that for buyers it is important to have very convenient ways of discovering and buying products. For sellers it is important to get their products in front of as many relevant customers as possible and make the sales process as painless as possible.

But more important that anything else for both parties is trust: easy to understand product information, high quality, relevant reviews by other customers, that the seller is reputable and has a history of delivery, and that the buyer will only be charged for his exact usage. For the seller it removes the burden of having to manage customers, measuring their usage and collecting payments for it.

The AWS Marketplace is a great step forward in making easier to buy and deploy software. It also makes it dead simple for ISVs for add hourly billing to their offerings and get their software in from of hundreds of thousands of active AWS customers.

For more information see the announcement at the AWS website, the "Introducing AWS Marketplace" video, the posting on the AWS blog and off course visit the AWS Marketplace for a test drive. Happy shopping!

Over the past several years I’ve spent much of my time traveling around the world speaking about distributed systems. From building infinitely scalable data stores, architectures for high performance computing, to the challenges imposed by the CAP theorem, there are wonderful, complex, fascinating problems to be solved in the area of distributed computing. During my travels I’ve met thousands of brilliant engineers who are leveraging the cloud to deliver exciting new products and revolutionize IT as we know it. One thing that’s become obvious to me is that there are innovative, inspiring developers in every corner of the planet from Australia to Iceland and from Israel to Peru.

And that leads me to another distributed problem – finding good engineers to help AWS build the next generation of cloud computing services. We’ve got a big vision and to realize it we need to find qualified engineers to join us on our journey. A quick look at the AWS career web sites reveals that we are hiring hundreds of people around the world.

Click here for our current job openings in the U.S.

Click here for our current job openings in Europe, Asia, and South Africa

Distributed problems call for innovative solutions. So next month we will be taking a distributed approach to finding engineers who want to join AWS. On May 17th and 18th we will be traveling to Houston, Minneapolis, and Nashville to interview candidates who want to join the AWS team. If you live in or near one of those cities and are interested in a meeting with us about careers in AWS check out this page. You can also simply email your resume to aws-recruiting@amazon.com

Today Amazon Web Services is introducing Amazon CloudSearch, a new web service that brings the power of the Amazon.com’s search technology to every developer. Amazon CloudSearch provides a fully-featured search engine that is easy to manage and scale. It offers full-text search with features like faceting and user-defined rank functions. And like most AWS services, Amazon CloudSearch scales automatically as your data and traffic grow, making it an easy choice for applications small to large. With Amazon CloudSearch, developers just create a Search Domain, upload data, and start querying.

Why Search?

Search is an essential part of many of today's cloud-centric applications. While in our daily lives we are mostly familiar with the search functionality offered by web search, there are in fact many more cases where search is a fundamental component of an application. Search is a much broader technology than just the indexing of large collections of web pages. Many organizations have large collections of documents, structured and unstructured, that can benefit from a specialized search service. With the rise of the App developer culture there is an increasing number of consumer data sources that cannot be simply queried with a web search engine. Using specialized ranking functions these apps can give their customers a highly specialized search experience.

And increasingly, search is applied to data that, though called a "document" for the purposes of search, is really just a record in a database or an object in a NoSQL system. On the query side, we are used to seeing search results as users, but search results are increasingly being used at the core of complex distributed systems where the results are consumed by machines, not people.

With these applications in mind, our customers have told us that a cloud-based managed search service is high on their wish lists. Their main motivation is that existing search technologies, both commercial and open source, have proven to be hard to manage and complex to configure.

Amazon CloudSearch will have democratization effect as it offers features that have been out of reach for many customers. With Amazon CloudSearch, a powerful search engine is now in the hands of every developer, at our familiar low prices, using a pay-as-you-go model. It will allow developers to improve functionality of their products, at lower costs with almost zero administration. It is very simple to get started; customers can create a Search Domain, upload their documents, and can immediately start querying.

How it Works

Developers set up a Search Domain -- a set of resources in AWS that will serve as the home for one collection of data. Developers then access their domain through two HTTP-based endpoints: a document upload endpoint and a query endpoint. As developers send documents to the upload endpoint they are quickly incorporated into the searchable index and become searchable.

Developers can upload data either through the AWS console, from the command-line tools, or by sending their own HTTP POST requests to the upload endpoint.

There are three features that make it easy to configure and customize the search results to meet exactly the needs of the application.

Filtering: Conceptually, this is using a match in a document field to restrict the match set. For example, if documents have a "color" field, you can filter the matches for the color "red".

Ranking: Search has at least two major phases: matching and ranking. The query specifies which documents match, generating a match set. After that, scores are computed (or direct sort criterion is applied) for each of the matching documents to rank them best to worst. Amazon CloudSearch provides the ability to have customized ranking functions to fine tune the search results.

Faceting: Faceting allows you to categorize your search results into refinements on which the user can further search. For example, a user might search for ‘umbrellas’, and facets allow you to group the results by price, such as $0-$10, $10-$20, $20-$40, etc. Amazon CloudSearch also allows for result counts to be included in facets, so that each refinement has a count of the number of documents in that group. The example could then be: $0-$10 (4 items), $10-$20 (123 items), $20-$40 (57 items), etc.

For more information on the different configuration possibilities visit the Amazon CloudSearch detail page.

Automatic Scaling

Amazon CloudSearch is itself built on AWS, which enables it to handle scale.

Amazon CloudSearch supports both horizontal and vertical scaling. The main search index is kept in memory to ensure that requests can be served at very high rates. As developers add data, CloudSearch increases either the size of your underlying node or it increases the number of nodes in the cluster. To handle growing request rates, the service autoscales the number of instances handling queries.

Amazon CloudSearch is based on more than a decade of developing high quality search technologies for Amazon.com. It has been developed by A9, the Amazon.com subsidiary that focuses on search technologies. The technology that is used at all the different places where you can search on Amazon.com is also at the core of at Amazon CloudSearch.

Summary

With the launch of Amazon CloudSearch the Amazon Web Services remove yet another pain point for developers. Almost every application these days needs some form of search and as such every developer has to spend significant time implementing it. With Amazon CloudSearch developers can now simply focus on their application and leave the management of search to the cloud.

For more information see the Amazon CloudSearch detail pages, the Amazon CloudSearch Developer Guide and the posting on the AWS developer blog.

You can sign up for the Introduction To Amazon CloudSearch webinar on May 10.

In yesterday’s blog post, Making the HTTP Archive faster, one of the biggest speedups came from not using a script loader. It turns out that script loader was using document.write to load scripts dynamically. I wrote about the document.write technique in Loading Script Without Blocking back in April 2009, as well as in Even Faster Web Sites (chapter 4). It looks something like this:

document.write('<script src="http://www.stevesouders.com/blog' + src + '" type="text/javascript"></script>'):

The problem with document.write for script loading is:

• Every DOM element below the inserted script is blocked from rendering until the script is done downloading (example).
• It blocks other dynamic scripts (example). One exception is if multiple scripts are inserted using document.write within the same SCRIPT block (example).

Because the script loader was using document.write, the page I was optimizing rendered late and other async scripts in the page took longer to download. I removed the script loader and instead wrote my own code to load the script asynchronously following the createElement-insertBefore pattern popularized by the Google Analytics async snippet:

var sNew = document.createElement("script");
sNew.async = true;
sNew.src="http://ajax.googleapis.com/ajax/libs/jquery/1.5.1/jquery.min.js";
var s0 = document.getElementsByTagName('script')[0];
s0.parentNode.insertBefore(sNew, s0);

Why does using document.write to dynamically insert scripts produce these bad performance effects?

It’s really not surprising if we walk through it step-by-step: We know that loading scripts using normal SCRIPT SRC= markup blocks rendering for all subsequent DOM elements. And we know that document.write is evaluated immediately before script execution releases control and the page resumes being parsed. Therefore, the document.write technique inserts a script using normal SCRIPT SRC= which blocks the rest of the page from rendering.

On the other hand, scripts inserted using the createElement-insertBefore technique do not block rendering. In fact, if document.write generated a createElement-insertBefore snippet then rendering would also not be blocked.

At the bottom of my Loading Script Without Blocking blog post is a decision tree to help developers choose which async technique to use under different scenarios. If you look closely you’ll notice that document.write is never recommended. A lot of things change on the Web, but that advice was true in 2009 and is still true today.

This week I finally got time to do some coding on the HTTP Archive. Coincidentally (ironically?) I needed to focus on performance. Hah! This turned out to be a good story with a few takeaways – info about the HTTP Archive, some MySQL optimizations, and a lesson learned about dynamic script loaders.

Setting the stage

The HTTP Archive started in November 2010 by analyzing 10K URLs and storing their information (subresource URLs, HTTP headers, sizes, etc.) in a MySQL database. We do these runs twice each month. In November 2011 we began increasing the number of URLs to 25K, 50K, 75K, and finally hit 100K this month. Our goal is to hit 1M URLs by the end of 2012.

The MySQL schema in use today is by-and-large the same one I wrote in a few hours back in November 2010. I didn’t spend much time on it – I’ve created numerous databases like this and was able to quickly get something that got the job done and was fast. I knew it wouldn’t scale as the size of the archive and number of URLs grew, but I left that for another day.

That day had arrived.

DB schema

The website was feeling slow. I figured I had reached that curve in the hockey stick where my year-old schema that worked on two orders of magnitude less data was showing its warts. I saw plenty of slow queries in the log. I occasionally did some profiling and was easily able to identify queries that took 500 ms or more; some even took 10+ seconds. I’ve built big databases before and had some tricks up my sleeve so I sat down today to pinpoint the long poles in the tent and cut them down.

The first was pretty simple. The urls table has over 1M URLs. The only index was based on the URL string – a blob. It took 500-1000 ms to do a lookup. The main place this happens is looking up the URL’s rank, for example, in the last crawl Whole Foods was ranked 5,872 (according to Alexa). This is a fairly non-critical piece of information, so slowing down the page 500-1000 ms wasn’t acceptable. Plus this seems like a simple lookup ripe for optimizing.

When I described this problem to my Velocity co-chair, John Allspaw, he suggested creating a hash for the URL that would be faster to index. I understood the concept but had never done this before. I didn’t find any obvious pointers out there on “the Web” so I rolled my own. I started with md5(), but that produced a fairly long string that was alphanumeric (hex):

select md5("http://www.wholefoodsmarket.com/");
=> 0a0936fe5c690a3b468a6895efaaff83

I didn’t think it would be that much faster to index off the md5() hex string (although I didn’t test this). Assuming that md5() strings are evenly distributed, I settled on taking a substring:

select substring(md5("http://www.wholefoodsmarket.com/"), 1, 4);
=> 0a09

This was still hex and I thought an int would be a faster index (but again, I didn’t test this). So I added a call to conv() to convert the hex to an int:

select conv(substring(md5("http://www.wholefoodsmarket.com/"), 1, 4), 16, 10);
=> 2569

I was pretty happy. This maps URLs across 64K hashes. I’m assuming they’re evenly distributed. This conversion is only done a few times per page so the overhead is low. If you have a better solution please comment below, but overall I thought this would work – and it did! Those 500+ ms queries went down to < 1 ms. Yay!

But the page was still slow. Darn!

Duh – it’s the frontend

This and a few other MySQL changes shaved a good 2-3 seconds of the page load time but the page still felt slow. The biggest problem was rendering – I could tell the page arrived quickly but something was blocking the rendering. This is more familiar performance territory for me so I gleefully rolled up my sleeves and pulled out my WPO toolbox.

The page being optimized is viewsite.php. I used WebPagetest to capture a waterfall chart and screenshots for Chrome 18, Firefox 11, IE 8, and IE 9. The blocking behavior and rendering times were not what I consider high performance. (Click on the waterfall chart to go to the detailed WebPagetest results.)

Chrome 18:

Firefox 11:

Internet Explorer 8:

Internet Explorer 9:

These waterfall charts looked really wrong to me. The start render times (green vertical line) were all too high: Chrome 1.2 seconds, Firefox 2.6 seconds, IE8 1.6 seconds, and IE9 2.4 seconds. Also, too many resources were downloading and potentially blocking start render. This page has a lot of content, but most of the scripts are loaded asynchronously and so shouldn’t block rendering. Something was defeating that optimization.

Docwrite blocks

I immediately honed in on jquery.min.js because it was often in the critical path or appeared to push out the start render time. I saw in the code that it was being loaded using Google Libraries API. Here’s the code that was being used to load jquery.min.js:

<script src="http://www.google.com/jsapi"></script>
<script>
google.load("jquery", "1.5.1");
</script>

I’ve looked at (and built) numerous async script loaders and know there are a lot of details to get right, so I dug into the jsapi script to see what was happening. I saw the typical createElement-insertBefore pattern popularized by the Google Analytics async snippet. But upon walking through the code I discovered that jquery.min.js was being loaded by this line:

m.write('<script src="http://www.stevesouders.com/blog'+b+'" type="text/javascript"></script>'):

The jsapi script was using document.write to load jquery.min.js. While it’s true that document.write has some asynchronous benefits, it’s more limited than the createElement-insertBefore pattern. Serendipitously, I was just talking with someone a few weeks ago about deprecating the jsapi script because it introduces an extra HTTP request, and instead recommend that people just load the script directly. So that’s what I did.

We don’t need no stinkin’ script loader

In my case I knew that jquery.min.js could be loaded async, so I replaced the google.load code with this:

var sNew = document.createElement("script");
sNew.async = true;
sNew.src="http://ajax.googleapis.com/ajax/libs/jquery/1.5.1/jquery.min.js";
var s0 = document.getElementsByTagName('script')[0];
s0.parentNode.insertBefore(sNew, s0);

This made the start render times and waterfall charts look much better:

Chrome 18:

Firefox 11:

Internet Explorer 8:

Internet Explorer 9:

There was better parallelization of downloads and the start render times improved. Chrome went from 1.2 to 0.9 seconds. Firefox went from 2.6 to 1.3 seconds. IE8 went from 1.6 to 1.1 seconds. IE9 went from 2.4 to 1.0 seconds.

This was a fun day spent making the HTTP Archive faster. Even though I consider myself a seasoned veteran when it comes to web performance, I still found a handful of takeaways including some oldies that still ring true:

• Even for web pages that have significant backend delays, don’t forget to focus on the frontend. After all, that is the Performance Golden Rule.
• Be careful using script loaders. They have to handle diverse script loading scenarios across a large number of browsers. If you know what you want it might be better to just do it yourself.
• Be careful using JavaScript libraries. In this case jquery.min.js is only being used for the drop down About menu. That’s 84K (~30K compressed) of JavaScript for a fairly simple behavior.

If you’re curious about why document.write results in worse performance for dynamic script loading, I’ll dig into that in tomorrow’s blog post. Hasta mañana.

What a crazy few weeks. Since my crazy, fuck you to homophobia coming out post almost a month ago, I’ve seem to have unintentionally gone into full overdrive with the social issues in technology culture posts: discussing how sexism in technology conferences insults everybody by making women and gay men invisible, and by portraying straight men as stupid, misogynistic idiots who think only with their dicks. Then there was the Brendan Eich post. Which wasn’t really about Brendan Eich so much as about whether it’s legitimate to call Eich out and the tired old trope about how the oppressed become the oppressors by, err, talking about homophobia on Twitter. Then to round it off, a post on the reaction to Ryan Funduk’s post about drinking in tech culture.

I wanted to share a little reading list: Derailing for Fun and Profit by Peter Aronoff, which deals with the ever so tiresome response “oh, but they have a right to their opinion”. Which is really a big old red herring. Sure, people have the right to an opinion. You have the right in the strict legal sense to believe that Queen Elizabeth II is actually a shape-shifting lizard from the Draco constellation. Only I also have the right to consider that absolutely fucking crazy and to think that you are off your rocker.

Chris Heilmann has written a post discussing whether Twitter is a good place to have these kinds of arguments, and includes the excellent TEDx video from Jay Smooth on racism.

Chris is obviously right about the potential for grandstanding and sloganeering on platforms like Twitter. But, I think he goes too far. DISREGARD THAT, I’m an idiot. Chris wasn’t saying what I think he was saying. Apologies.

I think that with a few obvious limits,¹ honesty is a better policy than hushing things up in order to give outsiders the view that the tech community is free of disagreement. However painful talking about things like sexism and homophobia and the social issues around geek culture can be, it’s better to have the conversation than keep quiet.

Finally, read Natalie Reed’s Hipster Misogyny. Because the reaction to the Boston API Hackathon thing was so clearly hipster misogyny. I didn’t cover it in my post as that wasn’t what the post was about. Remember what they said in their non-apology apology? “While we thought this was a fun, harmless comment poking fun at the fact that hack-a-thons are typically male-dominated, others were offended.”

Yeah, LOL GUYZ WHY SO SERIOUS ON THE SEXISM SHIT? That about sums it up.

Amazon ElastiCache makes it easy for you to deploy, scale, and run a cloud-based in-memory cache that is protocol-compliant with Memcached. ElastiCache improves the performance of web applications and reduces the load on your databases by retrieving data from a fast, managed, Memcached-compatible, in-memory caching system, instead of relying entirely on disk-based storage. It can significantly improve throughput for read-heavy or compute-intensive workloads including Social Networking, Mobile and Social Gaming, E-Commerce Sites, Media Sites, and Recommendation Engines.

In order to make ElastiCache an even better value, we are adding a full suite of Reserved Cache Node options -- Light, Medium, and Heavy with both 1 and 3 year terms. See the ElastiCache pricing for additional information. Reserved Cache Nodes can provide savings of up to 70% compared to On-Demand pricing. More information, including pricing, is available on our new Reserved Cache Nodespage.

You can easily migrate from Memcached to ElastiCache using our How Do I Migrate FAQ as a guide; you can also use the ElastiCache CloudFormation template to launch a cache cluster.

Finally, you'll also find useful information in the recorded version of our "Turbo-charge Your Apps Using Amazon ElastiCache" webinar:

-- Jeff;

From tax preparation to safe social networks, Amazon RDS brings new and innovative applications to the cloud

Empowering innovation is at the heart of everything we do at Amazon Web Services (AWS). I often get to meet, discuss, and learn from innovators how they are using AWS to deliver transformative applications to their users, customers and partners. Often we think about innovation as doing 'new things' or based on revolutionary new technologies such as DynamoDB, but it is more important to ensure that one can also innovate based on existing paradigms. One of the services that is very successful in driving innovation at our customers in this context is Amazon RDS, the Relational Database Service. Amazon RDS removes the headaches of running a relational database service reliably at scale, allowing Amazon RDS customers to focus on innovation for their customers.

Recently I had great conversations with Troy Otillio, Senior Development Manager at Intuit and Jack Murgia, Senior DevOps Engineer at Edmodo. Troy and his team have added a contextual social offering to the popular TurboTax and Intuit applications. Jack and his engineers have created a safe social app for teachers and students. These innovators use Amazon RDS in conjunction with other Amazon Web Services to build, scale and operate their applications. Below is my dialog with them. Read on.

Note: If you want to see how Amazon RDS can enable your creative agenda, sign up for the 60 day free trial.

Troy, Jack, Tell me a little bit about your app. What's unique and innovative about your service?

Troy: Live Community Platform is Intuit's flagship Contextual Social offering – Live Community makes it easy to find answers when and where you need them. This is a unique and innovative platform.

• Intelligent Social network - Facilitate topical Q&A conversations among employees, customers and our most valued super contributors.
• Large Seasonal Peaks – Our largest community supports TurboTax where the peak traffic during February or April is often 100's of times greater than a quiet day in June. Live Community Experience is deeply integrated into the tax experience, so we built a highly responsive and reliable web experience.
• Read-your-mind contextual integration – Our core innovation and underlying secret sauce involves selecting the most relevant content for a given page if not given user – to provide the right answer at the right time to our users.

Jack: Edmodo is the safe social network for education used by a network of over 6 million teachers and students worldwide that allows teachers to create and maintain their classroom communities. Some unique and innovative characteristics of Edmodo are:

• Edmodo is as easy to use as other social network sites, but secure - the teacher has the same control over access, content and behavior in Edmodo as he/she does in the classroom.
• Students gain experience they need for the modern workplace, learning how to work responsibly and effectively in a collaborative, project-based manner on a social website.
• Teachers can use Edmodo to share educational content, manage projects and assignments, handle notifications, and conduct quizzes and events.
• Teachers can interact with their colleagues in professional learning networks.
• Schools and districts can claim unique Edmodo web addresses for added communication and customization.

How are your users adopting and responding to your service?

Troy: We moved our service from internal servers to AWS. Our 25+ million strong TurboTax and Intuit user community grows every year and Live Community is an integral component of the overall product experience. Moving to AWS has enabled us with operational agility to deliver more value to those customers without having to worry about scale and infrastructure maintenance. We now have more time to focus on innovation while being confident that when demand increases we can easily add more capacity.

Jack:Since our launch in late 2008, we've grown to over 6 million teachers and students globally primarily through word of mouth of teachers who have shared Edmodo with each other. In addition to using Edmodo to engage students in classroom activities, teachers all over the world build profile pages on Edmodo, which they use to discover and share content, meet and stay in contact with other educators, and best practices and top resources.

Jack, how did this idea come about? How did you choose a SQL approach to solve this problem?

Jack: After years of seeing teachers struggle to share the web with their classroom, Edmodo founders Nic Borg and Jeff O'Hara knew there was a need for a highly scalable, secure social network targeted at K-12. SQL was the right choice because it was an established and proven technology for use in similar environments, and the massive knowledge base that exists around it.

And how about you Troy? Why did you choose a SQL approach to build your social community app?

Troy: The initial architecture was based on MySQL– we've continued with use of SQL but are now leveraging RDS. Of course, with as much textual data as we have we are leveraging Lucene/SOLR (a NoSQL solution) for Search and Semantic processing. More recently we've expanded our platform to include additional forms of user interaction observation in support of our real-time analytics – here we've begun to leverage NoSQL technologies like Redis. Going forward we'll continue to employ a hybrid approach using RDS for the necessary transactional computation and services like DynamoDB for high performance and scalability for structured data.

What did you find unique about RDS? What has been your experience so far?

Troy: We love RDS – it's reduced our operational workload by a noticeable factor but even more exciting is the benefits around fast recovery enabled by the Multi-Availability Zone capability. My team often brags about the one-click creation of read replicas, ability to upsize or downsize the database without downtime and automatic back-up. However, the shining moment occurred just last month – during peak load there was a hardware failure on the Server powering a RDS Master Database – RDS automatically failed over to the alternate zone within minutes and our customers experience was fully functional shortly thereafter. The best part was that the entire process was what I call Òhands freeÓ and took near zero development effort. With self-hosted databases we would have invested considerable engineering effort to implement, test and retest failover – to achieve fast recovery with RDS we simply changed our configuration. And when the actual production event took place the recovery required no manual intervention – the response from our CTO after hearing what happened: "that's cool".

We encountered a few situations that required help from the Amazon team – for example, we didn't know that the I/O capacity of the Server is governed by the size of storage and size of the server. When we first attempted to load our production database it took 28 hours – after a few days of attempts to reduce the load time through well-known optimizations (mostly documented on the RDS website) we were stuck at 8 hours. We consulted directly with Amazon and learned that the storage and the DB Server size affected I/O throughput – after altering our size we dropped our load time to 1 hour which was within expectations relative to native database.

Jack: Based on our experience during this period of phenomenal growth while our team productivity is stretched to the max, we see that:

• RDS is a huge time-saver
• RDS provides peace of mind about our data

Anything that saves time and simplifies processes for employees of a young startup has a positive affect that CAN NOT be overstated. The peace of mind part needs no explanation. Nobody on our team regrets moving to RDS MySQL - quite the opposite; we all agree we don't want to think about where we would have been without RDS. We have been able to meet our goal of architecting our application for 0% "maintenance downtime"

Out of the box, RDS' CloudWatch data and graphs speed up the troubleshooting process.

• Complete certainty in a DB environment is VERY unique- we never worry that:
• our DB parameters are identical across replicas, and changes propagate at a time of my choosing
• the recoverability of data

This is great to hear. We are glad we RDS meets your needs. Now, what's next on your innovation agenda?

Jack: We want to deliver an even more performance and rock solid experience for our global user base of teachers, students, administrators and parents. We will be:

• Building "incident managers" which utilize the cloud watch data and AWS APIs to automatically replace servers and/or re-deploy when problems arise.
• Building "incident creators" - servers which test our ability to maintain peak performance. 

We believe that by leveraging the services that Amazon provides to the fullest we can continue to scale our exceptional user experience so that Edmodo can be the platform for classrooms around the world on devices of all shapes and sizes.

Troy: At Intuit, we can go further to leverage the benefits of elasticity and further improve our resiliency. We are investing in use of CloudFormation coupled with Chef – the result will enable us to lower costs and further reduce risk.

Prior to AWS we had several tiers that we now think can be delegated to AWS services – this should free up our team to focus on our domain problems. For example, we are intending to replace our EC2/Memcache tier with ElastiCache, our batch processing with Simple Workflow and Web servers with CloudFront.

With our newfound agility we can launch new services quickly and there are a few on our plate in the near term. In some cases we are refactoring our system into smaller, discrete services, while in other cases we are creating wholesale new services Our core problem domain consists of extracting greater value out of textual and behavioral data which means that use of EMR and even the newly released Workflow should enable us to focus more on the domain and less on the system engineering.

Troy, Jack, Thank you both very much for sharing your unique experience. I look forward to hearing your progress.

Jack: Thank you. This has been a great dialog.

Troy: Thank you Werner. We appreciate the opportunity.

As I noted before, it's a great pleasure to talk to these innovators and how AWS helps their journey. If you have never used RDS before, you can sign up for a 60 day free trial What innovation will you bring to market? How will it change the world? We won't know until you try and build something.

We hung out with the DreamHost team for the first time at HostingCon in August 2011. They threw a great party, had awesome t-shirts, and exuded the kind of excitement and passion for hosting that CloudFlare has for making websites faster and more secure. We immediately knew we wanted to partner with them.
 
Fast forward nine months to today and we are happy to announce that DreamHost is now an official Certified Hosting Provider. Beginning today they're offering CloudFlare to all their customers with a one-click-simple integration. Prior to this partnership, we had thousands of DreamHost customers who signed up for CloudFlare directly through our site. Now, every DreamHost customer has simple, easy access to CloudFlare with a click of a button and without having to mess with their DNS. Other bells and whistles like mod_cloudflare are now included in all the default DreamHost configurations, so even existing CloudFlare users on the DreamHost network will benefit.

CloudFlare Plus

DreamHost is the latest CloudFlare Certified Hosting Partner, a program that makes CloudFlare one-click simple for any hosts to provide to their customers. We're trying a new experiment and allowingDreamHost to offer a special plan they've dubbed CloudFlare Plus. We worked with them to create this custom CloudFlare plan with the features they thought would be the most interesting for their customers and a price point lower than our current Pro product. For those folks for whom CloudFlare Pro is a bit more than they need, DreamHost now offers another option with some of our most popular paid features.

Just Sayin': CloudFlare ≈ Voltron

I do have to say that DreamHost will also always hold a special place in my heart for what must be the most fun press release I've ever seen, a full copy of which is below. If you only read one part, read the quote near the end that I bolded which is certifiably awesome.

FOR IMMEDIATE RELEASE

DreamHost Partners With CloudFlare

CloudFlare to Provide Free Site Performance Optimization and Security Services for all DreamHost customers

LOS ANGELES, California—April 5th, 2012—DreamHost, a global full-service web hosting company, has today announced a partnership with leading Internet web performance and security company, CloudFlare. DreamHost customers now have immediate access to CloudFlare's robust infrastructure at little or no cost as a standard feature of their hosting plans.

Shared web hosting customers, for many years, have rarely been able to take advantage of the benefits provided by Content Delivery Networks as a result of either the cost or complexity involved. CloudFlare has removed both barriers to entry by distilling the entire setup and configuration process down to a single checkbox and removing the cost entirely. CloudFlare brings the performance and security tools previously available only to Internet giants to anyone with a website.

Hundreds of nodes around the globe power CloudFlare's network ensuring that websites load quickly and consistently, regardless of where in the world users happen to be. CloudFlare's Anycast technology works with static and dynamic sites, routing users to the node on their network for the fastest performance — all without breaking a sweat!

CloudFlare's “Always Online” technology ensures that sites taking advantage of the CloudFlare platform will remain online, continuing to serve cached content, even if the hosting servers on which they are housed become temporarily unreachable. If the hosting industry had a holy grail, it might look a little something like CloudFlare.

In addition to CloudFlare's free offering, DreamHost has also worked with the CloudFlare team to create “CloudFlare Plus,” a bundling of CloudFlare's most popular features available exclusively to DreamHost customers. CloudFlare Plus is an optional paid upgrade, weighing in at $9.95 per month, and adds automatic image optimization and support for Secure Socket Layer (SSL) connections. Provisioning of either option has been integrated within the DreamHost customer control panel.

“When we first met with the CloudFlare team at HostingCon 2011 we had no idea if what they were telling us was true,” said Kathy Brahm, DreamHost's Vice President of Customer Experience and Partnerships. “You know how typical sales people can be — schmoozy, smiley, 'Let-me-buy-you-dinnery', handsy … all the while making outrageous claims about their product. CloudFlare's team has been the complete opposite of those — and a true pleasure to deal with. We've spent the past few weeks putting CloudFlare through its paces and running some tests of our own. Some of us nearly fainted when the first speed tests came back. One guy cried. Me? I buried my feelings deep inside so I don't have to deal with them. It's just what I do.”

“From days of long ago, from uncharted regions of the web, comes a legend; the legend of CloudFlare, Defender of the Interwebs: a mighty robot, loved by good, feared by evil,” explains Matthew Prince, co-founder and CEO of CloudFlare, doing his best Peter Cullen impersonation. “As CloudFlare's legend grew, peace settled across the network. From Los Angeles, an Interweb Alliance was led by DreamHost. Together with other good hosts of the network, DreamHost helped maintain peace throughout the Interwebs, until a new horrible menace threatened. A closer relationship with CloudFlare was needed. This is the story of the super force of web explorers, specially trained by DreamHost, to more tightly integrate CloudFlare, Defender of the Interwebs!”

CloudFlare's free offering and the DreamHost-exclusive “CloudFlare Plus” are now available to all DreamHost customers.

Permalink | Leave a comment  »

Read more »

I've been following the products of WonderNetwork for a while as they do some interesting stuff with servers around the world. I particularly like Wonder VPN as a drop dead simple and reliable VPN is very handy for any mobile user who wants some security when using a wireless network in Starbucks!

Recently they have been working on a new product called Natural Load Testing which is intended to make load testing your web application very simple. It's a very friendly service which is immediately apparent on the home page where it says "not logged in, no greeting. how sad". It's a little touch that appeals to me.

Natural Load Testing is currently in private beta and the WonderNetwork guys invited me into the beta and have been generous enough to spend a small part of their marketing budget to enable me to be able to publish this article. They've even managed to cope with my list of complaints and still talk to me! In this tutorial I will walk through how to use the Natural Load Testing product (as is today) to get some results. Interpreting the results and improving your website and server infrastructure is your job though :)

Getting Started

Natural Load Testing (shortened to NLT in this article!) consists of 4 main areas where you interact with it:

• Creating tests
• Configuring test suites
• Running test suites
• Reviewing results

Upon log in, the home page consists of a set of friendly buttons:

We start at the beginning: creating tests.

Creating tests

When load testing you need to configure your testing tool with the URLs of the pages and resources the you want to be tested. Natural Load Testing makes this job trivial by cleverly leveraging their proxy technology. By configuring your browser to use the NLT proxy, it will record every page you visit and provide you with a list of URLs which you can then use to create tests.

Obviously, as load testing involves hitting the server with a lot of requests, by the time NLT is out of beta, you will need to authorise each domain that you want to use. This is done in the "Manage Authorised Domains" section.

There is a helpful page on the NLT site that explains how to do this. Don't forget to turn off the proxy after recording. The WonderNetwork people have also remembered that you may want to record HTTPS traffic for load testing, and have provided a root certificate that you'll need to install. Again, there's help available on how to install this.

Once you have recorded some URLs, you can then click on the "Create" button and you are presented with a list of the most recent URLs you have recorded.

To create the test, you select the URLs that you require to be included (the checkbox in the header selects all within the group, which is very handy!). You then choose a name that will help you remember what this test does and create the test.

In order to create some interesting load tests on our site, I created a good few tests which I could then group into test suites.

Configuring test suites

Your tests are grouped into suites that are then run either sequentially or as a set of random loads. To create a suite, you click on the "Configure" button which then shows you the list of the suites that you have already created.

In this screenshot you can see two test suites that I've created. The second test "ZF2 Tutorial only" is the simplest suite possible as it contains a single page load within a single step. The top suite shows a more complex suite which hints at the complexity of the suites that you can create.

A suite is composed of one or more steps which are run in sequence. Each step can have multiple tests. If there are multiple test in the same step, then NLT will randomly pick one test for each run.
In this suite I have four tests in step 1 which are a set of pages within the BRI website. Step 2 is the contact page and so this test suite is measuring visiting one of 4 pages within the site and then choosing to go to the contact page.

Creation of a new test suite is done via drag and drop:

You simply drag your green tests over to the list of steps on the right hand side. It's all quite easy. Rather weirdly, you can't reuse the same test in multiple steps, so if you're testing a cycle, then you need to duplicate the same test so that you can place it into two different steps. I haven't found a way to edit the steps within a test suite once created either, so make sure you get it right!

Once you have added tests to steps, you can then tell NLT to send data whilst load testing. This is useful for filling in forms or logging into the website, for instance. I haven't used this section as I haven't yet tested any pages with forms or requiring login.

Rather useful fully, you can change the domain that a test suite uses. This enables you to duplicate a test suite and then change the domain to test domain that's running on another server or using a different configuration which makes side-by-side testing a little easier.

Calibration

Once we have created a test suite, the next step is to calibrate it. NLT will not allow you to run a test suite before calibration, so don't forget this bit! Calibration is done from the test suites list page where there is a drop down box of operations you can do on each suite:

Simply pick Calibrate and press Go. NLT will then perform a single run over your suite and present the standard results page. Ideally, I would like to see this page look different from the standard results page as it's a calibration run, not a standard test and so I was slightly confused at this point. Also, whilst the test is running, you simply see one of those "spinning gifs" to let you know that something is happening. I ran into a bug in this section which resulted in the calibration run failing, but I was never notified on the page that an error had occurred.

The calibration results on my ZF2 tutorial page suite look like this:

The intent of calibration is to provide a base-line to the NLT of the performance of the suite when there is no load. i.e. the idea is that this single run has provided an indication of the optimal conditions and we are expecting to be within 15% of this performance when testing under load.

Running test suites

Now that we have calibrated test suites, we can run some load tests and see what happens. In NLT parlance, we run test suites by pressing the Play button at which point the terminology changes to from play to execute. Simply select your test suite and press the Execute button. You are then presented with a form in order to set the run parameters:

There are three parameters you need to set:

We set these numbers up to create the required load testing profile. I've been using 100 users at 250ms spin-up for 2000 total executions as this takes around 30 to 60 seconds to complete a run and seems a reasonable profile for my blog given its traffic levels.

Upon pressing execute, NLT will spin up a number of worker processes on its servers and then display a graph that updates every few seconds showing you what happening:

As you can see in the screenshot above, you get this information:

• A bar showing number of active requests (blue) on top of number of requests initiated.
• A line chart of median response time.

The x-axis is in seconds. Hence the red bar chart shows the number of requests initiated during each second and the blue bar shows the number of requests that were still active at the end of the second. Hence a request that completed within a second is counted in the red bar, but not in the blue. Ideally, therefore we would like to see smaller blue bars than red bars and we definitely don't want to see blue bars getting bigger.

The green line shows us the average time it takes to server all requests in that second. Ideally, we want this number to be lower. Clearly, if there are any errors serving a request (e.g. an nginx bad gateway error), the rather fast failure time is not counted as it would bring the average down!

We can also view the data as a table:

Obviously, this is right at the start of the run, and so all look good. Clicking on a given run id will provide us with more information. This is a run that's later in the test:

As you can see, the server was struggling now! The red background means that the request took more than 15% longer than the calibration run and the difference from calibration is shown in brackets in the Response Time column. In this particular run, the main HTML took 760ms to deliver which as 411ms longer than the calibration run. We're also struggling to serve images in a sensible time. This is clearly not ideal!

Looking at what's going on on the server whilst running a load test is also instructive. A good introduction to the server tools that are useful is 16 Linux server monitoring commands you really need to know by Steven Vaughan-Nichols.

Reviewing results

Once you have a few runs under your belt, the Review page becomes useful. In this page we can see a list of all our previous runs and most importantly, we can edit the title of each run and give it a useful name. It would be nice to be able to store additional notes about each run though.

Adding the title makes it much easier to remember why a given run was performed and I've found it useful.

Using Natural Load Testing to improve performance

The obvious first target for testing NLT, was my blog at akrabat.com. This is a simple WordPress blog which recently moved to a new server. I haven't been particularly worried about performance and don't really expect to ever be slash-dotted (or is it Cal-dotted, nowadays!) any time soon.

However, as I've done nothing to tune the system, this is a good time to see how it behaved. The first run produced these results:

This isn't good! The only good sign was that none of the 2000 runs actually resulted in an error. A median response time around 1.5 seconds didn't sound good and the maximum run time was 13 seconds!

Something had to be done!

A little bit of investigating showed that neither APC or WP Super Cache was installed on the server. So I turned them on.

The effect of APC can be seen here:

We now have an average response time of much less than 500ms which seems much saner. There were a number of very long requests of over 1 second though.

To see if I could improve this further, I then installed and enabled WP Super Cache in PHP mode and re-ran Natural Load Testing. The results were:

Note that you need to be careful when comparing graphs as the axes are automatically scaled. Looking at the results with WP Super Cache was enabled, I can see that the response time display of the graph is much smoother. The tabular data backs this up as it shows that the vast majority of requests were taking less than 200ms with significantly fewer runs taking much longer. Also, the curve of the number of requests initiated vs request active at the end of each second is much smoother, which indicates that with WP Super Cache enabled, the server should be able to hold its own over a longer time period much more easily.

I tested this hypothesis by doing 5000 runs rather than the 2000 shown above. The result was:

Again, the median response time is nicely under 150ms for most of the time, but we have more variance in the response time, with a couple of seconds where the median was significantly over 250ms.

On the whole, my investigations with NLT have made my blog much more responsive and more likely to be able to handle more traffic than before I started this process.

Other thoughts

Having walked though what Natural Load Testing does, I also need to point out that when using it, you can tell that this is a product still in beta! Incidentally, the WonderNetwork people have responded to my reports quickly, even if half the time it was to let me know that my idea or complaint was already on their list!

Nearly all the issues I have with NLT at the moment are related to usability, which I'm sure will be ironed out over time. My "favourite" annoyance as noted above is on the create test suite page where the Save button is above the section where you set up the suite's steps. If you accidentally press Save after entering a title, then you find you can't actually add any tests to the steps! There's also no menu bar or easy way to get from one section to another. You very quickly learn to click on the NLT logo at the top which takes you to the home page.

It would also be good if NLT would allow me to store "execution profiles" so that I don't have to keep typing in that I want 100 users for 2000 total runs.

I would also like to see more aggregate results. In particular, I'd like to have stats on the response time for say the 50th and 95th percentile. I'd also like to graph these against multiple runs along with the run title so I can see how the site's performance changed over time. This then leads to the idea that it would be nice if I could schedule a load test once a month at 3am local time!

Conclusion

Natural Load Testing is the first product I've used that makes me actually want to do load testing. It makes it easy to run a specific load test and repeat exactly the same test as frequently as you need to. Tweaking the server to see the effect that any given change has becomes an interesting task and your websites can only benefit as you reduce and remove the bottle necks that you find.

On the whole, I'm quite impressed with Natural Load Testing and can see that it could turn into a very useful tool in the web developer's toolbox.

As I mentioned at the top, NLT is currently in private beta. If you want to get an invite, head over to the sign up form and fill in your details. You also have to read the thank you for signing up message!

This post tells a story.

A long time ago, I set out to write my own blog platform. Yes, WordPress is a fine blogging platform, as is Serendipity (aka "s9y", and my previous platform). And yes, I know about Habari. And, for those of you skimming ahead, yes, I'm quite aware of Jekyll, thank you anyways.

Why write something of my own? Well, of course, there's the fact that I'm a developer, and have control issues. Then there's also the fact that a blog is both a simple enough domain to allow easily experimenting with new technology and paradigms, while simultaneously providing a complex enough domain to expose non-trivial issues.

When I started this project, it was a technology-centered endeavor; I wanted to play with document databases such as CouchDB and MongoDB, and with caching technologies like memcached and redis.

Not long after I started, I also realized it was a great playground for me to prototype ideas for ZF2; in fact, the original DI and MVC prototypes lived as branches of my blog. (My repository is still named "zf2sandbox" to this day, though it technically houses just my site.)

Over time, I had a few realizations. First, my actual blog was suffering. I wasn't taking the time to perform security updates, nor even normal upgrades, and was so far behind as to make the process non-trivial, particularly as I had a custom theme, and because I was proxying to my blog via a ZF app in order to facilitate a cohesive site look-and-feel. I needed to either sink time into upgrading, or finish my blog.

My second realization, however, was the more important one: I wanted a platform where I could write how I want to write. I am a keyboard-centric developer and computer user, and while I love the web, I hate typing in its forms. Additionally, my posts often take longer than a typical browser session -- which leaves me either losing my work in a GUI admin, or having to write first in my editor of choice, and then cut-and-paste it to the web forms. Finally, I want versions I can easily browse with standard diffing tools.

When it came down to it, my blog content is basically static. Occasionally, I'll update a post, but it's rare. Comments are really the only dynamic aspect of the blog... and what I had with s9y was not cutting it, as I was getting more spam than I could keep up with. New commenting platforms such as Livefyre and Disqus provide more features than most blogging platforms I know, and provide another side benefit: because they are javascript-based, you can simply drop in a small amount of markup into your post once -- meaning your pages can be fully static!

Add these thoughts to the rise of static blogging platforms such as the aforementioned Jekyll, and I had a kernel of an idea: take the work I'd done already, and create a static blog generator.

Unlike Zend Framework 1, the view layer in Zend Framework 2 separates the variables assigned to each view model. This means that when you are in the layout view script, you don't automatically have access to variables that were assigned the the action's view model and vice versa.

Accessing action variables in the layout

Consider this controller code:

class IndexController extends ActionController
{
    public function indexAction()
    {
        return array('myvar' => 'test');
    }
}

If you are in the layout.phtml, then to retrieve this value you do:

layout.phtml:

<?php
$children = $this->viewModel()->getCurrent()->getChildren();
$child = $children[0];
?>
<!-- some HTML -->
<?php echo $this->escape($child->myvar);?>

If you really want to make sure you collect the correct child view model, then you could iterate over $children and look for the child that has the correct captureTo name set. For the action's view model, this defaults to content:

layout.phtml:

<?php
$children = $this->viewModel()->getCurrent()->getChildren();
foreach($children as $child) {
    if ($child->captureTo() == 'content') {
        break;
    }
}
?>
<!-- some HTML -->
<?php echo $this->escape($child->myvar);?>

Accessing layout variables in the action view

If you have assigned a variable to the layout's view model in, say, an event listener within Module.php:

Module.php:

    public function onBootstrap($e)
    {
        $application = $e->getParam('application');
        $viewModel = $application->getMvcEvent()->getViewModel();
        $viewModel->some_config_var = '12345';
    }

This is how you access some_config_var in the action view:

view/index/index.html:

<?php echo $this->escape($this->layout()->some_config_var); ?>

Another, more long winded way is to use the getRoot() method on the viewModel view helper:

view/index/index.html:

<?php
$layoutViewModel = $this->viewModel()->getRoot(); 
?>
<!-- Some HTML -->
<?php echo $this->escape($layoutViewModel->some_config_var); ?>

Setting configuration variables into the view

It therefore follows that if you need to set a variable that could be accessed from any view script, it's easiest to set it into the layout's view model and then access it via the layout() view script. This is handy for view layer config variables that you want to store in your config files, such as the Google search API key.

Application/config/module.config.php:

<?php
return array(
    'layout' => array(
        'google_search_api_key' => '1234567890',
    ),

Application/Module.php:

    public function onBootstrap($e)
    {
        $application = $e->getParam('application');
        $config = $e->getParam('config');

        $viewModel = $application->getMvcEvent()->getViewModel();
        $viewModel->config = $config->layout;
    }

view/search/index.html:

<?php echo $this->layout()->config->google_search_api_key; ?>

In Making a mobile connection I describe how after just a few seconds of inactivity your mobile phone demotes the radio link to your carrier network. It typically takes 1-2 seconds to re-establish the radio link to full bandwidth capacity. This is a huge delay!

A few days ago I was discussing desktop vs. mobile page load times with some web performance wonks. These times were gathered from real users via the W3C Nav Timing API. We started chatting about why the mobile times were worse – slower connection speeds, less cache space, etc. – and it hit me that taking 2 seconds to re-establish the radio link might account for much of what makes mobile sites slower, especially in RUM (Real User Monitoring) vs. synthetic testing. And I wondered:

After some initial testing it looks like the answers are:

I started by creating a Nav Timing test page that shows the values from Nav Timing. If you load the page you’ll see something like this. (Please look at page source to see how I calculate these conceptual time values.)

total time = 239 ms
dns = 119 ms
connect = 16 ms
ttfb = 61 ms
HTML = 0 ms
frontend = 42 ms

NOTE: Nav timing is available in Android 4. I’m not aware of any other mobile platform that has it, so you’ll need an Android 4 device to run these tests. You should close all/most currently running apps on your mobile device as they might be keeping the radio link alive in the background. On Android 4 this is done under Settings | Apps | Running. I had to stop Google Services.

You can determine if the radio link promotion delay occurs based on whether any of the times are greater than 2 seconds. Here’s a key:

no 2 second times

If all of the times are less than 2 seconds then the radio link was already active. You can create this result by loading the page multiple times in quick succession. All the times should be pretty fast because you have a radio link, the DNS resolution is cached, and you have a persistent connection to the web server.

dns > 2 seconds

If you wait 10-20 seconds (and closed all background apps) the radio link gets demoted. At this point clicking on one of the buttons to open the test page on another domain will force a DNS lookup. Normally the DNS lookup should take a few hundred milliseconds, but if the radio link needs to be promoted the DNS time jumps to 2000+ milliseconds. This page is hosted on three different domains. If you use all three pages thus caching all three DNS resolutions, the only way I know of to clear the DNS cache is to power cycle the phone.

connect > 2 seconds

If you allow the radio link to be demoted by waiting 10-20 seconds and reload the page (or click the button for the same page) you might see the connect time is greater than 2 seconds. This happens when the DNS is cached but there’s no persistent connection to the server. This is harder to reproduce – it depends on the browser’s policy for closing persistent connections.

ttfb > 2 seconds

If the radio link is demoted, the DNS is cached, and there’s a persistent connection to the server you’ll see the time-to-first-byte (ttfb) is greater than 2 seconds. This is what happens most frequently when you load the same page multiple times with a 10-20 second gap in-between.

It’s important that developers focusing on performance be aware of the impact of radio link promotion on nav timing for mobile traffic so you don’t waste time solving the wrong problem: If you’re gathering RUM data via nav timing and see slow DNS times, you might think about investing in your DNS infrastructure – even though those slow DNS times might be caused by radio link promotion. Similarly, if you see long connection times it might not make sense to investigate how your servers manage persistent connections. And slow time-to-first-byte values may or may not indicate a backend app layer performance problem.

My website doesn’t generate enough mobile traffic to verify this theory, but I believe that websites with enough mobile nav timing data will see bimodal distributions of their timing data for dns, connection, and ttfb where the modes are ~2 seconds apart. If anyone has enough data (you know who you are) please take a look and comment below. It might be possible to develop heuristics that help us determine when radio link delays are having an impact. I’d love to get some stats on the percentage of page views that incur this delay.

Following yesterday's article on returning JSON from a ZF2 controller action, Lukas suggested that I should also demonstrate how to use the Accept header to get JSON. So this is how you do it!

Set up the JsonStrategy

We set up the JsonStrategy as we did in returning JSON from a ZF2 controller action.

Return a ViewModel from the controller

As we're letting the JsonStrategy intercede for us, we don't need to do anything special in our controller at all. In this case, we simply return a normal ViewModel for use by either the JsonRenderer or PhpRenderer as required:

module/Application/src/Application/Controller/IndexController.php:

<?php

namespace ApplicationController;

use ZendMvcControllerActionController,
    ZendViewModelViewModel;

class IndexController extends ActionController
{
    public function anotherAction()
    {
        $matches[] = array('distance' => 10, 'playground' => array('a'=>1));
        $matches[] = array('distance' => 20, 'playground' => array('a'=>2));
        $matches[] = array('distance' => 30, 'playground' => array('a'=>3));

        $result = new ViewModel(array(
            'success'=>true,
            'results' => $matches,
        ));
        return $result;
    }
}

with our HTML view script:

module/Application/view/index/another.phtml:

<?php if ($success): ?>

<h2>Results</h2>

<ul>
<?php foreach ($results as $row): ?>
    <li>Distance: <?php echo $this->escape($row['distance']);?>m</li>
<?php endforeach; ?>
</ul>

<?php endif; ?>

So if you set up a route and browse to it, you'll see a nicely rendered page.

Retrieving the data as JSON

To retrieve the data via JSON, we need a client where we can set the Accept header. We'll use curl for this test. When doing anything with APIs and testing, we head over to LornaJane's blog for the Curl Cheat Sheet and use this command line:

curl -H "Accept: application/json" http://zf2test.dev/json/another

and you should see the output of:

{
  "content":{
    "success":true,
    "results": [
      {"distance":10,"playground":{"a":1}},
      {"distance":20,"playground":{"a":2}},
      {"distance":30,"playground":{"a":3}}
    ]
  }
}

(Formatted for readability - you get the result back on a single line from curl.)

This way you can use the same controllers for your HTML views and for returning JSON to those clients that can use it.

The new view layer in Zend Framework 2 can be set up to return JSON rather than rendered HTML relatively easily. There are two steps to this:

Set up the JsonStrategy

Firstly we need to set up the view's JsonStrategy to check to a situation when returning JSON is required and then to render out JSON for us. The JsonStrategy will cause the JsonRenderer to be run in two situations:

1. The view model returned by the controller action is a JsonModel
2. The HTTP Accept header sent in the Request include "application/json"

The enable the JsonStrategy, we simply attach it to the view's event manager with a reasonably high priority. This can be done in our Application's Module class. Firstly we create an onBootstrap() callback on the bootstrap event and then we implement onBootstrap() to attaché the JsonStrategy:

module/Application/Module.php:

class Module implements AutoloaderProvider
{
    public function init(Manager $moduleManager)
    {
        $events = StaticEventManager::getInstance();
        $events->attach('bootstrap', 'bootstrap',
            array($this, 'onBootstrap'));
    }

    public function onBootstrap(Event $e)
    {
        $application  = $e->getParam('application'); 
        /* @var $application ZendMvcApplication */
        $locator      = $application->getLocator();
        $view         = $locator->get('ZendViewView');
        $jsonStrategy = $locator->get('ZendViewStrategyJsonStrategy');
        $view->events()->attach($jsonStrategy, 100);        

    }

    // more methods such as getConfig() and getAutoloaderConfig()
}

As you can see, in init() we grab the StaticEventManager to attach our onBootstrap() method to the bootstrap event. Then, within onBootstrap(), we grab the view and the JsonStrategy from the locator (via application) and attach the JsonStrategy to the view's events() event manager.

Return a JsonModel from the controller action

To send JSON to the client when the Accept header isn't application/json, we use a JsonModel in a controller action like this:

module/Application/src/Application/Controller/IndexController.php:

namespace ApplicationController;

use ZendMvcControllerActionController,
    ZendViewModelViewModel,
    ZendViewModelJsonModel;

class IndexController extends ActionController
{
    public function indexAction()
    {
        $result = new JsonModel(array(
        'some_parameter' => 'some value',
            'success'=>true,
        ));

        return $result;
    }
}

The output will now be JSON. Obviously, if you're sending JSON back based on the Accept header, then you can return a normal ViewModel.

This past December I contributed an article called Frontend SPOF in Beijing to PerfPlanet’s Performance Calendar. I hope that everyone who reads my blog also read the Performance Calendar – it’s an amazing collection of web performance articles and gurus. But in case you don’t I’m cross-posting it here. I saw a great presentation from Pat Meenan about frontend SPOF and want to raise awareness around this issue. This post contains some good insights.

Make sure to read PerfPlanet – it’s a great aggregator of WPO blog posts.

Now – flash back to December 2011…

I’m at Velocity China in Beijing as I write this article for the Performance Calendar. Since this is my second time to Beijing I was better prepared for the challenges of being behind the Great Firewall. I knew I couldn’t access popular US websites like Google, Facebook, and Twitter, but as I did my typical surfing I was surprised at how many other websites seemed to be blocked.

Business Insider

It didn’t take me long to realize the problem was frontend SPOF – when a frontend resource (script, stylesheet, or font file) causes a page to be unusable. Some pages were completely blank, such as Business Insider:

figure 1: The dreaded “blank white screen” due to a blocking Twitter script.

Firebug’s Net Panel shows that anywhere.js is taking a long time to download because it’s coming from platform.twitter.com – which is blocked by the firewall. Knowing that scripts block rendering of all subsequent DOM elements, we form the hypothesis that anywhere.js is being loaded in blocking mode in the HEAD. Looking at the HTML source we see that’s exactly what is happening:

<head>
...
<!-- Twitter Anywhere -->
<script src="https://platform.twitter.com/anywhere.js?id=ZV0...&v=1" type="text/javascript"></script>
<!-- / Twitter Anywhere -->
...
</head>
...
<body>

If anywhere.js had been loaded asynchronously this wouldn’t happen. Instead, since anywhere.js is loaded the old way with <SCRIPT SRC=..., it blocks all the DOM elements that follow which in this case is the entire BODY of the page. If we wait long enough the request for anywhere.js times out and the page begins to render. How long does it take for the request to timeout? Looking at the “after” screenshot of Business Insider we see it takes 1 minute and 15 seconds for the request to timeout. That’s 1 minute and 15 seconds that the user is left staring at a blank white screen waiting for the Twitter script!

figure 2: Business Insider finally renders after 1 minute 15 seconds.

CNET

CNET has a slightly different experience; the navigation header is displayed but the rest of the page is blocked from rendering:

figure 3: CNET rendering is blocked by ads from eyewonder.com.

Looking in Firebug we see that wrapper.js from cdn.eyewonder.com is “pending” – this must be another domain that’s blocked by the firewall. Based on where the rendering stops our guess is that the wrapper.js SCRIPT tag is immediately after the navigation header and is loaded in blocking mode thus preventing the rest of the page from rendering. The HTML confirms that this is indeed what’s happening:

<header>
...
</header>
<script src="http://cdn.eyewonder.com/100125/771933/1592365/wrapper.js"></script>
<div id="rb_wrap">
<div id="rb_content"> <div id="contentMain">

O’Reilly Radar

Everyday I visit O’Reilly Radar to read Nat Torkington’s Four Short Links. Normally Nat’s is one of many stories on the Radar front page, but going there from Beijing shows a page with only one story:

figure 4: O’Reilly Radar rendering is blocked by Twitter widget.

At the bottom of this first story there’s supposed to be a Tweet button. This button is added by the widgets.js script fetched from platform.twitter.com which is blocked by the Great Firewall. This wouldn’t be an issue if widgets.js was fetched asynchronously, but sadly a peek at the HTML shows that’s not the case:

<a href="http://www.stevesouders.com/blog...">Comment</a>
&nbsp;|&nbsp;
<span class="social-counters">
<span class="retweet">
<a href="http://twitter.com/share" class="twitter-share-button"
   data-count="horizontal"
   data-url="http://radar.oreilly.com/2011/12/four-short-links-6-december-20-1.html"
   data-text="Four short links: 6 December 2011" data-via="radar"
   data-related="oreillymedia:oreilly.com">Tweet</a>
<script src="http://platform.twitter.com/widgets.js"
   type="text/javascript"></script>
</span>

The cause of frontend SPOF

One possible takeaway from these examples might be that frontend SPOF is specific to Twitter and eyewonder and a few other 3rd party widgets. Sadly, frontend SPOF can be caused by any 3rd party widget, and even from the main website’s own scripts, stylesheets, or font files.

Another possible takeaway from these examples might be to avoid 3rd party widgets that are blocked by the Great Firewall. But the Great Firewall isn’t the only cause of frontend SPOF – it just makes it easier to reproduce. Any script, stylesheet, or font file that takes a long time to return has the potential to cause frontend SPOF. This typically happens when there’s an outage or some other type of failure, such as an overloaded server where the HTTP request languishes in the server’s queue for so long the browser times out.

The true cause of frontend SPOF is loading a script, stylesheet, or font file in a blocking manner. The table in my frontend SPOF blog post shows when this happens. It’s really the website owner who controls whether or not their site is vulnerable to frontend SPOF. So what’s a website owner to do?

Avoiding frontend SPOF

The best way to avoid frontend SPOF is to load scripts asynchronously. Many popular 3rd party widgets do this by default, such as Google Analytics, Facebook, and Meebo. Twitter also has an async snippet for the Tweet button that O’Reilly Radar should use. If the widgets you use don’t offer an async version you can try Stoyan’s Social button BFFs async pattern.

Another solution is to wrap your widgets in an iframe. This isn’t always possible, but in two of the examples above the widget is eventually served in an iframe. Putting them in an iframe from the start would have avoided the frontend SPOF problems.

For the sake of brevity I’ve focused on solutions for scripts. Solutions for font files can be found in my @font-face and performance blog post. I’m not aware of much research on loading stylesheets asynchronously. Causing too many reflows and FOUC are concerns that need to be addressed.

Call to action

Business Insider, CNET, and O’Reilly Radar all have visitors from China, and yet the way their pages are constructed delivers a bad user experience where most if not all of the page is blocked for more than a minute. This isn’t a P2 frontend JavaScript issue. This is an outage. If the backend servers for these websites took 1 minute to send back a response, you can bet the DevOps teams at Business Insider, CNET, and O’Reilly wouldn’t sleep until the problem was fixed. So why is there so little concern about frontend SPOF?

Frontend SPOF doesn’t get much attention – it definitely doesn’t get the attention it deserves given how easily it can bring down a website. One reason is it’s hard to diagnose. There are a lot of monitors that will start going off if a server response time exceeds 60 seconds. And since all that activity is on the backend it’s easier to isolate the cause. Is it that pagers don’t go off when clientside page load times exceed 60 seconds? That’s hard to believe, but perhaps that’s the case.

Perhaps it’s the way page load times are tracked. If you’re looking at worldwide medians, or even averages, and China isn’t a major audience your page load time stats might not exceed alert levels when frontend SPOF happens. Or maybe page load times are mostly tracked using synthetic testing, and those user agents aren’t subjected to real world issues like the Great Firewall.

One thing website owners can do is ignore frontend SPOF until it’s triggered by some future outage. A quick calculation shows this is a scary choice. If a 3rd party widget has a 99.99% uptime and a website has five widgets that aren’t async, the probability of frontend SPOF is 0.05%. If we drop uptime to 99.9% the probability of frontend SPOF climbs to 0.5%. Five widgets might be high, but remember that “third party widget” includes ads and metrics. Also, the website’s own resources can cause frontend SPOF which brings the number even higher. The average website today contains 14 scripts any of which could cause frontend SPOF if they’re not loaded async.

Frontend SPOF is a real problem that needs more attention. Website owners should use async snippets and patterns, monitor real user page load times, and look beyond averages to 95th percentiles and standard deviations. Doing these things will mitigate the risk of subjecting users to the dreaded blank white page. A chain is only as strong as its weakest link. What’s your website’s weakest link? There’s a lot of focus on backend resiliency. I’ll wager your weakest link is on the frontend.

[Originally posted as part of PerfPlanet's Performance Calendar 2011.]

My previous blog post, Cache them if you can, suggests that current cache sizes are too small – especially on mobile.

Given this concern about cache size a relevant question is:

If a response is compressed, does the browser save it compressed or uncompressed?

Compression typically reduces responses by 70%. This means that a browser can cache 3x as many compressed responses if they’re saved in their compressed format.

Note that not all responses are compressed. Images make up the largest number of resources but shouldn’t be compressed. On the other hand, HTML documents, scripts, and stylesheets should be compressed and account for 30% of all requests. Being able to save 3x as many of these responses to cache could have a significant impact on cache hit rates.

It’s difficult and time-consuming to determine whether compressed responses are saved in compressed format. I created this Caching Gzip Test page to help determine browser behavior. It has two 200 KB scripts – one is compressed down to ~148 KB and the other is uncompressed. (Note that this file is random strings so the compression savings is only 25% as compared to the typical 70%.) After clearing the cache and loading the test page if the total cache disk size increases ~348 KB it means the browser saves compressed responses as compressed. If the total cache disk size increases ~400 KB it means compressed responses are saved uncompressed.

The challenging part of this experiment is finding where the cache is stored and measuring the response sizes. Firefox, Chrome, and Opera save responses as files and were easy to measure. For IE on Windows I wasn’t able to access the individual cache files (admin permissions?) but was able to measure the sizes based on the properties of the Temporary Internet Files folder. Safari saves all responses in Cache.db. I was able to see the incremental increase by modifying the experiment to be two pages: the compressed response and the uncompressed response. You can see the cache file locations and full details in the Caching Gzip Test Results page.

Here are the results for top desktop browsers:

* Chrome and Firefox cache size is a percentage of available disk space. Chrome is capped at 320 MB. I don’t know what Firefox’s cap is; on my laptop with 50 GB free the cache size is 830 MB.

We see that Chrome 17, Firefox 11, and Opera 11 store compressed responses in compressed format, while IE 8&9 and Safari 5 save them uncompressed. IE 8&9 have smaller cache sizes, so the fact that they uncompress responses before caching further reduces the number of responses that can be cached.

What’s the best choice? It’s possible that reading cached responses is faster if they’re already uncompressed. That would be a good next step to explore. I wouldn’t prejudge IE’s choice when it comes to performance on Windows. But it’s clear that saving compressed responses in compressed format increases the number of responses that can be cached, and this increases cache hit rates. What’s even clearer is that browsers don’t agree on the best answer. Should they?

Why do we have to bother about built-in GTID support in MySQL 5.6 at all? Sure, it is a tremendous step forward for a lazy primary copy system like MySQL Replication. Period. GTIDs make server-side failover easier (slides). And, load balancer, including PECL/mysqlnd_ms as an example of a driver integrated load balancer, can use them to provide session consistency. Please, see the slides. But…

… the primary remains a single point of failure. GTIDs can be described as cluster-wide transaction counters generated on the master. In case of a master failure, the slave that has replicated the highest transaction counter shall be promoted to become the master. Its the most current slave. Failover made easy - no doubt! Adequately deployed, you should reach very reasonable availability.

Know the limits of replicated systems

A multi-master (update anywhere) design does not have a single point of failure. But among the biggest is scaling a multi-master solution. Jim Gray and Pat Helland concluded 1996 in "The Dangers of Replication and a Solution": Update anywhere-anytime-anyway transactional replication has unstable behavior as the workload scales up: a ten-fold increase in nodes and traffic gives a thousand fold increase in deadlocks or reconciliations.. N^3 - buuuhhhh, anything worse than linear scale is not appreciated. Guess what: Microsoft SQL Azure is using primary copy combined with partitioning.

In practice things are not that bad, particulary not for a small number of nodes and recent algorithms. For example, MySQL Cluster (related webinar on March 29) is a true multi-master solution - even eager/synchronous. To overcome the write-scale limitations it has built-in partitioning (sharding). The two classical scale-out solutions - replication and partitioning - are combined in one product. If you want extreme performance and are ready to pay for the costs of partitioning… try it.

Anything to learn from the NoSQL kids on the block?

Some other kids offer relaxed eventual consistency just as MySQL Replication does. Sometimes the CAP theorem is cited as an excuse for it . Some leave conflict resolution, even conflict detection to the application developer . A massively scalabale, high available, synchronous update anywhere solution with built-in conflict resolution - the big thing we all dream of - is hard to create.

In the meanwhile… - maybe custer-aware APIs

While we all wait for the one-fits-all solution, there is something we can do. We can start to tell our load balancers precisely what we need and request no higher level of service than needed. Consistency - as in CAP - is one aspect of service quality. We should start to have cluster-aware APIs abstracting the details of replication architectures. Then, our load balancers, including PECL/mysqlnd_ms can hide everything that makes working with a cluster complicated (connection pooling, request splitting and redirection, failover, node selection, load distribution, …). Also, vendors can start to play with consistency to improve performance without messing up application logic.

Below is how you use the PECL/mysqlnd_ms 1.2+ function mysqlnd_ms_set_qos() to switch between eventual consistency (stale data allowed) and session concistency (read-your-writes). MySQL Replication details hidden behind a function call.

$mysqli = new mysqli("myapp", "username", "password", "database");
if (!$mysqli)
  /* Of course, your error handling is nicer... */
  die(sprintf("[%d] %sn", mysqli_connect_errno(), mysqli_connect_error()));
	
/* read-write splitting: master used */
if (!$mysqli->query("INSERT INTO orders(order_id, item) VALUES (1, 'christmas tree, 1.8m')")) {
   /* Please use better error handling in your code */
  die(sprintf("[%d] %sn", $mysqli->errno, $mysqli->error));
}
	
/* Request session consistency: read your writes */
if (!mysqlnd_ms_set_qos($mysqli, MYSQLND_MS_QOS_CONSISTENCY_SESSION))
  die(sprintf("[%d] %sn", $mysqli->errno, $mysqli->error));
	
/* Plugin picks a node which has the changes, here: master */
if (!$res = $mysqli->query("SELECT item FROM orders WHERE order_id = 1"))
  die(sprintf("[%d] %sn", $mysqli->errno, $mysqli->error));
	
var_dump($res->fetch_assoc());
	
/* Back to eventual consistency: stale data allowed */
if (!mysqlnd_ms_set_qos($mysqli, MYSQLND_MS_QOS_CONSISTENCY_EVENTUAL))
  die(sprintf("[%d] %sn", $mysqli->errno, $mysqli->error));
	
/* Plugin picks any slave, stale data is allowed */
if (!$res = $mysqli->query("SELECT item, price FROM specials"))
  die(sprintf("[%d] %sn", $mysqli->errno, $mysqli->error));

GTID for clients? Buzz alarm!

PECL/mysqlnd_ms 1.3 does not bring any ground breaking changes with regards to consistency or GTIDs. It can now either use the driver built-in GTID emulation (1.2+) or the server-side GTID feature (1.3+, MySQL 5.6) for session consistency. That’s all. I confess, the slide title is pure buzz. But in every tale is some truth.

Cluster-aware APIs and better load balancer? Follow up!

I’m convinced that good load balancers can make application developers life much easier. Read-your-writes and session consistency is an example how new API calls may come handy. Transparently replacing remote slave accesses with client-side cache accesses (coming with 1.3) is an example how load balancers can optimize overall cluster performance.

Whoever designs a replication solution in 2012 should include the load balancer into his considerations… - even for multi-master.

Happy hacking!

@Ulf_Wendel 

One thing that I've wanted to implement for a while now is automatic vhosts on my dev box. The idea is that I want to drop a folder into a directory and have it automatically turned into a vhost for me accessible at http://foldername.dev. It turns out that this isn't nearly as hard as expected which is usually the case with things that I've been putting off!

This is how to do it.

Apache configuration

The Apache magic is in an extension called mod_vhost_alias which you may need to enable in your httpd.conf file.

You can then set up the VirtualHost wherever you keep such things. On a stock OS X, the extras/httpd-vhosts.conf file is used.

Add the following to the bottom:

<Virtualhost *:80>
    VirtualDocumentRoot "/www/dev/%1/public"
    ServerName vhosts.dev
    ServerAlias *.dev
    UseCanonicalName Off
    LogFormat "%V %h %l %u %t "%r" %s %b" vcommon
    ErrorLog "/www/dev/vhosts-error_log"
    <Directory "/www/dev/*">
        Options Indexes FollowSymLinks MultiViews
        AllowOverride All
        Order allow,deny
        Allow from all
    </Directory>
</Virtualhost>

In the VirtualHost configuration, I have used the ServerAlias and VirtualDocumentRoot directives to map http://foldername.dev to the directory /www/dev/foldername/public. Hence, any folder that I place in /www/dev will have its own virtual host. Alter these appropriately for your set-up.

Don't forget to restart Apache.

Unfortunately, the computer hasn't a clue how to handle http://foldername.dev and the obvious solution is to run a local DNS server. Another solution is to use a PAC file.

DNS server configuration

This is easy enough with dnsmasq. On OS X, use Homebrew to install like this: brew install dnsmasq. On Linux, use your package manager; on Windows, you're own your own!

Note that on OS X, you should set it to start up automatically using launchd as noted in the instructions after installation. You also need to copy the configuration file to /etc using: cp /usr/local/Cellar/dnsmasq/2.57/dnsmasq.conf.example /usr/local/etc/dnsmasq.conf (or whatever the latest version number is). on Linux, I would guess that your package manager provides a dnsmasq.conf file in /etc or /etc/dnsmasq.

Next, edit dnsmasq.conf file and added the following lines to the bottom:

listen-address=127.0.0.1
address=/.dev/127.0.0.1

Add the name server to your network configuration

On OS X, Go to System Preferences -> Network -> {Wifi or Ethernet} -> Advanced… -> DNS and click on + button at the bottom of the left hand panel and add 127.0.0.1 to the list of DNS servers. Drag 127.0.0.1 at the top of the list.

On Linux, you should use the appropriate GUI tools for your distribution or potentially edit etc/dhcp/dhclient.conf and uncomment the domain-name-servers 127.0.0.1; on line 20 (on Ubuntu).

Restart dnsmasq and you should now be able to execute host test.dev on the command line and see 127.0.0.1 as the resultant address.

Alternative to DNS server: PAC file

Since publishing this article, Chris Morell pointed out that you can also use PAC files rather than install a DNS server. Details are on his blog post.

Check it works

Create a directory called test in your dev directory. Within test, create public/index.php and within index.php add some code to prove it works. e.g. < ;?php echo "Hello World"; ?>;

If you navigate to http://test.dev, you should see "Hello World" displayed.

Caveats

A couple of caveats:

• DOCUMENT_ROOT is not /www/dev/test as you'd expect. Instead it is the global document root. See this gist for a neat way to solve this using a prepend file.
• If you use mod_rewrite, then you'll need a RewriteBase / in your .htaccess file. Alternatively, you can change the Directory section of your vhost to do the rewriting for you if all your projects are alike. Something like this should work:

      <Directory "/www/dev/*">
          Options Indexes FollowSymLinks MultiViews
          AllowOverride None
          Order allow,deny
          Allow from all
  
          RewriteEngine On
          RewriteBase /
          RewriteCond %{REQUEST_FILENAME} -s [OR]
          RewriteCond %{REQUEST_FILENAME} -l [OR]
          RewriteCond %{REQUEST_FILENAME} -d
          RewriteRule ^.*$ - [NC,L]
          RewriteRule ^.*$ index.php [NC,L]
      </Directory>
  

All done

That's it. You can now create as many projects as you like without having to worry about setting up new virtual hosts or modifying you hosts file!

I've just finished presenting my talk on how I currently work on Puppet modules at Puppetcamp here in Edinburgh where I've been for the week talking on both FlossUK 2012 and Puppetcamp.

Earlier this week I opened FlossUK 2012 with my talk on 7 tools for your devops stack

“The fastest HTTP request is the one not made.”

I always smile when I hear a web performance speaker say this. I forget who said it first, but I’ve heard it numerous times at conferences and meetups over the past few years. It’s true! Caching is critical for making web pages faster. I’ve written extensively about caching:

• Call to improve browser caching
• (lack of) Caching for iPhone Home Screen Apps
• Redirect caching deep dive
• Mobile cache file sizes
• Improving app cache
• Storager case study: Bing, Google
• App cache & localStorage survey
• HTTP Archive: max-age

Things are getting better – but not quickly enough. The chart below from the HTTP Archive shows that the percentage of resources that are cacheable has increased 10% during the past year (from 42% to 46%). Over that same time the number of requests per page has increased 12% and total transfer size has increased 24% (chart).

Perhaps it’s hard to make progress on caching because the problem doesn’t belong to a single group – responsibility spans website owners, third party content providers, and browser developers. One thing is certain – we have to do a better job when it comes to caching. 

I’ve gathered some compelling statistics over the past few weeks that illuminate problems with caching and point to some next steps. Here are the highlights:

• 55% of resources don’t specify a max-age value
• 46% of the resources without any max-age remained unchanged over a 2 week period
• some of the most popular resources on the Web are only cacheable for an hour or two
• 40-60% of daily users to your site don’t have your resources in their cache
• 30% of users have a full cache
• for users with a full cache, the median time to fill their cache is 4 hours of active browsing

Read on to understand the full story.

My kingdom for a max-age header

Many of the caching articles I’ve written address issues such as size & space limitations, bugs with less common HTTP headers, and outdated purging logic. These are critical areas to focus on. But the basic function of caching hinges on websites specifying caching headers for their resources. This is typically done using max-age in the Cache-Control response header. This example specifies that a response can be read from cache for 1 year:

Cache-Control: max-age=31536000

Since you’re reading this blog post you probably already use max-age, but the following chart from the HTTP Archive shows that 55% of resources don’t specify a max-age value. This translates to 45 of the average website’s 81 resources needing a HTTP request even for repeat visits.

Missing max-age != dynamic

Why do 55% of resources have no caching information? Having looked at caching headers across thousands of websites my first guess is lack of awareness – many website owners simply don’t know about the benefits of caching. An alternative explanation might be that many resources are dynamic (JSON, ads, beacons, etc.) and shouldn’t be cached. Which is the bigger cause – lack of awareness or dynamic resources? Luckily we can quantify the dynamicness of these uncacheable resources using data from the HTTP Archive.

The HTTP Archive analyzes the world’s top ~50K web pages on the 1st and 15th of the month and records the HTTP headers for every resource. Using this history it’s possible to go back in time and quantify how many of today’s resources without any max-age value were identical in previous crawls. The data for the chart above (showing 55% of resources with no max-age) was gathered on Feb 15 2012. The chart below shows the percentage of those uncacheable resources that were identical in the previous crawl on Feb 1 2012. We can go back even further and see how many were identical in both the Feb 1 2012 and the Jan 15 2012 crawls. (The HTTP Archive doesn’t save response bodies so the determination of “identical” is based on the resource having the exact same URL, Last-Modified, ETag, and Content-Length.)

46% of the resources without any max-age remained unchanged over a 2 week period. This works out to 21 resources per page that could have been read from cache without any HTTP request but weren’t. Over a 1 month period 38% are unchanged – 17 resources per page.

This is a significant missed opportunity. Here are some popular websites and the number of resources that were unchanged for 1 month but did not specify max-age:

• http://www.toyota.jp/ – 172 resources without max-age & unchanged for 1 month
• http://www.sfgate.com/ – 133
• http://www.hasbro.com/ – 122
• http://www.rakuten.co.jp/ – 113
• http://www.ieee.org/ – 97
• http://www.elmundo.es/ – 80
• http://www.nih.gov/ – 76
• http://www.frys.com/ – 68
• http://www.foodnetwork.com/ – 66
• http://www.irs.gov/ – 58
• http://www.ca.gov/ – 53
• http://www.oracle.com/ – 52
• http://www.blackberry.com/ – 50

Recalling that “the fastest HTTP request is the one not made”, this is a lot of unnecessary HTTP traffic. I can’t prove it, but I strongly believe this is not intentional – it’s just a lack of awareness. The chart below reinforces this belief – it shows the percentage of resources (both cacheable and uncacheable) that remain unchanged starting from Feb 15 2012 and going back for one year.

The percentage of resources that are unchanged is nearly the same when looking at all resources as it is for only uncacheable resources: 44% vs. 46% going back 2 weeks and 35% vs. 38% going back 1 month. Given this similarity in “dynamicness” it’s likely that the absence of max-age has nothing to do with the resources themselves and is instead caused by website owners overlooking this best practice.

3rd party content

If a website owner doesn’t make their resources cacheable, they’re just hurting themselves (and their users). But if a 3rd party content provider doesn’t have good caching behavior it impacts all the websites that embed that content. This is both bad a good. It’s bad in that one uncacheable 3rd party resource can impact multiple sites. The good part is that shifting 3rd party content to adopt good caching practices also has a magnified effect.

So how are we doing when it comes to caching 3rd party content? Below is a list of the top 30 most-used resources according to the HTTP Archive. These are the resources that were used the most across the world’s top 50K web pages. The max-age value (in hours) is also shown.

1. http://www.google-analytics.com/ga.js (2 hours)
2. http://ssl.gstatic.com/s2/oz/images/stars/po/Publisher/sprite2.png (8760 hours)
3. http://pagead2.googlesyndication.com/pagead/js/r20120208/r20110914/show_ads_impl.js (336 hours)
4. http://pagead2.googlesyndication.com/pagead/render_ads.js (336 hours)
5. http://pagead2.googlesyndication.com/pagead/show_ads.js (1 hour)
6. https://apis.google.com/_/apps-static/_/js/gapi/gcm_ppb,googleapis_client,plusone/[...] (720 hours)
7. http://pagead2.googlesyndication.com/pagead/osd.js (24 hours)
8. http://pagead2.googlesyndication.com/pagead/expansion_embed.js (24 hours)
9. https://apis.google.com/js/plusone.js (1 hour)
10. http://googleads.g.doubleclick.net/pagead/drt/s?safe=on (1 hour)
11. http://static.ak.fbcdn.net/rsrc.php/v1/y7/r/ql9vukDCc4R.png (3825 hours)
12. http://connect.facebook.net/rsrc.php/v1/yQ/r/f3KaqM7xIBg.swf (164 hours)
13. https://ssl.gstatic.com/s2/oz/images/stars/po/Publisher/sprite2.png (8760 hours)
14. https://apis.google.com/_/apps-static/_/js/gapi/googleapis_client,iframes_styles[...] (720 hours)
15. http://static.ak.fbcdn.net/rsrc.php/v1/yv/r/ZSM9MGjuEiO.js (8742 hours)
16. http://static.ak.fbcdn.net/rsrc.php/v1/yx/r/qP7Pvs6bhpP.js (8699 hours)
17. https://plusone.google.com/_/apps-static/_/ss/plusone/[...] (720 hours)
18. http://b.scorecardresearch.com/beacon.js (336 hours)
19. http://static.ak.fbcdn.net/rsrc.php/v1/yx/r/lP_Rtwh3P-S.css (8710 hours)
20. http://static.ak.fbcdn.net/rsrc.php/v1/yA/r/TSn6F7aukNQ.js (8760 hours)
21. http://static.ak.fbcdn.net/rsrc.php/v1/yk/r/Wm4bpxemaRU.js (8702 hours)
22. http://static.ak.fbcdn.net/rsrc.php/v1/yZ/r/TtnIy6IhDUq.js (8699 hours)
23. http://static.ak.fbcdn.net/rsrc.php/v1/yy/r/0wf7ewMoKC2.css (8699 hours)
24. http://static.ak.fbcdn.net/rsrc.php/v1/yO/r/H0ip1JFN_jB.js (8760 hours)
25. http://platform.twitter.com/widgets/hub.1329256447.html (87659 hours)
26. http://static.ak.fbcdn.net/rsrc.php/v1/yv/r/T9SYP2crSuG.png (8699 hours)
27. http://platform.twitter.com/widgets.js (1 hour)
28. https://plusone.google.com/_/apps-static/_/js/plusone/[...] (720 hours)
29. http://pagead2.googlesyndication.com/pagead/js/graphics.js (24 hours)
30. http://s0.2mdn.net/879366/flashwrite_1_2.js (720 hours)

There are some interesting patterns.

• simple URLs have short cache times – Some resources have very short cache times, e.g., ga.js (1), show_ads.js (5), and twitter.com/widgets.js (27). Most of the URLs for these resources are very simple (no querystring or URL “fingerprints”) because these resource URLs are part of the snippet that website owners paste into their page. These “bootstrap” resources are given short cache times because there’s no way for the resource URL to be changed if there’s an emergency fix – instead the cached resource has to expire in order for the emergency update to be retrieved.
• long URLs have long cache times – Many 3rd party “bootstrap” scripts dynamically load other resources. These code-generated URLs are typically long and complicated because they contain some unique fingerprinting, e.g., http://pagead2.googlesyndication.com/pagead/js/r20120208/r20110914/show_ads_impl.js (3) and http://platform.twitter.com/widgets/hub.1329256447.html (25). If there’s an emergency change to one of these resources, the fingerprint in the bootstrap script can be modified so that a new URL is requested. Therefore, these fingerprinted resources can have long cache times because there’s no need to rev them in the case of an emergency fix.
• where’s Facebook’s like button? – Facebook’s like.php and likebox.php are also hugely popular but aren’t in this list because the URL contains a querystring that differs across every website. Those resources have an even more aggressive expiration policy compared to other bootstrap resources – they use no-cache, no-store, must-revalidate. Once the like[box] bootstrap resource is loaded, it loads the other required resources: lP_Rtwh3P-S.css (19), TSn6F7aukNQ.js (20), etc. Those resources have long URLs and long cache times because they’re generated by code, as explained in the previous bullet.
• short caching resources are often async – The fact that bootstrap scripts have short cache times is good for getting emergency updates, but is bad for performance because they generate many Conditional GET requests on subsequent requests. We all know that scripts block pages from loading, so these Conditional GET requests can have a significant impact on the user experience. Luckily, some 3rd party content providers are aware of this and offer async snippets for loading these bootstrap scripts mitigating the impact of their short cache times. This is true for ga.js (1), plusone.js (9), twitter.com/widgets.js (27), and Facebook’s like[box].php.

These extremely popular 3rd party snippets are in pretty good shape, but as we get out of the top widgets we quickly find that these good caching patterns degrade. In addition, more 3rd party providers need to support async snippets.

Cache sizes are too small

In January 2007 Tenni Theurer and I ran an experiment at Yahoo! to estimate how many users had a primed cache. The methodology was to embed a transparent 1×1 image in the page with an expiration date in the past. If users had the expired image in their cache the browser would issue a Conditional GET request and receive a 304 response (primed cache). Otherwise they’d get a 200 response (empty cache). I was surprised to see that 40-60% of daily users to the site didn’t have the site’s resources in their cache and 20% of page views were done without the site’s resources in the cache.

Numerous factors contribute to this high rate of unique users missing the site’s resources in their cache, but I believe the primary reason is small cache sizes. Browsers have increased the size of their caches since this experiment was run, but not enough. It’s hard to test browser cache size. Blaze.io’s article Understanding Mobile Cache Sizes shows results from their testing. Here are the max cache sizes I found for browsers on my MacBook Air. (Some browsers set the cache size based on available disk space, so let me mention that my drive is 250 GB and has 54 GB available.) I did some testing and searching to find max cache sizes for my mobile devices and IE.

• Chrome: 320 MB
• Internet Explorer 9: 250 MB
• Firefox 11: 830 MB (shown in about:cache)
• Opera 11: 20 MB (shown in Preferences | Advanced | History)
• iPhone 4, iOS 5.1: 30-35 MB (based on testing)
• Galaxy Nexus: 18 MB (based on testing)

I’m surprised that Firefox 11 has such a large cache size – that’s almost close to what I want. All the others are (way) too small. 18-35 MB on my mobile devices?! I have seven movies on my iPhone – I’d gladly trade Iron Man 2  (1.82 GB) for more cache space.

Caching in the real world

In order to justify increasing browser cache sizes we need some statistics on how many real users overflow their cache. This topic came up at last month’s Velocity Summit where we had representatives from Chrome, Internet Explorer, Firefox, Opera, and Silk. (Safari was invited but didn’t show up.) Will Chan from the Chrome team (working on SPDY) followed-up with this post on Chromium cache metrics from Windows Chrome. These are the most informative real user cache statistics I’ve ever seen. I strongly encourage you to read his article.

Some of the takeaways include:

• ~30% of users have a full cache (capped at 320 MB)
  
• for users with a full cache, the median time to fill their cache is 4 hours of active browsing (20 hours of clock time)
• 7% of users clear their cache at least once per week
• 19% of users experience “fatal cache corruption” at least once per week thus clearing their cache

The last stat about cache corruption is interesting – I appreciate the honesty. The IE 9 team experienced something similar. In IE 7&8 the cache was capped at 50 MB based on tests showing increasing the cache size didn’t improve the cache hit rate. They revisited this surprising result in IE9 and found that larger cache sizes actually did improve the cache hit rate:

In IE9, we took a much closer look at our cache behaviors to better understand our surprising finding that larger caches were rarely improving our hit rate. We found a number of functional problems related to what IE treats as cacheable and how the cache cleanup algorithm works. After fixing these issues, we found larger cache sizes were again resulting in better hit rates, and as a result, we’ve changed our default cache size algorithm to provide a larger default cache.

Will mentions that Chrome’s 320 MB cap should be revisited. 30% seems like a low percentage for full caches, but could be accounted for by users that aren’t very active and active users that only visit a small number of websites (for example, just Gmail and Facebook). If possible I’d like to see these full cache statistics correlated with activity. It’s likely that user who account for the biggest percentage of web visits are more likely to have a full cache, and thus experience slower page load times.

Next steps

First, much of the data for this post came from the HTTP Archive, so I’d like to thank our sponsors: Google, Mozilla, New Relic, O’Reilly Media, Etsy, Strangeloop, dynaTrace Software, and Torbit.

The data presented here suggest a few areas to focus on:

Website owners need to increase their use of a Cache-Control max-age, and the max-age times need to be longer. 38% of resources were unchanged over a 1 month period, and yet only 11% of resources have a max-age value that high. Most resources, even if they change, can be refreshed by including a fingerprint in the URL specified in the HTML document. Only bootstrap scripts from 3rd parties should have short cache times (hours). Truly dynamic responses (JSON, etc.) should specify must-revalidate. A year from now rather than seeing 55% of resources without any max-age value we should see 55% cacheable for a month or more.

3rd party content providers need wider adoption of the caching and async behavior shown by the top Google, Twitter, and Facebook snippets.

Browser developers stand to bring the biggest improvements to caching. Increasing cache sizes is a likely win, especially for mobile devices. Data correlating cache sizes and user activity is needed. More intelligence around purging algorithms, such as IE 9′s prioritization based on mime type, will help when the cache fills up. More focus on personalization (what are the sites I visit most often?) would also create a faster user experience when users go to their favorite websites.

It’s great that the number of resources with caching headers grew 10% over the last year, but that just isn’t enough progress. We should really expect to double the number of resources that can be read from cache over the coming year. Just think about all those HTTP requests that can be avoided!

Today’s AWS Elastic Beanstalk announcement of PHP and Git support reminded me of the post where I mentioned that we want to let a thousand platforms bloom on AWS. Some might ask why AWS would want a thousand platforms.

One of the most important AWS principles is flexibility. Flexibility is in the choice of software and languages running on AWS, in the tools and interfaces available to manipulate resources and applications, and in the ability to leverage services from other providers. One of our customers I met last week was talking about his application and how it runs on AWS; He collects geo-location data, analyzes and crunches this data using Elastic Map Reduce, stores the data for quick access in DynamoDB, runs his user interface on Heroku and his web services layer for mobile devices on Elastic Beanstalk. This application is a great way to highlight how developers might leverage different services, abstractions, and tools to deliver the most value to their customers.

If you’re seeking ultimate flexibility, AWS allows you to interact with services such as Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3) directly and to piece these services together in a building block fashion. This might incur some initial groundwork, especially if you just want to deploy a simple application. AWS CloudFormation can help bring the building blocks together through its template mechanism. This simplifies the provisioning and updates, but you’re still responsible for the operational aspects of running your application.

If you don’t need control over the software stack, you can use development platforms such as AppFog, Engine Yard, and Heroku to help you manage, deploy, and monitor your applications on AWS more easily. We’ve seen some newcomers in this space over the last year such as Stackato and NodeJitsu, and each platform continues to add value through highly curated software stacks and a set of management automation.

AWS Elastic Beanstalk is another abstraction on top of the core AWS building blocks. It takes a different approach than most other development platforms by exposing the underlying resources. This approach provides the simplicity to quickly get started for application developers, but it also allows them to modify the stack to meet their goals. For example, one customer needed extensive Apache rewrite rules and a few other mods to meet his security requirements. He simply created a new AMI to use as his base for his Elastic Beanstalk container. Another pattern I have seen is customers attaching a debugger to the JVM running in their EC2 instance so that they can debug particular interaction patterns between their code and the JVM.

So is there a “one-size-fits-all” in the development platform space? No, each platform fits the needs of different developers, applications, and use cases. Preference and familiarity also play a role in why some developers choose one over the other. Ultimately, we want developers to successfully run and manage reliable, highly scalable applications on AWS, irrespective of the abstraction that their development platform of choice offers.

We will continue to work closely together with all current and future platform partners. Based on their feedback, we will develop new features and services that can help them be more successful by allowing them to focus on their customers instead of the infrastructure on which they run. This will also make it easier for new platforms to be developed such that developers will have more choice and flexibility, and they can really find the exact tools that make them most productive. AWS Elastic Beanstalk can play an important role there, too, because it is a good base for building new platforms. We are looking forward to seeing a thousand platforms bloom.

AWS Elastic Beanstalk now supports PHP applications (in addition to Java) and the ability to deploy through the popular Git version control system. To get started using PHP and Git on AWS Elastic Beanstalk, visit Deploying PHP Applications Using Git in the AWS Elastic Beanstalk Developer Guide. More details about the release at the AWS developer blog.

Both the Module Manager and the MVC system use the Event Manger extensively in order to provide "hook points" for you to add your own code into the application flow. This is a list of the events triggered by each class during a standard request with the Skeleton Application:

Module Manager

• ZendModuleManager: loadModules.pre
• For every module:
  • ZendModuleManager: loadModule.resolve
  • ZendModuleManager: loadModule
• ZendModuleManager: loadModules.post

Bootstrap

• ZendMvcBootstrap: bootstrap

Application

Successful:

• ZendMvcApplication: route
• ZendMvcApplication: dispatch
  • ZendMvcControllerActionController: dispatch (if controller extends this class)
• ZendMvcApplication: render
• ZendViewView: renderer
• ZendViewView: response
• ZendMvcApplication: finish

• ZendMvcApplication: dispatch.error
• ZendMvcApplication: render
• ZendViewView: renderer
• ZendViewView: response
• ZendMvcApplication: finish

Note that routing and dispatching is also implemented using these registered events, so you can implement "pre" and "post" hooks by changing the priority of the listener that you register.

Ralph Schindler has posted PHP Constructor Best Practices And The Prototype Pattern

If your knowledge of constructors ends with “the place where I put my object initialization code,” read on. While this is mostly what a constructor is, the way a developer crafts their class constructor greatly impacts the initial API of a particular class/object; which ultimately affects usability and extensibility. After all, the constructor is the first impression a particular class can make.

In case you missed this last Friday, this is an in-depth look at how to construct an object in PHP whilst adhering to SOLID principles. If you missed this last week, read it now! Get a coffee first.

Read more »

The long lasting MySQL replication failover issue is cured. MySQL 5.6 makes master failover easy, PECL/mysqlnd_ms assists with the client/connection failover. Compared to the past this is a significant step towards improving MySQL replication cluster availability, eleminating the need to use 3rd party tools in many cases. The slides illustrate the basic idea, as blogged about before.

There is not much to say about the feature as such. Slave to master promotion works without hassles, finally. Regardless if you do failover because of an error of the current master or switchover because you want to change the master server, its easy now. Congratulations to the replication team!

Limitations of the current server implementation

The global transaction identifier implementation in MySQL 5.6 has a couple of limitations, though. Its not hard to guess that mixing transactional and non-transactional updates in one transaction can cause problems. Its pretty much the first pitfall I ran into when trying to setup a MySQL 5.6.5-m8 (not 5.6.4-m8…) slave using a mysqldump generated SQL dump. MySQL bailed at me and stopped me from failing.

Let a master run a transaction which first updates an InnoDB table. followed by an update of a MyISAM table, followed by another update: t(UInnoDB, UMyISAM, UX). Let the binary log settings be so that this transaction is written as one to the binary log (binlog_format=statement, binlog_direct_non_transactional_updates=0). It is then copied “as is” to the relay log of a slave. Assume that the slave runs with different binary log settings so that t(UInnoDB, UMyISAM, UX) is split up to t(UMyISAM), t(UInnoDB[, …])and logged as distinct transactions in the slaves binary log.

Because slaves must preserve global transaction identifiers they got from their master, the two resulting transactions are given the same identifier. The transaction identifier in the slaves binary log is no longer unique, it now refers to two transactions not just one (issue #1). Any slave that would read from the binary log of the above slave may loose the InnoDB transaction because it may refuse to execute a transaction using an id that has been executed already (issue #2).

The workaround? Don’t mix InnoDB and MyISAM updates in one transaction. To me it does not sound that much of an issue in 2012. Please note, I’m describing my experience with MySQL 5.6.5-m8, which is a development version.

The load balancer update

MySQL Replication takes a primary copy approach to replication. A primary/master handles all the updates. Read-only replicas/slaves replicate from the primary. The primary is a single point of failure.

The failure of a slave is unproblematic. A client usually has plenty of other slaves to start reading from. If no slave is available, reads can even be forwarded to the master.

All a PECL/mysqlnd_ms user has to do is check for an error after statement execution. If there’s one and the error code hints that the server has gone away, the user reruns the statement. The connection handle remains useable all the time. Upon rerun, PECL/mysqlnd_ms openes a new connection to another slave.

do {
  $res = $mysql->query("SELECT id, title FROM news");
} while (isset($connection_error_codes[$mysql->errno]));
	
if (!$res) {
  bail("SQL error", $mysql->errno, $mysql->error);
}

A master failure is much more problematic. There is no server to send a write to but the master. The master is a single point of failure. The new global transaction identifier help to reduce the time it takes to put a new master in place after a failure.

After a master failure some process needs to promote a former slave to the new master and, preferrable atomically, update all other slaves to start replicating from the new master. The below illustration is a bit confusing. It is intentionally. What happens during the simple to say “slave to master promotion” is a complete restructuring of the cluster.

Don’t forget to update the load balancer

After the cluster has been reorganized, the load balancer configurations must be updated. PECL/mysqlnd_ms happens to be a driver integrated load balancer. However, other than that, it is not different from a classical load balancer. Whatever process restructures the cluster it must take care of deploying the load balancer configurations afterwards.

Global transaction identifiers are a great help for the biggest part of the failover job - the server side. But, they are no swiss army knife. Don’t forget to update your load balancer configuration - the client side. No matter where it is. Whether it is part of the application code, driver integrated or you are using MySQL Proxy. As long as we are talking primary copy, a master failure will always be a major pain.

Happy hacking!

@Ulf_Wendel 

Pádraic Brady has posted A Hitchhiker’s Guide to Cross-Site Scripting (XSS) in PHP (Part 1): How Not To Use Htmlspecialchars() For Output Escaping:

Always set the third parameter to htmlspecialchars(), set it correctly, and make sure your document is never served with a mismatched or invalid character encoding! Don’t expect some theoretically perfect world to magically appear - browsers are filthily efficient at doing weird things you don’t expect.

With a nod to the anniversary of Douglas Adams' death on Sunday, Pádraic Brady has written possibly the definitive guide to the htmlspecialchars() function.

Read it. Then read it again.

(Photo credit: bertboerland)

In recent weeks, I consulted with the second most intelligent species on the planet: Dolphins. Dolphins are renowned across the known Universe for their awesome programming skills. After all, it was they who developed such insightful works as “Evolution By Example”, “Dude! We Wrote The Laws Of Physics!”, and “How Many Humans Does It Take To Screw Up A Planet?”. The answer to the last will be published on 01/01/2013 after the experiment is shut down and sent to a landfill site assuming the Supreme Spaghetti Monster signs off on the permit.

Dolphins think we are really dumb and theorise that this level of stupidity has one obvious cause: self-imposed ignorance. We are, after all, only the third most intelligent species on Earth and appear to have aspirations to lower our IQ just a bit more.

While it’s no harm poking fun at ourselves, in PHP we do have a serious problem. Cross-Site Scripting (XSS) remains one of the most significant classes of security problems afflicting PHP applications. Despite years of education, community awareness and the development of frameworks which can offer a huge boost in consistent practices – things are not getting any better.

So, I finally figured out what the core problem is: PHP programmers are completely clueless about XSS. It’s that simple. Instead of going out and studying the topic, we blindly follow some preferred herd of people offering advice with heartfelt conviction despite the fact that they are probably just as ignorant as the rest of us. Does that sound like the behaviour of something which allegedly evolved into an intelligent species? The result is a mix of ignorance and stagnant knowledge that leaves PHP in an unenviable position beset by wrongheaded zealots.

To get the ball rolling, this two-part article series is a tour of how NOT to use the htmlspecialchars() function that is typically pressed ganged into service as PHP’s universal output escaper. By offering an example based guide, I hope it will illustrate just how many ways a prospective attacker using XSS can exploit this function’s misuse to pull off a successful attack. The examples were written for PHP 5.3, so 5.4 users may need to imagine they still have 5.3 installed and/or lodge an official complaint with somebody who looks like they keep a complaints box handy (your local fast food restaurant is a good start).

This example led approach has another motive. Simple examples can be translated into unit tests. Ideally, many of the current crop of frameworks can use this article as a guide to what their unit tests should be looking for. This also makes it far easier for everyday programmers to consume the article and run around the place, drunk with ungodly power, identifying issues in the libraries, frameworks and other projects that they rely on.

To help us on the path of enlightenment before it’s too late (I’d lodge an appeal with the Supreme Spaghetti Monster but apparently the Mayans already tried and failed), I also invite other PHP programmers to blog about a security topic over the next month or two. Give programmers one last chance to get it right before the Planet is demolished by the Vogon destructor fleet. Just pick a topic that drives you up the walls in defiance of gravity and spend an hour writing something useful and (optionally) expletive filled. Every little bit helps.

What Is Htmlspecialchars()?

According to many programmers from Earth, htmlspecialchars() is a function used to escape output to prevent XSS. This is however a completely wrong definition. The function was actually co-opted by programmers to combat XSS because it was either that or create slow userland functions for which the internals developers might get around to creating, when the full moon coincided with the right planetary alignment in another 314 years, a speedier C alternative to. The actual definition (along with a half-hearted self-doubting nod to preventing XSS) is as follows:

Certain characters have special significance in HTML, and should be represented by HTML entities if they are to preserve their meanings. This function returns a string with some of these conversions made; the translations made are those most useful for everyday web programming. If you require all HTML character entities to be translated, use htmlentities() instead. This function is useful in preventing user-supplied text from containing HTML markup, such as in a message board or guest book application.

Note that this hints at, but does not explicitly use, the terms Cross-Site Scripting, XSS or even Security. Then again, it does refer to guest book applications so it was probably written in 1790 by the Dolphin who created PHP v86 and who then got around to backporting version 1.0 for Humans in the late 20th Century out of extreme pity for our reliance on CGI. No, not the let’s take an action movie and turn it into a plotless eyesore with computer generated fake stuff style CGI – though memories of both are comparably bad.

Does this make htmlspecialchars() terrible at preventing XSS? No. As part of a comprehensive well-understood strategy to prevent XSS, the function is very useful. However, in PHP it is frequently overused, misused, abused, confused and…. Darn it, ran out of rhyming words again. Suffice it to say that a good description of htmlspecialchars() is that it’s an unsuitable tool for preventing XSS that has slowly evolved into a better suited tool over the years. I keep telling myself that, at least.

The function, htmlspecialchars(), accepts four parameters. Here is its function prototype as of PHP 5.4:

string htmlspecialchars ( string $string [, int $flags = ENT_COMPAT | ENT_HTML401 [, string $encoding = 'UTF-8' [, bool $double_encode = true ]]] )

The first parameter accepts a string whose special HTML characters will be converted to HTML entities. The second accepts one or more flags which defaults to using ENT_COMPAT (does not convert single quotes to entities) but should be set to use ENT_QUOTES (does convert single quotes to entities). You can include another flag, in PHP 5.4, called ENT_SUBSTITUTE which is not a bad idea for UTF-8, i.e. ENT_QUOTES | ENT_SUBSTITUTE. You can pretend that all the other constants don’t exist. The third parameter accepts a string indicating the character encoding of the string being processed and defaults to ISO-8859-1 for PHP 5.3, and UTF-8 for PHP 5.4. Don’t ever set the fourth parameter to TRUE when escaping unless your filtering logic was written by an Über Dolphin – always keep filtering and escaping separate from each other to avoid confusing the two and then having to pointlessly argue why your way is better in defiance of all logic.

The function, if correctly configured using this super simple article for guidance, will now convert the following characters to entities: <, >, ‘, ” and &. These characters make sense to escape since they are used to construct HTML tags, delineate attribute values or reference HTML entities – none of which we want users to be able to do!

If you want some very good advice before your brain implodes from too much reading, a good way to potentially make yourself vulnerable to XSS is to not explicitly set the first two optional parameters ($flags and $encoding) to an appropriate value. In fact, if you see htmlspecialchars() missing any of those two parameters in someone’s source code, you should request that they fix it or, at the very least, curse their name and pray for the Supreme Spaghetti Monster to label them as biohazardous waste in need of emergency disposal.

Now, let’s get down to overloading your brain with information. I’m told that this part is like being sucked into the Total Perspective Vortex machine on Frogstar World B.

To Quote Or Not To Quote. How Is That A Question?

As it turns out, HTML is not simply a popular markup language, it is a popular markup language designed by a bureaucratic species of transdimensional beings seeking to drive Humanity insane by inventing the most impossible-to-secure markup language known in 172 Universes which is then interpreted by “browsers” written by Mice to test the patience of security professionals and keep the really intelligent Humans distracted from the truth of their soon-to-end existence as they search out ever more ludicrous examples of parsing weirdness. Excuse me, I held my breath writing that and need to fetch my Oxygen tank…

Consider the following example. If you want to see whether they work without copy pasting, you can clone all examples from my ominously titled xss repository on Github into a webroot somewhere to read or execute them.

Single Quoted Attributes

1. <?php header('Content-Type: text/html; charset=UTF-8'); ?>

2. <!DOCTYPE html>

3. <?php

4. $input = <<<INPUT

5. ' onmouseover='alert(/Meow!/);

6. INPUT;

7. /**

8.  * NOTE: This is equivalent to using htmlspecialchars($input, ENT_COMPAT)

9.  */

10. $output = htmlspecialchars($input);

11. ?>

12. <html>

13. <head>

14.     <title>Single Quoted Attribute</title>

15.     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

16. </head>

17. <body>

18.     <div>

19.         <span title='<?php echo $output ?>'>

20.             What's that latin placeholder text again?

21.         </span>

22.     </div>

23. </body>

24. </html>

If you run the example from a browser and pass your mouse pointer over the text, you will get a popup saying “/Meow!/”. Granted, this is hardly the most impressive XSS ever but remember that the Javascript executed could be a lot more ingenious and damaging. The reason you see alert() used everywhere in XSS examples is to prove that Javascript was executable – a real attacker will hardly advertise his success like this.

In this case, the htmlspecialchars() function call omits the second parameter which defaults to using the ENT_COMPAT flag. With this setting, the function does not convert single quotes to entities, allowing us to inject an unescaped single quote (to close the title attribute value) and another to start a new attribute and value which will be closed by the final single quote used in the template.

We can fix this problem in one of two ways:

1. Use double quotes which will prevent user input from breaking out of the HTML attribute value context using single quotes; or

2. Set the second parameter to htmlspecialchars() to use the ENT_QUOTES flag which will escape any single quotes a user tries to inject.

The moral of the story can be made even clearer by another example. In this case we use another perfectly validating means of delineating attribute values in HTML5 – we just don’t bother using quotes at all!

Quoteless Attributes

1. <?php header('Content-Type: text/html; charset=UTF-8'); ?>

2. <!DOCTYPE html>

3. <?php

4. $input = <<<INPUT

5. faketitle onmouseover=alert(/Meow!/);

6. INPUT;

7. $output = htmlspecialchars($input, ENT_QUOTES);

8. ?>

9. <html>

10. <head>

11.     <title>Quoteless Attribute</title>

12.     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

13. </head>

14. <body>

15.     <div>

16.         <span title=<?php echo $output ?>>

17.             What's that latin placeholder text again?

18.         </span>

19.     </div>

20. </body>

21. </html>

Without quotes delineating the attribute value, any space character (including any character a browser might interpret as a space – there are a lot!) allows the user to inject new attributes and values. As from the above, converting all quotes to entities is pointless if there are no quotes to start with! Our escaping doesn’t convert spaces or other space-interpreted characters into entities at all.

By now, you should see the obvious. All HTML attribute values MUST be quoted, and preferably DOUBLE quoted, in any scenario where you suspect untrusted input will be injected into an attribute value, or where htmlspecialchars() calls do not set the second parameter to use ENT_QUOTES. Believe it or not, using single quotes or no quotes remains popular and is perfectly valid under the new HTML5 spec. Some people even celebrate this new insanity. Keep an eye on any designers who look a bit wild eyed or spend too much time smiling while staring into empty space.

Excuse Me, Sir, But Someone Ate My Quotes

One of the great mysteries in escaping output is a common myth known as the Great ASCII Delusion (GAD). Those under the influence of this delusion, besides hearing voices in their head, have arrived at a belief that many character encodings are equivalent for the purposes of escaping those characters which have a special meaning for HTML, e.g ISO-8859-1 and UTF-8. Alas, this is untrue because the Mice created something called Internet Explorer 6 – a thoroughly shameful (but still commonly used) browser which corporations across the Planet continue to insist on using because buying new computers and upgrading operating systems just to use some fancy new Microsoft Office version is seen as a waste of shareholder funds.

Internet Explorer 6 is the bad boy of the XSS world since it’s vulnerable to ridiculous exploits no decent modern browser would dare associate with. Even Netscape would probably spit on it from beyond the grave. For example, have a go with this example using IE6 and PHP 5.3. If you need a testing version of all IE browsers since IE 5.5, you can download IETester from http://www.my-debugbar.com/ietester/index_all.php and use it from Windows. Try hard, I know Windows is bad and the new Tablet makeover for Windows 8 makes you feel ill, but it’s important to see these examples in action.

Source code

1. <?php header('Content-Type: text/html; charset=UTF-8'); ?>

2. <!DOCTYPE html>

3. <?php

4. /**

5.  * You could also subsititute xC0 or any other impacted character

6.  * above ASCII number 192

7.  */

8. $input1 = 'fakeimage'.chr(192);

9. $input2 = <<<INPUT2

10. onerror=alert(/Meow!/)//

11. INPUT2;

12. $output1 = htmlspecialchars($input1, ENT_QUOTES);

13. $output2 = htmlspecialchars($input2, ENT_QUOTES);

14. ?>

15. <html>

16. <head>

17.     <title>Swallowed Quotes</title>

18.     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

19. </head>

20. <body>

21.     <div>

22.         <img src="http://example.com/images/<?php echo $output1 ?>"

23.         title="<?php echo $output2 ?>">

24.     </div>

25. </body>

26. </html>

With the above example, something very weird happens. Using ASCII character number 192 just before a double quote in a document being interpreted as UTF-8 results in the double quote…vanishing in IE6. Seriously, it’s there but not there. Obviously the Mice are behind it – no Human could possibly defy Physics like this!

This allows an attacker to once again break out of the HTML attribute they can inject values into. Using a coincidental opportunity to inject a second free text string nearby which a browser will concatenate to the broken out attribute value of the first, you get an effective XSS combo attack.

This IE6 quirk even bypasses the call to htmlspecialchars() which, as explained above, defaults to the ISO-8859-1 character encoding for PHP 5.3 or less. If the Great ASCII Delusion were not a fabrication of someone’s imaginative wishful thinking, this should not be possible. Not to be too harsh though, this weirdness is due primarily to a bug in IE6′s treatment of the various character encodings where you can trick the browser into thinking something like xC0 (in hex) is the start of a multi-byte character thus swallowing the next ASCII character (the double quote).

To fix the above weirdness, you must make sure that escaping is done using the same character encoding that the document is being served as. The above HTML document is identifying itself as being UTF-8 but the default htmlspecialchars() encoding is ISO-8859-1 in PHP 5.3 – there’s obviously something not agreeing there! This brings us to the absolutely perfect use (well, almost) of htmlspecialchars(), the golden rule, the Word of The Supreme Spaghetti Monster, the bringer of frustration to XSS attackers:

Always set the third parameter to htmlspecialchars(), set it correctly, and make sure your document is never served with a mismatched or invalid character encoding! Don’t expect some theoretically perfect world to magically appear – browsers are filthily efficient at doing weird things you don’t expect.

I suppose I have to mention that most versions of IE have similar issues with other character encodings such as BIG5 and Shift-JIS. You can test your IE versions using http://ha.ckers.org/weird/variable-width-encoding.cgi to see what characters can be used across different character encodings. Believe it or not, these character encodings are actually still being used and, for some strange reason, people from China and Japan do use PHP.

If you want to be completely paranoid, you can either check the input for invalid UTF-8 (Drupal and HTMLPurifier have reusable functions/classes for this), and/or run it through a conversion function which should theoretically filter out the naughty bits:

$input = mb_convert_encoding($input, 'UTF-8', 'UTF-8');

This is probably a good idea for older PHP versions pre 2010 or earlier but recent PHP versions have specifically improved htmlspecialchars() to disallow invalid characters such as the above (if you set the right character encoding!). You should be aware, though, that htmlspecialchars() may still return blank strings on certain malformed input and, since PHP 5.4, will not issue any warnings about this.

I Broke It! I Broke It!

Before you think htmlspecialchars() is getting off lightly, there is one minor quibble. We’ll keep picking on Internet Explorer 6 for the rest of this article since it’s so easy to exploit.

Source code

1. <?php header('Content-Type: text/html; charset=UTF-8'); ?>

2. <!DOCTYPE html>

3. <?php

4. $input1 = 'fakeimage'."xC0";

5. $input2 = <<<INPUT2

6. onerror=alert(/Meow!/)//

7. INPUT2;

8. /**

9.  * If you think PHP 5.4 will save you - empty strings make it guess the encoding

10.  * or use the default_charset value from php.ini. You sure everyone on the whole

11.  * planet uses UTF-8? Under 5.3 - empty strings === default encoding.

12.  */

13. $encoding = ''; // from outside source or unvalidated variable

14. $output1 = htmlspecialchars($input1, ENT_QUOTES, $encoding);

15. $output2 = htmlspecialchars($input2, ENT_QUOTES, $encoding);

16. ?>

17. <html>

18. <head>

19.     <title>Swallowed Quotes</title>

20.     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

21. </head>

22. <body>

23.     <div>

24.         <img src="http://example.com/images/<?php echo $output1 ?>"

25.         title="<?php echo $output2 ?>">

26.     </div>

27. </body>

28. </html>

Setting the third $encoding parameter of htmlspecialchars() to an empty string in PHP 5.4 will set the encoding to be auto-detected, grabbed from the php.ini value of default_charset, or guessed from the current locale (in that order). Be very careful under PHP 5.4 NEVER to let this happen. Don’t leave your escaping parameters to chance.

Use empty() or strlen(), for example, to spot this issue if accepting encodings from another source or variable that might allow for empty strings. Again, this behaviour is very secure and there’s nothing wrong with it whatsoever. Oh, who am I kidding… This is the dumbest parameter behaviour ever invented. NULL means use the default encoding; blank string means play a guessing game. Even Vogon poetry pales in comparison to such nonsense. One slip and an empty parameter string can rip apart this house of cards because who knows which character encoding will be used.

Oooh, I wonder what this does under PHP 5.3… Yes, er, don’t allow blank encoding parameter strings under PHP 5.3 either. Setting an empty string in PHP 5.3 is interpreted as setting the default character encoding, i.e. ISO-8859-1, instead of triggering the expected warning about an unsupported encoding.

So, be careful kids. When setting the encoding for htmlspecialchars() do a safety check to make sure it’s not an empty string you are passing in. Keep it predictable and consistent.

There’s also one other curious behaviour when using htmlspecialchars().

Source code

1. <?php header('Content-Type: text/html; charset=UTF-8'); ?>

2. <!DOCTYPE html>

3. <?php

4. error_reporting(E_ALL);

5. ini_set('display_errors', 1);

6. $input1 = 'fakeimage'."xC0";

7. $input2 = <<<INPUT2

8. onerror=alert(/Meow!/)//

9. INPUT2;

10. /**

11.  * Invalid encoding makes htmlspecialchars() throw a warning but it continues

12.  * the current operation anyway using the default encoding even if the default

13.  * is an unsafe choice for the application. Don't allow invalid encodings!

14.  */

15. $encoding = 'invalid-encoding'; // from outside source or unvalidated variable

16. $output1 = htmlspecialchars($input1, ENT_QUOTES, $encoding);

17. $output2 = htmlspecialchars($input2, ENT_QUOTES, $encoding);

18. ?>

Author: Interesting Blog Posts