Adam Leach

Measuring mobile performance is hard

When Amazon announced their Silk browser I got excited reading about the “split architecture”. I’m not big on ereaders but I pre-ordered my Kindle Fire that day. It arrived a week or two ago. I’ve been playing with it trying to find a scientific way to measure page load times for various websites. It’s not easy.

• Since it’s a new browser and runs on a tablet we don’t have plugins like Firebug.
• It doesn’t (yet) support the Navigation Timing spec, so even though I can inspect pages using Firebug Lite (via the Mobile Perf bookmarklet) and Weinre (I haven’t tried it but I assume it works), there’s no page load time value to extract.
• Connecting my Fire to a wifi hotspot on my laptop running tcpdump (the technique evangelized by pcapperf) doesn’t work in accelerated mode because Silk uses SPDY over SSL. This technique works when acceleration is turned off, but I want to see the performance optimizations.

While I was poking at this problem a bunch of Kindle Fire reviews came out. Most of them talked about the performance of Silk, but I was disappointed by the lack of scientific rigor in the testing. Instead of data there were subjective statements like “the iPad took about half as long [compared to Silk]” and “the Fire routinely got beat in rendering pages but often not by much”. Most of the articles did not include a description of the test procedures. I contacted one of the authors who confided that they used a stopwatch to measure page load times.

If we’re going to critique Silk and compare its performance to other browsers we need reproducible, unbiased techniques for testing performance. Using a stopwatch or loading pages side-by-side and doing a visual comparison to determine which is faster are not reliable methods for measuring performance. We need better tools.

Introducing Loadtimer

Anyone doing mobile web development knows that dev tools for mobile are lacking. Firebug came out in 2006. We’re getting close to having that kind of functionality in mobile browsers using remote debuggers, but it’s pretty safe to say the state of mobile dev tools is 3-5 years behind desktop tools. It might not be sexy, but there’s a lot to be gained from taking tools and techniques that worked on the desktop and moving them to mobile.

In that vein I’ve been working the last few days to build an iframe-based test harness similar to one I built back in 2003. I call it Loadtimer. (I was shocked to see this domain was available – that’s a first.) Here’s a screenshot:

The way it works is straightforward:

• It’s preloaded with a list of popular URLs. The list of URLs can be modified.
• The URLs are loaded one-at-a-time into the iframe lower in the page.
• The iframe’s onload time is measured and displayed on the right next to each URL.
• If you check “record load times” the page load time is beaconed to the specified URL. The beacon URL defaults to point to loadtimer.org, but you can modify it if, for example, you’re testing some private pages and want the results to go to your own server.
• You can’t test websites that have “framebusting” code that prevents them from being loaded in an iframe, such as Google, YouTube, Twitter, and NYTimes.

There are some subtle optimizations worth noting:

• You should clear the cache between each run (unless you explicitly want to test the primed cache experience). There’s no way for the test harness to clear the cache, but it does have a check that helps remind you to clear the cache. (It loads a script that is known to take 3 seconds to load – if it takes less than 3 seconds it means the cache wasn’t cleared.)
• It’s possible that URL 1′s unload time could make URL 2′s onload time be longer than it actually should be. To avoid this about:blank is loaded between each URL.
• The order of the preset URLs is randomized to mitigate biases across URLs, for example, where URL 1 loads resources used by URL 2.

Two biases that aren’t addressed by Loadtimer:

• DNS resolutions aren’t cleared. I don’t think there’s a way to do this on mobile devices short of power cycling. This could be a significant issue when comparing Silk with acceleration on and off. When acceleration is on there’s only one DNS lookup, whereas when acceleration is off there’s a DNS lookup for each hostname in the page (13 domains per page on average). Having the DNS resolutions cached gives an advantage to acceleration being off.
• Favicons aren’t loaded for websites in iframes. This probably has a negligible impact on page load times.

 Have at it

The nice thing about the Loadtimer test harness is that it’s web-based – nothing to install. This ensures it’ll work on all mobile phones and tablets that support JavaScript. The code is open source. There’s a forum for questions and discussions.

There’s also a results page. If you select the “record load times” checkbox you’ll be helping out by contributing to the crowdsourced data that’s being gathered. Getting back to what started all of this, I’ve also been using Loadtimer the last few days to compare the performance of Silk to other tablets. Those results are the topic of my next blog post – see you there.

 

In my previous blog post I announced Loadtimer – a mobile test harness for measuring page load times. I was motivated to create Loadtimer because recent reviews of the Kindle Fire lacked the quantified data and reliable test procedures needed to compare browser performance.

Most performance evaluations of Silk that have come out since its launch have two conclusions:

1. Silk is faster when acceleration is turned off.
2. Silk is slow compared to other tablets.

Let’s poke at those more rigorously using Loadtimer.

Test Description

In this test I’m going to compare the following tablets: Kindle Fire (with acceleration on and off), iPad 1, iPad 2, Galaxy 7.0, and Galaxy 10.1.

The test is based on how long it takes for web pages to load on each device. I picked 11 URLs that are top US websites:

• http://www.yahoo.com/
• http://www.amazon.com/
• http://en.wikipedia.org/wiki/Flowers
• http://www.craigslist.com/
• http://www.ebay.com/
• http://www.linkedin.com/
• http://www.bing.com/search?q=flowers
• http://www.msn.com/
• http://www.engadget.com/
• http://www.cnn.com/
• http://www.reddit.com/

Some popular choices (Google, YouTube, and Twitter) weren’t selected because they have framebusting code and so don’t work in Loadtimer’s iframe-based test harness.

The set of 11 URLs were loaded 9 times on each device. The set of URLs was randomized for each run. All the tests were conducted on my home wifi over a Comcast cable modem. (Check out this photo of my test setup.) All the tests were done at the same time of day over a 3 hour period. I did one test at a time to avoid bandwidth contention, and rotated through the devices doing one run at a time. I cleared the cache between each run.

Apples and Oranges

The median page load time for each URL on each device is shown in the Loadtimer Results page. It’s a bit complicated to digest. The fastest load time is shown in green and the slowest is red – that’s easy. The main complication is that not every device got the same version of a given URL. Cells in the table that are shaded with a gray background were cases where the device received a mobile version of the URL. Typically (but not always) the mobile version is lighter than the desktop version (fewer requests, fewer bytes, less JavaScript, etc.) so it’s not valid to do a heads up comparison of page load times between desktop and mobile versions.

Out of 11 URLs, the Galaxy 7.0 received 6 that were mobile versions. The Galaxy 10.1 and Silk each received 2 mobile versions, and the iPads each had only one mobile version across the 11 URLs.

In order to gauge the difference between the desktop and mobile versions, the results table shows the number of resources in each page. eBay, for example, had 64 resources in the desktop version, but only 18-22 in the mobile version. Not surprisingly, the three tablets that received the lighter mobile version had the fastest page load times. (If a mobile version was faster than the fastest desktop version, I show it in non-bolded green with a gray background.)

This demonstrates the importance of looking at the context of what’s being tested. In the comparisons below we’ll make sure to keep the desktop vs mobile issue in mind.

Silk vs Silk

Let’s start making some comparisons. The results table is complicated when all 6 rows are viewed. The checkboxes are useful for making more focused comparisons. The Silk (accel off) and Silk (accel on) results show that indeed Silk performed better with acceleration turned off for every URL. This is surprising, but there are some things to note.

First, this is the first version of Silk. Jon Jenkins, Director of Software Development for Silk, spoke at Velocity Europe a few weeks back. In his presentation he shows different places where the split in Silk’s split architecture could happen (slides 26-28). He also talked about the various types of optimizations that are part of the acceleration. Although he didn’t give specifics, it’s unlikely that all of those architectural pieces and performance optimizations have been deployed in this first version of Silk. The test results show that some of the obvious optimizations, such as concatenating scripts, aren’t happening when acceleration is on. I expect we’ll see more optimizations rolled out during the Silk release cycle, just as we do for other browsers.

A smaller but still important issue is that although the browser cache was cleared between tests, the DNS cache wasn’t cleared. When acceleration is on there’s only one DNS lookup needed – the one to Amazon’s server. When acceleration is off Silk has to do a DNS lookup for every unique domain – an average of 13 domains per page. Having all of those DNS lookups cached gives an unfair advantage to the “acceleration off” page load times.

I’m still optimistic about the performance gains we’ll see as Silk’s split architecture matures, but for the remainder of this comparison we’ll use Silk with acceleration off since that performed best.

Silk vs iPad

I had both an iPad 1 and iPad 2 at my disposal so included both in the study. The iPad 1 was the slowest across all 11 URLs so I restricted the comparison to Silk (accel off) and iPad 2.

The results are mixed with iPad 2 being faster for most but not all URLs. The iPad 2 is fastest in 7 URLs. Silk is fastest in 3 URLs. One URL (eBay) is apples and oranges since Silk gets a mobile version of the site (18 resources compared to 64 resources for the desktop version).

Silk vs Galaxy

Comparing the Galaxy 7.0 to any other tablet is not fair since Galaxy 7.0 receives a lighter mobile version in 6 of 11 URLs. The Galaxy 7.0 has the slowest page load time in 3 of the 4 URLs where it, Galaxy 10.1, and Silk all receive the desktop version. Since it’s slower head-to-head and has mobile versions in the other URLs, I’ll focus on comparing Silk to the Galaxy 10.1.

Silk has the fastest page load time in 7 URLs. The Galaxy 10.1 is faster in 3 URLs. One URL is mixed as Silk gets a mobile version (18 resources) while the Galaxy 10.1 gets a desktop version (64 resources).

 Takeaways

These results show that, as strange as it might sound, Silk appears to be faster when acceleration is turned off. Am I going to turn off acceleration on my Kindle Fire? No. I don’t want to miss out on the next wave of performance optimizations in Silk. The browser is sound. It holds its own compared to other tablet browsers. Once the acceleration gets sorted out I expect it’ll do even better.

More importantly, it’s nice to have some real data and to have Loadtimer to help with future comparisons. Doing these comparisons to see which browser/tablet/phone is fastest makes for entertaining reading and heated competition. But all of us should expect more scientific rigor in the reviews we read, and push authors and ourselves to build and use better tools for measuring performance. I hope Loadtimer is useful. Loadtimer plus pcapperf and the Mobile Perf bookmarklet are the start of a mobile performance toolkit. Between the three of them I’m able to do most of what I need for analyzing mobile performance. It’s still a little clunky, but just as it happened in the desktop world we’ll see better tools with increasingly powerful features across more platforms as the industry matures. It’s still early days.

 

So you’ve been building some responsive designs and you’ve been working through your checklist of things to do:

• You started with the content and designed around it, with mobile in mind first.
• You’ve gone liquid and there’s nary a px value in sight; % is your weapon of choice now.
• You’ve baked in a few media queries to adapt your layout and tweak your design at different window widths.
• You’ve made your images scale to the container width using the fluid Image technique.
• You’ve even done the same for your videos using a nifty bit of JavaScript.

You’ve done a good job so pat yourself on the back. But there’s still a problem and it’s as tricky as it is important: image resolutions.

HTML has an <img> problem

CSS is great at adapting a website design to different window sizes – it allows you not only to tweak layout but also to send rescaled versions of the design’s images. And you want to do that because, after all, a smartphone does not need a 1,900-pixel background image¹.

HTML is less great. In the same way that you don’t want CSS background images to be larger than required, you don’t want that happening with <img>s either. A smartphone only needs a small image but desktop users need a large one. Unfortunately <img>s can’t adapt like CSS, so what do we do?

Well, you could just use a high resolution image and the fluid image technique would scale it down to fit the viewport; but that’s sending an image five or six times the file size that’s really needed, which makes it slow to download and unpleasant to use. Smartphones are pretty impressive devices – my ancient iPhone 3G is more powerful in every way than my first proper computer – but they’re still terribly slow in comparison to today’s desktop machines. Sending a massive image means it has to be manipulated in memory and redrawn as you scroll. You’ll find phones rapidly run out of RAM and slow to a crawl.

Well, OK. You went mobile first with everything else so why not put in mobile resolution <img>s too? Because even though mobile devices are rapidly gaining share in your analytics stats, they’re still not likely to be the major share of your user base. I don’t think desktop users would be happy with pokey little mobile resolution images, do you? What we need are adaptive images.

Adaptive image techniques

There are a number of possible solutions, each with pros and cons, and it’s not as simple to find a graceful solution as you might expect.

Your first thought might be to use JavaScript to trawl through the markup and rewrite the source attribute. That’ll get you the right end result, but it’ll have done it in a way you absolutely don’t want. That’s because of the way browsers load resources. It starts to load the HTML and builds the page on-the-fly; as soon as it finds an <img> element it immediately asks the server for that image. After the HTML has finished loading, the JavaScript will run, change the src attribute, and then the browser will request that new image too. Not instead of, but as well as. Not good: that’s added more bloat instead of cutting it.

Plain JavaScript is out then, which is a problem, because what other tools do we have to work with as web designers? Let’s ignore that for now and I’ll outline another issue with the concept of serving different resolution images for different window widths: a basic file management problem. To request a different image, that image has to exist on the server. How’s it going to get there? That’s not a trivial problem, especially if you have non-technical users that update content through a CMS. Let’s say you solve that – do you plan on a simple binary switch: big image|little image? Is that really efficient or future-proof? What happens if you have an archive of existing content that needs to behave this way? Can you apply such a solution to existing content or markup?

There’s a detailed round-up of potential techniques for solving the adaptive images problem over at the Cloud Four blog if you fancy a dig around exploring all the options currently available. But I’m here to show you what I think is the most flexible and easy to implement solution, so here we are.

Adaptive Images

Adaptive Images aims to mitigate most of the issues surrounding the problems of bringing the venerable <img> tag into the 21st century. And it works by leaving that tag completely alone – just add that desktop resolution image into the markup as you’ve been doing for years now. We’ll fix it using secret magic techniques and bottled pixie dreams. Well, fine: with one .htaccess file, one small PHP file and one line of JavaScript. But you’re killing the mystique with that kind of talk.

So, what does this solution do?

• It allows <img>s to adapt to the same break points you use in your media queries, giving granular control in the same way you get with your CSS.
• It installs on your server in five minutes or less and after that is automatic and you don’t need to do anything.
• It generates its own rescaled images on the server and doesn’t require markup changes, so you can apply it to existing web content.
• If you wish, it will make all of your images go mobile-first (just in case that’s what you want if JavaScript and cookies aren’t available).

Sound good? I hope so. Here’s what you do.

Setting up and rolling out

I’ll assume you have some basic server knowledge along with that wealth of front-end wisdom exploding out of your head: that you know not to overwrite any existing .htaccess file for example, and how to set file permissions on your server. Feeling up to it? Excellent.

1. Download the latest version of Adaptive Images either from the website or from the GitHub repository.
2. Upload the included .htaccess and adaptive-images.php files into the root folder of your website.
3. Create a directory called ai-cache and make sure the server can write to it (CHMOD 755 should do it).
4. Add the following line of JavaScript into the <head> of your site:

<script>document.cookie='resolution='+Math.max(screen.width,screen.height)+'; path=/‘;</script>

That’s it, unless you want to tweak the default settings. You likely do, but essentially you’re already up and running.

How it works

Adaptive Images does a number of things depending on the scenario the script has to handle, but here’s a basic overview of what it does when you load a page running it:

1. A session cookie is written with the value of the visitor’s screen size in pixels.
2. The HTML encounters an <img> tag and sends a request to the server for that image. It also sends the cookie, because that’s how browsers work.
3. Apache sits on the server and receives the request for the image. Apache then has a look in the .htaccess file to see if there are any special instructions for files in the requested URL.
4. There are! The .htaccess says “Hey, server! Any request you get for a JPG, GIF or PNG file just send to the adaptive-images.php file instead.”
5. The PHP file then does some intelligent thinking which can cover a number of scenarios, but I’ll illustrate one path that can happen:

• The PHP file looks for the cookie and finds out that the user has a maximum screen width of 480px.
• The PHP has a look at the available media query sizes that were configured and decides which one matches the user’s device.
• It then has a look inside the /ai-cache/480/ folder to see if a rescaled image already exists there.
• We’ll pretend it doesn’t – the PHP then goes to the actual requested URI and finds that the original file does exist.
• It has a look to see how wide that image is. If it’s already smaller than the user’s screen width it sends it along and stops there. But, let’s pretend the image is 1,000px wide.
• The PHP then resizes the image and saves it into the /ai-cache/480 folder ready for the next time someone needs it.

It also does a few other things when needs arise, for example:

• It sends images with a cache header field that tells proxies not to cache the image, while telling browsers they should. This avoids problems with proxy servers and network caching systems grabbing the wrong image and storing it.
• It handles cases where there isn’t a cookie set, and you can choose whether to then send the mobile version or the largest configured media query size.
• It compares timestamps between the source image and the generated cache image – to ensure that if the source image gets updated, the old cached file won’t be sent.

Customizing

There are a few options you can customize if you don’t like the default values. By looking in the PHP’s configuration section at the top of the file, you can:

• Set the resolution breakpoints to match your media query break points.
• Change the name and location of the ai-cache folder.
• Change the quality level any generated JPG images are saved at.
• Have it perform a subtle sharpen on rescaled images to help keep detail.
• Toggle whether you want it to compare the files in the cache folder with the source ones or not.
• Set how long the browser should cache the images for.
• Switch between a mobile-first or desktop-first approach when a cookie isn’t found.

More importantly, you probably want to omit a few folders from the AI behaviour. You don’t need or want it resizing the images you’re using in your CSS, for example. That’s fine – just open up the .htaccess file and follow the instructions to list any directories you want AI to ignore. Or, if you’re a dab hand at RewriteRules you can remove the exclamation mark at the start of the rule and it’ll only apply AI behaviour to a given list of folders.

Caveats

As I mentioned, I think this is one of the most flexible, future-proof, retrofittable and easy to use solutions available today. But, there are problems with this approach as there are with all of the ones I’ve seen so far.

This is a PHP solution

I wish I was smarter and knew some fancy modern languages the cool kids discuss at parties, but I don’t. So, you need PHP on your server. That said, Adaptive Images has a Creative Commons licence² and I would welcome anyone to contribute a port of the code³.

Content delivery networks

Adaptive Images relies on the server being able to: intercept requests for images; do some logic; and send one of a given number of responses. Content delivery networks are generally dumb caches, and they won’t allow that to happen. Adaptive Images will not work if you’re using a CDN to deliver your website.

A minor but interesting cookie issue.

As Yoav Weiss pointed out in his article Preloaders, cookies and race conditions, there is no way to guarantee that a cookie will be set before images are requested – even though the JavaScript that sets the cookie is loaded by the browser before it finds any <img> tags. That could mean images being requested without a cookie being available. Adaptive Images has a two-fold mechanism to avoid this being a problem:

1. The $mobile_first toggle allows you to choose what to send to a browser if a cookie isn’t set. If FALSE then it will send the highest configured resolution; if TRUE it will send the lowest.
2. Even if set to TRUE, Adaptive Images checks the User Agent String. If it discovers the user is on a desktop environment, it will override $mobile_first and set it to FALSE.

This means that if $mobile_first is set to TRUE and the user was unlucky (their browser didn’t write the cookie fast enough), mobile devices will be supplied with the smallest image, and desktop devices will get the largest.

The best way to get a cookie written is to use JavaScript as I’ve explained above, because it’s the fastest way. However, for those that want it, there is a JavaScript-free method which uses CSS and a bogus PHP ‘image’ to set the cookie. A word of caution: because it requests an external file, this method is slower than the JavaScript one, and it is very likely that the cookie won’t be set until after images have been requested.

The future

For today, this is a pretty good solution. It works, and as it doesn’t interfere with your markup or source material in any way, the process is non-destructive. If a future solution is superior, you can just remove the Adaptive Images files and you’re good to go – you’d never know AI had been there.

However, this isn’t really a long-term solution, not least because of the intermittent problem of the cookie and image request race condition. What we really need are a number of standardized ways to handle this in the future.

First, we could do with browsers sending far more information about the user’s environment along with each HTTP request (device size, connection speed, pixel density, etc.), because the way things work now is no longer fit for purpose. The web now is a much broader entity used on far more diverse devices than when these technologies were dreamed up, and we absolutely require the server to have better knowledge about device capabilities than is currently possible. Relying on cookies to do this job doesn’t cut it, and the User Agent String is a complete mess incapable of fulfilling the various purposes we are forced to hijack it for.

Secondly, we need a W3C-backed markup level solution to supply semantically different content at different resolutions, not just rescaled versions of the same content as Adaptive Images does.

I hope you’ve found this interesting and will find Adaptive Images useful.

Footnotes

¹ While I’m talking about preventing smartphones from downloading resources they don’t need: you should be careful of your media query construction if you want to stop WebKit downloading all the images in all of the CSS files.

² Adaptive Images has a very broad Creative Commons licence and I warmly welcome feedback and community contributions via the GitHub repository.

³ There is a ColdFusion port of an older version of Adaptive Images. I do not have anything to do with ported versions of Adaptive Images.

Using :before and :after pseudo-elements to produce a rollover menu.

Using :target to produce a click open/close dropline menu.

In responsive web design we’ve found a technique that allows us to design for the web as a medium in its own right: one that presents a fluid, adaptable and ever changing canvas.

Until this point, we gave little thought to the environment in which users will experience our work, caring more about the aggregate than the individual. The applications we use encourage rigid layouts, whilst linear processes focus on clients signing off paintings of websites that have little regard for behaviour and interactions. The handover of pristine, pixel-perfect creations to developers isn’t dissimilar to farting before exiting a crowded lift, leaving front-end developers scratching their heads as they fill in the inevitable gaps. If you haven’t already, I recommend reading Drew’s checklist of things to consider before handing over a design.

Somehow, this broken methodology has survived for the last fifteen years or so. Even the advent of web standards has had little impact. Now, as we face an onslaught of different devices, the true universality of the web can no longer be ignored.

Responsive web design is just the thin end of the wedge. Largely concerned with layout, its underlying philosophy could ignite a trend towards interfaces that adapt to any number of different variables: input methods, bandwidth availability, user preference – you name it!

With such adaptability, a collaborative and iterative process is required. Ethan Marcotte, who worked with the team behind the responsive redesign of the Boston Globe website, talked about such an approach in his book:

The responsive projects I’ve worked on have had a lot of success combining design and development into one hybrid phase, bringing the two teams into one highly collaborative group.

Whilst their process still involved the creation of desktop-centric mock-ups, these were presented to the entire team early on, where questions about how pages might adapt and behave at different sizes were asked. Mock-ups were quickly converted into HTML prototypes, meaning further decisions could be based on usage rather than guesswork (and endless hours spent in Photoshop).

Regardless of the exact process, it’s clear that the relationship between our two disciplines is more crucial than ever. Yet, historically, it seems a wedge has been driven between us – perhaps a result of segregation and waterfall-style processes – resulting in animosity.

So how can we improve this relationship? Ultimately, we’ll need to adapt, but even within existing workflows we can start to overlap. Simply adjusting our attitude can effect change, and bring design and development teams closer together.

Good design is constant contact.

Mark Otto

The way we work needs to be more open and inclusive. For example, ensuring members of the development team attend initial kick-off meetings and design workshops will not only ensure technical concerns are raised, but mean that those implementing our designs better understand the problems we’re trying to solve.

It can also be useful at this stage to explain how you work and the sort of deliverables you expect to produce. This will give developers a chance to make recommendations on how these can be optimized for their own needs.

You may even find opportunities to share the load. On a recent project I worked on, our development partners offered to produce the interactive prototypes needed for user testing. This allowed us to concentrate on refining the experience, whilst they were able to get a head start on building the product.

While developers should be involved at the beginning of projects, it’s also important that designers are able to review and contribute to a product as it’s being built. Any handover should be done in person, and ideally you’ll have a day set aside to do so. Having additional budget available for follow-up design reviews is also recommended. Learning how to use version control tools like Subversion or Git will allow you to work within the same environment as developers, and allow you to contribute code or graphic assets directly to a project if needed.

Don’t underestimate the benefits of designer and developer sitting next to each other. Subtle nuances can be explored far more easily than if they were conducted over email or phone. As Ethan writes, “‘Design’ is the means, not merely the end; the path we walk over the course of a project, the choices we make”.

It’s from collaboration like this that I’ve become fond of producing visual style guides. These demonstrate typographic treatments for common markup and patterns (blockquotes, lists, pagination, basic form controls and so on). Thinking in terms of components rather than individual pages not only fits in better with how a developer will implement a site, but can also ensure your design works as a coherent whole.

Despite the amount of research and design produced, when it comes to the crunch, there will always be a need for compromise. As the old saying goes, ‘fast, cheap and good – pick two.’ It’s important that you know which pieces are crucial to a design and which areas can allow for movement. Pick your battles wisely. Having an agreed set of design principles can be useful when making such decisions, as they help everyone focus on the goals of the project.

The best compromises are reached when both sides understand the issues of the other.

Richard Rutter

Ultimately, better collaboration comes through a shared understanding of the different competencies required to build a website. Instead of viewing ourselves in terms of discrete roles, we should instead look to emphasize our range of abilities, and work with others whose skills are complementary.

Perhaps somebody who actively seeks to broaden their knowledge is the mark of a professional. Seek these people out.

The best developers I’ve worked with have a respect for design, probably having attempted to do some themselves! Having wrangled with a few MySQL databases myself, I certainly believe the obverse is true. While knowing HTML won’t necessarily make you a better designer, it will help you understand the issues being faced by a front-end developer and, more importantly, allow you to offer solutions or alternative approaches.

So take a moment to think about how you work with developers and how you could improve your relationship with them. What are you doing to ease the path towards our collaborative future?

Yesterday I had to convert a few mp3 files to wavs. I also needed to crop them a bit on both ends. This is how I did it:

mplayer -ss 0.5 -endpos 1:14 -ao pcm:file="newfile.wav" oldfile.mp3

This is how it works: -ss says how many seconds of the file to skip at the start. If you don't know the time, but the number of bytes instead, use -sb instead. Note that -ss only works with seconds, but accepts fractional seconds. -endpos says at what point to stop playing. This accepts time as hh:mm:ss.ms or in bytes. If you specify only one number, it's seconds. -ao is a special instruction that specifies a plugin, and everything following the plugin are plugin specific options. pcm is the format for wave files. You could use any format you like. mplayer -ao help will tell you which codecs are supported. I haven't figured out all the suboptions for pcm yet.

Evan has posted the first of a series of articles on his blog about using git and github successfully:

Let’s face it, nobody likes a dirty fork. In this series, I’ll show you some of the tricks I’ve learned over the years to successfully maintain a clean fork on GitHub for projects I actively contribute to.

The first article covers creating your clone, adding remotes, keeping things up to date and working in topic branches. It’s definitely worth a read and I’m looking forward to the subsequent articles in the series.

We are pleased to announce availability of an early access version of Percona’s PAM Authentication plugin for MySQL. This plugin supports MySQL-5.5.x, Percona Server 5.5.x and MariaDB 5.2.x. The PAM Authentication plugin can be used for:

• MySQL authentication using operating system users (pam_unix)
• MySQL authentication from LDAP server (pam_ldap)
• authentication against RSA SecurID server
• any other authentication methods that provides access via PAM

We name it early access as it does not yet have the full list of features we want to implement. It is currently functional and we want to make it available for everybody. In this version you still need to create individual users in MySQL (even though they will be authenticated via PAM), in the final (non-early access) version this restriction will be removed. Percona PAM Authentication Plugin for MySQL as always, is fully open source, free of charge and can be used on an unlimited amount of servers. Resources:

• Documentation
• Binaries and source code
• Launchpad project
• Bug tracker
• Discussion group

What if your PHP application could tell the mysqlnd library what service quality you need when using a MySQL replication cluster? If you wanted read-your-writes, the driver would select replication nodes for you which can offer it. If you can allow replication lag but no more than three seconds, the driver would select… One function call and you get the service you need. That’s what version 1.2 of PECL/mysqlnd_ms is about.

The quality of service filter

In the world of PECL/mysqlnd_ms, the free and Open Source replication and load balancing plugin for mysqlnd, a so-called filter is responsible for choosing nodes for statement execution. A filter looks at the SQL statement to be executed and picks a capable server. The current production ready 1.1 release has three filters. Two load balancing filter (random, round-robin) and a user hook filter (user).

For example, if you run a SELECT statement, the filters of PECL/mysqlnd_ms ensure it ends up on a slave.

query(SELECT id FROM test)
|
mysqli	PDO_MySQL	mysql
mysqlnd library
PECL/mysqlnd_ms magic
|
	Master
Slave 1	Slave 2	Slave 3

PECL/mysqlnd_ms 1.2 brings a new quality-of-service filter. The quality of service filter can be configured in the plugins configuration file but also at runtime. The latter is new and unique to this filter.

Reading from slaves no more than n seconds behind

Certain elements on a web site qualify for simple TTL based caching. Others don’t. One could say, that the quality of service demands for cacheable contents are lower than for non-cacheable. The same goes for SQL statements. Stale data may be OK, but it may not be older than n seconds, you may want to set a time-to-live, so to say.

MySQL Replication is asynchronous. It takes some time until a write operation on the master has been replicated to all slaves. Slaves may not always serve current data. The MySQL administrative statement SHOW SLAVE STATUS give a hint how many seconds a slave is behind the master (Seconds_Behind_Master). It is a rather rough estimation that MySQL makes but its the best built-in logic I am aware of. Please, check the MySQL reference manualfor details.

	Master
Slave 1	Slave 2	Slave 3
Seconds_Behind_Master = 0	Seconds_Behind_Master = 3	Seconds_Behind_Master = 5

Say your application is fine with reading from slaves that lag no more than three seconds behind the master. With PECL/mysqlnd_ms 1.2 it will be one line to set the required service quality.

mysqlnd_ms_set_qos($link,
  MYSQLND_MS_QOS_CONSISTENCY_EVENTUAL,
  MYSQLND_MS_QOS_OPTION_AGE,
  3);

After the function call, PECL/mysqlnd_ms checks for every statement nodes quality for executing the statement. If the statement is a write, the quality-of-service (qos) filter, returns a list of all masters. If the statement is a read, the qos filter returns all masters and all slaves for which Seconds_Behind_Master <= 3 is true. In the example, a SELECT statement would be run on the master, slave 1 or slave 2, depending on the choice of the load balancing filter. Slave 3 is not used because its replication lag is too high.

To be very clear here: the qos filter may execute SHOW SLAVE STATUS on all slaves. This is an expensive and slow operation. But it is the only option availablewith todays MySQL replication server features. One may try to cache the SHOW SLAVE STATUS results but there’s no fundamental change until the MySQL replication cluster can tell clients which nodes to use.

Don’t drop and forget the idea after this warning. The cool thing is, that if you tell PECL/mysqlnd_ms it may replace the slave access with a local TTL cache access… allow disabling the slow status query, focus on the caching idea, and … But that’s for sure not for 1.2.

Reading from slaves which have replicated a global transaction ID

Setting the maximum age using the qos filter gives you eventual consistency. Using global transaction ids, the qos filter can also offer session consistency or read-your-wrtites. PECL/mysqlnd_ms 1.2 can do global transaction id injection and the MySQL replication team has published a global transaction id feature preview release in October. Because the server preview is not fully functional, 1.2 focusses on its client-side emulation and injection.

$link->query("INSERT INTO test(id) VALUES (123)"); $gtid = mysqlnd_ms_get_last_gtid($link);
|
	Master
	GTID = 27263
Slave 1	Slave 2	Slave 3
GTID = 27263	GTID = 27251	GTID = 27263

Whoever does maintain the global transaction ids, some SQL exists to check if a node has replicated a certain id or not. The qos filter knows the SQL and checks if a node has replicated the transaction in question. If so, the node can be used to achieve read-your-writes.

mysqlnd_ms_set_qos( $link, MYSQLND_MS_QOS_CONSISTENCY_SESSION, MYSQLND_MS_QOS_OPTION_GTID, 27263); $link->query("SELECT * FROM test ");
|
	Master
	GTID = 27263
Slave 1	Slave 2	Slave 3
GTID = 27263	GTID = 27251	GTID = 27263

Is it worth it?

The price of the qos filter is high. Checking nodes for every statement is slow. However, if you hit scalability limits of your master because of too many read-your-write requests, there is no choice . PECL/mysqlnd_ms, a driver-based solution, does not do worse than an application-based solution. But, it takes work from you, the application developer.

If that’s not convincing, OK, but what about the caching idea…

Happy hacking!

@Ulf_Wendel

PS: Do you speak portuguese? If so, you may want to check out this PHP Conference Brasil presentation from Airton Lastori.

For past few days (days!!!) I’m not able to log into my account at Google (except GMail, thanks for that! – Voice/Blogger/Plus/Docs/… are all broken though). I’m either getting this message:

Or I’m sent to infinite redirect loop. None of their self-help pages are useful or have this error mentioned.

Apparently if you get such message, you should read this message in opposite way:

“Sorry (or you should be sorry), there seems to be a problem. This service is looking for you and you seem to be temporarily unavailable. We’re not working on this at all, it may show up on some report at the end of the year though as lost revenue, sad, we see you spent few hundred dollars on our services, so you trust us, but of course we have enough money from everyone else. You may try again in few minutes, few hours or few days, it won’t help you though. Though we appreciate your patience, you should start researching and debugging this yourself instead, thank you.”

I cleared all Google cookies on my browser and it started working. Apparently this has happened to others too.

 

The QuickEdit plugin for YUI 3 DataTable makes it easy to edit an entire page of records as an atomic operation. However, sometimes you need to do even more. For example, you might have to simultaneously edit more records than you can comfortably fit on a single page. Or you might need to support adding, duplicating, and removing records as part of the atomic operation. Or you might wish to visually group fields by placing them in a single table cell. The Bulk Editor widget supports all these possibilities.

(Click the screenshot to play with this example.)

Overview

The Bulk Editor widget consists of three components:

Data source

This wraps a YUI DataSource and manages the changes: insertions, removals, and changed values.

Base widget

This provides the basic structure for editing by managing records and fields within each record. Derived classes are responsible for rendering each record into a separate row, which could be a div, a tbody, or some other container.

HTML table implementation

This extends the base widget to render each record into a tbody in an HTML table. The column configuration determines which field is displayed in each column of the table. A custom cell formatter can be used to render multiple fields in a single table cell.

Configuration

In the example that generated the above screenshot, the configuration has been kept as simple as possible:

fields defines the editable values in each record. The default type is input. The other valid types are select and textarea. (select requires a list of values.) Basic validation is provided by Form Manager gallery module. This covers required fields, length restrictions, and numeric ranges. More complex validation can be performed by specifying regex or your own function (fn). Here is an excerpt from the live example:

var fields =
{
	title:
	{
		type: 'textarea'
	},
	year:
	{
		validation:
		{
			css: 'yiv-integer:[1500,2100]'
		}
	},
	color:
	{
		type: 'select',
		values:
		[
			{ value: 'red',   text: 'Red'   },
			{ value: 'green', text: 'Green' },
			{ value: 'blue',  text: 'Blue'  }
		]
	}
};

Y.BulkEditDataSource requires an instance of Y.DataSource and the following parameters:

uniqueIdKey

The name of a key which uniquely identifies each record.

generateRequest

A function to generate request parameters for Y.DataSource. (This is empty in the example, because Y.DataSource.Local always returns all the data.)

extractTotalRecords

A function to extract the total number of records from the Y.DataSource response.

Since the example uses Y.DataSource.Local, totalRecordsReturnExpr is also required. This OGNL expression specifies where in the response to store the total number of records. (Notice that extractTotalRecords reads this value.)

var ds = new Y.BulkEditDataSource(
{
	ds:                     raw_ds,
	uniqueIdKey:            'id',
	generateRequest:        function() { },
	totalRecordsReturnExpr: '.meta.totalRecords',
	extractTotalRecords:    function(response)
	{
		return response.meta.totalRecords;
	}
});

Y.HTMLTableBulkEditor requires the data source, the field configuration, and the column configuration. In the column configuration, the key is the field name, unless you specify a custom formatter. The label is used as the column title. Here is an excerpt from the live example:

var columns =
[
	{
		key:       'checkbox',
		label:     '<input type="checkbox" id="select-all" />',
		formatter: function(o)
		{
			var markup = '<input type="checkbox" class="record-select" id="{id}" />';
			o.cell.set('innerHTML', Y.Lang.sub(markup,
			{
				id: this.getRecordId(o.record)
			}));
		}
	},
	{ key: 'title', label: 'Title' },
	{ key: 'year', label: 'Year' },
	{ key: 'color', label: 'Color' }
];

(Note that the live example defines a minor extension to Y.HTMLTableBulkEditor to handle the checkbox column.)

You can also pass an instance of Y.Paginator to Y.BulkEditDataSource. This is demonstrated in a separate, more complicated live example.

Local vs. Remote Data Sources

When deciding whether to use a local or a remote datasource, you must carefully consider the trade-offs. The obvious trade-off is that a local datasource is faster when paginating, but the initial page load will take longer, and it requires more memory on the client.

The Bulk Editor widget imposes additional trade-offs, however.

First, the YUI DataSource must return immutable data. This is automatic for local data sources, but can be tricky to implement for remote data sources. You will need to lock the rows in your database table for the duration of the bulk edit operation if more than one user is allowed to modify them.

Second, the choice between local and remote data source affects how you are allowed to save the data. When you use a local data source, you can do best effort saving, i.e., save all the valid records to the server, remove them from the local datasource, and allow the user to focus on the records that have invalid values. When you use a remote data source, the immutability requirement only allows you to do all or nothing saving, i.e., the data can only be saved after all the data is valid.

Real-world Use Case

The original motivation for the Bulk Editor widget was to allow post-processing of an uploaded spreadsheet. Introducing a post-processing step removes the need for the spreadsheet values to be perfect. Errors can be flagged and fixed in the Bulk Editor widget instead of rejecting the entire upload. In addition, processing on the server can do best-guess assignment of additional values required for each record, and the user can check and fix these extra values before saving. This simplifies the initial creation of the spreadsheet.

In this scenario, a remote data source is the best choice. The uploaded data is stored in a scratch space, and is therefore guaranteed immutable, since no other user can see it. “All or nothing” saving is appropriate: Once all the errors have been fixed, the save operation is atomic, just like a standard upload operation.

About the author: John Lindal (@jafl5272 on Twitter) is one of the lead engineers constructing the foundation on which Yahoo! APT is built. Previously, he worked on the Yahoo! Publisher Network.

Using :target to produce a click maze puzzle. Get to the end without turnng left.

On July 14, 1789, citizens of Paris stormed the Bastille, igniting a revolution that toppled the French monarchy. On July 14 of this year, there was a less dramatic (though more tweeted) takedown: The Deck network, which delivers advertising to some of the most popular web design and culture destinations, was down for about thirty minutes. During this period, most partner sites running ads from The Deck could not be viewed as result.

A few partners were unaffected (aside from not having an ad to display). Fortunately, Dribbble, was one of them. In this article, I’ll discuss outages like this and how to defend against them. But first, a few qualifiers: The Deck has been rock solid – this is the only downtime we’ve witnessed since joining in June. More importantly, the issues in play are applicable to any web widget you might add to your site to display third-party content.

Down and out

Your defense is only as good as its weakest link. Web pages are filled with links, some of which threaten the ability of your page to load quickly and correctly. If you want your site to work when external resources fail, you need to identify the weak links on your site. In this article, we’ll talk about web widgets as a point of failure and defensive JavaScript techniques for handling them.

Widgets 101

Imagine a widget that prints out a Pun of the Day on your site. A simple technique for both widget provider and consumer is for the provider to expose a URL:

http://widgetjonesdiary.com/punoftheday.js

which returns a JavaScript file like this:

document.write("<h2>The Pun of the Day</h2><p>Where do frogs go for beers after work? Hoppy hour!</p>");

The call to document.write() injects the string passed into the document where it is called. So to display the widget on your page, simply add an external script tag where you want it to appear:

<div class="punoftheday">
  <script src="http://widgetjonesdiary.com/punoftheday.js"></script>
  <!-- Content appears here as output of script above -->
</div>

This approach is incredibly easy for both provider and consumer. But there are implications…

document.write()… or wrong?

As in the example above, scripts may perform a document.write() to inject HTML. Page rendering halts while a script is processed so any output can be inlined into the document. Therefore, page rendering speed depends on how fast the script returns the data. If an external JavaScript widget hangs, so does the page content that follows. It was this scenario that briefly stalled partner sites of The Deck last summer.

The elegant solution

To make our web widget more robust, calls to document.write() should be avoided. This can be achieved with a technique called JSONP (AKA JSON with padding). In our example, instead of writing inline with document.write(), a JSONP script passes content to a callback function:

publishPun("<h2>Pun of the Day</h2><p>Where do frogs go for beers after work? Hoppy hour!</p>");

Then, it’s up to the widget consumer to implement a callback function responsible for displaying the content. Here’s a simple example where our callback uses jQuery to write the content into a target <div>:

1. <!-- Where widget content should appear -->
2. <div class="punoftheday"></div>
3.  
4. ...
5.  
6. <!-- Callback function, must be defined before the widget script tag -->
7. <script>
8. function publishPun(content) {
9. $('.punoftheday').html(content); // Writes content display location
10. }
11. </script>
12.  
13. <!-- Link to widget which, in turn, invokes callback -->
14. <script src="http://widgetjonesdiary.com/punoftheday-jsonp.js"></script>

View Example 1

Even if the widget content appears at the top of the page, our script can be included at the bottom so it’s non-blocking: a slow response leaves page rendering unaffected. It simply invokes the callback which, in turn, writes the widget content to its display destination.

The hack

But what to do if your provider doesn’t support JSONP? This was our case with The Deck. Returning to our example, I’m reminded of computer scientist David Wheeler’s statement, “All problems in computer science can be solved by another level of indirection… Except for the problem of too many layers of indirection.”

In our case, the indirection is to move the widget content into position after writing it to the page. This allows us to place the widget <script> tag at the bottom of the page so rendering won’t be blocked, but still display the widget in the target. The strategy:

1. Load widget content into a hidden <div> at the bottom of the page.
2. Move the loaded content from the hidden <div> to its display location.

and the code:

1. <!-- Where widget content should appear -->
2. <div class="punoftheday"></div>
3.  
4. ...
5.  
6. <!-- Hidden div at the bottom of the page, before the closing body tag -->
7. <div class="loading-dock hidden">
8. <!-- Widget content is written here at the bottom of the page, but hidden from view. -->
9. <script src="http://widgetjonesdiary.com/punoftheday.js"></script>
10. </div>
11.  
12. <!-- Move content from hidden loading div to display div -->
13. <script>
14. $('.punoftheday').append($('.loading-dock').children(':gt(0)'));
15. </script>

View Example 2

After the external punoftheday.js script has processed, the rendered HTML will look as follows:

<div class="loading-dock hidden">
  <script src="http://widgetjonesdiary.com/punoftheday.js"></script>
  <h2>Pun of the Day</h2>
  <p>Where do frogs go for beers after work? Hoppy hour!</p>
</div>

The ‘loading-dock’ <div> now includes the widget content, albeit hidden from view (if we’ve styled the ‘hidden’ class with display: none). There’s just one more step: move the content to its display destination. This line of jQuery (from above) does the trick:

$('.punoftheday').append($('.loading-dock').children(':gt(0)'));

This selects all child elements in the ‘loading-doc’ <div> except the first – the widget <script> tag which generated it – and moves it to the display destination. Worth noting is the :gt(0) jQuery selector extension, which allows us to exclude the first (in a 0-based array) child element – the widget <script> tag – from selection.

Since all of this happens at the bottom of the page, just before the </body> tag, no rendering has to wait on the external widget script. The only thing that fails if our widget hangs is… the widget itself. Our weakest link has been strengthened and so has our site. DE-FENSE!

During some interesting discussions on Twitter yesterday I found that there is now more than ever a confusion about JavaScript dependence in web applications and web sites. This is a never ending story but it seems to me to flare up ever time our browsing technology leaps forward.

I encountered this for the first time back in the days of DHTML. We pushed browsers to their limits with our lovely animated menus and 3D logos (something we of course learned not to do again, right?) and we were grumpy when people told us that there are environments out there where JavaScript isn’t available.

Who turns off JavaScript?

The first question we need to ask about this is what these environments are. There are a few options for that:

• Security systems like noscript or corporate proxies that filter out JavaScript
• Feature phones like old Blackberries (I remember switching to Opera Mini on mine to have at least a bearable surfing experience)
• Mobile environments where carriers proxy images and scripts and sometimes break them
• People on traffic-limited or very slow connections
• People who turn off JavaScript for their own reasons
• People sick of modal pop-ups and other aggressive advertising

As you can see some of them are done to our end users (proxying my companies or mobile provider), some are probably temporary (feature phones) and some are simply their own choice. So there is no way to say that only people who want to mess with our cool web stuff are affected.

Why do they turn off JavaScript?

As listed above, there are many reasons. When it comes to deliberately turning off JavaScript, I’d wager to guess that the main three are security concerns, advertising fatigue and slow connectivity.

Security is actually very understandable. Almost every attack on a client machine happens using JavaScript (in most cases in conjunction with plugin vulnerabilities). Java of course is the biggest security hole at the moment but there is a lot of evil you can do with JavaScript via a vulnerable web site and unprotected or outdated browser and OS.

Slow connectivity is a very interesting one. Quite ironic – if you think about it – as most of what we use JavaScript for is to speed up the experience of our end users. One of the first use cases for JS was client side validation of forms to avoid unnecessary server roundtrips.

Now when you are on a very flaky connection (say a free wireless or bad 3G connectivity or at any web development conference) and you try to use for example Google Reader or Gmail you’ll end up with half broken interfaces. If the flakiness gets caught during first load you actually get offered a “HTML only low version” that is very likely to work better.

The best of both worlds

This is totally fine – it tries to give an end user the best experience depending on environment and connectivity. And this is what progressive enhancement is about, really. And there is nothing evangelical about that – it is plain and pure pragmatism.

It seems just not a good plan under any circumstances to give people an interface that doesn’t work. So to avoid this, let’s generate the interface with the technologies that it is dependent on.

With techniques like event delegation this is incredibly simple. You add click handlers to the parent elements and write out your HTML using innerHTML or other, newer and faster techniques.

So why is this such a problem?

Frankly, I really don’t know. Maybe it is because I am old school and like my localhost. Maybe it is because I have been disappointed by browsers and environments over and over again and like to play it safe. I just really don’t get why someone would go for a JS-only solution when the JS is really only needed to provide the enhanced experience on top of something that can work without it.

The mythical edge case application

A big thing that people keep coming up with are the “applications that need JavaScript”. If we are really honest with ourselves, then these are very rare. If pushed, I could only think of something like photoshop in the browser, or any other editor (video, IDE in the browser, synth) that would be dependent on JavaScript. All the others can fall back to a solution that requires a reload and server-side component.

And let’s face it – in the times of Node.js the server side solution can be done in JavaScript, too. Dav Glass of Yahoo 2 years ago showed that if a widget library is written to be independent of its environment, you can re-use the same rich widget client and server side.

The real reasons for the “App that needs JavaScript” seems to be a different, non-technical ones.

The real reasons for “Apps that need JavaScript”

Much like there are reasons for not having JavaScript there are reasons for apps that need JavaScript and deliver broken experiences.

• You only know JS and think people should upgrade their browsers and stop being pussies. This is fine, but doesn’t make you the visionary you think you are as it is actually a limited view. We called that DHTML and it failed once – it can fail again
• You are building an app with a team without server side skills and want to get it out cheaply. This can work, but sounds to me like apps that “add accessibility later”, thus quadrupling the time and money needed to make that happen. Plan for that and all is good.
• You want to get the app out quickly and you know you’ll have to re-write it later. This is actually a pretty common thing, especially when you get highly successful or bought by someone else. Good luck to you, just don’t give people the impression that you are there to stay.
• Your app will run in a pure JS environment. Of course this means there is no need to make it work without JS. One example of this would be Air applications. Just make sure you bet on tech and environments that will stay on the radar of the company selling it.
• Your app really needs JS to work. If that is the case, just don’t offer it to people without it. Explain in a nice fashion the whys and hows (and avoid telling people they need to turn it on as they may not be able to and all you do is frustrate even more) and redirect with JS to your app.

In summary – sort of

All in all, the question of JavaScript dependence reaches much further than just the technical issues. It questions old best practices and has quite an impact on maintainability (I will write about this soon).

Let’s just say that our discussions about it would be much more fruitful if we started asking the “what do we need JS for” question rather than the “why do people have no JS”. There is no point in blaming people to hold back the web when our techniques are very adaptive to different needs.

There is also no point in showing people you can break their stuff by turning things in your browser on and off. That is not a representation of what happens when a normal visitor gets stuck in our apps.

Maybe all of this will be moot when node.js matures and becomes as ubiquitous as the LAMP stack is now. I’d like to see that.

Pat Cavit (@tivac), a frontend engineer at ArenaNet and an active YUI contributor and community member, joined us at YUIConf 2011 to give this talk on automating build-time website optimizations such as file concatenation, minification, renaming, and more using the Ant build tool.

Links

• View in HD on YouTube
• Full YUIConf 2011 Playlist on YouTube

Online account compromise occurs regularly and will probably continue in the foreseeable future. Often hackers employ botnets to hijack accounts from millions of unsuspecting consumers. In turn, these hijacked accounts are sold and used to initiate email scams, with messages seemingly sent from the legitimate account owners to their contacts. Scams can range from the less harmful “check this cool stuff” advertising spam to the more damaging “send money to help your friend” email. Worse still, the scam emails may contain malicious links that lead to the installation of malware on computers, which then makes the computers part of the ever-growing botnet networks.

To thwart account compromise, Yahoo! is introducing a stronger user authentication feature that aims to prevent account hijackers with a stolen password from accessing a person's account. If you have a Yahoo! account, you can now further protect it by activating this new second sign-in verification feature from Yahoo! Account Info. As part of the process, you will be required to add a mobile phone number to your account and verify it via SMS.

Once the feature is turned on, any suspicious account sign-in attempt will be challenged by a second sign-in verification beyond the initial password validation. To confirm the legitimacy of the sign-in attempt, you or the hijacker will have to answer your account security question or enter a verification code that will be sent to your mobile phone. Presumably, only you, as the legitimate user, can sign in. Account hijackers will be blocked since they neither know your security answer nor possess your mobile phone. In short, this second sign-in verification step acts as an additional, stronger “beyond-the-password” challenge against any unauthorized access attempt. And our systems are also capable of other refinements to accomplish our ultimate goal: to block all account hijackers from accessing Yahoo! accounts.

Given the complexity of behind-the-scenes working systems, we are offering this feature first to users residing in the United States, Canada, India, and the Philippines. We will be extending this feature gradually to all worldwide users by March 2012. As we continue to improve how to better prevent account compromise and keep you aware of any suspicious account activity, you can enact stronger protection on your account by turning on the second sign-in verification feature and by monitoring your recent sign-in activity.

Yahoo! is no stranger to hack events. Between our internal Hack Days, Open Hacks, and the HackU events at universities around the world, we have introduced thousands of engineers to the joys of creating cool projects in 24 hours. We are also quite proud of our participation in one of the greatest hack movements: Random Hacks of Kindness (RHoK).

RHoK started after a discussion panel in 2009 about how to use web technology to save lives during a crisis. It's core members include Yahoo, Microsoft, Google, the World Bank, NASA, and HP. RHoK provides support and coordination for global and local hacking events for humanity.

RHoK Global - December 2011

Over 1,000 engineers in 34 cities around the world worked together December 3 and 4 to solve pre-defined problems. These solutions ranged from augmented communication devices for those unable to speak to seed banks for developing countries. Here are just a few of the projects that have been saved as open source solutions.

• Appcessible: This Chrome extension is for people with low vision and/or reading difficulties. It reorganizes a web page to separate the important content from the secondary modules. It also integrates text to speech.
• The Collaboration Project: With help from NASA, this project creates a dashboard for Spaceship Earth and provides a single repository for civic projects.
• Implement Humidity Slide Functionality with Software: Some of the folks at the National Meteorology department of the Gambia need to calculate Dew Point Temperature, Vapor Pressure and Relative Humidity. They had previously been using a piece of equipment called a Humidity Slide Rule to accomplish this task. The Humidity Slide Rules that they have now are old, no longer readable and not easily distributable. This project replaces the outdated tool with a software solution.
• RHoK London was convened by MyBnk and focused on building apps relating to financial illiteracy and financial exclusion among U.K. youth. Resulting hacks included Pound Around the World, an online game teaching kids purchasing power parity (PPP) by educating them on the value of a pound in different parts of the world.
• RHoK Montreal’s first event was won by the aptly-named Bacteria Detecto-Droid Team for an Android app that will allow users to run a photo of bacteria plates through an image recognition software to provide easy and low-cost water monitoring, particularly in developing countries.

Random Hacks of Kindness has a complete summary of hacks from the event:

From Banjul to Bangalore, from Porto Alegre to Pretoria, tech-savvy do-gooders collaborated with subject-matter experts in a wide variety of fields to build software applications for the benefit of their local communities and the world.  Random Hacks of Kindness hackers came from diverse backgrounds: in addition to computer programmers, RHoK events included designers, project managers, PR and marketing professionals, UI/UX specialists, GIS mappers, and many more people volunteering their professional skills for a good cause.

Events around the world started off with anticipation high as participants followed other global locations via Ustream and Twitter and shared videos of RHoK introductions in other locations. Many events began with a Friday night meet-up and problem brainstorming session where experts and hackers began building out concepts to respond to proposed problem definitions.  The experts proposing problems for RHoK Global included, among others, Oxfam, UN OCHA, Doctors Without Borders, the World Bank, InSTEDD, Ushahidi, Fragile Oasis, as well as numerous challenges posed by local, municipal and national government agencies.

RHoK Around the World - Random Hacks of Kindness

Join the next event

The next global RHoK event will be in June 2012. However, RHoK helps groups coordinate events throughout the year based on local or issue specific problems. Visit RHoK's upcoming events page for more information.

We all know that feeling: some time after we launch a site, new designers and developers come in and make adjustments. They add styles that don’t fit with the content, use typefaces that make us cringe, or chuck in bloated code. But if we didn’t leave behind any documentation, we can’t really blame them for messing up our hard work.

To counter this problem, graphic designers are often commissioned to produce style guides as part of a rebranding project. A style guide provides details such as how much white space should surround a logo, which typefaces and colours a brand uses, along with when and where it is appropriate to use them.

Design guidelines

Some design guidelines focus on visual branding and identity. The UK National Health Service (NHS) refer to theirs as “brand guidelines”. They help any designer create something such as a trustworthy leaflet for an NHS doctor’s surgery. Similarly, Transport for London’s “design standards” ensure the correct logos and typefaces are used in communications, and that they comply with the Disability Discrimination Act.

Some guidelines go further, encompassing a whole experience, from the visual branding to the messaging, and the icon sets used. The BBC calls its guidelines a “Global Experience Language” or GEL. It’s essential for maintaining coherence across multiple sites under the same BBC brand.

The BBC’s Global Experience Language.

Design guidelines may be brief and loose to promote creativity, like Mozilla’s “brand toolkit”, or be precise and run to many pages to encourage greater conformity, such as Apple’s “Human Interface Guidelines”.

Whatever name or form they’re given, documenting reusable styles is invaluable when maintaining a brand identity over time, particularly when more than one person (who may not be a designer) is producing material.

Code standards documents

We can make a similar argument for code. For example, in open source projects, where hundreds of developers are submitting code, it makes sense to set some standards. Drupal and Wordpress have written standards that make editing code less confusing for users, and more maintainable for contributors.

Each community has nuances: Drupal requests that developers indent with two spaces, while Wordpress stipulates a tab. Whatever the rules, good code standards documents also explain why they make their recommendations.

The front-end developer’s style guide

Design style guides and code standards documents have been a successful way of ensuring brand and code consistency, but in between the code and the design examples, web-based style guides are emerging. These are maintained by front-end developers, and are more dynamic than visual design guidelines, documenting every component and its code on the site in one place.

Here are a few examples I’ve seen in the wild:

Natalie Downe’s pattern portfolio

Natalie created the pattern portfolio system while working at Clearleft. The phrase describes a single HTML page containing all the site’s components and styles that can act as a deliverable.

Pattern portfolio by Natalie Downe for St Paul’s School, kept up to date when new components are added. The entire page is about four times the length shown.

Each different item within a pattern portfolio is a building block or module. The components are decoupled from the layout, and linearized so they can slot into anywhere on a page.

The pattern portfolio expresses every component and layout structure in the smallest number of documents. It sets out how the markup and CSS should be, and is used to illustrate the project’s shared vocabulary.

Natalie Downe

By developing a system, rather than individual pages, the result is flexible when the client wants to add more pages later on.

Paul Lloyd’s style guide

Paul Lloyd has written an extremely comprehensive style guide for his site. Not only does it feature every plausible element, but it also explains in detail when it’s appropriate to use each one.

Paul’s style guide is also great educational material for people learning to write code.

Oli Studholme’s style guide

Even though Oli’s style guide is specific to his site, he’s written it as though it’s for someone else. It’s exhaustive and gives justifications for all his decisions. In some places, he links to browser bug tickets and makes recommendations for cross-browser compatibility.

Oli has released his style guide under a Creative Commons Attribution Share-alike license, and encourages others to create their own versions.

Jeremy Keith’s pattern primer

Front-end style guides may have comments written in the code, annotations that appear on the page, or they may list components alongside their code, like Jeremy’s pattern primer.

You can watch or fork Jeremy’s pattern primer on Github.

Linearizing components like this resembles a kind of mobile first approach to development, which Jeremy talks about on the 5by5 podcast: The Web Ahead 3.

The benefits of maintaining a front-end style guide

If you still need convincing that producing documentation like this for every project is worth the effort, here are a few nice side-effects to working this way:

Easier to test

A unified style guide makes it easier to spot where your design breaks. It’s simple to check how components adapt to different screen widths, test for browser bugs and develop print style sheets when everything is on the same page. When I worked with Natalie, she’d resize the browser window and bump the text size up and down during development to see if anything would break.

Better workflow

The approach also forces you to think how something works in relation to the whole site, rather than just a specific page, making it easier to add more pages later on. Starting development by creating a style guide makes a lot more sense than developing on a page-by-page basis.

Shared vocabulary

Natalie’s pattern portfolio in particular creates a shared vocabulary of names for components (teaser, global navigation, carousel…), so a team can refer to different regions of the site and have a shared understanding of its meaning.

Useful reference

A combined style guide also helps designers and writers to see the elements that will be incorporated in the site and, therefore, which need to be designed or populated. A boilerplate list of components for every project can act as a reminder of things that may get missed in the design, such as button states or error messages.

Creating your front-end style guide

As you’ve seen, there are plenty of variations on the web style guide. Which method is best depends on your project and workflow. Let’s say you want to show your content team how blockquotes and asides look, when it’s appropriate to use them, and how to create them within the CMS. In this case, a combination of Jeremy’s pattern primer and Paul’s descriptive style guide – with the styled component alongside a code snippet and a description of when to use it – may be ideal.

Start work on your style guide as soon as you can, preferably during the design stage:

Simply presenting flat image comps is by no means enough - it’s only the start… As layouts become more adaptable, flexible and context-specific, so individual components will become the focus of our design. It is therefore essential to get the foundational aspects of our designs right, and style guides allow us to do that.

Paul Lloyd on Style guides for the Web

1. Print out the designs and label the unique elements and components you’ll need to add to your style guide. Make a note of the purpose of each component. While you’re doing this, identify the main colours used for things like links, headings and buttons.
   

   I draw over the print-outs on to tracing paper so I can make more annotations. Here, I’ve started annotating the widths from the designer’s mockup so I can translate these into percentages.

2. Start developing your style guide with base styles that target core elements: headings, links, tables, blockquotes, ordered lists, unordered lists and forms. For these elements, you could maintain a standard document to reuse for every project.
3. Next, add the components that override the base styles, like search boxes, breadcrumbs, feedback messages and blog comments. Include interaction styles, such as hover, focus and visited state on links, and hover, focus and active states on buttons.
4. Now start adding layout and begin slotting the components into place. You may want to present each layout as a separate document, or you could have them all on the same page stacked beneath one another.

Document code practices

Code can look messy when people use different conventions, so note down a standard approach alongside your style guide. For example, Paul Stanton has documented how he writes CSS.

The gift wrapping

Presenting this documentation to your client may be a little overwhelming so, to be really helpful, create a simple page that links together all your files and explains what each of them do.

This is an example of a contents page that Clearleft produce for their clients. They’ve added date stamps, subversion revision numbers and written notes for each file.

Encourage participation

There’s always a risk that the person you’re writing the style guide for will ignore it completely, so make your documentation as user-friendly as possible. Justify why you do things a certain way to make it more approachable and encourage similar behaviour.

As always, good communication helps. Working with the designer to put together this document will improve the site. It’s often not practical for designers to provide a style for everything, so drafting a web style guide and asking for feedback gives designers a chance to make sure any default styles fit in.

If you work in a team with other developers, documenting your code and development decisions will not only be useful as a deliverable, but will also force you to think about why you do things a certain way.

Future-friendly

The roles of designer and developer are increasingly blurred, yet all too often we work in isolation. Working side-by-side with designers on web style guides can vastly improve the quality of our work, and the collaborative approach can spark discussions like “how would this work on a smaller screen?”

Sometimes we can be so focused on getting the site ready and live, that we lose sight of what happens after it’s launched, and how it’s going to be maintained. A simple web style guide can make all the difference.

If you make your own style guide, I’d love to add it to my stash of examples so please share a link to it in the comments.

I am sure I have raised this before but we have recently had cause to discuss this with RIPE NCC, and their answers are rather telling.

Basically PI space is IP address space (in this case IPv4) which has been assigned by RIPE to end organisations or individuals for their use. It is normally assigned in order to let people multi-home their network, but the rules allow for it to be used in contexts that are not even connected to the Internet as such.

Traditionally this has been assigned to end users without imposing any contract terms on those end users, and understandably, especially with IPv4 running out, RIPE decided that there should be a clear contract in place for the PI assignments - either directly with RIPE or with an LIR that have a contract with RIPE.

So far, so good.

The bit i think that is wrong in my view is that they are trying to retrospectively apply this requirement to existing assignees. This is where you get a bit of a catch 22...

Because there is no existing contract, you have no legal basis to force the existing assignees to do anything (such as enter a new contract). It is because of this lack of control that RIPE want to have contracts in place. But if they had the control needed to force such contracts they would have a contract in place already.

I think that is the problem, and what is rather telling is that in discussions with RIPE NCC regarding one of our customers long established PI assignment, RIPE NCC have not been able to answer simple questions such as "what legal basis is there compelling our customer to enter in to a new contract". They just keep spouting that it is RIPE policy (now). They can't even tell me what RIPE policy "at the time the assignment was made" would allow the new policy to be imposed on existing assignees. They are just waffling and waffling. I half expected them to come up with some clear legal basis, but they have not. If there is one, someone tell me (and tell RIPE NCC as they seem not to know).

So, if there is no legal basis for compelling the end user to sign a new contract, what is there? You are left with the simple fact that RIPE could simply remove the route record from their database. That would mean the assignment could not be used on the Internet. So RIPE can cut off the Internet for this customer.

Sorry, but that sounds like a typical scam or protection racket to me. RIPE provide lots of records which are key to the operation of the Internet and in many cases do not charge for them. They do charge for membership (ISPs typically). Many people and organisations provide key parts of the Internet infrastructure with not contract in place with the people that use those services. It is not like we contract with someone to provide root name servers, for example.

So surely, threatening to delete the route record is much the same as blackmail? After all, the assignment has no expiry date - it is an assignment and has been made - that is now a simple historical fact, and is recorded in the database as a record of fact. Is that not the case?


Maybe that is, in this context, completely legal, and blackmail is indeed the legal basis by which RIPE can force existing assignees to enter a new contract. Though it sounds like entering in to a contract under duress to me (i.e. not enforceable).

From the customers point of view, they dealt with us, and we arranged for them to be assigned PI space, job done, contract with us over, all obligations completed... Then some third party (RIPE NCC) is asking them for ongoing money for something they already own and to enter in to a new contract.

This also raises other interesting issues as to why a Dutch company is controlling a key part of a UK business and can just dictate terms and force them to do things? Scarily, I start to see where the ITU are coming from on this now - and that is a tad worrying.

It does make you wonder how much of the Internet has no clear contractual conditions or enforcement. I mean, what is to stop RIPE saying PI shall be £1M/year suddenly? Or what is to stop Whoever runs the root servers deciding they want contracts with all ISPs and charging lots of money? Can't happen? I would have thought it can't happen until this whole PI charging thing came up, and now I wonder...

The Mozilla Developer Network holiday calendar, each day is a useful resource to enjoy.

WebGL Aquarium.

The lifecycle of a web page on StumbleUpon.

HTML5 essentials and good practices.

What is Web Accessibility?

30 days to learn HTML & CSS.

A (very) comprehensive article on efficient JavaScript coding.

Validate the HTML markup of your whole website with just one click.

Colrd, a nice color palette extractor and generator.

A collection of nice JavaScript animations.

Top 20 Google logos of 2011.

When I was asked to write an article for 24 ways I jumped at the chance, as I’d been wanting to write about some fun hacks for responsive images and related parsing behaviours. My heart sank a little when Matt Wilcox beat me to the subject, but it floated back up when I realized I disagreed with his method and still had something to write about.

So, Matt Wilcox, if that is your real name (and I’m pretty sure it is), I disagree. I see your dirty server-based hack and raise you an even dirtier client-side hack. Evil laugh, etc., etc.

You guys can stomach yet another article about responsive design, right? Right?

Half the room gets up to leave

Whoa, whoa… OK, I’ll cut to the chase…

TL;DR

In a previous episode, we were introduced to Debbie and her responsive cat poetry page. Well, now she’s added some reviews of cat videos and some images of cats. Check out her new page and have a play around with the browser window. At smaller widths, the images change and the design responds. The benefits of this method are:

• it’s entirely client-side
• images are still shown to users without JavaScript
• your media queries stay in your CSS file
• no repetition of image URLs
• no extra downloads per image
• it’s fast enough to work on resize
• it’s pure filth

What’s wrong with the server-side solution?

Responsive design is a client-side issue; involving the server creates a boatload of problems.

• It sets a cookie at the top of the page which is read in subsequent requests. However, the cookie is not guaranteed to be set in time for requests on the same page, so the server may see an old value or no value at all.
• Serving images via server scripts is much slower than plain old static hosting.
• The URL can only cache with vary: cookie, so the cache breaks when the cookie changes, even if the change is unrelated. Also, far-future caching is out for devices that can change width.
• It depends on detecting screen width, which is rather messy on mobile devices.
• Responding to things other than screen width (such as DPI) means packing more information into the cookie, and a more complicated script at the top of each page.

So, why isn’t this straightforward on the client?

Client-side solutions to the problem involve JavaScript testing user agent properties (such as screen width), looping through some images and setting their URLs accordingly. However, by the time JavaScript has sprung into action, the original image source has already started downloading. If you change the source of an image via JavaScript, you’re setting off yet another request.

Images are downloaded as soon as their DOM node is created. They don’t need to be visible, they don’t need to be in the document.

new Image().src = url

The above will start an HTTP request for url. This is a handy trick for quick requests and preloading, but also shows the browser’s eagerness to download images.

Here’s an example of that in action. Check out the network tab in Web Inspector (other non-WebKit development aids are available) to see the image requests.

Because of this, some client-side solutions look like this:

<img src="t.gif" data-src="real-image.jpg" data-bigger-src="real-bigger-image.jpg">

where t.gif is a 1×1px tiny transparent GIF.

This results in no images if JavaScript isn’t available. Dealing with the absence of JavaScript is still important, even on mobile. I was recently asked to make a website work on an old Blackberry 9000. I was able to get most of the way there by preventing that OS parsing any JavaScript, and that was only possible because the site didn’t depend on it.

We need to delay loading images for JavaScript users, but ensure they load for users without JavaScript. How can we conditionally parse markup depending on JavaScript support?

Oh yeah! <noscript>!

<noscript>
    <img src="image.jpg">
</noscript>

Whoa! First spacer GIFs and now <noscript>? This really is the future! The image above will only load for users without JavaScript support. Now all we need to do is send JavaScript in there to get the textContent of the <noscript> element, then we can alter the image source before handing it to the DOM for parsing.

Here’s an example of that working … unless you’re using Internet Explorer.

Internet Explorer doesn’t retain the content of <noscript> elements. As soon as it’s parsed, it considers it an empty element. FANKS INTERNET EXPLORER. This is why some solutions do this:

<noscript data-src="image.jpg">
    <img src="image.jpg">
</noscript>

so JavaScript can still get at the URL via the data-src attribute. However, repeating stuff isn’t great. Surely we can do better than that.

A dirty, dirty hack

Thankfully, I managed to come up with a solution, and by me, I mean someone cleverer than me. Pornel’s solution uses <noscript>, but surely we don’t need that.

Now, before we look at this, I can’t stress how dirty it is. It’s so dirty that if you’ve seen it, schools will refuse to employ you.

<script>document.write('<' + '!--')</script>
<img src="image.jpg">
<!---->

Phwoar! Dirty, isn’t it? I’ll stop for a moment, so you can go have a wash.

Done? Excellent.

With this, the image is wrapped in a comment only for users with JavaScript. Without JavaScript, we get the image. Unlike the <noscript> example above, we can get the text content of the comment pretty easily.

Hurrah! But wait… Some browsers are sometimes downloading the image, even with JavaScript enabled. Notably Firefox. Huh?

Images are downloaded in comments now? What?

No. What we’re seeing here is the effect of speculative parsing. Here’s what’s happening:

While the browser is parsing the script, it parses the rest of the document. This is usually a good thing, as it can download subsequent images and scripts without waiting for the script to complete. The problem here is we create an unbalanced tree.

An unbalanced tree, yesterday.

In this case, the browser must throw away its speculative parsing and reparse from the end of the <script> element, taking our document.write into consideration. Unfortunately, by this stage it may have already discovered the image and sent an HTTP request for it.

A dirty, dirty hack… that works

Pornel was right: we still need the <noscript> element to cater for browsers with speculative parsing.

<script>document.write('<' + '!--')</script><noscript>
    <img src="image.jpg">
</noscript -->

And there we have it. We can now prevent images loading for users with JavaScript, but we can still get at the markup.

We’re still creating an unbalanced tree and there’s a performance impact in that. However, the parser won’t have got far by the time our script executes, so the impact is small. Unbalanced trees are more of a concern for external scripts; a lot of parsing can happen by the time the script has downloaded and parsed.

Using dirtiness to create responsive images

Now all we need to do is give each of our dirty scripts a class name, then JavaScript can pick them up, grab the markup from the comment and decide what to do with the images.

This technique isn’t exclusively useful for responsive images. It could also be used to delay images loading until they’ve scrolled into view. But to do that you’ll need a bulletproof way of detecting when elements are in view. This involves getting the height of the viewport, which is extremely unreliable on mobile devices.

Here’s a hastily thrown together example showing how it can be used for responsive images.

I adjust the end of the image URLs conditionally depending on the result of media queries. This is done on page load, and on resize.

I’m using regular expressions to alter the URLs. Using regex to deal with HTML is usually a sign of insanity, but parsing it with the browser’s DOM parser would trigger the download of images before we change the URLs. My implementation currently requires double-quoted image URLs, because I’m lazy. Wanna fight about it?

Media querying via JavaScript

Jeremy Keith used document.documentElement.clientWidth in his example, which is great as a proof of concept, but unfortunately is rather unreliable across mobile devices.

Thankfully, standards are coming to the rescue with window.matchMedia, which lets us provide a media query string and get a boolean result. There’s even a great polyfill for browsers that don’t support it (as long as they support media queries in CSS).

I didn’t go with that for three reasons:

1. I’d like to keep media queries in the CSS file, if possible.
2. I wanted something a little lighter to keep things speedy while resizing.
3. It’s just not dirty enough yet.

To make things ultra-dirty, I add a test element to the page with a specific class, let’s say media-test. Then, I control the width of it using media queries in my CSS file:

@media all and (min-width: 640px) {
    .media-test {
        width: 1px;
    }
}
@media all and (min-width: 926px) {
    .media-test {
        width: 2px;
    }
}

The JavaScript part changes the URL suffix depending on the width of media-test. I’m using a min-width media query, but you can use others such as pixel-ratio to detect high DPI displays. Basically, it’s a hacky way for CSS to set a value that can be picked up by JavaScript. It means the bit that signals changes to the images sits with the rest of the responsive code, without duplication.

Also, phwoar, dirty!

The API

I threw a script together to demonstrate the technique. I’m not particularly attached to it, I’m not even sure I like it, but here’s the API:

responsiveGallery({
    // Class name of dirty script element(s) to target
    scriptClass: 'dirty-gallery-script',
    // Class name for our test element
    testClass: 'dirty-gallery-test',
    // The initial suffix of URLs, the bit that changes.
    initialSuffix: '-mobile.jpg',
    // A map of suffixes, for each width of 'dirty-gallery-test'
    suffixes: {
        '1': '-desktop.jpg',
        '2': '-large-desktop.jpg',
        '3': '-mobile-retina.jpg'
    }
});

The API can cover individual images or multiple galleries at once. In the example I gave at the start of the article I make two calls to the API, one for both galleries, and one for the single image above the video reviews. They’re separate calls because they respond slightly differently.

The future

Hopefully, we’ll get a proper solution to this soon. My favourite suggestion is the <picture> element that Bruce Lawson covers.

<picture alt="Angry pirate">
    <source src="hires.png" media="min-width:800px">
    <source src="midres.png" media="min-width:480px">
    <source src="lores.png">
    <!-- fallback for browsers without support -->
    <img src="midres.png" alt="Angry pirate">
</picture>

Unfortunately, we’re nowhere near that yet, and I’d still rather have my media queries stay in CSS. Perhaps the source elements could be skipped if they’re display:none; then they could have class names and be controlled via CSS. Sigh.

Well, I’m tired of writing now and I’m sure you’re tired of reading. I realize what I’ve presented is a yet another dirty hack to the responsive image problem (perhaps the dirtiest?) and may be completely unfeasible in professional situations. But isn’t that the true spirit of Christmas?

No.

If you are close to Washington DC, you should plan to attend our Percona Live event on January 11th at the DC Convention Center. We have three tracks of expert speakers on core MySQL-related topics.

This event will not be as large as some of the other events we’ve run, with a capacity of about 200. Frankly, we are targeting this to more local attendees. We know that a lot of people simply can’t travel out to Santa Clara for the main MySQL conference every year, and we want to make sure there is great technical content for them. This is part of our idea to host smaller regional events periodically to try to make sure top-quality MySQL events are more accessible to all, not just the privileged.

We have the entire hallway, with a number of large rooms, immediately off the Metro stop. This event couldn’t be easier to attend if you’re inside the Beltway. We are right outside the Metro doors. If you’re from outside the area, we suggest taking the Metro in for the day. Detailed travel instructions and hotel suggestions (if you need them) are listed on the event venue page. (I’ve personally stayed in the hotels listed there and found them to be great.)

This is a great opportunity to learn from experts, recruit, and network in Washington DC. I personally will be there because I’m close and this is something of a pet project for me. I hope to see you there, and would appreciate it if you tell your friends about the event.

Boss.yahoo.com launches with new offerings for site owners, developers and publishers; Yahoo! unveils new updates for the BOSS API

Today, Yahoo! announced a new home for Search BOSS, boss.yahoo.com, and three new BOSS offerings: BOSS Hosted Search, BOSS Site Search and BOSS Shortcuts. The new offerings address user engagement, traffic recirculation, monetization, and high development and maintenance costs.

Whether you’re a site owner looking for a robust customizable search solution, a developer that needs the utmost flexibility for search as a service, or a publisher that needs quality solutions that deliver higher user engagement, boss.yahoo.com has it all.

Check out the video below, as well as the Yahoo! Search Blog post for more information.

The argument about whether or not it’s okay to extend JavaScript natives tends to focus too much on whether you should or shouldn’t, instead of on when you should and shouldn’t, which would be a much more useful discussion.

The ability to extend natives is a powerful feature of the language, and it’s one that can be used to do fantastic things. When used in the wrong places or for the wrong reasons, it can seriously fuck shit up and make developers cry.

Generally speaking, there are two kinds of JavaScript code that you run on any website you build: there’s the code you (or your team) write yourself, and there’s the code that comes from libraries someone else wrote.

Extending natives in your own code is a bit like decorating your living room. It makes perfect sense. You live there, you have opinions about how it should look and where things should go, and you’re the one who either benefits or suffers as a result of your choices, so you should feel free to go nuts.

Extending natives in library code that will be used on someone else’s pages is a bit like being invited into someone else’s house and — without asking permission — repainting the walls, moving furniture around, adding new furniture, breaking expensive vases, and just generally being an asshole.

You could argue that the person who invited you into their home knew you were an asshole before they invited you, so they knew what they were getting themselves into, but that would just make you a victim-blaming asshole, which is even worse than a normal asshole. So don’t argue that.

Sure, some libraries are so well-known for extending natives that anyone who uses them can be said to be opting into those risks. But later, when the developer adds another library to the page to help them build some new functionality and the second library expects natives to behave like real natives when they actually behave however the first library thinks they should behave, it’s the developer who suffers.

And that developer will inevitably file a bug against the second library, because things broke when they started using it. And then that library’s authors suffer.

And then the authors of the second library end up writing code like this to protect users from brokenness caused by the first library, and the fiery heat death of the universe draws just a little bit closer.

All because some asshole thought repainting someone else’s living room was a good idea.

I've been catching up on my technical reading, and this weekend was spent on Responsive Enhancement¹. I'd read about it before on Jeremy Keith's blog and his comments on proportion perfection over pixel perfection² made me think. Finally, Kayla's report³ on Smashing Magazine about responsive web design coming up as I was thinking about making bluesmoon.info more mobile friendly is what prompted me to study it in detail.

I'm not going to go into the details of responsive enhancement, the references at the end of this article serve that purpose. This article lists what I think are best practices and my reasons for them.

@media queries

As a web designer or developer, you want your page to be easily viewable across different devices and screen sizes. It shouldn't matter whether your user uses a 21" desktop monitor, a 13" laptop, a 10" iPad or a much smaller smartphone. Responsive web design uses @media queries to change the layout of the page using CSS based on browser width. You might have CSS that looks like this:

/* Default wide-screen styles */

@media all and (max-width: 1024px) {
    /* styles for narrow desktop browsers and iPad landscape */
}

@media all and (max-width: 768px) {
    /* styles for narrower desktop browsers and iPad portrait */
}

@media all and (max-width: 480px) {
    /* styles for iPhone/Android landscape (and really narrow browser windows) */
}

@media all and (max-width: 320px) {
    /* styles for iPhone/Android portrait */
}

@media all and (max-width: 240px) {
    /* styles for smaller devices */
}

And yes, you could go smaller than that, or have intermediate sizes, but I'll cover that later.

viewports

Now this works reasonably well when you resize desktop browsers⁴, but not so much for mobile browsers. The problem is that mobile browsers (iPhone/Safari, Android/Chrome and Fennec) assume that the page were designed for a wide screen, and shrink it to fit into the smaller screen. This means that even though users could have had a good customised experience for their smaller devices, they won't because the device doesn't know about this⁵. The trick is to use Apple's viewport^{6, 7, 8} meta tag in your document's head in conjunction with @media queries⁹:

<meta name="viewport" content="...">

I've left the content attribute empty for now because this is where I see confusion... which is what we'll talk about now.

width=device-width

Most sites that I've seen advise you to set the content attribute to width=device-width. This tells the browser to assume that the page is as wide as the device. Unfortunately, this is only true when your device is in the portrait orientation. When you rotate to landscape, the device-width remains the same (eg: 320px), which means that even if your page were designed to work well in a 480px landscape design, it would still be rendered as if it were 320px.

It's tempting to use the orientation media query to solve this problem, but orientation doesn't really tell you the actual width of the device. All it tells you is whether the width is larger than or smaller than the device's height. As ppk points out⁵, since most pages tend to scroll vertically, this is irrelevant.

Use this if you use the same page styles in portrait and landscape orientation. Also note that using width=device-width is the only way to tell android devices to use the device's width¹².

initial-scale=1.0,maximum-scale=1.0

Setting initial-scale=1 tells the browser not to zoom in or out regardless of what it thinks the page width is. This is good when you've designed your page to fit different widths since the browser will use the appropriate CSS rules for its own width, and initial-scale stops the zooming problem that we faced without the viewport meta tag.

Unfortunately a bug, or more likely a mis-feature, in mobile safari messes this up when a device is rotated from portrait to landscape mode. initial-scale is honoured only on full page load. On rotate from portrait to landscape mode, the browser assumes that the page width stays the same and scales accordingly (1.5) to make 320 pixels fit into 480pixels. However, as far as @media queries go, it reports a 480px width, and uses the appropriate CSS rules to render the page. This results in a page designed for 480px rendered scaled up 1.5 times. It's not horrible, but it's not desirable. Fennec claims⁸ that it does the right thing in this case. The Android emulator is impossible to work with and I haven't tested on mobile Opera yet.

To get around this bug, the pixel perfection camp suggests also setting maximum-scale=1. This stops the page zoom in on rotate, but it has the undesired side effect of preventing the user from zooming the page. This is a problem from the accessibility point of view. Zooming in is a very valid use case for users with bad eyesight, and in some cases, even users with good eyesight who just want a closer look at some part of your page. Do this only if you hate your users. It goes without saying that setting user-scalable=no should also not be used on most general purpose pages.

A better solution may be design your page to use the same styles in portrait and landscape orientation and set width=device-width. This way even if it does zoom, it will still be proportionate. See Lanyrd¹⁰ for an example of this design.

width=<actual width>

Some sites advise using a specific viewport width and designing your pages for that width. This is fine if you're building a separate page for each device class, but that doesn't flow with the concept of responsive design. Fixed width layouts are for print. The web is fluid and adapts to its users. Your site should too. Don't use this.

@media all and (device-width:480)

While this is a media query rather than an option to the viewport meta tag, I've seen it at various locations, and don't think it's the best option around. Here's why. According to the CSS3 media queries spec¹¹, the device-width media feature describes the width of the rendering surface of the output device. For continuous media, this is the width of the screen. For paged media, this is the width of the page sheet size.

We're dealing with continuous media here (on-screen as opposed to printed), in which case the spec states that this is the width of the screen. Unless the browser window is maximised, this might be larger than the viewport with. My tests show that most desktop browsers treat device-width and width as synonyms. Mobile browsers seem a little confused on the matter. As far as the viewport meta tag goes, device-width is the width of the device in portrait orientation only. For a 320x480 device, device-width is always 320px regardless of orientation. For CSS media queries, however, device-width is the width of the screen based on its current orientation.

If you are going to use this, use it in conjunction with the orientation media feature. Never use max-device-width and min-device-width. It's better to use max-width and min-width instead. Also remember that device widths may change with newer models. You want your design to be future proof.

Intermediate widths

I'd mentioned above that you could design for any number of widths. The important thing is to test your page for different browser widths. This is fairly easy to do just by resizing your browser window. Test, and whenever you find your page layout break, either fix the layout for all widths, or build a new layout for smaller widths.

On bluesmoon.info, I change many parts of the page depending on page width. The default design (at the time of this article) has 5% empty space around the content. This is fine really wide screens (1152px or more), but as you get smaller, the empty space becomes a waste. Below 1152px, I shrink this to 2% and below 1024px, I get rid of it completely. You could say that my page content was really built for 1024px. This design also works for the iPad in landscape mode.

Below 1000px, all 3 column pages switch to a 2 column layout. Below 580px, I move the right column on all pages below the main content. All pages that initially had 3 columns now have 2 columns below the main content.

As we get smaller, I reduce the space used by non-essential content like the footer, the sidebars and the menu at the top, leaving as much space as possible for main content. Finally, when we get below 380px, the whole page turns into a single column layout.

This is of course, just an example. Your own site may have a layout that works perfectly at all screen widths, or you may need to design only two or three layouts. It's easy to test and design, so there's no reason not to. Designing for multiple widths took me just a couple of hours, and a lot of it was spent reading the articles below.

Recommendations

So finally, this is what I recommend.

1. DO use the viewport meta tag
2. DO use media queries to render your page appropriately for various widths ranging from under 200px to 1024px or more
3. DO use width=device-width,initial-scale=1 in your viewport meta tag OR use width=device-width alone¹².
4. DO NOT use maximum-scale=1 or user-scalable=no
5. DO NOT use width=<specific width>
6. DO NOT use @media all and (*-device-width: xxx)

Remember that using initial-scale=1.0 throws you open to a zooming bug in mobile Safari. Push Safari to fix this bug. Finally, David Calhoun has a great summary¹³ of all options to the viewport meta tag, and alternate meta tags for older phones. Well worth a read. Also note that Mozilla's documentation⁸ of the viewport meta tag is far better than Safari's⁷.

Footnotes & References

1. Ethan Marcotte. 2010. Responsive Web Design. In A List Apart #306. ISSN: 1534-0295.
2. Jeremy Keith. 2010. Responsive Enhancement. In adactio.
3. Kayla Knight. 2011. Responsive Web Design: What It Is and How To Use It. In Smashing Magazine.
4. Webkit based desktop browsers re-render the page correctly as you resize the browser, however they have a minimum width of 385px (on MacOSX) and I was unable to shrink the browser below this. Firefox 4 re-renders the page correctly until the width gets too narrow to fit the navigation toolbar. At that point the viewport width stays fixed even if you shrink the browser. The page is re-rendered if you type something (anything) into the URL bar. Opera 10/11 re-render correctly at all sizes.
5. Peter Paul Koch. 2010. A tale of two viewports — part two. In Quirksmode.
6. Using the Viewport on Safari. In Safari Web Content Guide.
7. The viewport meta tag. In Safari HTML Reference.
8. MDC. 2010. Using the viewport meta tag to control layout on mobile browsers. In Mozilla Developer Network.
9. Peter Paul Koch. 2010. Combining meta viewport and media queries. In Quirksmode.
10. Willison & Downe. Lanyrd.
11. Lie et al. 2010. Media Queries. W3C Candidate Recommendation 27 July 2010.
12. If you design your page for the narrow view and expect it to scale when rotated, then use width=device-width and nothing else. If, instead, you design your page for either width, then use width=device-width,initial-scale=1. This is the only way to get the android browser to render a page with the intended width. Mobile Safari will render the page exactly as if initial-scale=1 were specified alone. You will still end up with the zoom on rotate bug.
13. David Calhoun. 2010. The viewport metatag (Mobile web part I).

One of the more common questions I get asked is which Linux distribution I would use for a MySQL database server. Bearing the responsibility for someone else’s success means I should advise something that is stable, reliable, easy to manage and has plenty of resources available online. It should also allow running MySQL without too much hassle. Unless there are individual circumstances, it actually makes the decision quite easy.

There are probably only a few distributions, which can be considered: CentOS, Debian, RedHat Enterprise Linux, SuSE Linux and Ubuntu. Of course CentOS and Ubuntu derive from RedHat and Debian respectively, but their install bases are large enough to mention them separately. Running MySQL won’t be much different whether one or another distribution is used. All use common Linux kernel – the heart of Linux operating system – which in principle will behave the same way in all cases. The kernel versions may be different in different distributions as for example RedHat is very conservative in that area for the sake of compatibility with drivers and applications throughout a release lifetime, which can be even several years. Newer kernel versions may carry new features or slight performance improvements, however these days it is rarely important to MySQL users. If it is to you, then you probably did your own research and benchmarks already and this post is not for you.

For most people, managing a database server during its lifetime comes down to this rather boring process – install, configure, tune, start/stop MYSQL, upgrade MySQL, 10x start/stop MySQL, upgrade MySQL, downgrade MySQL, upgrade MySQL, install a security fix for something, 20x start/stop MySQL, upgrade MySQL, … . Restarting and changing MySQL version might just be the most critical operations you ever do once server is moved into production. Success and efficiency of these operations may directly impact service availability, which means that what you could wish for the most are quick and problem free restarts and upgrades (or downgrades).

From my experience, and I have done thousands upgrades and downgrades in my life, the least number of problems come from RPM packages available in RedHat, CentOS and SuSE. In fact, I cannot recall encountering any serious problem with that package management system. Moreover, I have not seen broken systems, where installing or updating a RPM package would be impossible without resolving tons of problems first. It can obviously mean that RPM has flaws and does not verify consistency very carefully, but it never turned out to be any problem.

On the other end there are Debian and Ubuntu. Both use tool called dpkg for package management. There isn’t a month that I log in to a system based on either distribution where there are no issues with packages consistency. Unfinished installations, unresolved conflicts are so common that it’s just beyond simple negligence. The packaging system is just not robust enough. Another problem is that one broken package may block you from installing or uninstalling anything else. Imagine that someone left system in such shape, you prepared for downtime, stopped MySQL and… error – text editor has not been properly installed, so you cannot upgrade MySQL either until the problem is fixed. In a stressful situation when downtime clock ticks – annoying at best. Resolving problems can easily lead to unexpected consequences. Here’s a different scenario:

# apt-get install binutils
E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem.


While everyone should stop and check what this means instead of following the advice blindly, many still would just do as they were instructed. In this particular case the problems were with MySQL packages upgraded from 5.1 to 5.5. The former were not fully uninstalled, the latter were in half-configured state. Obviously binutils has nothing to do with MySQL, but it would not install anyway. The larger problem – chances are that fixing this with ‘dpkg –configure -a’ would cause MySQL to shut down. The MySQL package scripts for Debian force MySQL stop/start upon completion and unlike RPM, dpkg does not have any opt-out option.

More problems can come from the standard Debian init (startup) script for MySQL. By default it not only starts database, but also updates system tables (if needed), scans all tables for consistency problems, etc. I mean, that works great for a computer at home or a tiny and unimportant system, but any of these operations may have severe consequences on a large production system. This is why all of this extra functionality was stripped from Percona Server. This behavior actually appears to be a general problem with Debian – it wants to be smarter than you. This might work for desktops, but anywhere else it is plain stupid and makes you waste time on outsmarting a wise-ass system. Of course I do know some people who like such approach, but he is Belgian and they were unable to form a government for 541 days… (congrats that you finally made it this week, btw! )

The init scripts for RedHat, CentOS and SuSE are simple and do only what’s required of them – stop or start MySQL. No problems there.

It’s now clear that I never recommend Debian or Ubuntu, because I do not like some of the “mechanics” and I feel people are generally safer if they do not use any of these two. With the choice left between RedHat/CentOS and SuSE, I lean towards the former. Why? RedHat and CentOS are the only platforms getting packages from all MySQL and MySQL-fork vendors – Oracle, MariaDB and Percona. By my observations RedHat and CentOS are also much more frequently used with MySQL, so there will be more resources available online.

But if you are a skilled systems administrator or your company hires one, then you could use pretty much anything, including my favorites such as Slackware and Gentoo.

In the beginning I mentioned that individual circumstances may influence the decision. One example of such specific case, which limits your options, is for example a requirement for commercial support for the operating system. You will have to choose between RedHat or SuSE. Another such case can be related to hardware. Any newly released component may simply lack support from the Linux kernel, but this can also be the case for any high-end/enterprise class equipment. Hardware vendors may release Linux drivers on their own, but often only for very specific Linux distributions.

Back in November, the SQL Server team witnessed an outpouring of support for the#SQLFamily meme started by Thomas LaRock (aka@SQLRockstar) and Karen Lopez (@datachick). For an installment of SQL Rockstar’s “Meme Monday,” they asked the SQL community to share the meaning behind #SQLFamily. The responses were heartwarming and painted a picture of a passionate community who help and support each other both professionally and personally, as well as share a laugh and pint (or two) when they get together.

The SQL Server team would like to do a little something to help recognize all of you who make the #SQLFamily amazing.

Keeping in line with the theme of giving back, we’d like to keep the stories of the #SQLfamily going through December. What other stories do you have about ways someone in the #SQLFamily community has gone above and beyond to help you or another member of the community? For instance…

• Did you ask a question of the #sqlhelp hashtag at 1am and get an answer in 5 minutes?
• Did a member of the #SQLFamily take time out of their day to help you debug something you just couldn’t figure out?

We just learned about thePragmatic Works Foundation, a non-profit that provides free technical training to veterans, the jobless, and underemployed.  In 2012 the foundation will be embarking on a campaign to bring better jobs for returning veterans. The classes will be taught at military base hotspots around the United States and will primarily focus on introductory technical training for SQL Server and soft skill training and interview guidance. At the end of the class, job placement assistance is given to help the veterans find jobs. The cost to train each veteran for a week is approximately $50. 

So, for the first 400 submissions*, the SQL Server team will donate $50 per submission to the Pragmatic Works Foundation. You can submit your #SQLFamily stories tosqlfamilysubmission@live.com, along with your name and email address. Or, send us a link to your blog or your Twitter handle if you post your story online. We will feature a selection of submitted stories weekly on this blog into the New Year.

Lastly, if this giving “campaign” is well received by the #SQLFamily, we’d like to do this again to celebrate SQL Server 2012 RTM. If you would like to see us do this again, we are now accepting suggestions for charitable organizations that you are familiar with that could benefit from a donation. We may select one or more charities based on their charter, use of Microsoft Technology, and history of impact, as well as eligibility to receive Microsoft donations. Charitable organization suggestions can also be sent to sqlfamilysubmission@live.com, but of course we can’t guarantee that your submission will be selected.

Thank you.

* Microsoft will donate $50 per qualified story, up to $20,000, toPragmatic Works Foundation during this promotion. To qualify, the story must be truthful and relevant to the theme. This offer ends December 31st, 2011 or when 400 qualified stories are received. Microsoft reserves the right to alter or cancel this offer at any time. Limit one submission per person and email address. By sending your story tosqlfamilysubmission@live.com you agree that Microsoft may read, share, modify and post your story on the SQL Server team blog and identify you as the writer by first initial and last name or as anonymous. When submitting their story, senders may indicate that they wish their story to be anonymous if published on this blog.  The names and email addresses submitted will not be used for any other purpose.  Microsoft is not responsible for verifying the validity of submissions.  Microsoft is not responsible for the content of public commentary about this offer.  Any other inquiries about this offer should be directed tosqlfamilysubmission@live.com.

In this talk from YUIConf 2011, YUI engineer Allen Rabinovich (@allenr) shares the process he used to architect and build the new Calendar widget in YUI 3, and explains how you can use a similar process to build your own widgets. He also shows off the new component and reveals a clever performance trick used to speed up the rendering of multiple calendars.

Links

• View in HD on YouTube
• Full YUIConf 2011 Playlist on YouTube

Ohne Musik wäre das Leben ein Irrtum
—Friedrich NIETZSCHE, Götzen-Dämmerung, Sprüche und Pfeile 33, 1889

Somehow, music is hardcoded in human beings. It is something we understand and respond to without prior knowledge. Music exercises the emotions and our imaginative reflex, not just our hearing. It behaves so much like our emotions that music can seem to symbolize them, to bear them from one person to another. Not surprisingly, it conjures memories: the word music derives from Greek μουσική (mousike), art of the Muses, whose mythological mother was Mnemosyne, memory. But it can also summon up the blood, console the bereaved, inspire fanaticism, bolster governments and dissenters alike, help us learn, and make web designers dance. And what would Christmas be without music?

Music moves us, often in ways we can’t explain. By some kind of alchemy, music frees us from the elaborate nuisance and inadequacy of words. Across the world and throughout recorded history – and no doubt well before that – people have listened and made (and made out to) music.

[I]t appears probable that the progenitors of man, either the males or females or both sexes, before acquiring the power of expressing their mutual love in articulate language, endeavoured to charm each other with musical notes and rhythm.
—Charles DARWIN, The Descent of Man, and Selection in Relation to Sex, 1871

It’s so integral to humankind, we’ve sent it into space as a totem for who we are. (Who knows? It might be important.) Music is essential, a universal compulsion; as Nietzsche wrote, without music life would be a mistake.

Music, design and web design

There are some obvious and notable similarities between music and visual design. Both can convey mood and evoke emotion but, even under close scrutiny, how they do that remains to a great extent mysterious. Each has formal qualities or parts that can be abstracted, analysed and discussed, often using the same terminology: composition, harmony, rhythm, repetition, form, theme; even colour, texture and tone.

A possible reason for these shared aspects is that both visual design and music are means to connect with people in deep and lasting ways. Furthermore, I believe the connections to be made can complement direct emotional appeal. Certain aesthetic qualities in music work on an unconscious and, it could be argued, universal level. Using musical principles in our designs, then, can help provide the connectedness between content, device and user that we now seek as web designers.

Yet, when we talk about music and web design, the conversation is almost always about the music designers listen to while working, a theme finding its apotheosis in Designers.MX. Sometimes, articles in that dreary list format seek inspiration from music industry websites. There’s even a service offering pre-templated web designs for bands, and at least one book surveyed the landscape back in 2006. Occasionally, discussions broaden somewhat into whether and how different kinds of music can inspire and influence the design work we produce.

Such enquiries, it seems to me, are beside the point. Do I really design differently when I listen to Bach rather than Bacharach? Will the barely restrained energy of Count Basie’s The Kid from Red Bank mean I choose a lively colour palette, and rural, autumnal shades when inspired by Fleet Foxes? Mahler means a thirteen-column layout? Gillian Welch leads to distressed black and white photography? While reflecting the importance we place in music and how it seems to help us in our work, surveys on musical taste and lists of favourite artists fail to recognize that some of the fundamental aesthetic characteristics of music can be adapted and incorporated into modern web design.

Antiphonal geometry

Over recent years, web designers have embraced grid systems as powerful tools to help create good-looking and intuitive user experiences. With the advent of responsive design, these grids and their contents must adapt to the different screen sizes and properties of all kinds of user devices. Finding and using grid values that can scale well and retain or enhance their proportions and relationships while making the user experience meaningful in several different contexts is more important than ever.

In print, this challenge has always started with the dimensions and proportions of the page. Content can thereby be made to belong inside the page and be bound to it. And music has been used for centuries to further this aim. As Robert Bringhurst says in The Elements of Typographical Style:

Indeed, one of the simplest of all systems of page proportions is based on the familiar intervals of the diatonic scale. Pages that embody these basic musical proportions have been in common use in Europe for more than a thousand years.

Very well. But while he goes on to list (from the full chromatic scale, rather than just diatonic) the proportions and the musical intervals they’re based on, Bringhurst fails to mention what they’re ratios of or their potential effects. Shame. In his favour, however, he later touches on how proportions in print might be considered to work:

The page is a piece of paper. It is also a visible and tangible proportion, silently sounding the thoroughbass of the book. On it lies the textblock, which must answer to the page. The two together – page and textblock – produce an antiphonal geometry. That geometry alone can bond the reader to the book. Or conversely, it can put the reader to sleep, or put the reader’s nerves on edge, or drive the reader away.

So what does Bringhurst mean by antiphonal geometry, a phrase that marries the musical to the spatial? By stating that the textblock “must answer to the page”, the implication is that the relationship between the proportions of the page and the shape of the textblock printed on it embodies a spatial (geometrical) call-and-response (antiphony) that can be appealing or not.

Boulton’s new canon

But, as Mark Boulton has pointed out, on the web “there are no edges. There are no ‘pages’. We’ve made them up.” So, what is to be done? In January 2011 at the New Adventures in Web Design conference, Boulton presented his vision of a new canon of web design, a set of principles to guide us as we design the web. There are three overlapping tenets:

1. design from the content out
2. create connectedness between the different content elements
3. bind the content to the web device

Rather than design from the edges in, we need to design layout systems from the content out. To this end, Boulton asserts that grid system design should begin with a constraint, and he suggests we use the size of a fixed content element, such as an advertising unit or image, as a starting point for online grid calculations. Khoi Vinh advocates the same approach in his book, Ordering Disorder: Grid Principles for Web Design.

Boulton’s second and third tenets, however, are more complex and overlap significantly with each other. Connecting the different parts of the content and binding the content to the device share many characteristics and solutions:

• adopting ems and percentages as units of size for layout elements
• altering text size, line length and line height for different viewport dimensions
• providing higher resolution images for devices with greater pixel densities
• fluid layout grids, flexible images and responsive design

All can help relate the presentation of the content to its delivery in a certain context.

But how do we determine the relationship between one element of a layout and another? How can we avoid making arbitrary decisions about the relative sizes of parts of our designs? What can we use to connect the parts of our design to one another, and how can we bind the presentation of the content to the user’s device?

Tim Brown’s application of modular typographic scales hints at an answer. In the very useful tool he created for calculating such scales, Brown includes two musical ratios: the perfect fifth (2:3); and the perfect fourth (3:4). Why? Where do they come from? And what do they mean?

Harmonies musical and visual

Fundamental to music are rhythm and harmony.

As any drummer will tell you, without rhythm there is no music. Even when there’s no regular beat, any tune follows a rhythm, however irregular, simply because a change of note is a point of change in the music. Although rhythm, timing and pacing are all relevant to interaction design, right now it’s harmony we’re interested in.

Sometimes harmony is called the vertical aspect of music, and melody the horizontal. But this conceit overlooks the fact that harmony is both vertical and horizontal. A single melodic line, as it is played, implies various sets of harmonies on which it is grounded, whether or not those harmonies are played. So, harmony doesn’t just sit vertically beneath the horizontal melody, but moves horizontally as well, through harmonic progression.

To stretch this arrangement pixel-thin, we could argue that in onscreen design melody is the content, and the layout and arrangement of the content is the harmony. We sometimes say a design is harmonious when the interplay of different elements of a design is pleasing or balanced or in proportion, and the content (the melody) is set off or conveyed well by or appropriate to the design.

We seem to know instinctively whether a layout is harmonious…

In the design of The Great Discontent, the relationships between different elements combine to form a balanced whole.

…or not.

There’s no harmony in the Department for Education’s website because the different parts of the content don’t feel related to one another.

What is it that makes one design harmonious and another dissonant? It’s not just whether things line up, though that’s a start. I believe there are much deeper aesthetic forces at work, forces we can tap into in our onscreen designs. Now, I’m not going start a difficult discussion about aesthetics. For our purposes, we just need to know that it’s the branch of philosophy dealing with the nature of beauty, and the creation and perception of beauty. And among the key components in the perception of beauty are harmony and proportion. These have been part of traditional western aesthetics since Plato (about 2,500 years).

One of the ways we appreciate the beauty of music is through the harmonic intervals we hear. A musical interval is a combination of two notes and it describes the distance between the two pitches. For example, the distance between C and the G above it (if we take C as the tonic or root) is called a perfect fifth.

Left: C to G, a perfect fifth. Right: C and G, not a perfect fifth.

And, to get superficially scientific for a moment, each musical interval can be expressed as a ratio of the wavelength frequencies of the notes; for our perfect fifth, with every two wavelengths of C, there are three of G. And what is a ratio, if not a proportion of one thing to another?

So, simple musical harmony (using what’s known as just intonation¹) affords us a set of proportions, expressed as ratios. Where better to apply these ideas of harmony and proportion from music in web design, than grid systems?

A digression: whither φ?

Quite often in our discussions of pure design and aesthetics, we mention the golden ratio and regurgitate the same justifications for its use: roots in antiquity; embodied in classical and Renaissance architecture and art; occurrence in nature; the New Twitter, and so forth (oh, really?).

Yet the ratios of musical intervals from just intonation are equally venerable and much more widespread: described by Pythagorus; employed in Palladian architecture, and printing, books and art from the Renaissance onwards; in modern times, film and television dimensions; standard international paper sizes (ISO 216, the A and B series); and, again and again, screen dimensions – chances are that screen you’re probably looking at right now has the proportions 2:3 (iPhone and iPod Touch), 3:4 (iPad and Kindle), 3:5 (many smartphones), 5:8 or 16:9 (many widescreen monitors), all ratios of musical intervals.

Back to our theme…

Musical interval ratios

Let’s take a look at most of the ratios within a couple of octaves and crunch some numbers to generate some percentages and other values that we can use in our designs. First, the intervals and their ratios in just intonation and expressed as ratios of one:

Name	Interval in C	Ratio	Ratio (1:x)
unison	C→C	1:1	1:1
minor second	C→D♭	15:16	1:1.067
major second	C→D	8:9	1:1.125
minor third	C→E♭	5:6	1:1.2
major third	C→E	4:5	1:1.25
perfect fourth	C→F	3:4	1:1.333
augmented fourth or diminished fifth	C→F♯/G♭	1:√2	1:1.414
perfect fifth	C→G	2:3	1:1.5
minor sixth	C→A♭	5:8	1:1.6
major sixth	C→A	3:5	1:1.667
minor seventh	C→B♭	9:16	1:1.778
major seventh	C→B	8:15	1:1.875
octave	C→C↑	1:2	1:2
major tenth	C→E↑	2:5	1:2.5
major eleventh	C→F↑	3:8	1:2.667
major twelfth	C→G↑	1:3	1:3
double octave	C→C↑	1:4	1:4
Name	Interval in C	Ratio	Ratio (1:x)

And now as percentages, of both the larger and smaller values in the ratios:

Name	Ratio	% of larger value	% of smaller value
unison	1:1	100%	100%
minor second	15:16	93.75%	106.667%
major second	8:9	88.889%	112.5%
minor third	5:6	83.333%	120%
major third	4:5	80%	125%
perfect fourth	3:4	75%	133.333%
augmented fourth or diminished fifth	1:√2	70.711%	141.421%
perfect fifth	2:3	66.667%	150%
minor sixth	5:8	62.5%	160%
major sixth	3:5	60%	166.667%
minor seventh	9:16	56.25%	177.778%
major seventh	8:15	53.333%	187.5%
octave	1:2	50%	200%
major tenth	2:5	40%	250%
major eleventh	3:8	37.5%	266.667%
major twelfth	1:3	33.333%	300%
double octave	1:4	25%	400%
Name	Ratio	% of larger value	% of smaller value

As you can see, the simple musical intervals are expressed as ratios of small whole numbers (integers). We can then normalize them as ratios of one, as well as derive percentage values, both in terms of the smaller value to the larger, and vice versa. These are the numbers we can incorporate into our designs. If you’ve ever written something like body { font: 100%/1.5 "Museo Sans", Helvetica, sans-serif; } in your CSS, you’re already using a musical ratio: the perfect fifth.

Modular scales allow us to generate a set of numbers based on a musical interval that can be used for all kinds of typographic and layout decisions to create harmony in a visual design for the web. As Tim Brown said at the 2010 Build conference:

I think that from that most atomic unit – type – whole experiences can resonate, whole experiences can be harmonious. And whole experiences can have a purpose suited to our design intentions.

Once more, with feeling: connectedness

As well as modular scales, there are other methods of incorporating musical interval ratios into our work. Setting the ratio of font size to line height in CSS is one such example. We could also create a typographic hierarchy using the same principle and combining several ratios that we know harmonize well musically in a chord:

body { font-size: 75%; } /* =12px = base size or tonic */

h1 { font-size: 32px; font-size: 2.667rem; }
   /* =32px = 3:8 = major eleventh (C→F↑) */

h2 { font-size: 24px; font-size: 2rem; }
   /* =24px = 1:2 = octave (C→C↑) */

h3 { font-size: 20px; font-size: 1.667rem; }
   /* =20px = 3:5 = major sixth (C→A) */

figcaption, small { font-size: 9px; font-size : 0.75rem }
   /* =9px = 3:4 = perfect fourth (C→F) */

Whoa! Hold your reindeer, Santa! How can we know what interval combinations work well together to form chords? Well, I’m a classically trained musician, so perhaps I have an advantage. To avoid a long, technically complex digression into musical harmony, here are a few basic combinations of intervals that are harmonious in one way or another:

• unison; major third; perfect fifth; octave
• unison; perfect fourth; major sixth; octave
• unison; minor third; minor sixth; octave
• unison; minor third; diminished fifth; major sixth; octave

This isn’t to say that other combinations can’t be used to interesting effect and particular purpose – they surely can – but I have to make sure there’s something left for you to experiment with in the wee small hours over the holiday. Bear in mind, though, were I to play you two notes from the same scale to form a minor second, for example, you’d probably say it was dissonant and maybe that quality of the 15:16 ratio would be translated to the design.

In the typographic hierarchy above, you’ll notice I used an interval in the higher octave, which affords a broader range of ratios while retaining the harmony. Thus, a perfect fifth (2:3) becomes a major twelfth (1:3), or a major sixth (3:5) becomes a major thirteenth (3:10).

The harmonic ratios can obviously be used as proportions for layout as well, in several different ways:

• image width and height (for example, 450×800px = 9:16 = minor seventh)
• main content to page width (67%:100% = 2:3 = perfect fifth)
• page width to viewport width (80%:100% = 4:5 = major third)

One great benefit of using such ratios in web design work is that they can be applied in responsive web design. Proportional values, based on percentages or equivalent em units, will scale with changing viewports, so your layout and image proportions can be maintained or deliberately changed, as we’re about to find out, across devices.

Small speakers, tall speakers: binding to the device

The musical interval ratios also provide an opportunity, not only to create connectedness between the parts of a layout, but to bind the content to a device – that elusive antiphonal geometry. Just as a textblock and page resonate together, so too can web content and the screen. Earlier, I mentioned that several common screen aspect ratios match musical interval ratios. It would seem, then, that we have a set of proportions that we can use in different ways to establish and retain a sense of harmony that can be based on and change with those contexts. Using musical interval ratios, we can bind the display of our content to the device it’s displayed on.

If you haven’t met already, let me introduce you to the device-aspect-ratio property of CSS media queries.

@media only screen and (device-aspect-ratio: 3/4) { }
@media only screen and (device-aspect-ratio: 480/640) { }
@media only screen and (device-aspect-ratio: 600/800) { }
@media only screen and (device-aspect-ratio: 768/1024) { }

Regardless of the precise pixel values, each of these media queries would apply to devices whose display area has an aspect ratio of 3:4. It works by comparing the device-width with the device-height. (It’s not to be confused with aspect-ratio, which is defined by the width and height of the browser within the device.) The values in the media query are always presented as width/height, with portrait being the default orientation for smartphones and tablets; that is, to match an iPhone screen, you’d use device-aspect-ratio: 2/3, not 3/2, which won’t work.

Here’s a table of the musical intervals with their corresponding screens.

Name	device-aspect-ratio	Screens	Common resolutions (pixels)
major third	5/4	TFT LCD computer screens	1,280×1,024
perfect fourth	3/4 or 4/3	iPad, Kindle and other tablets, PDAs	320×240, 768×1,024
perfect fifth	2/3	iPhone, iPod Touch	320×480, 640×960
minor sixth	8/5 (16/10)	Many widescreens	1,152×720, 1,440×900, 1,920×1,200
major sixth	3/5	Many smartphones	240×400, 480×800
minor seventh	16/9 or 9/16	Many widescreens and some smartphones	720×1,280, 1,366×768, 1,920×1,080, 2,560×1,440

[You might argue that I’m playing fast and loose with the ratios. I suppose, mathematically speaking, 9:16 is not the same as 16:9: I’m no expert. But let’s not throw the baby out with the bath water, particularly at Christmas.]

With this in mind, we can begin to write media queries that will influence various typographic and layout values in line with the aspect ratios of specific screens and in combination with em-based min-width queries that work from smaller, mobile screens to larger, desktop screens.

Here’s a very simple demo page with only some text, an image with a caption and a little basic layout: no seasonal overindulgence here.

Demo: Sample page using device-aspect-ratio media queries based on musical interval ratios

Our initial styles for all devices are based on the perfect fifth, with the major third and octave rounding things out into a harmonious whole, whether or not media queries are supported. For example:

html { font-size: 100%; line-height: 1.5; }
   /* font-size:line-height = 16:24 = 2:3 = perfect fifth */

h1 { font-size: 32px; font-size: 2rem; line-height: 1.25; }
   /* font-size:line-height = 32:40 = 4:5 = major third
      body:h1 = 16:32 = 1:2 = octave */

While we should really consider methods of delivering images appropriate to the screen size, let’s just stick to a single image for all devices. But why don’t we change its aspect ratio from 4:3 to 3:2, to fit with our harmonic scheme? It’s easy enough to do with overflow:hidden on the <figure> element to hide a part of the image, and a negative margin fudge:

figure img { margin: -8.5% 0 0 0; width: 100%; max-width: 100%; }

Our first break point targets devices 320 pixels wide with an aspect ratio of 2:3, namely the iPhone and iPod Touch:

/* 320px = 20×16 */
@media only screen and (min-width: 20em) and (device-aspect-ratio: 2/3) { }

We’re actually already there, of course, as the intervals we’ve chosen resonate with this aspect ratio – the content is already bound to the device.

Our next media query, then, will make some changes to match a different ratio, the major sixth (3:5), which is same as that of many smartphones:

/* 480px = 30×16 */
@media only screen and (min-width: 30em) and (device-aspect-ratio: 3/5) { }

A different aspect ratio might require a change in harmony. For devices with these proportions, we’ll now use the perfect fourth (3:4) and the major sixth (3:5) along with the octave we already have to create a new resonating harmony. For instance, a slightly wider screen means we can increase the line-height to aid the legibility of longer lines:

html { line-height: 1.667; }
   /* font-size:line-height = 16:26.672 = 3:5 = major sixth */

h1 { font-size: 32px; font-size: 2rem; line-height: 1.667; }
   /* font-size:line-height = 32:53.333 = 3:5 = major sixth
      body:h1 = 16:32 = 1:2 = octave */

and we can remove the negative margin to display our 4:3 image in its entirety.

Each screen displays content styled using relationships related to its own proportions. On the left, an iPhone 4 (2:3); on the right, a Samsung Nexus S (3:5). Your mileage may vary.

Another device, another media query. At 768 pixels, screens are wide enough to add columns. The ratios we’ve used for the 3:5 screens include the perfect fourth (3:4) so we don’t need to change any of the font measurements, but we can base the proportions of the columns on the major sixth interval:

article { float: left; width: 56%; }
   /* width of main column 3:5 (60% of 100%, major sixth)
      incorporating gutter width */

aside { float : right; width : 36%; }

On devices with a 3:4 aspect ratio, this works even better in landscape orientation.

While not every screen over 768 pixels wide will have 3:4 proportions, the range of intervals informing the design ensure harmonious relationships between the different parts of the layout.

For wide screens proper (break point at 1,280 pixels) we can employ a new set of harmonious intervals. Many laptop and desktop screens have a 16:10 aspect ratio, which boils down to 8:5, equivalent to the minor sixth (5:8). Combined with a minor third (5:6) and the octave (1:2), this creates a new harmony appropriate to these devices. Let’s increase the font size and change the image’s aspect ratio to match:

html { font-size: 120%; line-height: 1.6; }
   /* font-size increased for wider screens from 16px to 19.2px
      (5:6 = minor third)
      font-size:line-height = 19.2:30.72 = 5:8 = minor sixth */

figure img { margin: -12.5% 0 0 ; }	
   /* using -ve margin combined with overflow:hidden
      on the figure element
      to crop the image from 4:3 to 8:5 = minor sixth */

A wide screen with a 8:5 (16:10) aspect ratio and an image to match.

With more pixels at our disposal, we can also now use the musical interval ratios to determine the width of the layout, and change the column proportions as well:

section { margin: 0 auto; width: 83.333%; }
   /* content width:screen width = 5:6 = minor third */

article { width: 60%; }
   /* width of main column 5:8 (62.5% of 100%, minor sixth)
      incorporating gutter width */

aside { width: 35%; }

With some carefully targeted media queries, we can begin to reach towards fulfilling the second and third tenets of Boulton’s new canon for web design: connecting the parts of content through relationships embodied in the layout design; and binding the content to the devices people use to access it.

Coda

Musical interval ratios and screen aspect ratios reveal more than convenient correspondence. These proportions work on a deep aesthetic level. Much is claimed for the golden ratio φ, but none of the screens pervading our lives use it. Perhaps that’s an accident of technology, but can making screens to φ’s proportions be more difficult or expensive than 2:3 or 3:4 or 16:10? Here, then, is not just one but a set of proportions with a uniquely human focus, originating in nature, recognized in antiquity, fundamental still.

We find music to be an art steeped with meaning, yet, unlike literary and representational arts, purely instrumental music has no obvious semantic content. It boasts an ability to express emotions while remaining an abstract art in some sense, which makes it very like design. These days, we’re rightly encouraged to design for emotion, to make our users’ experience meaningful, seductive, delightful. Using musical ideas and principles in our designs can help achieve those ends.

Let’s not be naïve, of course; designing web pages is even less like composing music than it’s like designing for print. In visual design, the eye will always be sovereign to the ear; following these principles will only get us so far. We cannot truly claim that a carefully composed web page layout will have the same qualities and effect as any musical patterns that inform it. In music, a set of intervals is always harmonious in relation to other sets of intervals: music rarely stands still. What aspect ratios will future screens take? Already today there is great variation in devices and support for media queries (and within that, support for device-aspect-ratio). And what of non-western musical traditions? Or rhythm, form, tempo and dynamics? What I’ve demonstrated above is only a suggestion, a tentative exploration of one possible way forward.

But as our discipline matures and we become more articulate about what we do, we must look longer and deeper into areas of human endeavour already rich with value. Music is a fertile ground to explore and has the potential to yield up new approaches for web design.

Footnotes

1. Just intonation is a system of tuning that uses small integers to describe the musical intervals, based initially on the perfect fifth, that most consonant of intervals. Simple instruments such as vibrating strings and natural horns, as well as unaccompanied voices, tend to fall into just intonation naturally.

Many, many years ago, before web design became my proper job, I trained and worked as a journalist. I studied publishing in London and spent three fun years learning how to take a few little nuggets of information and turn them into a story. I learned a bunch of stuff that has all been a huge help to my design career. Flatplanning, layout, typographic theory. All of these disciplines have since translated really well to web design, but without doubt the most useful thing I learned was how to ask difficult questions.

Pretty much from day one of journalism school they hammer into you the importance of the Five Ws. Five disarmingly simple lines of enquiry that eloquently manage to provide the meat of any decent story. And with alliteration thrown in too. For a young journo, it’s almost too good to be true.

Who? What? Where? When? Why? It seems so obvious to almost be trite but, fundamentally, any story that manages to answer those questions for the reader is doing a pretty good job. You’ll probably have noticed feeling underwhelmed by certain news pieces in the past – disappointed, like something was missing. Some irritating oversight that really lets the story down. No doubt it was one of the Ws – those innocuous little suckers are generally only noticeable by their absence, but they sure get missed when they’re not there.

Question everything

I’ve always been curious. An inveterate tinkerer with things and asker of dopey questions, often to the point of abject annoyance for anyone unfortunate enough to have ended up in my line of fire. So, naturally, the Five Ws started drifting into other areas of my life. I’d scrutinize everything, trying to justify or explain my rationale using these Ws, but I’d also find myself ripping apart the stuff that clearly couldn’t justify itself against the same criteria.

So when I started working as a designer I applied the same logic and, sure enough, the Ws pretty much mapped to the exact same needs we had for gathering requirements at the start of a project. It seemed so obvious, such a simple way to establish the purpose of a product. What was it for? Why we were making it? And, of course, who were we making it for? It forced clients to stop and think, when really what they wanted was to get going and see something shiny. Sometimes that was a tricky conversation to have, but it’s no coincidence that those who got it also understood the value of strategy and went on to have good solid products, while those that didn’t often ended up with arrogantly insular and very shiny but ultimately unsatisfying and expendable products. Empty vessels make the most noise and all that…

Content first

I was both surprised and pleased when the whole content first idea started to rear its head a couple of years back. Pleased, because without doubt it’s absolutely the right way to work. And surprised, because personally it’s always been the way I’ve done it – I wasn’t aware there was even an alternative way. Content in some form or another is the whole reason we were making the things we were making. I can’t even imagine how you’d start figuring out what a site needs to do, how it should be structured, or how it should look without a really good idea of what that content might be. It baffles me still that this was somehow news to a lot of people. What on earth were they doing? Design without purpose is just folly, surely?

It’s great to see the idea gaining momentum but, having watched it unfold, it occurred to me recently that although it’s fantastic to see a tangible shift in thinking – away from those bleak times, where making things up was somehow deemed an appropriate way to do things – there’s now a new bad guy in town.

With any buzzword solution of the moment, there’s always a catch, and it seems like some have taken the content first approach a little too literally. By which I mean, it’s literally the first thing they do. The project starts, there’s a very cursory nod towards gathering requirements, and off they go, cranking content. Writing copy, making video, commissioning illustrations.

All that’s happened is that the ‘making stuff up’ part has shifted along the line, away from layout and UI, back to the content.

Starting is too easy

I can’t remember where I first heard that phrase, but it’s a great sentiment which applies to so much of what we do on the web. The medium is so accessible and to an extent disposable; throwing things together quickly carries far less burden than in any other industry. We’re used to tweaking as we go, changing bits, iterating things into shape. The ubiquitous beta tag has become the ultimate caveat, and has made the unfinished and unpolished acceptable. Of course, that can work brilliantly in some circumstances. Occasionally, a product offers such a paradigm shift it’s beyond the level of deep planning and prelaunch finessing we’d ideally like. But, in the main, for most client sites we work on, there really is no excuse not to do things properly. To ask the tricky questions, to challenge preconceptions and really understand the Ws behind the products we’re making before we even start.

The four Ws

For product definition, only four of the five Ws really apply, although there’s a lot of discussion around the idea of when being an influencing factor. For example, the context of a user’s engagement with your product is something you can make a call on depending on the specifics of the project.

So, here’s my take on the four essential Ws. I’ll point out here that, of course, these are not intended to be autocratic dictums. Your needs may differ, your clients’ needs may differ, but these four starting points will get you pretty close to where you need to be.

Who

It’s surprising just how many projects start without a real understanding of the intended audience. Many clients think they have an idea, but without really knowing – it’s presumptive at best, and we all know what presumption is the mother of, right? Of course, we can’t know our audiences in the same way a small shop owner might know their customers. But we can at least strive to find out what type of people are likely to be using the product. I’m not talking about deep user research. That should come later.

These are the absolute basics. What’s the context for their visit? How informed are they? What’s their level of comprehension? Are they able to self-identify and relate to categories you have created? I could go on, and it changes on a per-project basis. You’ll only find this out by speaking to them, if not in person, then indirectly through surveys, questionnaires or polls. The mechanism is less important than actually reaching out and engaging with them, because without that understanding it’s impossible to start to design with any empathy.

What

Once you become deeply involved directly with a product or service, it’s notoriously difficult to see things as an outsider would. You learn the thing inside and out, you develop shortcuts and internal phraseology. Colloquialisms creep in. You become too close. So it’s no surprise when clients sometimes struggle to explain what it is their product actually does in a way that others can understand.

Often products are complex but, really, the core reasons behind someone wanting to use that product are very simple. There’s a value proposition for the customer and, if they choose to engage with it, there’s a value exchange. If that proposition or exchange isn’t transparent, then people become confused and will likely go elsewhere. Make sure both your client and you really understand what that proposition is and, in turn, what the expected exchange should be. In a nutshell: what is the intended outcome of that engagement? Often the best way to do this is strip everything back to nothing. Verbosity is rife on the web. Just because it’s easy to create content, that shouldn’t be a reason to do so. Figure out what the value proposition is and then reintroduce content elements that genuinely help explain or present that to a level that is appropriate for the audience.

Why

In advertising, they talk about the truths behind a product or service. Truths can be both tangible or abstract, but the most important part is the resonance those truths hit with a customer. In a digital product or service those truths are often exposed as benefits. Why is this what I need? Why will it work for me? Why should I trust you? The why is one of the more fluffy Ws, yet it’s such an important one to nail. Clients can get prickly when you ask them to justify the why behind their product, but it’s a fantastic way to make sure the value proposition is clear, realistic and meets with the expectations of both client and customer.

It’s our job as designers to question things: we’re not just a pair of hands for clients. Just recently I spoke to a potential client about a site for his business. I asked him why people would use his product and also why his product seemed so fractured in its direction. He couldnt answer that question so, instead of ploughing on regardless, he went back to his directors and is now re-evaluating that business. It was awkward but he thanked me and hopefully he’ll have a better product as a result.

Where

In this instance, where is not so much a geographical thing, although in some cases that level of context may indeed become a influencing factor… The where we’re talking about here is the position of the product in relation to others around it. By looking at competitors or similar services around the one you are designing, you can start to get a sense for many of the things that are otherwise hard to pin down or have yet to be defined. For example, in a collection of sites all selling cars, where does yours fit most closely? Where are the overlaps? How are they communicating to their customers? How is the product range presented or categorized?

It’s good to look around and see how others are doing it. Not in a quest for homogeneity but more to reference or to avoid certain patterns that may or may not make sense for your own particular product. Clients often strive to be different for the sake of it. They feel they need to provide distinction by going against the flow a bit. We know different. We know users love convention. They embrace familiar mental models. They’re comfortable with things that they’ve experienced elsewhere. By showing your client that position is a vital part of their strategy, you can help shape their product into something great.

To conclude

So there we have it – the four Ws. Each part tells a different and vital part of the story you need to be able to make a really good product. It might sound like a lot of work, particularly when the client is breathing down your neck expecting to see things, but without those pieces in place, the story you’re building your product on, and the content that you’re creating to form that product can only ever fit into one genre. Fiction.

Remember when you accidentally deleted that master to-do list in your Basecamp project? Or when Bob thought he was looking at the old Sales project and deleted the new one by mistake?

These things happen, and they definitely used to ruin your day. We felt your pain as well, since these things quickly found their way into our support queue. But Basecamp now sports a spiffy new way to recover things that you (or anyone else on your account) deleted recently.

Recovering a deleted project

If it is a project that was deleted, any of your admins will see a new link on the sidebar of the dashboard:

Clicking that link will take them to a page showing all recently deleted projects, including when they were deleted, and who deleted them:

To recover any of those projects, simply click the corresponding “restore” link, and you’ll be back in business!

Recovering a deleted data within a project

Similarly, within any project, there is a link at the bottom of each page visible to any admin:

Following that link will take you to a page listing all data recently deleted from that project, again with when it was deleted and who deleted it:

Clicking the restore link there will let you recover any of that data as well.

The deleted data is only retained for a short time (no more than 30 days). Hopefully this will reduce heartburn when someone accidentally deletes that crucial message or mock up!

This year I spoke at the Content Strategy Forum about the language of interfaces. Usability problems usually fall into two categories; either it’s not clear how to do something, or something is too cumbersome to do. The latter is fixed by a better understanding of what the key tasks are, and the former is usually resolved by adding clarity. Often the best way to do this is through the writing of the interface.

Wireframing is a low fidelity way of exploring ideas; it’s not a good time to obsess over the words. There is a time and place for obsessing over the writing in an interface, and that’s what this presentation was about.

Writing Microcopy

Writing an interface has similar traits with writing anything else. Some pieces are obvious, some flow naturally with the application, and some you’ll read back and wonder who was dropping acid in your coffee. It’s convenient to break down a design into components like visual, layout, typography, and content; but this division often ignores the inter-dependencies. If your visual designer only leaves you 10 characters of space to explain a button the content, and therefore the usability, can suffer because of it.

That said when I review all the copy/microcopy/labels/whatever in an application there is always low hanging fruit. Quick wins that everyone will appreciate. I usually work with the designer or app owner to re-write pieces. If I have little domain knowledge I just ask a lot of questions.

Questioning your copy

1. Who is it for? – Any user, new users only, old users, free users? New users benefit from more precise instruction. Telling a new sign up to change their notifications “in preferences” isn’t clear; most likely they don’t know where preferences is or what icon represents it.
2. When they do see this message? – Does this appear as a reaction to a user action, or is it something that is processed behind the scenes (e.g. charging a credit card). Does it appear immediately? Does the user expect it?
3. What do they need to know? Self explanatory.
4. What must they do now, if anything? Often messages have a next step attached; something you’d like the user to do or decide. This should be the logical conclusion of the message.
5. How is this communicated? On screen, pop-up, by email, iOS notification, SMS? Everything that’s good for something is bad for something else.
6. What tone does this app speak in? This is driven by the brand. Some apps are cute. Some are fun. Some are exact and precise. OKCupid can be flirty, and Mailchimp can be fun. A investment website should probably be neither. Productivity apps are best kept clear and precise. The tone of your app is an easy way to stand out, just be sure to stand out for the right reasons.

All that for a string of text?

This may seem like a laborious list to go through for each label and message, and of course many will done in microseconds, e.g. Cancel, Edit, Save, etc. The above questions are for the labels and messages that aren’t coming to you easily.

Video and Slides

The video is available of my talk, there is a small amount of profanity and plenty of laughter thanks the good folks at the conference.

Every now and then I come across a problem with PHPs autoloading whereby class_exists fails to autoload classes. I’ve noticed this with a number of PHP versions, most recently with 5.3.2-1ubuntu4.9. Typically I’ve resolved this by simply upgrading the PHP version. In the following example the initial class_exists fails until you instantiate the object and [...]

I’ve been a professional graphic designer for fourteen years and for just under four of those a professional web designer. Like most designers I’ve learned a lot in my time, both from a design point of view and in business as freelance designer. A few of the things I’ve learned stick out in my mind, so I thought I’d share them with you. They’re pretty random and in no particular order.

1. Becoming the designer you want to be

When I started out as a young graphic designer, I wanted to design posters and record sleeves, pretty much like every other young graphic designer. The problem is that the reality of the world means that when you get your first job you’re designing the back of a paracetamol packet or something equally weird. I recently saw a tweet that went something like this: “You’ll never become the designer you always dreamt of being by doing the work you never wanted to do”. This is so true; to become the designer you want to be, you need to be designing the things you’re passionate about designing. This probably this means working in the evenings and weekends for little or no money, but it’s time well spent. Doing this will build up your portfolio with the work that really shows what you can do! Soon, someone will ask you to design something based on having seen this work. From this point, you’re carving your own path in the direction of becoming the designer you always wanted to be.

2. Compete on your own terms

As well as all being friends, we are also competitors. In order to win new work we need a selling point, preferably a unique selling point. Web design is a combination of design disciplines – user experience design, user interface Design, visual design, development, and so on. Some companies will sell themselves as UX specialists, which is fine, but everyone who designs a website from scratch does some sort of UX, so it’s not really a unique selling point. Of course, some people do it better than others.

One area of web design that clients have a strong opinion on, and will judge you by, is visual design. It’s an area in which it’s definitely possible to have a unique selling point. Designing the visual aesthetic for a website is a combination of logical decision making and a certain amount of personal style. If you can create a unique visual style to your work, it can become a selling point that’s unique to you.

3. How much to charge and staying motivated

When you’re a freelance designer one of the hardest things to do is put a price on your work and skills. Finding the right amount to charge is a fine balance between supplying value to your customer and also charging enough to stay motivated to do a great job. It’s always tempting to offer a low price to win work, but it’s often not the best approach: not just for yourself but for the client as well.

A client once asked me if I could reduce my fee by £1,000 and still be motivated enough to do a good job. In this case the answer was yes, but it was the question that resonated with me. I realized I could use this as a gauge to help me price projects. Before I send out a quote I now always ask myself the question “Is the amount I’ve quoted enough to make me feel motivated to do my best on this project?” I never send out a quote unless the answer is yes. In my mind there’s no point in doing any project half-heartedly, as every project is an opportunity to build your reputation and expand your portfolio to show potential clients what you can do. Offering a client a good price but not being prepared to put everything you have into it, isn’t value for money.

4. Supplying the right design

When I started out as a graphic designer it seemed to be the done thing to supply clients with a ton of options for their logo or brochure designs. In a talk given by Dan Rubin, he mentioned that this was a legacy of agencies competing with each other in a bid to create the illusion of offering more value for money. Over the years, I’ve realized that offering more than one solution makes no sense. The reason a client comes to you as a designer is because you’re the person than can get it right. If I were to supply three options, I’d be knowingly offering my client at least two options that I didn’t think worked.

To this day I still get asked how many homepage design options I’ll supply for the quoted amount. The answer is one. Of course, I’m more than happy to iterate upon the design to fine-tune it and, on the odd occasion, I do revisit a design concept if I just didn’t nail the design first time around. Your time is much better spent refining the right design option than rushing out three substandard designs in the same amount of time.

5. Colour is key

There are many contributing factors that go into making a good visual design, but one of the simplest ways to do this is through the use of colour. The colour palette used in a design can have such a profound effect on a visual design that it almost feels like you’re cheating. It’s easy to add more and more subtle shades of colour to add a sense of sophistication and complexity to a design, but it dilutes the overall visual impact. When I design, I almost have a rule that only allows me to use a very limited colour palette. I don’t always stick to it, but it’s always in mind and something I’m constantly reviewing through my design process.

6. Creative thinking is central to good or boundary-pushing web design

When we think of creativity in web design we often link this to the visual design, as there is an obvious opportunity to be creative in this area if the brief allows it. Something that I’ve learnt in my time as a web designer is that there’s a massive need for creative thinking in the more technical aspects of web design. The tools we use for building websites are there to be manipulated and used in creative ways to design exciting and engaging user experiences. Great developers are constantly using their creativity to push the boundaries of what can be done with CSS, jQuery and JavaScript.

Being creative and creative thinking are things we should embrace as an industry and they are qualities that can be found in anyone, whether they be a visual designer or Rails developer.

7. Creative block: don’t be afraid to get things wrong

Creative block can be a killer when designing. It’s often applied to visual design, which is more subjective. I suffer from creative block on a regular basis. It’s hugely frustrating and can screw up your schedule. Having thought about what creative block actually is, I’ve come to the conclusion that it’s actually more of a lack of direction than a lack of ideas. You have ideas and solutions in mind but don’t feel committed to any of them. You’re scared that whatever direction you take, it’ll turn out to be wrong. I’ve found that the best remedy for this is to work through this barrier. It’s a bit like designing with a blindfold on – you don’t really know where you’re going. If you stick to your guns and keep pressing forward I find that, nine times out of ten, this process leads to a solution. As the page begins to fill, the direction you’re looking for slowly begins to take shape.

8. You get better at designing by designing

I often get emails asking me what books someone can read to help them become a better designer. There are a lot of good books on subjects like HTML5, CSS, responsive web design and the like, that will really help improve anyone’s web design skills. But, when it comes to visual design, the best way to get better is to design as much as possible. You can’t follow instructions for these things because design isn’t following instructions. A large part of web design is definitely applying a set of widely held conventions, but there’s another part to it that is invention and the only way to get better at this is to do it as much as possible.

9. Self-belief is overrated

Throughout our lives we’re told to have self-belief. Self-belief and confidence in what we do, whatever that may be. The problem is that some people find it easier than others to believe in themselves. I’ve spent years trying to convince myself to believe in what I do but have always found it difficult to have complete confidence in my design skills. Self-doubt always creeps in.

I’ve realized that it’s ok to doubt myself and I think it might even be a good thing! I’ve realized that it’s my self-doubt that propels me forward and makes me work harder to achieve the best results. The reason I’m sharing this is because I know I’m not the only designer that feels this way. You can spend a lot of time fighting self-doubt only to discover that it’s your body’s natural mechanism to help you do the best job possible.

Facebook Software Engineer and HipHop for PHP team member Jason Evans provides details on Facebook’s move to a new high-performance PHP virtual machine. Described by Evans is ”a new PHP execution engine based on the HipHop language runtime that we call the HipHop Virtual Machine (hhvm).” He sees it as replacement for the HipHop PHP interpreter (hphpi). He continues:

We have long been keenly aware of the limitations to static analysis imposed by such a dynamic language as PHP, not to mention the risks inherent in developing software with hphpi and deploying with hphpc. Our experiences with hphpc led us to start experimenting with dynamic translation to native machine code, also known as just-in-time (JIT) compilation … we developed a high-level stack-based virtual machine specifically tailored to PHP that executes HipHop bytecode (HHBC). hhvm uses hphpc’s PHP>AST implementation and extends the pipeline to PHP>AST>HHBC.

He estimates the hhvm bytecode interpreter is approximately 1.6X faster for certain Facebook-specific benchmarks, with still better performance in the offing. But, as described in his blog post on the PHP compilation innovations, there is still work ahead. You can view HipHop-related information at GitHub.

Vagrant is simply a wrapper around Virtualbox headless that allows for provisioning virtual machines with support for puppet, chef-solo, chef, and bash. This allows you to automate the deployment and sandboxing of development sites. Additional base box images can be found at vagrantbox.es. Installing Vagrant apt-get install -y ruby1.9.1 ruby1.9.1-dev ln -svf /usr/bin/ruby1.9.1 /etc/alternatives/ruby gem [...]

Assuming you have a curl compiled with IPv6, if you wanted to hit a page using its IPv6 address rather than its hostname, you have to do it as follows:

curl "http://[2600:xxx:yyy::zzz]/page.html"

The square brackets are required to tell curl that it's an IPv6 address and not a host:port pair. The quotes are required to stop the shell from treating the square brackets as a glob. The backslash is required to stop curl from treating the square brackets as a range specification. The http:// is optional, but good form. This isn't required if you use a hostname or an IPv4 address.

Finally, an issue lots of my fellow techies have been droning on about for years has reached public consciousness: the lamentable state of ICT lessons in schools and the lack of any programming or computer science stuff for learners before university age.

John Naughton in the Guardian last week:

What’s missing from both sides of this campaign is any appreciation of the real significance of introducing children to programming. Messrs Livingstone and Hope take an instrumental view of the matter, which is understandable given their industrial backgrounds. They are concerned that UK universities are not producing graduates with the skills that their industry needs now. […] But in a way they’re making the same mistake as those who saw ICT as a way of preparing kids for the world of work by training them to use Microsoft Office […] What governments don’t seem to understand is that software is the nearest thing to magic that we’ve yet invented. It’s pure “thought stuff” – which means that it enables ingenious or gifted people to create wonderful things out of thin air. All you need to change the world is imagination, programming ability and access to a cheap PC.

So, here we’ve got two justifications: Livingstone and Hope have a business justification. Naughton thinks we should teach proper computing in schools because it potentially enables kids to produce wonderful things that might change the world. I have a funny feeling the former might be more convincing for the Tories, and I’ve got no objection to fitting the message to the target audience.

But I’ve got a better reason than both, I think. The reason I think programming ought to be taught in schools is simple: it is real, practical, applied logic. There are people on philosophy courses who really struggle with the logic, and battle through logic textbooks, eventually mastering it. But if you write code, everything you learn in logic gets felt deep in your bones. The process of taking some kind of hand-wavy specification and turning it into a working machine, where the cogs and gears and pipes of that machine are pure logic? The government seem to want vocationally useful training that still has academic quality: unlike “leisure and tourism studies” or an A-level from McDonald’s, computer science gives you that.

Teaching people to code might lead them to create something cool, and it might mean that software companies can hire lots of talented programmers in the future. But the primary reason we should teach programming, in my opinion, is simply because it is a bloody good way of instilling logic into one’s mental toolkit, along with that nice, vague “problem solving” that is so often considered a “soft skill” or “key skill” or whatnot. There is a valuable moral lesson too: when you want to solve problems, you have to break them down and solve each step as logically as straight-forwardly as possible. That kind of attitude might help contribute to making society less dominated by utter bullshit.

Structuring a solution to a programming problem is essentially a pragmatic form of conceptual analysis. Writing decent, non-bloated code? Ockham’s Razor. Plenty of logic. Unit testing? Well, when did you last read an analytic philosophy paper that didn’t include a whole stack of intuition-prodding examples with which to compare proposed theories? If all the philosophers haven’t been forced to drink hemlock in order to save money in the higher education budget, and they find that they are particularly pedantic, they could even go study philosophy…

Traditionally, bitmap formats such as PNG have been the standard way of delivering iconography on websites. They’re quick and easy, and it also ensures they’re as pixel crisp as possible. Bitmaps have two drawbacks, however: multiple HTTP requests, affecting the page’s loading performance; and a lack of scalability, noticeable when the page is zoomed or viewed on a screen with a high pixel density, such as the iPhone 4 and 4S.

The requests problem is normally solved by using CSS sprites, combining the icon set into one (physically) large image file and showing the relevant portion via background-position. While this works well, it can get a bit fiddly to specify all the positions. In particular, scalability is still an issue. A vector-based format such as SVG sounds ideal to solve this, but browser support is still patchy.

The rise and adoption of web fonts have given us another alternative. By their very nature, they’re not only scalable, but resolution-independent too. No need to specify higher resolution graphics for high resolution screens!

That’s not all though:

• Browser support: Unlike a lot of new shiny techniques, they have been supported by Internet Explorer since version 4, and, of course, by all modern browsers. We do need several different formats, however!
• Design on the fly: The font contains the basic graphic, which can then be coloured easily with CSS – changing colours for themes or :hover and :focus styles is done with one line of CSS, rather than requiring a new graphic. You can also use CSS3 properties such as text-shadow to add further effects. Using -webkit-background-clip: text;, it’s possible to use gradient and inset shadow effects, although this creates a bitmap mask which spoils the scalability.
• Small file size: specially designed icon fonts, such as Drew Wilson’s Pictos font, can be as little as 12Kb for the .woff font. This is because they contain fewer characters than a fully fledged font. You can see Pictos being used in the wild on sites like Garrett Murray’s Maniacal Rage.

As with all formats though, it’s not without its disadvantages:

• Icons can only be rendered in monochrome or with a gradient fill in browsers that are capable of rendering CSS3 gradients. Specific parts of the icon can’t be a different colour.
• It’s only appropriate when there is an accompanying text to provide meaning. This can be alleviated by wrapping the text label in a tag (I like to use <b> rather than <span>, due to the fact that it’s smaller and isn’ t being used elsewhere) and then hiding it from view with text-indent:-999em.
• Creating an icon font can be a complex and time-consuming process. While font editors can carry out hinting automatically, the best results are achieved manually.
• Unless you’re adept at creating your own fonts, you’re restricted to what is available in the font. However, fonts like Pictos will cover the most common needs, and icons are most effective when they’re using familiar conventions.

The main complaint about using fonts for icons is that it can mean adding a meaningless character to our markup. The good news is that we can overcome this by using one of two methods – CSS generated content or the data-icon attribute – in combination with the :before and :after pseudo-selectors, to keep our markup minimal and meaningful.

Our simple markup looks like this:

<a href="/basket" class="icon basket">View Basket</a>

Note the multiple class attributes. Next, we’ll import the Pictos font using the @font-face web fonts property in CSS:

@font-face {
    font-family: 'Pictos';
    src: url('pictos-web.eot');
    src: local('☺'), 
    url('pictos-web.woff') format('woff'), 
    url('pictos-web.ttf') format('truetype'),
    url('pictos-web.svg#webfontIyfZbseF') format('svg');
}

This rather complicated looking set of rules is (at the time of writing) the most bulletproof way of ensuring as many browsers as possible load the font we want. We’ll now use the content property applied to the :before pseudo-class selector to generate our icon. Once again, we’ll use those multiple class attribute values to set common icon styles, then specific styles for .basket. This helps us avoid repeating styles:

.icon {
    font-family: 'Pictos';
    font-size: 22px:
}

.basket:before {
    content: "$";
}

What does the :before pseudo-class do? It generates the dollar character in a browser, even when it’s not present in the markup. Using the generated content approach means our markup stays simple, but we’ll need a new line of CSS, defining what letter to apply to each class attribute for every icon we add.

data-icon is a new alternative approach that uses the HTML5 data- attribute in combination with CSS attribute selectors. This new attribute lets us add our own metadata to elements, as long as its prefixed by data- and doesn’t contain any uppercase letters. In this case, we want to use it to provide the letter value for the icon. Look closely at this markup and you’ll see the data-icon attribute.

<a href="/basket" class="icon" data-icon="$">View Basket</a>

We could add others, in fact as many as we like.

<a href="/" class="icon" data-icon="k">Favourites</a>
<a href="/" class="icon" data-icon="t">History</a>
<a href="/" class="icon" data-icon="@">Location</a>

Then, we need just one CSS attribute selector to style all our icons in one go:

.icon:before {
    content: attr(data-icon);
    /* Insert your fancy colours here */
    }

By placing our custom attribute data-icon in the selector in this way, we can enable CSS to read the value of that attribute and display it before the element (in this case, the anchor tag). It saves writing a lot of CSS rules. I can imagine that some may not like the extra attribute, but it does keep it out of the actual content – generated or not.

This could be used for all manner of tasks, including a media player and large simple illustrations. See the demo for live examples. Go ahead and zoom the page, and the icons will be crisp, with the exception of the examples that use -webkit-background-clip: text as mentioned earlier.

Finally, it’s worth pointing out that with both generated content and the data-icon method, the letter will be announced to people using screen readers. For example, with the shopping basket icon above, the reader will say “dollar sign view basket”. As accessibility issues go, it’s not exactly the worst, but could be confusing. You would need to decide whether this method is appropriate for the audience. Despite the disadvantages, icon fonts have huge potential.

Most people haven't thought about philosophy since high school. Most people think philosophy is stuffy and too old fashioned to be relevant in today's world. Most people are wrong. Truth is, a major part of philosophy is critical thinking, and critical thinking is one of the most important tools in an entrepreneur's tool box.

Wikipedia says critical thinking reveals goals, examines assumptions, discerns hidden values, evaluates evidence, accomplishes actions, and assesses conclusions. All good things, right? Problem is our sneaky brains have ways to set traps for us, and these nasty traps can impair our ability to think rationally and make good decisions. Here's a list of 6 things that will challenge your brain into thinking critically: