I original wrote this as a comment on Victor Nicollet's blog, but I thought I should document it here too so that I can refer other people to the information.

These are some tips and tricks when using PDT with Zend Framework that make my life easier:

Autocompletion for dynamic properties

Zend_Db_Table_Row_Abstract uses __get() and __set() magic in order to map to the underlying database table row in question. This means that you can use property autocompletion on an instance of it. You can however tell your your table class which class to use for the row objects:

class Application_Model_Users extends Zend_Db_Table_Abstract
{
    protected $_name = 'users'; // database table name
    protected $_rowClass = 'Application_Model_User'; // row class to use
}

With your row class, you can take advantage of the @property docblock element to document the fields in the class:

/**
 * @property string $user_id
 * @property string $username
 * @property string $password
 */
class Application_Model_User extends Zend_Db_Table_Row_Abstact
{
}

Autocomplete on instances of Application_Model_User will now work.

Zend_View scripts are more complicated as they use $this to access magic properties. You can do however use the @var trick at the top of the .phtml file:

$user = $this->user; /* @var $user Application_Model_User */
$page = $this->page; /* @var $user Application_Model_Page */
etc...

You can now autocomplete on $user and $page. This also has the side-effect of documenting which view properties are used in this view script.

Opening an arbitrary method

View helpers are also magic methods on Zend_View's $this within a script file. This means that you can't cmd+click( or press F3) on the view helper's method name to jump directly to the code for that method. You can however take advantage of PDT's Navigation->Open Method feature. Simply select the method name and then press shift+cmd+m. The "Open Method" dialog appears and usually the view helper is selected as the first item. Just press return to go directly to the code for that view helper.

Opening an arbitrary file

For opening up an arbitrary file that's in my project, I find that PDT's Navigation->Open Resource (shift+cmd+r) is handy. For TextMate users, this is similar to the cmd+t feature.

Opening an arbitrary class

Using Navigation->Open Type (shift+cmd+t), you can also jump to classes easily. e.g. I regularly hit shift+cmd+t followed by "*Model_" and I get a list of all my models and can jump to the one I want using the down arrow key and then return.

Finding the current open file in the project tree

Just press cmd+option+w. You have to be careful here as it's very similar keyboard shortcut to close and close all! However if you have an entity file open and you name your files like I do, then the mapper is next to it in the project list, so it's only a couple of keystrokes to open it up.

Conclusion

These tips obviously work with all frameworks and any PHP coding :) Netbeans also has similar features, so I would guess all IDEs do. If you are wedded to vim or such like, then ctags is helpful for jumping around your code.

If you have a favourite tip that I haven't mentioned, let me know in the comments!

We've had more than our fair share of technical challenges along the way, but the time is right for me to talk about our newest product, the Quantum Compute Cloud, or QC2 for short.

This is the first production-ready quantum computer. You can use it to solve certain types of math and logic problems with breathtaking speed.

Ordinary computers use collections of bits to represent their state. Each bit is definitively 0 or 1, and the number of possible states is 2ⁿ. 1 bit can be in either of 2 states, 2 bits can be in any one of 4 states, and so forth.

Quantum computers such as the QC2 use a more sophisticated data representation known as a qubit or quantum bit. Each qubit exists in all of it's possible states simultaneously, but the probability that a qubit can be in any of the states can change. Quantum computers work by manipulating the probability distribution of each state.

How do you program a quantum computer? With quantum algorithms, of course. Pretty much everything that you know about traditional programming becomes obsolete when you step up to the QC2. You need to think in terms of probabilities, distributions of probabilities, and so forth. Take a look at Shor's Algorithm for finding prime factors to get a better idea of the power of a quantum computer.

We are also planning to support Bernhard Omer's QCL programming language. Take a look at his thesis on Structured Quantum Programming to learn more. Here's a QCL code sample to get you started:

Once you've launched a QC2 instance and loaded up your algorithm, you must sample the output (also known as "collapsing the quantum state") in order to retrieve the probability distribution which represents your answer. You'll want to do this more than once for any particular problem in order to increase your confidence in the solution. Collapsing the quantum state is a destructive operation (much like reading from a magnetic core memory); be sure to account for this in your algorithm. In effect, the answer doesn't exist until you ask for it.

Until now, the largest quantum computer contained less than 8 qubits. Because we're really, really smart, we've been able to push this all the way to 32 in the first-generation QC2. This will allow you to represent problems with up to 2³² distinct states.

We're launching the QC2 in the US East Region in multiple Availability Zones. using the amazing "spooky action at a distance" property of quantum entanglement, you can actually replicate QC2 instances across Zones.

The QC2 beta is limited, and will definitely close before the end of the day.

-- Jeff;

PS - We need to hire lots of world-class people to help us with leading edge technologies like QC2, EC2, and the like. Please check out our AWS jobs page.

Today Amazon Web Services takes another big step in making it easier to migrate legacy storage systems to the cloud through AWS Import/Export support for ingesting Punch Cards. AWS Import/Export accelerates moving large amounts of data into and out of AWS using portable storage media for transport. Punch cards are paper-based storage media that represent data using the presence or absence of holes in specific positions. With AWS Import/Export for Punch Cards, enterprises can begin using the service to preserve and unlock the large volumes of data that have accumulated over the last century on what was the first broadly adopted digital storage medium.

"Enterprises spent the better half of the 20th century accumulating critical data on punch cards," said Alyssa Henry, General Manager of Amazon S3. "Now that data can be easily transferred into Amazon S3, making it instantly accessible to customers and applications around the world."

AWS Import/Export for Punch Cards initially supports importing 80 and 96 column punch cards that conform to ANSI X3.26 and are encoded using ISO 6586 compliant 7-bit or 8-bit character sets. Customers may send up to 572 sorted cards per package and may send as many packages to AWS as they like. AWS Import/Export loads each card into Amazon Simple Storage Service (Amazon S3) as a distinct object, uniquely identified by batch and card number.

"We have decades of valuable data stored on punch cards that sit idle in storage cabinets, readable only by specialized, aging systems," said Matt Hunter, CTO of Hollerith Inc. "With AWS Import/Export for Punch Cards, not only are we eliminating large amounts of physical storage space, but we're actively leveraging that data for the first time in years."

For general information on AWS Import/Export, see http://aws.amazon.com/importexport. For details on supported punch card configurations, boxing requirements, and pricing of AWS Import/Export for Punch Cards, see http://aws.amazon.com/importexport/punchcards.

Ever wanted to use SVN to grab code from GitHub? Well, now you can, and just like that GitHub is the world's biggest Subversion host. Here's a few things you might like to do with it:

• Use any GitHub repository as an SVN external.
• Use TortoiseSVN or your favorite SVN tool to view Git repositories.
• Use existing build tools or continuous integrations systems that only work with SVN.

How's it work? Just checkout your project from svn.github.com:

svn checkout http://svn.github.com/schacon/simplegit.git

Give it a shot! For now it's only read-only, but who knows what will happen in the future!

In this video I walk through the basics of building Adobe AIR desktop applications in Flash Builder 4 using the Flex 4 SDK. Let me know what you think.

Load balancing is one of the technologies that virtually all our customers are using within EC2, and there is an increasing set of options available for doing it. We’ve been giving advice to our customers for years on what we’ve seen work but we finally decided to spend some time and do a real A-B benchmark comparison of a number of solutions. The result is a white paper Load Balancing in the Cloud: Tools, Tips, and Techniques we just published.

The test we ran compared the following solutions:

• HAproxy on an EC2 instance
• Zeus on an EC2 instances
• aiCache on an EC2 instance
• Amazon’s Elastic Load Balancing service

We focused purely on request rate scalability, i.e., how many requests per second the load balancer can sustain. We didn’t focus on feature set, bandwidth, or other metrics. So we ended up requesting a tiny web page over and over as fast as the system under test would serve up and we measure requests/second, or rather, responses/second. We also didn’t take advantage of more advanced features, such as caching in the load balancers, so we ran aiCache in a pure LB mode, for example.

Cutting to the chase, we ran the HAproxy, Zeus, and aiCache tests on an m1.large EC2 instance. After chasing down all kinds of options, trying to tune the kernel, trying other instance types, and finally conferring with AWS– the result is 100,000 packets per second in+out! I know, that’s not requests/sec or responses/sec so let me explain. Basically, with the current virtualization technology implemented in EC2 the network speed of light on an instance is getting about 100,000 packets per second through the two networking stacks of the host OS and the guest OS. Your load balancing solution and the tests you run may use these 100K pps to requests and responses in various ways which give you slightly different performance as measured in resp/sec. On average you can get about 5000 requests/sec through a load balancer. If you use HTTP1.1 persistent connections you get a few more resp/sec because there are a couple fewer packets per request, but the difference is not all that dramatic. If you turn some form of caching on you can roughly double the resp/sec because you’re eliminating the packets to the back-end servers. Tuning kernel params has very little effect, but pinning the load balancer process to a specific core does help quite a bit and makes the performance a lot more even. But in the end it’s all about pps (packets per second).

We then turned to ELB, the Amazon Load Balancing service. It operates differently from the other solutions in that it is a scalable service provided by Amazon. Everything ELB does for you can be replicated using the above solutions running on EC2 instances, but of course that requires extra work. Unfortunately benchmarking ELB is really tricky. One has to use many clients applying load, ensure that they requery DNS frequently, and run tests for a long time so ELB scales up. In the end we produced some pretty graphs like this, showing the requests/sec handled over time:

This shows how our tests ramped up to around 20K requests/sec over the course of about three hours (note that we ramped up the load slowly to see the progress, so this is not all time taken by ELB to ramp up). We could have continued higher but lost interest . I would prefer it if ELB were more transparent and easier to test, but it certainly delivers real world performance!

The whole benchmarking project was interesting in that it once again showed that until you really understand what is going on your benchmark is not done. We chased down more supposed performance bottlenecks than we care to remember and we drove the helpful folks at aiCache batty because they expected to see better performance given their results on non-virtualized machines. But in the end the results make a lot of sense and 100K pps is easy to remember.

I hope all this has whetted your appetite for the full white paper Load Balancing in the Cloud: Tools, Tips, and Techniques.

Apparently I started a collection ..
Of DNS books signed by their author ...

As already mentioned on twitter, Jan-Piet Mens kindly gave me a signed copy of his alternative DNS Servers book, last week at the UKUUG 2010 Spring Conference

When I got that book I didn't even know yet that this morning I was going to be at a seminar on DNS sec by Cricket Liu
and because of having registered so late for the event I certainly wasn't expecting to get one of the books that were going to be handed out to the first 50 people that signed up ... but I was lucky ..

So which book should I try to get next ? :)

Technorati Tags: authors books dns dnsproble dnssec eiafdp ukuuug

Share with Shareomatic!

A couple of members of the AWS Developer Support team put together the following tips and tricks to help you get the most from the Elastic Load Balancer.

-- Jeff;

Are you thinking about using Amazon EC2 with Elastic Load Balancing, but want to make sure you set it up right the first time? Are you already using an ELB but are seeing intermittent problems with your page loads? Well, you've come to the right place! Let's uncover a couple of common pitfalls. They're easy to avoid, once you know about them.

For those of you who aren't familiar, Elastic Load Balancing helps you distribute incoming network traffic across multiple Amazon EC2 instances. Your Elastic Load Balancer (ELB) will automatically route traffic to only the EC2 instances it deems to be healthy, so you needn't worry about manually enforcing which instances handle the traffic. With ELB, it's all manged for you. Furthermore, your ELB will also scale itself up and down to meet the demands of your traffic load. You can ensure that the EC2 instances themselves do the same by using Amazon Auto Scaling, but that's beyond the scope of today's discussion. You can read more about Auto Scaling here.

A key feature of ELB is that it will distribute incoming traffic equally across all of the Availability Zones you've configured it to use. This means that if you enabled, say, Availability Zones us-east-1a and us-east-1d, but only registered instances in us-east-1a, half of your traffic will go to us-east-1d, but there will be no EC2 instances there to handle it. The traffic will be redirected back to us-east-1a, but this redirection could increase latency for your users. Thus, you'll want to keep track of which Availability Zones your ELB is set up to use. You can use the ELB API command line tools to do this:

elb-describe-lbs --show-xml

If you don't already have the ELB command line API tools, then you can grab them here. Once you know which Availability Zones are enabled for your ELB, you can run this next command to see which instances are currently registered with your ELB:

elb-describe-instance-health Your_ELB_Name_Goes_Here

The command above will return the instance IDs of the registered instances, which you can then use to determine which Availability Zones each is in:

ec2-describe-instances instance_ID_1 instance_ID_2 ...

You can glean a lot of potentially useful information at this point regarding each of the instances behind your ELB. Here are some things to check:

1) Does each enabled Availability Zone contain at least one instance registered with your ELB?

If not, you have two approaches to remedy the situation. The quick fix is to simply disable the empty Availability Zones:

elb-disable-zones-for-lb  ELB_Name_Goes_Here -z Availability_Zone_Name_Goes_Here

The better fix is to register instances with your ELB in the empty Availability Zones:

elb-register-instances-with-lb  ELB_Name_Goes_Here -instances instance_ID_5 instance_ID_6

Great, so at this point you should have at least one instance in each of your ELB's Availability Zones. But could we still strengthen the setup even further? How about digging into the details of the individual instances behind your ELB? How robust is your configuration? This brings us to a second important item to check:

2) Do you have an equal number of instances in each Availability Zone? And are they the same type?

Since your ELB will distribute incoming traffic equally across your Availability Zones, you really don't want to have, say, one m1.large instance in us-east-1a and five c1.medium instances in us-east-1d. The single m1.large instance will receive roughly 50% of all of your traffic and, under high traffic volume, may not be able to keep up. Meanwhile, your five c1.medium instances are each under a much lower load. This is definitely a suboptimal arrangement.

The ec2-describe-instances command above returned not only the Availability Zone of each instance but also its instance type. We suggest populating each Availability Zone with an equal number of instances of the same type. You may even want to check if they are all based on the same AMI. Cycling out older instances for replacements based on your most recent AMI will help ensure that your instances remain up-to-date and service requests in a consistent manner, and can simplify debugging in the future.

We hope this helps you understand Elastic Load Balancing. Do you have more questions? Post them to the AWS forums!

For the past month, I've been immersed in PHP 5.3 as I and my team have started work on Zend Framework 2.0. PHP 5.3 offers a slew of new language features, many of which were developed to assist framework and library developers. Most of the time, these features are straight-forward, and you can simply use them; in other cases, however, we've run into behaviors that were unexpected. This post will detail several of these, so you either don't run into the same issues -- or can capitalize on some of our discoveries.

(I’m in the middle of traveling, but I know that a lot of people will be interested in the news that Google is incorporating site speed as one of the over 200 signals that we use in determining search rankings. I wanted to jot down some quick thoughts.)

The main thing I want to get across is: don’t panic. We mentioned site speed as early as last year, and you can watch this video from February where I pointed out that we still put much more weight on factors like relevance, topicality, reputation, value-add, etc. — all the factors that you probably think about all the time. Compared to those signals, site speed will carry much less weight.

In fact, if you read the official blog post, you’ll notice that the current implementation mentions that fewer than 1% of search queries will change as a result of incorporating site speed into our ranking. That means that even fewer search results are affected, since the average search query is returning 10 or so search results on each page. So please don’t worry that the effect of this change will be huge. In fact, I believe the official blog post mentioned that “We launched this change a few weeks back after rigorous testing.” The fact that not too many people noticed the change is another reason not to stress out disproportionately over this change.

There are lots of tools to help you identify ways to improve the speed of your site. The official blog post gives lots of links, and some of the links lead to even more tools. But just to highlight a few, Google’s webmaster console provides information very close to the information that we’re actually using in our ranking. In addition, various free-to-use tools offer things like in-depth analysis of individual pages. Google also provides an entire speed-related mini-site with tons of resources and videos about speeding up websites.

I want to pre-debunk another misconception, which is that this change will somehow help “big sites” who can affect to pay more for hosting. In my experience, small sites can often react and respond faster than large companies to changes on the web. Often even a little bit of work can make big differences for site speed. So I think the average smaller web site can really benefit from this change, because a smaller website can often implement the best practices that speed up a site more easily than a larger organization that might move slower or be hindered by bureaucracy.

Also take a step back for a minute and consider the intent of this change: a faster web is great for everyone, but especially for users. Lots of websites have demonstrated that speeding up the user experience results in more usage. So speeding up your website isn’t just something that can affect your search rankings–it’s a fantastic idea for your users.

I know this change will be popular with some people and unpopular with others. Let me reiterate a point to the search engine optimizers (SEOs) out there: SEO is a field that changes over time, and the most successful SEOs embrace change and turn it into an opportunity. SEOs in 1999 didn’t think about social media, but there’s clearly a lot of interesting things going on in that space in 2010. I would love if SEOs dive into improving website speed, because (unlike a few facets of SEO) decreasing the latency of a website is something that is easily measurable and controllable. A #1 ranking might not always be achievable, but most websites can be made noticeably faster, which can improve ROI and conversion rates. In that sense, this change represents an opportunity for SEOs and developers who can help other websites improve their speediness.

I know that there will be a lot of discussion about this change, and some people won’t like it. But I’m glad that Google is making this step, both for the sake of transparency (letting webmasters know more about how to do better in Google) and because I think this change will make the web better. My takeaway messages would be three-fold: first, this is actually a relatively small-impact change, so you don’t need to panic. Second, speeding up your website is a great thing to do in general. Visitors to your site will be happier (and might convert more or use your site more), and a faster web will be better for all. Third, this change highlights that there are very constructive things that can directly improve your website’s user experience. Instead of wasting time on keyword meta tags, you can focus on some very easy, straightforward, small steps that can really improve how users perceive your site.

Amazon just released a new service, the Simple Notification Service (SNS), which is a distributed message delivery service roughly similar to AMQP or JMS message services. It uses a publish-subscribe paradigm and supports push delivery of notifications using HTTP and email. It seems mostly targeted at back-end applications, e.g. servers sending notifications to one another, but given that is has an email delivery mechanism it can also be used to deliver notifications to users.

Jeff Barr wrote an article describing SNS, the key points are:

• you can create topics and publish messages to these topics
• others can subscribe to the topics and they will get messages pushed to them
• messages can be pushed over http or smtp (email)
• using access control policies one can control who is allowed to subscribe to a topic
• the SNS system is redundant and retries message delivery if necessary
• the cost at volume is $0.06 per 100,000 messages published and $0.06 per 100,000 HTTP message pushes. Email pushes cost 33x more. Good news is the first 100,000 message publish and pushes are free.

It really is great that AWS provides such a service. It’s relatively easy to fire up a messaging server, like RabbitMQ, but it’s a different story to set up a redundant scalable messaging system. While this can be done with RabbitMQ, for many users having this provided as a service is the right way to go.

Unfortunately SNS does not use a standard messaging API, it’s all proprietary. This is a major weakness of SQS, SNS, and SDB: once you use their interface you’re locked in to using AWS. Granted, the SNS interface isn’t particularly big, but then why did they have to roll their own?

My biggest beef with SNS is what is being said, or more precisely, not being said about reliability. I have no reason to believe that SNS doesn’t do all the right things, but AWS isn’t very forthcoming with specifics. Here is what the SNS docs state:

• “Reliable – Amazon SNS runs within Amazon’s proven network infrastructure and datacenters, so topics will be available whenever applications need them. To prevent messages from being lost, all messages published to Amazon SNS are stored redundantly across multiple servers and data centers.” source
• “Although most of the time each message will be delivered to your application exactly once, the distributed nature of Amazon SNS and transient network conditions could result in occasional, duplicate messages at the subscriber end.” source

So there is talk about redundant storage, at least once delivery, and delivery retries. But what I’d really want to know is not all this fuzzy feel-good stuff. The question is not that difficult:

• If SNS returns a HTTP “200 OK” to my publish request, what is the probability that each subscriber will receive at least one delivery attempt?

ok, I guess I really need to factor in time, which would also give an indication of performance:

• If SNS returns a HTTP “200 OK” to my publish request, what is the probability distribution over time that each subscriber has received at least one delivery attempt?

This would let me reason about what I can use SNS for and what not, or whether I need a back-up synchronization mechanism or not, etc. If the story stays at the warm&fuzzy level, AWS could at least specify when messages are stored redundantly, e.g. is a redundant copy stored by the time I get an HTTP “200 OK” response? Also specifics about how long and how often retries are made. (I’m focusing on the HTTP delivery, I don’t think it makes much sense talking about email delivery reliability.)

I hope that many others will also ask AWS to be more specific about what in the end is really the SLA offered by SNS (same goes for some of the other AWS services). I’m not asking for damages if the SLA isn’t met, I just want to know what AWS is publicly holding itself accountable for and thus what I can design for and apply my “trust in AWS” judgement to. While the service is in beta, the SLA might be a target.

Yesterday, while at the MySQL Conference, Robert Scoble interviewed Mark Atwood, our Director of Community Development, and Joaquin Ruiz, our EVP of Products, about our lastest release of Gear6 Web Cache and Gear6 Cloud Cache.

I am excited that Amazon Cloudfront has been selected as one of the 10 Editor's pick of 2010 by Streaming Media. Amazon Cloudfront is the Content Delivery Network (CDN) that is dead simple to use both from a technology and a business point of view. From a technology point it literally takes a single button click in the console or a single API call to have your content CDN enabled. You can find several websites marveling about how simple it is to CDN enable your content with amazon Cloudfront. On the business side Amazon Cloudfront revolutionized the CDN business by providing a true pay-as-you-go service, no longer requiring upfront commitments that are commonplace in the CDN business and which opens up CDN for everyone to use. Customers can switch on Cloudfront whenever they want and disable it when no longer needed and only get charged for what they need.

The Streaming Media editors singled out Cloudfront Streaming Content service as possibly truly disruptive. Here are the laurels given by the editors:

Debuting in November 2008, Amazon's entry into the CDN market quickly became a major player. It's still not a threat to Akamai or Limelight, but the addition in December 2009 of Flash streaming to its offerings could truly disrupt the market. Even without Flash, though, CloudFront was winning customers based on its pay-as-you-go rate structure and its self-service model; take a look at Larry Bouthillier's "How to Get Started With Amazon CloudFront Streaming" to get an idea of just how easy the service is to use. As Dan Rayburn wrote on his Business of Video blog, "Amazon will be in the driver's seat to own the market for small and medium sized content owners who need simple delivery at a great price."

For more information on Cloudfront visit their product page.

Yesterday marked the end of my tenure as curator of the Yahoo! Design Pattern library, a role I held for more than three years, longer than any of my predecessors. Speaking of which, let me take a moment to thank the following people:

• Erin Malone, who envisioned the library, championed it, and brought into being
• Chanel Wheeler, who built the core infrastructure in Drupal for the first-gen library
• Matt Leacock, the first curator, who established the processes and set very high standards for pattern shepherding
• Bill Scott, my immediate predecessor as curator, who brought the library to YDN (and hence to the public) and produced the influential set of rich interaction patterns that established the library’s reputation

When I started in January of 2007, the library had gone for a few months without active curation. A lot of the processes used to generate and publish patterns had gotten rusty. I did what I could to reform these processes based on what worked and what didn’t, and spent a fair amount of time reviewing and cleaning up unfinished and obsolete patterns. Meanwhile, I worked at pushing more patterns through the pipeline and publishing as many as possible.

There were 24 patterns in the public version of the library when I took over. Today there are 59. I also took it upon myself to delineate and describe a subset of social design patterns, a project that led to a book and an independent wiki.

When the library started, it was intended as an integral part of Yahoo!’s internal design processes, guidelines, and standards. It was created to be more flexible than a style guide and more focused on fundamental principles, as well as a tool for promoting consistency at Yahoo!.

Over time we found that the pattern collection excelled in the area of identifying universal design patterns. In recent years, the pattern library has been liberated to focus on general web-wide design patterns. This has given me a great deal of freedom to track down and document the patterns that I’m observing, creating a valuable reference that will be as useful for designers everywhere else as it is continues to be for Yahoo! designers.

The patterns in this library are shared with a Creative Commons license that permits anyone to take, reuse, adapt, and even commercialize any of the patterns in the library. The YUI Blog also includes many entries about Yahoo! patterns.

Curating this library, being the “pattern detective” has — beyond a shadow of a doubt — been the best job I have ever done in my life. I am brimming with gratitude toward the many fantastic people at Yahoo! who have advised me, helped me, guided me, challenged me, entertained me, and educated me. I will always be proud of the work we do here on behalf of the community of people who make the Web.

Christian Crumlish, @mediajunkie
Pattern Detective, 2007-2010

I recently interviewed web designer Justin Shearer (@justinsane98), Senior Web Designer at VC3 (@VC3inc). Justin described himself in the third person — which is a little creepy — but here it is: "Justin began designing for the web over a decade ago in Columbia, South Carolina. He started a small design firm that was eventually gobbled up by another startup once he realized that he should stick to design and leave business matters to others. He is currently the senior web designer at VC3 where he tries to be innovative and to create new functionality that clients and users don't even know they want yet! He is a designer by trade and a geek at heart who is fascinated by code, disturbed by Comic Sans, and loathes IE6 on a daily basis. He has a pet 'jaguar,' drives a Jeep that is probably older than you, and, oh yeah... he got the girl too!"

You wrote a jQuery plugin for YQL, how hard was that?

Actually this was my first jQuery plugin. After I wrote a couple of YQL widgets in just JavaScript I realized there had to be a better way of doing things with less code. The main thing I wanted to do accomplish was to be able to just copy and paste the YQL statement from the YQL console and have the plugin create that crazy REST query for me. All in all it was pretty easy.

What about now there is a plugin, how easy is it to get YQL from jQuery?

Super easy. It really enables me to take advantage of the YQL console. The tree view allows me to easily find the data I am looking for and makes it very clear what the path is to the specific piece of data. Looking at raw jsonp can be a bit overwhelming and this alleviates that.

What's your favorite feature of YQL?

YQL enables me to grab third-party data with no server-side coding. Thus I can take my widgets from platform to platform and CMS to CMS. The flexibility is amazing.

Your site is http://justinshearer.com/yql/. What do you plan to do with YQL next?

I am currently working on a Facebook feed reader, but my overall goal is to eventually create an all-in-one social media stream. One stream to rule them all! Kind of like the Facebook live feed but with the latest data from all of your social sites.

What's your favorite YQL table?

I'd have to say the generic XML or CSV tables for their flexibility, but I use the weather.forecast table a lot. I do a bunch of municipal government websites and they all want weather on their site. It takes me 30 seconds to swap out the zip code and put it on their site.

Is there a YQL table you wish existed?

Well personally, I am a bit of a football fan so I would like NFL players statistics so I could do some type of mashup with my fantasy team. But from a business perspective I would like to see tide tables. Being in South Carolina we have a lot of coastal towns and cities that would love to have that information displayed on their site. I have scraped the data off the boat and beach forecast on weather.com using XPATH, but in my mind it is way too hacky for a production environment.

Thanks, Justin!

Try out Justin's social media dashboard and grab his jQuery code samples for every piece to use on your own site. Or use the YUI3 YQL module or the YUI2 widget.

Tom Hughes-Croucher (@sh1mmer)
Yahoo! Technology Evangelist

Preloading components in advance is good for performance. There are several ways to do it. But even the cleanest solution (open up an iframe and go crazy there) comes at a price - the price of the iframe and the price of parsing and executing the preloaded CSS and JavaScript. There's also a relatively high risk of potential JavaScript errors if the script you preload assumes it's loaded in a page different than the one that preloads.

After a bit of trial and lot of error I think I came up with something that could work cross-browser:

• in IE use new Image().src to preload all component types
• in all other browsers use a dynamic <object> tag

Code and demo

Here's the final solution, below are some details.

In this example I assume the page prefetches after onload some components that will be needed by the next page. The components are a CSS, a JS and a PNG (sprite).

window.onload = function () {

    var i = 0,
        max = 0,
        o = null,

        // list of stuff to preload
        preload = [
            'http://tools.w3clubs.com/pagr2/<?php echo $id; ?>.sleep.expires.png',
            'http://tools.w3clubs.com/pagr2/<?php echo $id; ?>.sleep.expires.js',
            'http://tools.w3clubs.com/pagr2/<?php echo $id; ?>.sleep.expires.css'
        ],
        isIE = navigator.appName.indexOf('Microsoft') === 0;

    for (i = 0, max = preload.length; i < max; i += 1) {

        if (isIE) {
            new Image().src = preload[i];
            continue;
        }
        o = document.createElement('object');
        o.data = preload[i];

        // IE stuff, otherwise 0x0 is OK
        //o.width = 1;
        //o.height = 1;
        //o.style.visibility = "hidden";
        //o.type = "text/plain"; // IE 
        o.width  = 0;
        o.height = 0;

        // only FF appends to the head
        // all others require body
        document.body.appendChild(o);
    }

};

A demo is here:
http://phpied.com/files/object-prefetch/page1.php?id=1
In the demo the components are delayed with 1 second each and sent with Expries header. Feel free to increment the ID for a new test with uncached components.

Tested in FF3.6, O10, Safari 4, Chrome 5, IE 6,7,8.

Comments

• new Image().src doesn't do the job in FF because it has a separate cache for images. Didn't seem to work in Safari either where CSS and JS were requested on the second page where they sould've been cached
• the dynamic object element has to be outside the head in most browsers in order to fire off the downloads
• dynamic object works also in IE7,8 with a few tweaks (commented out in the code above) but not in IE6. In a separate tests I've also found the object element to be expensive in IE in general.

That's about it. Below are some unsuccessful attempts I tried which failed for various reasons in different browsers.

Other unsuccessful attempts

1.
I was actually inspired by this post by Ben Cherry where he loads CSS and JS in a print stylesheet. Clever hack, unfortunately didn't work in Chrome which caches the JS but doesn't execute it on the next page.

2.
One of the comments on Ben's post suggested (Philip and Dejan said the same) using invalid type attribute to prevent execution, e.g. text/cache.

var s = document.createElement('script');
s.src = preload[1];
s.type = "text/cache";
document.getElementsByTagName('head')[0].appendChild(s);

That worked for the most parts but not in FF3.6 where the JavaScript was never requested.

3.
A dynamic link prefetch didn't do anything, not even in FF which is probably the only browser that supports this.

for (i = 0, max = preload.length; i < max; i += 1) {
    var link = document.createElement('link');
    link.href = preload[i];
    link.rel = "prefetch";
    document.getElementsByTagName('head')[0].appendChild(link);
}

Then it took a bit of trial/error to make IE7,8 work with an object tag, before I stumbled into IE6 and gave up in favor of image src.

In conclusion

I believe this is a solution I could be comfortable with, although it involves user agent sniffing. It certainly looks less hacky than loading JS as CSS anyways. And object elements are meant to load any type of component so no semantic conflict here I don't believe. Feel free to test and report any edge cases or browser/OS combos. (JS errors in IE on the second page are ok, because I'm using console.log in the preloaded javascript)

Thanks for reading!

It is Tech Thursday and here are some links for you to hone your geek knowledge:

• Peter-Paul Koch discusses the issues of Pixel definitions on handheld devices
• The lastest Webkit builds now allow for Desktop Notifications via JavaScript
• Facebook's new Graph API makes it much easier to automate tasks in Facebook
• If you ever wondered about the number 42 as the answer to life, the universe and everything in the Hitchhiker's Guide to the Galaxy here is the rationale behind it.
• The White House released some cool open source extensions for Drupal
• Mozilla, Microsoft and Opera submitted the WOFF file format for web fonts as a standard.
• Google released a drawing tool for Google Docs that can match Omnigraffle. However it lacked wireframes, so Morten Just created a wireframe kit for Google Drawings.
• Stoyan Stefanov explains how to preload CSS and JavaScript without executing it - and no, he doesn't use iframes.
• The World Bank now also released their data for developers.

The new Tips and tricks for improving JavaScript performance article in the Forum Nokia wiki is a great resource for web developers who want their Web Runtime widgets to shine. The article is organized along practical lines, including these top tips:

• Startup performance is critical for a great user experience. Try combining your style sheets to decrease loading time and learn to be brutal when distinguishing between "startup" and "right after startup". The more you can move to "right after startup", the faster the response.
• General performance  perception is heavily influenced by the way you code scrolls. The article explains three options and when to use them: full redraw, copy scroll, and tile scroll.
• Watch your DOM  (Document Object Model) manipulations. Only create DOM elements when you need them, and consider carefully when you decide to destroy them.

The ecosystem of new application and platform services in the cloud is the future of application development. It will drive rapid innovation and we'll see a wealth of mobile, web and desktop applications arrive that we couldn't dream about a few years ago, and these building blocks are the enablers of that. These services will be delivered not only by new startups but also by enterprises looking to capitalize on their IP.

As examples of such services I always use Twillio (voice &sms) and Simplegeo (location), but it is time to start building out my knowledge of all the different services that are in the ecosystem. If you run such a service or know of one that I should be checking out, please leave the info in the comments below. I'll be using that information in presentations and some future writings on this topic.

This week, the Apache Software Foundation Board promoted Traffic Server to a top-level project (TLP). This is a big step going forward, as it recognizes Traffic Server as an active and well-functioning open-source community. There is nothing to stop us now!

Traffic Server is a fast, scalable, and extensible HTTP/1.1-compliant caching proxy server, which Yahoo! donated to the Apache Foundation. Since incubation inception last July, six new committers joined the project and the v2.0.0-alpha version was released. Some 170 Jira bug tickets have been resolved so far, and numerous new bugs and enhancement requests have been filed.

The final 2.0.0 release is almost ready, and should be available soon. The v2.0.x releases are all very close to the same code Yahoo! has successfully been using internally for years. It's fair to say that it is well tested.

In addition to the stable 2.0.x releases, we have a new developer-only release about to hit the download servers. We have adopted the HTTPD (and formerly, Linux) notion of even major release numbers for production releases and odd numbers for developer releases. So the next release after v2.0.0 will be v2.1.0. We hope to achieve a v2.2.0-beta release sometime this summer.

So let's talk more about what has happened during our incubation period, particularly related to the code. There has been a lot of clean-up and a number of significant improvements:

• The trunk is now 64-bit compatible. Not only can this speed up the server, but it also allows much more RAM to be allocated to the caches and allow us to handle more connections.
• Portability - the 2.0.x releases should build on most common Linux flavors. The upcoming 2.1.0 release also builds on many more UNIX platforms, such as Solaris, FreeBSD, and MacOSX.
• Small performance and latency related improvements were made in the 2.0.x releases. The 2.1.x development release promises even better performance: We've seen 2x or greater improvements, hitting well over 75,000 requests/second on a single box.
• Large disk support is ready and will be available in the next (2.1.x) release. This allows up to 0.5PB disk partitions for cache.
• There were many documentation updates. We even have a new home page that looks awesome (our product manager made me say that).

There is a lot of development activity in the community right now, which promises more features, capabilities, and performance in several important areas:

• Cache improvements not only give us support for vastly larger caches, they also noticeably improve performance and reduce latency.
• New additions to APIs include more scalable statistics, easier fetches of HTTP pages, and other important changes to enable new, exciting plugins to be developed.
• Significant work has been done to make it easy to use and deploy Traffic Server in cloud environments. For instance, there is a Amazon EC2 AMI available.
• Continuous improvements were made in portability, ease of use, and code cleanup. One goal is to make the code base smaller, and therefore easier to work with.

In the coming months, look for some plugins to be released, further extending Traffic Server's capabilities (and showing how extensible the server can be). The community is thriving, and we have a lot of fun, so come visit us in IRC, mailing lists, and community blogs. Things will change a bit now after graduation. So make sure you check out our new home page as we make the transition from a podling to a TLP.

Finally, we'll be talking about Traffic Server at Velocity 2010 and OSCon 2010. Hope to see you at these events.

Leif Hedstrom, Miles Libbey
Yahoo! Traffic Server