Adam Leach

I recently was asked what maximal amount transactions per second we can get using MySQL and XtraDB / InnoDB storage engine if we have high-end server. Good questions, though not easy to answer, as it depends on:

- durability setting ( innodb_flush_log_at_trx_commit = 0 or 1 ) ?
- do we use binary logs ( I used ROW based replication for 5.1)
- do we have sync_binlog options.

So why would not take these as variable parameters and run simple benchmark.
I took sysbench update_key scenario ( update indexed field on simple table)
and used Dell PowerEdge R900 with 16 cores, FusionIO as storage for table and RAID 10 with BBU as storage for innodb log files, innodb system table space and binary logs. And I used Percon-XtraDB-5.1.43-9.1 for benchmarks. All used partitions are formatted in XFS and mounted with nobarrier option.

I run update key for various threads and with next parameters

• trx_commit=0 : innodb_flush_log_at_trx_commit = 0 and no binary logs
• trx_commit=1 : innodb_flush_log_at_trx_commit = 1 and no binary logs
• trx_commit=0 & binlog : innodb_flush_log_at_trx_commit = 0 and binary logs
• trx_commit=1 & binlog : innodb_flush_log_at_trx_commit = 1 and binary logs
• trx_commit=1 & binlog & sync_bin : innodb_flush_log_at_trx_commit = 1 and binary logs and sync_binlog=1

• There are results I get:
  

  I found results being quite interesting.
  with innodb_flush_log_at_trx_commit = 0 maximal tps is 36332.02 tps, which drops to 23115.04 tps as
  we switch to innodb_flush_log_at_trx_commit = 1. As we use RAID10 with BBU, I did not expect the drops is going to be significant. In second case InnoDB spends

  With enabling binary logs, the results drops to 17451.01 tps with innodb_flush_log_at_trx_commit = 0 and to 12097.39 tps with innodb_flush_log_at_trx_commit = 1. So with binary logs serialization is getting even worse.

  Enabling sync_binlog makes things really bad, and maximal results I have is
  3086.7 tps. So this is good decision if binary log protection is worth such drop.

  UPDATE ( 3/4/2010 )

  Results with innodb_flush_log_at_trx_commit = 2
  
  Results with innodb_flush_log_at_trx_commit = 2 and binlogs
  

  ---

  Entry posted by Vadim | 7 comments

  Add to: | | | |

• Multi-master - high-scale applications have users around the world. Latency is reduced by distributing databases and application servers around the world. Databases are rarely sharded by location so the data store must support multi-master deployments with conflict resolution and eventual consistency for some database tables. There is no support for conflict resolution in MySQL. It might be possible to do something with the output of row-based replication.
• SQL - this is a problem that MySQL cannot fix. SQL makes it easy to make mistakes. Mistakes include insert, update and delete statements that lock all rows in a table. Alas, the EXPLAIN statement in MySQL does not support insert, update and delete statements. Another serious mistake is a query that has a lousy response time when the database buffer cache is cold because it does many random disk reads. The EXPLAIN statement in MySQL does not provide an estimate for the worst-case number of random disk IOs and many people who write SQL don't know how to interpret it to get an estimate. Worst-case performance is critical for queries run during web requests.
• Write-optimization - several NoSQL systems are write-optimized including Cassandra, HBase and Bigtable. A write-optimized system makes it possible to use more indexes than an update-in-place data store. With more indexes it is more likely that there can be an index defined for every popular query and the index reduces the number of disk reads that must be done to evaluate the query.  This improves worst-case query response time and reduces the need to use memcached or a huge database buffer cache. Write-optimization has finally arrived for MySQL with the availability of TokuDB. I hope that RethinkDB provides a GA version in the future.
• Monitoring - without good monitoring you will either spend too much time fixing performance problems or never find them and buy too much hardware. I suspect that monitoring in MySQL is much better than anything in a NoSQL system but MySQL is missing features that make it easy to understand current and new sources of workload. I need to aggregate the overhead (CPU time, disk operations, rows read, ...) by database user, table and statement. It is extremely hard or not possible to do this by database user and table. It became possible in MySQL 5.1 to do this by statement for short periods of time by using the slow query log in MySQL 5.1. Prior to MySQL 5.1, the slow query log was limited to queries that ran for at least two seconds. The alternative is to use tcpdump with mk-query-digest. Despite all of the work that has gone into the performance schema, MySQL has yet to support anything like my favorite feature -- user and table monitoring.
• Crash proof slaves - replication slaves are not crash proof. The slave commits transactions to a storage engine and then updates a state file to maintain the replication offset. There is nothing to keep the state file and storage engine in sync. Until recently there wasn't even an option to force the state file to disk after it was updated. Unplanned hardware reboots are frequent when there are hundreds or thousands of slaves. If you are clever and use the right version of InnoDB it is possible in some cases to figure out the correct offset for the slave after a crash and repair it manually. This isn't a good use of DBA time. Otherwise DBAs must waste their time and network bandwidth to restore the slaves. The Google patch published two different fixes for this: rpl_transaction_enabled and global transaction IDs. MySQL is working on a fix.
• Automated failover - for MySQL deployments that have many slaves connected to one master it isn't possible to automate failover when a master crashes. Tungsten and DRBD might make this better. 
• Resharding - sharding is an excellent way to scale MySQL. Sharding usually requires resharding. Resharding is hard and must be done with minimal downtime. It might be possible to build a tool that uses row-based replication output to reshard a database in the background with little downtime. No such tool exists today.
• Replication lag - a slave with replication lag is useless for OLTP scaleout. The replication thread is single-threaded. MySQL is working on support for parallel execution on a slave. Until then we need to improve mk-slave-prefetch (Domas can you hear me).
• Schema change - these are frequently needed for growing high-scale applications. Long running schema changes in MySQL require downtime unless first done on a slave (assuming you have a spare slave and that slave can become the master after the change). Users don't like downtime. I think it is possible to do many of these on a master with minimal downtime using the output from row-based replication. Alas, there is no tool for that today.

• Crash safety - most NoSQL systems are crash safe but a few are not. I would limit the use of systems that are not crash safe to supporting batch workload. Unplanned server reboots are frequent for high-scale applications when a large number of servers is used. At least two prominent members of the NoSQL family are not crash safe. That should be documented in bold text on their project pages. It is not.
• Sharding - some NoSQL systems do sharding. BigTable and others do not. With sharding it is possible to support transactions and multiple-indexes on a table within the scope of a shard. That then requires support for resharding. It also requires that queries on secondary indexes to be run on all shards while queries on primary indexes can be limited to run on one shard which may limit the ability to use secondary indexes.
• Index types - Many NoSQL systems are limited to hash indexes. You can't do range scans on hash indexes. I wonder whether this leads to data redundancy when every query must be resolved by one index lookup.
• Secondary indexes - NoSQL systems like BigTable not only do not support transactions, they also do not support secondary indexes. You can explicitly maintain a secondary index but there is no support to make multiple-changes atomic and there can be a failure between the primary and secondary index updates which results in data drift. It is also difficult to do consistent reads between the two.
• Consistent reads - consistency is usually the responsibility of the client and done by using per-row timestamps.
• Single-node performance - I know that performance != scale-out but scale-out is not a substitute for lousy single-node performance in the high-scale application market. It might be acceptable to use 5X as many nodes because your data store is slow when you end up using 40 nodes. This becomes a show-stopper when you end up using thousands of nodes. One NoSQL system has accepted this compromise. While I know it has many other use cases I think that will limit the use of it for high-scale applications.
• Network efficiency - MySQL reduces use of the network because all query evaluation is done at the server. All NoSQL systems evaluate predicates implied by the index access. Only some NoSQL systems evaluate non-indexed predicates. This can result in more data returned to the client.
• Technology or solution - MySQL is more mature than the NoSQL systems. A lot of work remains to grow NoSQL from technology into solution with support for audit, backup, monitoring and all of the other things required to scale in a large company.

Git's branching model is one of it's best features. Branches are cheap, fast and extremely flexible. They're great for developing features, maintaining old releases, or just plain experimentation.

If you spend a lot of time with git, you'll also find that there's a lot of really useful information to be discovered in the way git can compare various branches. We're using some of this information to generate our new branch list page (you can get to this page from the 'Branch List' subnav item under Source).

Check in on your topic branches in one glance

Not only do these new branch list pages show you which branches exist on your remote, but you can see at a glance how they compare to any branch.

Each branch has what we call a divergence graph. On the left side of the black bar we show how many commits that branch is behind (commits in master not found in the branch). On the right side, we show how many commits that branch is ahead (commits found in the branch, but not in master). The colors of the bars indicate how recent the last commit was.

In that one graphic, you get an idea of when the last time each branch was updated with master, how far along that branch is, and if people have been working on it recently.

These graphs also have some implicit rules that can help when merging:

• No left side bar - This branch is safe to merge with your base branch and there will not be any conflicts.
• No right side bar - This branch has already been merged into your base branch, so it's probably ready to be deleted.
• Large left and right side bars - This branch is probably going to be difficult to merge. There's a lot of unique commits in the branch and in your base branch so the likelihood of merge conflicts is much higher.

Having fun with Rails releases

This view can also be fun to glean some information out of Rails releases. Rails keeps a branch for each point release. If we take a look at the branches with 2.2 as the base, we get a pretty interesting page

Using the divergence graphs, we can see how each of the point releases of Rails compare to the 2.2 release. You can also see that there was almost as many commits from 1.2->2.2 as there has been from 2.2->master (Rails 3 beta).

Compare View

The last piece of the branch lists page is the compare button on each branch. This is an awesome feature--but I think I'll leave it to Ryan to explain in more detail.

Picking up where Kyle left off in his Branch List post, we're all very excited to announce a new feature designed to ease the process of comparing two points in a repository's history. It's called GitHub Compare View and it's going to change the way you review code.

The Compare View brings all information needed to determine what changed over a series of commits onto a single page: a condensed commit list in chronological order, followed by a rollup diff of all changes between the two points, followed by any relevant commit comments. All in the same place and with a single well-defined URL.

It's a versatile feature with many potential uses. Some of our favorites are detailed below with links to live examples of Compare View in action.

Reviewing topic branches before merging

Reviewing topic branches is a fundamental activity for anyone maintaining an active open source project or working with a team on a private project. Until now, the easiest way to do a thorough review of a complex topic branch was to drop down to the shell and run some combination of the git log, git cherry, and git diff commands. With Compare View, we've taken that process and put it behind a single URL, so jumping into the review process is usually as simple as following a link.

Follow the examples below to see it live:

• jQuery require support branch (pictured above)
• Mustache pragma support branch
• Homebrew Pathname.rename branch

Generating comprehensive change logs for releases

While Compare View was initially designed to supplement our code review process, we quickly found that it was useful in a variety of other scenarios. Projects that are diligent about tagging can use Compare View to generate a comprehensive list of changes between any two releases.

A few examples in the wild:

• Changes between the Sinatra 0.9.4 and 1.0.a releases (pictured above)
• Changes between the Rails 3.0 beta and current master
• Changes slated for the next version of Git

How to customize the commit range

The commit range determines the starting and ending point to use in the comparison. It looks like this:

Here, we're reviewing the proxy_owner topic branch using the master branch as the starting point. Hitting the button switches the starting and ending points. Clicking the starting or ending point label brings up the ref selector:

Enter any branch, tag, or commit SHA1 in the little box there and get an instant preview of the newly selected revision. Hit Save & Refresh once you're happy with the selection.

Getting There

We've seen what the Compare View feels like and how to change the commit range once there, but how do you get to a Compare View in the first place?

• The Branch List page. Click the Compare button next to any branch to jump into a Compare View with that branch as the ending point. See Kyle's post introducing the Branch List page for more information.

• Push and Branch Create events. All push events with more than one commit now link to a Compare View over all commits included in the push. All create branch events now link to a Compare View between the repository's default branch (typically master) and the branch head. This effects dashboards, repository timelines, and activity feeds.

• Service Hooks. The IRC and Campfire service hooks now drop a Compare View link when more than one commit is included in the push. More service hooks will follow.

Compare View URLs

We wanted Compare View to be something we could link to from external sites and services anytime we were referring to a range of commits in a git repository. As such, we thought it would be worthwhile to document the basic structure of a Compare View URL:

http://github.com/<USER>/<REPO>/compare/[<START>...]<END>

Where <USER> and <REPO> are obvious, and <START> and <END> are branch names, tag names, or commit SHA1s specifying the range of history to compare. If <START> is omitted, the repository's default branch is assumed.

Big Plans

Compare View is the first of many code review related features we plan to introduce this year. We'll be incorporating Compare View into other areas of the site and developing entirely new features with Compare View as a core component.

That being said, we felt it was important that the basic task of comparing two points in a repository's history stand on its own and have a well defined URL so that the feature is useful in a broad number of circumstances. If everything goes to plan, you'll be seeing Compare View links tossed around on mailing lists, forums, IRC channels, Campfire, blog posts, release announcements, issue trackers, etc. in the very near future.

The first talk I gave after arriving at Yahoo! was entitled Maintainable JavaScript (video). As with most topics I write or speak about, I didn’t think it would be terribly controversial. The basis of the talk is that hacking around on your own and writing code in an enterprise environment are two different things. Web developers are truly unique in that none of us learned what we know in school; we all began as hobbyists one way or another and taught ourselves most (if not all) of what we know.

Professionalization

The professionalization of web development has been a difficult journey because of our disparate beginnings. Even those who end up at large companies such as Yahoo! inevitably began on their own, hacking around. Perhaps you were even “the web guy” at a small company and could do pretty much whatever you wanted. When the large companies started tapping this previously undiscovered resource, it brought a lot of hackers into a corporate environment where they were met with constraints. No longer a lone soldier in a small battle, all of these self-taught, self-directed individuals had to figure out how to work together as a team.

At the time that I gave the talk (2007), web development was evolving into front-end engineering and people were having trouble with the transition. Smart folks like Nate Koechley talked about the professionalization of front-end engineering (video) and how our discipline was evolving. My talk was aimed at the same goal: helping front-end engineers adapt to JavaScript development in a team environment by making sure that their code was as maintainable as possible.

Why can’t I modify objects I don’t own?

I still gets email and comments about Maintainable JavaScript, and the most popular question is, “why can’t I modify objects I don’t own?” JavaScript is, of course, a dynamic language that allows you to add and remove objects and their members at any point in time. For many, this is precisely why they enjoy the language: there are very few constraints imposed by the language. And I was telling them not to do this. Why?

Dependability

The simple explanation is that an enterprise software product needs a consistent and dependable execution environment to be maintainable. In other languages, you consider already-existing objects as libraries for you to use in order to complete your task. In JavaScript, people saw already-existing objects as a playground in which you could do anything you wanted. My point was that you should treat the already-existing JavaScript objects as you would a library of utilities. Don’t override methods, don’t add new methods, don’t remove existing methods.

When you’re the only one working on a project, it’s easy to get away with these types of modifications because you know them and expect them. When working with a team on a large project, making changes like this cause mass confusion and a lot of lost time. I still remember a bug that occurred while working on My Yahoo! because someone had overridden YAHOO.util.Event.stopEvent() to do something else. It took days to track this problem down because we all assumed that this method was doing exactly what it always did. Once we discovered this, we also found other bugs because the same method was being used in other places with its original intended usage…but of course, it wasn’t behaving in that way. Unraveling this was an incredible mess and I’d be very happy if no engineers ever had to go through a similar exercise.

Incompatible implementations

But developer confusion isn’t the only problem. Another peril of modifying objects that you don’t own is the possibility of naming collisions and incompatible implementations. Take a lesson from the history of the Prototype JavaScript library. John Resig wrote about this a while ago, so I’ll just quickly summarize. Prior to version 1.6, Prototype implemented its own document.getElementsByClassName() method long before it was part of HTML5 and long before any browser thought about implementing it natively. In addition, Prototype also added the each() method to Array objects. Thus, users of the Prototype library began writing code such as:

document.getElementsByClassName("myclass").each(doSomething);

This wasn’t a problem until the native document.getElementsByClassName() method was implemented. While Prototype’s version returned an instance of Array, the native implementation returns a NodeList object. Since NodeList doesn’t have an each() method, either natively or added by Prototype, the above coding pattern caused a JavaScript error when executed in browsers that had a native implementation of document.getElementsByClassName(). The end result is that users of Prototype had to upgrade both the library code and their own code; what a maintenance nightmare.

What if everyone did it?

Looking at a few isolated examples doesn’t really represent the enormity of the maintenance problem when you modify objects that you shouldn’t. To understand this point of view, it’s helpful to take a step back and look at moral philosophy (aka ethics). Moral philosophy is all about determining if an action is moral. There are many schools of thought on the topic, but I point towards a favorite modern philosopher, Immanuel Kant.

While I don’t want to get too deeply into moral philosophy and open this up for philosophical debate, Kant was famous for trying to determine “universal law” as the basis for moral action. In short, you can determine if an act is moral by asking, what would happen if everyone did it? For example, what if everyone cheated on a test? In that case, the test becomes useless, so this must not be a moral action.

Applying this same line of reasoning to the topic at hand, what if everyone on your team started modifying objects that they didn’t own? What if I went in and made modifications to document and so did everyone else on my team? What if everyone on the team created their own global variables? I hope that it’s obvious just how detrimental these actions could be to a team development environment.

Simply put: if everyone on your team modified objects that they didn’t own, you’d quickly run into naming collisions, incompatible implementations, and maintenance nightmares.

As a side note, I find Kant’s question incredibly relevant to any system that must scale. “What if everyone did it?” can really save you some trouble when considered as part of a technical design.

Conclusion

Maintainable code is code that you don’t need to modify when the browser changes. You don’t know how browser developers will evolve existing browsers and the rate at which those evolutions will take place. The code you write needs to continue working in future browsers and with future versions of JavaScript libraries without modification, and you cannot ensure that when you’re modifying objects that you didn’t create in the first place. The only code you can be certain will remain the same is the code you write yourself.

I can’t state this strongly enough: your code is not maintainable when it requires modifications to objects you didn’t create. Stepping down that path only leads to maintenance nightmares going forward.

P.S. If you’re interested in learning more, check out my presentation on Scalable JavaScript Application Architecture (video).

Related posts

• Answering Soshnikov’s quiz
• My JavaScript quiz - answers
• My JavaScript quiz
• Announcing High Performance JavaScript
• How Internet Explorer 8 document mode affects JavaScript
• Answering Baranovskiy’s JavaScript quiz

P.S. My new book, High Performance JavaScript is coming out in early 2010. Need help now? Check out Professional JavaScript, 2nd Edition.

Late last week I met Jim Kaskade of SIOS at a Seattle-area Starbucks for a meeting and a product demo. With the very cool (and appropriate) title "Chief of Cloud", Jim was the right person to demonstrate his company's new cloud-powered high availability and disaster recovery solution.

Jim's Mac laptop was running Centos. He used Xen and Red Hat's Virtual Machine Manager to host a couple of virtual machines representing the web, application, and database tiers of a SugarCRM installation. Each of the guest operating systems was running a copy of the new SIOS CloudStation product. Each copy of CloudStation was configured (using a web-based GUI) to replicate the state of the virtual machine to an Amazon EC2 instance running in a user-selected Region.

Once everything was up and running, Jim showed me how he could selectively kill the local virtual machines while keeping the application running. The demo was designed to feature a very short RPO (Recovery Point Objective) so that changes made locally just seconds before the database was killed were available from the cloud-based virtual mirror. Jim walked me through a number of different failure and recovery scenarios.

It was quite impressive and makes a great demo of the cloud-based DR (Disaster Recovery) and HA (High Availability) that I've been telling my audiences about for the last couple of years. Once configured, CloudStation can fail over from local processing to the cloud, from one cloud region to another, or even from one cloud provider to another. It can also be used as a migration tool, or what is sometimes calls P2V (Physical to Virtual) or P2C (Physical to Cloud).

Read more in the Solution Brief (PDF) or sign up for the March 24th webinar.

-- Jeff;

I was working on an internal product here at Yahoo! that required users to input time-of-day in a specific format. I decided that rather than force users to type exactly the right format it would be easier just to provide a UI widget for time input. I’ve always liked the jQuery timepicker; it’s a simple and fast way to input time and meets my use case perfectly. Of course, we were using YUI 3, so I decided to recreate the widget with YUI 3. (This is quick and easy with the YUI 3 Widget foundation.) I thought it might be useful to others working with YUI, so I decided to give it right back to the community for use in your own projects.

Using the picker should be pretty simple for you if you are familiar with the basics of YUI 3. (see a live version here).

To use the picker in your own project include the script:

        <script type="text/javascript" src="http://yui.yahooapis.com/combo?3.0.0/build/yui/yui-min.js&gallery-2010.02.25-22/build/gallery-timepicker/gallery-timepicker-min.js"></script>

Then instantiate and render the widget:

YUI().use('gallery-timepicker', function(Y){
        //Pass a configuration object to the timepicker
       var picker = new Y.Saw.Timepicker(
           {
                           //an element that will contain the timepicker
               contentBox: 'div.foo', 

                           /the initial time
               time:{
                   hour:0,
                   minute:0
               },
               strings:{
                   am:'AM',
                   pm:'PM',
                   seperator:':'
               },
               delay:5 //delay before selecting a box on mouseover
           }
       );
      picker.render();
});

Like all YUI 3 widgets the timepicker constructor takes a configuration object to control the initial display of the widget. Manipulating the widget is then done via the widget methods render, hide and show. The render method is where the actual DOM elements are created. hide and show simply add and remove the class yui-timepicker-hidden to the elements bounding box. This class (and the additional css classes for the widget) must be implemented for the widget to behave properly. For simplicity’s sake, here are the styles I am using on the running example:

        /* yui reset assumed */
        .yui-timepicker{
        display: block;
        margin: 5px;
        left: 0;
        position: relative;
        background: transparent;
    }

    /* standard for widgets, in this case just pushing the hidden one off the screen*/
    .yui-timepicker-hidden{
        left: -9999em;
        position: absolute;
    }

    .yui-timepicker{
        color: #000;
        font-family: verdana;
        text-align: left;
    }

/* the picker is actually two ordered lists */
    .yui-timepicker ol{
        display: block;
        position: relative;
        left: 0;
        .left: 5px;
        margin: 0px;
        padding: 0px;
        height: 24px;
        text-align: left;
        -webkit-transition: left .1s ease-in;
    }

    .yui-timepicker li{
        list-style: none;
        display: block;
        float: left;
        position: relative;
        left: 0;
        overflow: hidden;
        width: 19px;
        padding: 1px;
        margin: 0 2px 0 0;
        border: 1px solid #999;
        text-align: center;
    }

    .yui-timepicker li{
        -moz-border-radius: 2px;
        -webkit-border-radius: 2px;
    }

        /* highlight the selected times */
    .yui-timepicker li.yui-timepicker-active{
        background: #000;
        color: #fff;
        -moz-box-shadow: 2px 2px 2px #ccc;
        -webkit-box-shadow: 2px 2px 2px #ccc;
    }

I’m using webkit animations just for style; for your project, customize these styles as you see fit. For most purposes you will want to hide the picker initially. Calling the hide method just adds the yui-timepicker-hidden style to the widget’s bounding box. I’ve added a click handler to my wrapper element so that a click on the element with the id time will cause the widget to appear/disappear:

         picker.hide();
     Y.get('#main').on('click', function(e){
         var target =e.target;
         if(target.test('#time')){
             picker.toggle();
         }
     });

To make the picker actually useful I will listen to the

timeset

event, which returns an object with information about the selected time, I’m going to use the “s24hour” member of the object passed to the handler. That’s a string representation of the time in 24 hour format. (also available are hour, minute, ampm and s12hour):

        picker.subscribe('timeset', function(e){
        //timeset is a custom event that fires when the time is *set*
        //use this rather than timeChange
        Y.get('#time').set('value',e.s24hour);
    });

//add a handler to "cell click" to hide the picker when the user clicks on a cell
    picker.subscribe('cellclick', function(e){
       this.hide();
    },picker);

That’s all there is to it! Enjoy.

Because I'm becoming known as someone who knows something about "this whole NoSQL thing", people have started asking me to take a look at some of their systems or ideas, and tell them which NoSQL technology they should use.

To be fair, it is a confusing space right now, there are a LOT of NoSQL technologies showing up, and there is a lot of buzz from the tech press, and in blogs and on twitter. Most of that buzz is, frankly, ignorant and uninformed, and is being written by people who do not have enough experience running live systems.

Today, we are launching a new feature on the Yahoo! Developer Network called the Create Consumer Key API. A few weeks ago we posted a sneak peek. This blog post touches again on what it is, who should use it, and adds links to API documentation and example integrations.

About the CCK API

Many developers who use Yahoo! Web services and APIs distribute applications that allow their users to access their own Yahoo! user data. This requires you as the developer (the third party) to have your own OAuth Consumer Key and Secret, while ensuring each of your users (the fourth party) also have their own distinct OAuth credentials. The Consumer Key ensures that the right data, for the right user, is going to the right developer.

The Create Consumer Key (CCK) API allows developers to easily provide Yahoo!-issued Consumer Keys, Consumer Secrets, and Application IDs to their users.

The CCK API offers the following benefits to developers:

Streamlines the process of using your application: Your users do not separately request an OAuth Consumer Key from the Yahoo! Developer Network (YDN).
One call: Simply create one form providing the necessary information for a Consumer Key, such as access scopes and redirect URLs.
One-click authorization: Your users do not need to know about access scopes and you have the option to automatically send Yahoo! Information back to your application with no copy-and-pasting necessary for your users.

Use Cases

Developers building applications they plan to distribute (and which require access to private Yahoo! user data) should consider integrating with the CCK API. Examples of distributed applications might include a JavaScript sharing widget that interacts with Yahoo! Contacts or a Wordpress plugin that blasts new blog posts into Yahoo! Updates, etc.

Getting Started

The Create Consumer Key API has been documented on the YDN site and includes sample code to get you up and running quickly.

If you need help, or would like to suggest ways of improving this service, please post a message in the YDN OAuth forums or message us @YDN on Twitter.

Example Integrations

JanRain’s RPX is a hosted software solution that makes it easy to add Yahoo! based sign-in on your website using OpenID, adds social publishing from your website to Yahoo! Updates and Status APIs, and provides access to a Yahoo! Users Contacts. With the Create Consumer Key API, JanRain users now have a simple, streamlined flow for configuring their RPX integration with Yahoo!. JanRain posted about their integration here.

We've also created a WordPress plugin that connects your WordPress blog with your Yahoo! account. When you publish a new blog post it will share an update that's visible to your Yahoo! connections.

Daniel Raffel
Senior Product Manager
Yahoo! Open Strategy

One of the dangers of frameworks in general is that you forget that they do lots of handy things for you.

Consider Zend Framework's default error controller:

   public function errorAction()
    {
        $errors = $this->_getParam('error_handler');
        
        switch ($errors->type) {
            case Zend_Controller_Plugin_ErrorHandler::EXCEPTION_NO_ROUTE:
            case Zend_Controller_Plugin_ErrorHandler::EXCEPTION_NO_CONTROLLER:
            case Zend_Controller_Plugin_ErrorHandler::EXCEPTION_NO_ACTION:
        
                // 404 error -- controller or action not found
                $this->getResponse()->setHttpResponseCode(404);
                $this->view->message = 'Page not found';
                break;
            default:
                // application error
                $this->getResponse()->setHttpResponseCode(500);
                $this->view->message = 'Application error';
                break;
        }
        
        // Log exception, if logger available
        if ($log = $this->getLog()) {
            $log->crit($this->view->message, $errors->exception);
        }
        
        // conditionally display exceptions
        if ($this->getInvokeArg('displayExceptions') == true) {
            $this->view->exception = $errors->exception;
        }
        
        $this->view->request   = $errors->request;
    }

The error handler in ZF will catch any exceptions and route them to the error action in the error controller. This then sets the correct HTTP response code, logs the error and optionally displays it if a config setting is set.

Obviously on our production boxes, we don't display the exceptions, but we do on our local development machines.

IIS has the concept of custom error pages that it displays when the app returns a 4xx or 5xx status code:

There is also some settings for this page:

By default this is set so that if you access the site using the localhost domain, then you'll get the ZF error page and if you access the site remotely then you'll get the IIS custom page and won't see the error.

If, like me, you are developing with your IIS in a VM and using the host OS's browser and developer tools, then you need to change the setting to "Detailed":

Now you can see your exceptions and it doesn't look like PHP has crashed :)

I gave a number of talks this spring on jQuery and especially on some of the recent additions made in jQuery 1.4. Below are all the slides and demos that I've given.

The conferences / meetups that I spoke at (or will speak at, in the case of MIX), and the talks that I gave, are as follows:

• Webstock (Wellington, NZ) (Introduction to jQuery Workshop, Things You Might Not Know About jQuery)
• Future of Web Apps (Miami, FL) (Introduction to jQuery Workshop, Improve Your Web App with jQuery)
• jQuery Boston Meetup (Boston, MA) (Things You Might Not Know About jQuery)
• MIX (Las Vegas, NV) (Improve Your Web App with jQuery)

Introduction to jQuery Workshop

This workshop starts with an introduction to the fundamentals of jQuery (1 hour) and continues on with two pieces of hands-on coding (Todo list, 30 min, Social Networking Site, 1.5 hours).

Source Code

In the workshop I also had two pieces of hands-on coding. The first was an ajax-y todo list the second was converting a functional social networking site into a one page application (making significant use of jQuery UI).

Source Code Reset Demo Edit Demo

Source Code Reset Demo Edit Demo

Things You Might Not Know About jQuery

A variety of things that people don't know about in jQuery - including new things added in jQuery 1.4 (and newer), data bindings, custom events, and special events.

Source Code

For the first jQuery Boston Meetup I built a game using the avatars of everyone in attendance. Sort of a space shooter style game you need to kick and kill the advancing hordes of users. I used this game as a way of demonstrating constructing an application that makes use of custom events, data binding, and building applications in an event-centric manner.

Source Code

Improve Your Web App with jQuery

A different restructuring of the previous talk that emphasizes a more holistic approach to improving your web applications with jQuery

Source Code

I've been messing around with a new piece of presentation software that I wrote for these talks. It's still terribly crude and buggy (pretty much just got it working enough in order to run my talks in Firefox 3.6 and Chrome) - you've been warned. I hope to refine it at some point and release it for general consumption.

The need to save dependent or child models appears frequently in object oriented applications. In such cases, it’s usually desirable to group all the save operations for a parent and its children into a single atomic transaction. Many RDBMS brands support transactions, but few can handle these kind of “nested” transactions.

This article discusses how to extend Zend_Db_Adapter to give the illusion of nested transactions – a handy workaround for use in object oriented applications. Although I focus on MySQL, it should be possible to apply these concepts to any other database flavour which supports transactions. There are some limitations to this approach, but I’ve found it to be very useful in practice.

A Brief Introduction to Transactions

You can’t work with database backed web apps for very long without coming across the concept of database transactions.

Transactions (http://en.wikipedia.org/wiki/Database_transaction) are a way of encapsulating a group of database queries, typically INSERT and UPDATE statements, so that they all must complete successfully in order for the changes to be written (“committed”) to the database. If any single query fails, any changes to the database can be “rolled back” to how they were when the transaction began.

This has great practical value. Many operations consist of several steps where the changes should only be applied if every step is successful. The classic example is transferring funds between two bank accounts, which is a three stage process:

1. Checking that the sender’s account has a balance greater than the amount to be transferred.
2. Subtracting that amount from the sender’s account
3. And, finally, adding it to the recipient’s account.

The SQL for this transaction might go something like:

START TRANSACTION;
SELECT balance FROM account WHERE account_number = 123456789;
UPDATE account SET balance = balance - 100.00 WHERE account_number = 123456789;
UPDATE account SET balance = balance + 100.00 WHERE account_number = 987654321;
COMMIT;

If an error is thrown at any point in the process, the ROLLBACK statement can be used to revert the database to its state before the transaction began.

In an application that doesn’t use transactions, a typical approach might be to check the return value of each query or enclose it in a try…catch block:

try {
    $balance = $db->fetchOne( "SELECT `balance` FROM `account` WHERE `account_number` = 123456789" );
    $db->update( 'account', array( 'balance' => new Zend_Db_Expr( "`balance` + 100" ) ), "`account_number` = 123456789" );
    $db->update( 'account', array( 'balance' => new Zend_Db_Expr( "`balance` - 100" ) ), "`account_number` = 987654321" );
} catch ( Zend_Db_Exception $e ) {
    echo $e->getMessage();
}

The problem with this approach is that although we know the operation failed, we don’t know when it failed. Perhaps the first update statement was executed, but the second one wasn’t. Perhaps the database connection was lost before we even started. The end result is that we may have introduced inconsistencies into our data that, even with this simple example, are going to be hard to track down and correct.

Using transactions we can ensure that each step is carried out successfully or the whole operation fails.

try {
    $db->beginTransaction();
    $balance = $db->fetchOne( "SELECT `balance` FROM `account` WHERE `account_number` = 123456789" );
    $db->update( 'account', array( 'balance' => new Zend_Db_Expr( "`balance` + 100" ) ), "`account_number` = 123456789" );
    $db->update( 'account', array( 'balance' => new Zend_Db_Expr( "`balance` - 100" ) ), "`account_number` = 987654321" );
    $db->commit();
} catch ( Zend_Db_Exception $e ) {
    $db->rollBack();
    throw $e;
}

By calling the beginTransaction() method, we temporarily delay any actual modifications to the database. If each query is successful, the changes are committed using the commit() method. But if an exception is thrown, the changes are reverted using rollback().

The Problem

Although transactions are a useful tool, sadly they don’t always fit well into an object oriented application.

It’s common to delegate the actual persistence (inserting and updating) of models to a save method which, in turn, goes on to save child objects. Say we have a simple address book application which allows you to enter multiple notes for each contact. The save method for the Contact model might look something like this:

public function save()
{
    $db = $this->_db;
 
    try {
        $db->beginTransaction();
        if ( !$this->hasId() ) {
            // new model
            $db->insert( 'contact', $this->getData() );
        } else {
            // existing model
            $db->update( 'contact', $this->getData(), $db->quoteInto( '`id` = ?', $this->getId() ) );
        }
        // save the contact's note
        foreach ( $this->getNotes() as $note ) {
            $note->save();
        }
        $db->commit();
    } catch ( Zend_Db_Exception $e ) {
        $db->rollBack();
         throw $e;
    }
}

The save method for the Note class is similar, starting a transaction and saving the individual note.

Herein lies the problem: attempting to start an new transaction implicitly commits the current transaction (http://dev.mysql.com/doc/refman/5.0/en/implicit-commit.html). Without ugly hacks which tie the models directly to the database operations, it becomes difficult to establish which object started the transaction and when it should be committed.

The Solution

Disclaimer: This solution is adapted from the extended Pdo_MySql adapter in Varien’s Magento e-commerce product. A similar approach is adopted by Bryce Lohr’s Nested Table Support for Zend_Db proposal. You should read Bill Karwin’s comments about that proposal and understand the limitations of this method before implementing it. That said, I still believe this is a useful and practical way of simulating nested transactions and I have used it a number of times.

A simple solution to the problem is to keep track of the “depth” of the transaction, that is, how many times the beginTransaction() method has been called. That way, we can hold off committing the changes to the database until we are certain that all the save operations have completed successfully.

Since transactions apply to the whole database connection, the most logical place to manage this process is in the DB adapter class. To do this, we extend our adapter class like so:

class App_Zend_Db_Adapter_Mysqli extends Zend_Db_Adapter_Mysqli
{
    /**
     * Current Transaction Level
     *
     * @var int
     */
    protected $_transactionLevel = 0;
 
    /**
     * Begin new DB transaction for connection
     *
     * @return App_Zend_Db_Adapter_Mysqli
     */
    public function beginTransaction()
    {
        if ( $this->_transactionLevel === 0 ) {
            parent::beginTransaction();
        }
        $this->_transactionLevel++;
 
        return $this;
    }
 
    /**
     * Commit DB transaction
     *
     * @return App_Zend_Db_Adapter_Mysqli
     */
    public function commit()
    {
        if ( $this->_transactionLevel === 1 ) {
            parent::commit();
        }
        $this->_transactionLevel--;
 
        return $this;
    }
 
    /**
     * Rollback DB transaction
     *
     * @return App_Zend_Db_Adapter_Mysqli
     */
    public function rollback()
    {
        if ( $this->_transactionLevel === 1 ) {
            parent::rollback();
        }
        $this->_transactionLevel--;
 
        return $this;
    }
 
    /**
     * Get adapter transaction level state. Return 0 if all transactions are complete
     *
     * @return int
     */
    public function getTransactionLevel()
    {
        return $this->_transactionLevel;
    }
}

Update your bootstrap to use the extended class et voila – a single START TRANSACTION and COMMIT or ROLLBACK is sent to MySQL, regardless of how many levels of nested pseudo-transactions have been created.

Please Note:

• It’s important that each child save() method re-throws the exception so that transaction depth is reduced by successive calls to rollBack(). The end result is that the originally called save() method then performs the actual rollback.
• All the tables used in the transaction must use a storage engine that supports transactions. For MySQL, this will most likely mean using InnoDB. To convert a MyISAM or other table type to InnoDB, use “ALTER TABLE table ENGINE = InnoDB”. It could take some time to rebuild the indexes on large tables. There are other considerations about the use of InnoDB – please consult the MySQL manual.
• If the tables used don’t support transactions, it’ll just fail silently. Bad times. I highly recommend using Firebug and Zend_Db_Profiler to monitor database queries during development (see http://framework.zend.com/manual/en/zend.db.profiler.html);

I hope you find the above useful – if so, please leave a comment. If not, please be gentle… it’s my first blog post!

I recently was asked how many fsync / sec ( and therefore durable transactions / sec) we can get on FusionIO card.

It should be easy to test, let's take sysbench fileio benchmark and run, the next command should make it:


./sysbench --test=fileio --file-num=1 --file-total-size=50G --file-fsync-all=on --file-test-mode=seqrewr --max-time=100 --file-block-size=4096 --max-requests=0 run


So that's 9229.35 req/sec, which is pretty impressive.

For comparison the same run on PERC 6i RAID10 with BBU:

which gives us 5369.62 req/sec.

Note this is for single thread, and in MySQL/InnoDB multi-thread load you may get more transactions per second with group commit ( which is back to live in InnoDB-plugin / XtraDB )

Entry posted by Vadim | 11 comments

Add to: | | | |

We've just upgraded datatables.org with a few improvements. We know that people have had a few issues with the reliability so we are happy to say that we've moved datatables.org from its current hosting provider to dedicated servers in multiple Yahoo! data centers.

The YQL team pose with their YQL shirts on.

Most importantly we've improved the process which we use to process your contributions to Github. This makes it even easier for us to accept your contribution and roll it into the console and our environment file with key from the NYT, Amee and others (it's store://datatables.org/alltableswithkeys in case you were wondering).

Thanks to all the tables you've been contributing we now have a massive 700+ tables available to use today with 600+ community contributed tables. We aren't satisfied with that though, we still want more tables. So if you contribute a new table that we add to the Github repository we'll send you one of these awesome YQL t-shirts. You'll be the envy of all your (geek) friends.

We have tables for getting your status from Twitter, for making a blog post on Wordpress, for finding articles about President Obama on the NYT without signing up for a key. The list is huge, and the possibilities endless.

Don't forget we have docs for building tables yourself, and a guide to issuing a pull request to add your changes to the main repository.

select * from internet; just got even better.

Tom Hughes-Croucher (@sh1mmer)
select * from yahoo.geeks where job = "technology evangelist";

In previous articles, I've explored building service endpoints and RESTful services with Zend Framework. With RPC-style services, you get to cheat: the protocol dictates the content type (XML-RPC uses XML, JSON-RPC uses JSON, SOAP uses XML, etc.). With REST, however, you have to make choices: what serialization format will you support?

Why not support multiple formats?

There's no reason you can't re-use your RESTful web service to support multiple formats. Zend Framework and PHP have plenty of tools to assist you in responding to different format requests, so don't limit yourself. With a small amount of work, you can make your controllers format agnostic, and ensure that you respond appropriately to different requests.

A couple of weeks a go Brian Profitt pinged me for a chat about Devops , the result of that chat , his article can now be found on the Zenoss blog, it's titled Datacenter Barometer: Better days arrive when dev meets ops

It's a very nice read with some pointers to places regular readers of my blog should already know ;)
So with lots of leading Open Source infrastructure companies on different levels, such as config management (OpsCode and Reductive Labs) , monitoring (Zenoss) , deployment (openQRM, RPath, and obviously Consultancy companies , the upcoming Devops conferences around the planet promise to be a lot of fun ! ;)

Oh, and apparently there is some more on the story on /.

Technorati Tags: devops ha krisbuytaert monitoring opensource zenoss

Share with Shareomatic!

Two great Flex conferences have recently posted their video recordings:

• Flex Camp Wall Street
• Flex on the Cloud

There were numerous great sessions and speakers at each of these conferences. At Flex Camp Wall Street I presented “Flex Stuff I am Excited About”. Check it out and let me know what you think.

Things that made me happy this morning:

I just arrived in Atlanta for Georgia Tech University hack day and the weather is awesome. I spent the day in the sun in cafes writing my slides for the Mix10 conference next week and now I am going through my feeds. So time for another TTMMHTM:

• YouTube is changing its URLs – this didn’t really make me happy but it means I know why some people cannot use Easy YouTube at the moment and I know now that it is easy to fix.
• Google released a Public Data Directory taking government data and running it through the visualization API to allow you to directly embed charts into your pages. The Guardian’s World Government Data collection is still much more detailed but well done Google.
• The Humunga Stache Dog Toy makes me wish I had a dog to mess with.
• Carsonified/ThinkVitamin released my article on GeoExplorer and using Yahoo’s geo technology
• Creative people rock: 45 robots build with everyday trash
• Packrati.us is a service to automatically add URLs you tweet about to del.icio.us – this saves me some time!
• RadarVirtuel is a live map of aircraft and where they are right at this moment, where they came from and where they are headed.
• What do you suggest? takes a seed from you (or gives you something random) then guides you on a journey through language and the collective lives of Google users.
• GestureCons are a set of icons for touch-screen devices. My favourite is the two finger slide which in England doesn’t go down well as it also is the two finger salute

When I first started curating the Pattern Library, I put "tags" near the top of my list of user interaction patterns to investigate. By that time, Yahoo! had already acquired several pioneers in the tagging realm, Flickr and Delicious, and there were some subtle distinctions in how they implemented the experience.

We got down in the weeds on these and did a lot of research, ultimately settled on offering high-level guidance, and finished the patterns in the course of writing the social patterns book, where we filed tagging under the group of patterns known as Collecting, under Social Objects.

Tagging and other forms of collecting are also an example of social design patterns that mimic game dynamics. Collecting objects is a core "easy fun" activity in many games, and similarly these extremely lightweight social interactions around gathering or tagging objects enable a form of self-interested behavior that creates aggregate value and potentially richer forms of engagement.

Our three new tagging patterns are Tag an Object, Find with Tags, and the somewhat controversial Tag Cloud, which some people view as an "anti-pattern." Drop by, check them out, and let us know if we can make them any better.

Christian Crumlish
Curator, Design Pattern Library

Lose pounds of excess code in 1 easy step!

Slim controllers are easier to test, encourage the DRY principle and force business logic back into the model layer where it belongs.

In this first article, I’ll be looking at a quick way to de-bloat Zend Framework Action Controllers by using an Action Helper to automatically instatiate models and assign them to properties in the controller and view.

Lather, Rinse, Repeat

Controller actions often perform mundane, repetitive tasks. Checking for required parameters, creating objects and assigning variables (again and again) eats up line after line of code in a typical controller.

Using the address book example, a simple action in our Contact controller might look something like this:

class ContactController extends Zend_Controller_Action
{
    public function viewAction()
    {
        // check that we have a contact ID, otherwise redirect to index
        if ( !$this->_hasParam( 'id' ) || !Contact::exists( $this->_getParam( 'id' ) ) ) {
            $this->_helper->FlashMessenger( 'No contact specified or contact does not exist.' );
            $this->_redirect( 'contact' );
        }
        // lookup and instantiate the contact object
        $contact = Contact::find( $this->_getParam( 'id' ) );
 
        // assign contact to view
        $this->view->contact = $contact;
    }
}

A few things are happening here:

1. First, we check to see if the ‘id’ param exists in the request.
2. Then we use our model’s static exists() method to check that the contact ID corresponds to a real contact.
3. The contact object is then looked up using a static finder method.
4. …and finally we assign the contact to the view.

The exact code will vary depending on your model implementation, but this sort of process is pretty typical in my experience. In many actions, that’s all that’s required: the view then does its thing on the contact object that it’s been given.

Pretty boring stuff and chances are some or all of the above will be repeated in a handful of other actions in the same controller. That’s a “code smell” if ever I smelled one.

By automating these repetitive operations, there’s an opportunity to substantially reduce the size of our controllers without losing any functionality.

The Action Helper

Fortunately, Zend Framework has the perfect tool for this kind of job: an Action Helper.

By tapping into the preDispatch() hook provided and making a few basic assumptions, we can look for models specified in the request, instantiate them and then assign them to properties in both the action controller and the view.

Here’s the basic action helper class, which I’ve called “ModelWatcher”:

<?php
 
class Dog_Zend_Controller_Action_Helper_ModelWatcher extends Zend_Controller_Action_Helper_Abstract
{
    /**
     * @var string
     */
    protected $_controllerName;
 
    /**
     * @return void
     */
    public function preDispatch()
    {
        $request = $this->getRequest();
        $this->_controllerName = $request->getControllerName();
        $this->_view = $this->_actionController->view;
 
        try {
            $model = $this->_getModel();
 
            // assign to the controller and view
            $this->_actionController->{$this->_controllerName} = $model;
            $this->_view->{$this->_controllerName} = $model;
 
        } catch ( Exception $e ) {
            return;
        }
    }
 
    /**
     * @return object The instantiated model object
     */
    protected function _getModel()
    {
        $request = $this->getRequest();
 
        if ( NULL === $request->getParam( 'id' ) ) {
            throw new Exception( "ID parameter not found in request" );
        }
        $id = $request->getParam( 'id' );
 
        // attempt to find and instantiate the model object
        $modelClass = ucfirst( $this->_controllerName );
        $model = $modelClass::find( $id );
 
        return $model;
    }
}

Important: Don’t forget that you’ll need to add the helper to your bootstrap for this to work. Something like this should do the trick:

Zend_Controller_Action_HelperBroker::addHelper( new Dog_Zend_Controller_Action_Helper_ModelWatcher() );

When the helper’s preDispatch() method is called, it checks the request for the “id” parameter and attempts to find and create the model object. In the case of our address book, a request URL like http://www.example.com/contact/id/123 would try to retrieve an object of the class Contact whose ID is 123.

We wrap the call to _getModel() in a try…catch block to deal with two scenarios:

1. When no ID parameter exists in the request or
2. No record exists for the given ID

The latter assumes that the find() method will throw an exception if the record is not found.

If the lookup is successful, the newly created object is assigned to the controller and view with a property named after the controller. So with our address book example, you’d access the contact object using $this->contact from within both the controller and the view.

On failure, we do nothing – leaving the action controller to implement its own logic for dealing with non-existent objects.

The Final Weigh-In

All this means we’re left with a significantly slimmer controller action:

class ContactController extends Zend_Controller_Action
{
    public function viewAction()
    {
        // check that we have a contact object
        if ( !isset( $this->contact ) ) {
            $this->_helper->FlashMessenger( 'No contact specified or contact does not exist.' );
            $this->_redirect( 'contact' );
        }
    }
}

The controller action now only needs to check that a contact object has been created by the action helper. There’s no need for it to get its hands dirty with the request params, object generation or assigning variables to the view as it’s already been done.

This approach does require an adherence to some naming conventions… primarily that the model class name has to mirror the controller name and the identifying parameter is always passed as “id” in the URL. Provided they aren’t too wacky, it should be reasonably easy to re-jig the code to suit any conventions of your own.

Further Weight Loss

A slightly more flexible approach is to use verbose ID parameters like “contact-id”. This gives you the ability to have model classes which don’t match the controller name and also allows multiple models to be instantiated by the ModelWatcher in one request, both of which can be useful. I use something similar in my own implementation.

You might also want to consider adding other logic in to the ModelWatcher helper like record locking or fine-grained access control. Even if you decide to separate these activities into a separate helper class, you can still use the models inserted into the controller rather than going back to the request parameters again. Just make sure that the ModelWatcher is added to the action helper plugin stack before any helper that relies on it or the models it creates.

Final Thoughts

Hopefully it should be fairly clear that this simple action helper has the potential to dramatically reduce the amount of boiler plate code in your controllers. Eagle-eyed readers might also have noticed that automatically assigning models to the controller is a feature of CakePHP.

Look out for future posts where I’ll be discussing more ways to de-clutter and cut down your controllers.

P.S. Thanks to Rob Allen, Matthew Weier O’Phinney, Pádraic Brady, Wenbert Del Rosario and others for retweeting/linking to my first post

Fellow Nestors!

Without a doubt, the most often-used paradigm in JavaScript is events. Events are a manifestation of the observer pattern, a well-defined computer science design pattern for loose coupling. Loose coupling is incredibly important for creating maintainable, stable codebases. I talk a lot about loose coupling and its importance in my talk, Scalable JavaScript Application Architecture (video), so I won’t talk too much about it here. However, the concept is very important to grasp if you wish to progress as a software engineer.

Events

Unless you’ve never written any JavaScript before, you’ve used events at some point in time (admittedly, if you’ve never written JavaScript before, the chances of your reading my blog are probably pretty slim). Put quite simply: the way that you tie behavior to web pages is through events. Events are a way of letting interested parties know that an important moment has occurred in the lifecycle of the application. For instance:

window.onload = function(){
    Application.init();
};

In this example, the load event is the interesting moment. I want to know when the window is fully loaded so that I can initialized the JavaScript application. The onload event handler is the location to where an event handler is assigned. The brilliant part is that window doesn’t care what web page is loaded or who is writing the code; it just knows that there’s a function to call when load occurs. This is the essence of loose coupling: when parts of an application have very limited knowledge of one another.

The Browser Object Model (BOM) and Document Object Model (DOM) publish events to allow developers access to the interesting moments of the browser and web page, respectively.

Custom events

It’s no surprise that most JavaScript libraries rely heavily on custom events since this is a pattern that web developers are familiar with. Every major JavaScript library provides its own events, components to enable easy custom event definition, or both. This makes sense, of course, since libraries want to be loosely-coupled to the execution environment, and therefore, to your code.

There’s nothing magic about custom events, though, and there’s no need to load an entire library if you’d like to experiment with custom events. An object that supports custom events needs to be able to do a small set of things:

1. Assign an event handler for a particular event.
2. Remove an event handler for a particular event.
3. Fire an event and call all assigned event handlers.

The following implements all of this basic functionality:

//Copyright (c) 2010 Nicholas C. Zakas. All rights reserved.
//MIT License

function EventTarget(){
    this._listeners = {};
}

EventTarget.prototype = {

    constructor: EventTarget,

    addListener: function(type, listener){
        if (typeof this._listeners[type] == "undefined"){
            this._listeners[type] = [];
        }

        this._listeners[type].push(listener);
    },

    fire: function(event){
        if (typeof event == "string"){
            event = { type: event };
        }
        if (!event.target){
            event.target = this;
        }

        if (!event.type){  //falsy
            throw new Error("Event object missing 'type' property.");
        }

        if (this._listeners[event.type] instanceof Array){
            var listeners = this._listeners[event.type];
            for (var i=0, len=listeners.length; i < len; i++){
                listeners[i].call(this, event);
            }
        }
    },

    removeListener: function(type, listener){
        if (this._listeners[type] instanceof Array){
            var listeners = this._listeners[type];
            for (var i=0, len=listeners.length; i < len; i++){
                if (listeners[i] === listener){
                    listeners.splice(i, 1);
                    break;
                }
            }
        }
    }
};

The EventTarget type has three methods: addListener(), fire(), and removeListener.

The addListener() uses the private _listeners object to store event handlers for various events. When an event handler is added, the method first checks to see if there’s a named property for that event type on the _listeners object, and if not, creates one containing an array. The event handler function is then saved to the array for later.

The fire() method fires an event with a given name. In effect, this method’s only job is to execute each event handler for the given event type. The method accepts either an object, in which case it’s expected to have a type property, or a string, in which case a new object is created and the string is assigned as the value of type. Next, if the event object doesn’t have a target property assigned, it is set to the current instance. This effectively creates an event object similar to the one most are familiar with via the BOM and DOM. Once the event object is created, the _listeners object is checked for event handlers, and if found, they are executed. Note that in order to mimic the BOM/DOM approach, event handlers are executed in the scope of this via the call() method.

The last method, removeListener(), simply reverses the process of addListener(). It searches through the _listeners property for the given event type to locate the specified event handler. If found, the event handler is removed by using the array’s splice() method, and otherwise it exits without doing anything.

Basic usage:

var target = new EventTarget();
function handleEvent(event){
    alert(event.type);
};

target.addListener("foo", handleEvent);
target.fire({ type: "foo" });    //can also do target.fire("foo")
target.removeListener("foo", handleEvent);

Practically speaking, you’ll likely not want to use an instance of EventTarget directly, but rather inherit from it:

function MyObject(){
    EventTarget.call(this);
}

MyObject.prototype = new EventTarget();
MyObject.prototype.constructor = MyObject;
MyObject.prototype.foo = function(){
    this.fire("foo");
};

var o = new MyObject();

o.addListener("foo", function(){
    alert("Foo just happened.");
});

o.foo();

Typically, events are fired in reaction to some other method call, as in this example (events are usually not fired external to the object that is publishing the events).

What about…?

This is a pretty barebones implementation of a custom event providing object, so inevitably someone will come along and ask why I didn’t include one feature or another. There are, of course, a lot of enhancements you can make to custom events if you so desire. Some enhancements others have implemented:

• Bubbling of events
• Continue to execute event handlers even if one throws an error
• Allow event handlers to cancel further processing or default actions

Each of these can be built pretty easily on top of the base presented in this post.

Conclusion

Custom events are a very powerful and useful pattern in JavaScript programming, and your usage of them doesn’t have to rely on a large JavaScript library. Implementing your own custom events is easy. The implementation presented in this post is a minimum feature set that typically fulfills most requirements, but you can consider it as a starting point for more advanced functionality if your requirements are more complex.

Related posts

• Event delegation in JavaScript
• Event order of blur and change
• Web definitions: DOM, Ajax, and more
• XPath in JavaScript, Part 2
• XPath in JavaScript, Part 1
• Speed up your JavaScript, Part 4

P.S. My new book, High Performance JavaScript is coming out in early 2010. Need help now? Check out Professional JavaScript, 2nd Edition.

Last week, Yahoo! JavaScript architect Douglas Crockford delivered the fourth installment of his Crockford on JavaScript series:

1. Volume One: The Early Years
2. Chapter 2: And Then There Was JavaScript
3. Act III: Function the Ultimate
4. Episode IV: The Metamorphosis of Ajax
5. Part V: The End of All Things (March 31 — RSVP)

In this session, Douglas tackles the DOM. On the one hand there was JavaScript, he says, and JavaScript is “what made the browser work.”

On the other hand, there was the Document Object Model, also known affectionately as the DOM. It is what most people hate when they say they hate JavaScript. Most of the people who say they hate JavaScript don’t know JavaScript, might have never seen JavaScript, but they’ve felt the DOM alright. If you don’t know what the difference is and you say, “JavaScript is the stupidest thing I’ve ever seen,” you’re not talking about JavaScript, you’re talking about the DOM. The DOM is the browser’s API. It is the interface. It provides JavaScript for manipulating documents.

The DOM may be imperfect, but it’s nonetheless crucial to what frontend engineers do when they write web applications. In this talk, Douglas provides an overview, situated historically, of where the DOM came from, how it achieved ascendance with Ajax, and what the future might hold. In Douglas’s inimitable fashion, this history starts with Sir John Harrington and takes us up to the present day. A few choice words for CSS are among the many applause lines for veteran developers:

I find within the community of people who use CSS great affection for it. They’re totally invested in CSS, they love it. They can’t imagine any other way of doing formatting in a document. It’s it. It’s sort of like watching an episode of Cops where the cops come in and break up the family dispute, and there’s this “CSS ain’t bad, you just don’t understand it like I do. I know it hurts me, but I make mistakes, I’m wrong.” CSS is awful, and it amazes me the way people get invested in it. It’s like once you figure it out, kind of go “oh, OK, I see how I might be able to make it work,” then you flip from hating it to loving it, and despising anybody who hasn’t gone through what you’ve gone through. It doesn’t make sense to me.

If the video embed below doesn’t show up correctly in your RSS reader of choice, be sure to click through to watch the high-resolution version of the video on YUI Theater.

• Download HD video (480p ~720MB)
• Download video (m4v)
• Download slides
• A high-resolution, transcripted version of this talk is available on the YUI Theater site

Other Recent YUI Theater Videos:

• Douglas Crockford: Crockford on JavaScript — Act III: Function the Ultimate — Yahoo!’s JavaScript architect Douglas Crockford continues his lecture series on the JavaScript programming language with a discussion of functions in JavaScript. ‘Functions are the very best part of JavaScript,’ Crockford says. ‘It’s where the power is, it’s where the beauty is.’ Watch the video to learn why.
• Douglas Crockford: Crockford on JavaScript — Chapter 2: And Then There Was JavaScript — Yahoo!’s JavaScript architect Douglas Crockford surveys the features of the JavaScript programming language.
• Douglas Crockford: Crockford on JavaScript — Volume 1: The Early Years — Douglas Crockford puts the JavaScript programming language in its proper historical context, tracing the language’s structure and conventions (and some of its quirks) back to their roots in the early decades of computer science.
• Christian Heilmann: YQL and YUI: Building Blocks for Quick Applications — The Yahoo! Developer Network’s international evangelist Christian Heilmann discusses his philosophy for creating fast, powerful, compelling applications using the Yahoo Query Language (YQL) and the Yahoo User Interface Library (YUI).

Subscribing to YUI Theater:

• YUI Theater RSS feed
• YUI Theater on iTunes

This post is long overdue, as the idea struck me when dicussing with Lefred while preparing his Fosdem talk on Maintaining too big tables

I got triggered finishing this post by Mr BuidlDoctor

Fred has been struggling with a typical DevOps problem resulting in the most unmanageable database setup possible, there's little room for him to move but he managed is way out .. because he is good at his job

It set the mark for me that because in different organisations even the Opsteam is fragmented `in different groups that there also we need to get the Devops idea going.

Typical setups here are the Network guys vs the Platform guys , specially with the growth of virtualization where the network stack doesn't stop at the physcial switchport anymore but the vlan trunks go deep in to the VM's a lot of discussion happens. There where traditionally the story for the network engineer stopped at the switch they now want control much deeper in the infrastructure.

But an even bigger group that needs integration are the security folks, it's no secret that in some organisations the security guys job is to be the bad guy, their default reply to something is NO. Specially to people wanting to drill holes in their architecture .

Patrick wonders if its the specialist vs generalist dillemma, I think it's the Web vs Enterprise IT way of thinking ..
DevOps first gained ground in Web environments , the battle has only started ..

We still have a long way to go before in say a banking environment the Devs and SecOffs' and the DBA's and the Ops are on the same line ... they all need to break the walls of confusion, they all need to come out of their silos. And when you are a generalist in charge of a bunch of these things you have to make sure your tuesday afernoon security persona talks with his other persona's from time to time ... otherwise you are really gonna need those meds :)

Technorati Tags: DBAOps devops NetOps SecOPS

Share with Shareomatic!

Business Development Managers Required

There are currently 3 vacancies available -

• South East (inc. London)
• South West
• North of England & Scotland

The Role

In order to maximise revenues and exceed performance and financial targets, you will be wholly responsible for selling the benefits and features of the Look4AProperty Money portfolio of unique and exclusive products to key buyers, influencers and decision makers. Working within a defined territory, this exciting role exclusively involves the pursuit of new business. Account management will be handled by a separate dedicated team of professionals.

The Company

Look4aProperty.com founded in 2006 connects home hunters with the widest choice of properties on the market. We make the experience of finding suitable homes to view easier for the general public and we make the cost per enquiry for estate agents and developers dramatically lower than traditional newspaper based advertising.

James Caan

Look4AProperty.com displays over 200,000 UK sale and rental properties from more than 1,200 listing estate agents with over 1 million unique users using the site each month. Look4AProperty’s customers are the estate agents and new homes developers who list their properties on the site. They range from 7 of the 10 estate agent groups with more than 100 offices to individual owner-operated companies with just a single branch.

Look4AProperty.com is privately owned. The board of directors is currently Chaired by one of the UK’s leading and most successful entrepreneur’s James Caan also famous as one of the leading Dragon’s in BBC’s Dragons Den. You will have an opportunity to meet, work and hopefully impress James as your role progresses.

Our new and exclusive Look4AProperty Money financial products have taken the industry by storm and we are looking for highly skilled, motivated, proven and credible sales people to carry this estate agent revenue enhancing offering forwards.

The growth opportunities are enormous as well as are the very substantial financial rewards on offer as well as the satisfaction that bears in helping people both find and finance their home ownership aspirations.

Role Requirements and Responsibilities

• To pro-actively pursue new business opportunities for which you will be required to source and develop qualified sales leads into new sales as well as using your own networking and cold calling skills to develop business and surpass your performance and financial targets;
• To prepare and research appropriately in order to accelerate progress and/or gain commitment via face to face customer meetings and/or telephone calls. This includes lead generation, setting up appointments and creating excellent pitches, presentations and suitable proposals;
• To provide relevant information in response to enquiries regarding the Look4AProperty Money suite of products;
• To use the marketing mix and both your product and client knowledge to correctly identify and position your proposed solutions and also manage client expectations;
• To input and maintain lead progress records with accurate, high quality information in order to enable effective forward planning, maximise the success of leads generated and maintain up to date knowledge of customer activity. All client conversations and transactions must be clearly documented and e-filed;
• To identify new market trends and business opportunities as well as maintain an in-depth level of knowledge of the latest industry news and competitor activity;
• To attend, prepare for and participate in one to one and team meetings that will both review and guide performance (including appraisals), share and progress ideas and initiatives, communicate objectives and agree sales plans and focus;
• To recognise the importance and benefits of effective team working, assisting colleagues and providing cover as and when required. This also includes respecting and motivating your colleagues in other departments who are responsible for servicing and supporting your activities as well as maintaining effective communications with every other department across the business;
• To attend internal and/or external training sessions as and when appropriate to further develop skills level and expertise. Responsible for continually updating your product, market and competitor knowledge;
• To undertake and/or assist in special projects and tasks as and when required.

Candidate Requirements : Skills, Knowledge and Experience

• Experience of selling financial products and portal advertising into the property industry would be a distinct advantage;
• Outstanding inter-personal skills that enable you to work effectively with people and professionals at all levels. You must be credible and comfortable both dealing with and presenting to senior company executives and board directors;
• Outstanding selling, presentation, communication and negotiation skills that permit you to inform, advise and influence others clearly both face to face and over the telephone;
• Positive, determined and “can do” approach to both researching and analysing new business opportunities. You should be wholly comfortable “knocking on doors” and cold calling as appropriate;
• Outstandingly professional with solid presence with at least 4 years proven and demonstrable b2b solution sales experience;
• Structured sales and management trained is preferable;
• Organisational skills that permit you to manage and organise your time effectively as well as cope with competing demands and work under pressure to tight deadlines;
• Ability to be flexible and comfortable in a fast pasted performance led entrepreneurial environment. You are adaptive and able to clearly read a situation quickly and react accordingly as well as always creating the right impression;
• Genuine and constructive enthusiasm for the job and management;
• Demonstrate high levels of personal energy, commitment, loyalty, the right focused “can do” attitude and an unfailing interest to do better than best;
• Highly self-motivated, a self-starter, a strategic thinker, superior problem solving skills and able to quickly deliver the required solution to delight the customer;
• Sound ability to work using own initiative as well as realise and implement new ideas and smarter ways of working;
• Outstanding administrative and reporting skills with Microsoft Office proficiency at Advanced Level
• Demonstrable evidence of tangible career progression and stability;
• Professionally polished person with high levels of integrity and hardworking ethic;
• Hold a personal commitment to continuously developing one’s own knowledge and natural desire to share best practice with colleagues;
• Full clean UK driving license;
• Someone looking to smash norms and OTE’s and drive earnings radically through higher sales achievements;

Compensation & Benefits

• Basic Salary: £35,000 p.a.
• Realistic OTE (Uncapped) £115,000 p.a.
• BMW Company Car
• Laptop
• Mobile Telephone
• 25 Days Annual Holiday Entitlement
• Employee Contributory Pension Scheme

To Apply

Please send an up to date copy of your CV with a covering letter to careers@look4aproperty.com

Please note, due to the high level of applications we receive as a Company, we are only able to respond to those that have been successful at the initial screening stage. If you have not had a response from us within 14 days of submitting your CV, unfortunately you must assume that your application has been unsuccessful.

Thank You.

YQL is easy to use. How easy? Check out these three examples:

1) Fetch Twitter search results using YQL + JSONP + a simple JavaScript handler function

Feed readers, see the code on Github.

Run this query in the YQL console:
select * from json where url="http://search.twitter.com/search.json?q=puppy"

Note: You could also use YQL’s community table for Twitter search

2) Scrape content off the World Wildlife Fund site and pass it back to a jQuery handler

Feed readers, see the code on Github.

Run the YQL query in the console:
select * from html where url=”http://www.worldwildlife.org/sites/photos/galleries.html” and xpath=’//div[@class=”narrativeText”]/h4|//ul[@id=”speciesList”]’

Note: instead of running all this on the page, you might consider using YQL execute to generate your HTML on Yahoo! servers and pass it back to the handler for insertion into the page. You could also create an adhoc API that leans on Yahoo!'s cache to reduce WWF's bandwidth costs.

3) Convert XML to JSON in your sleep

This script instructs YQL to fetch the “UN News Centre - Top Stories” RSS feed and return it as JSON, and then drops the data on the page using YUI.

Feed readers, see the code on Github.

Run this query in the YQL console:
select * from rss where url=’http://www.un.org/apps/news/rss/rss_top.asp’

Erik Eldridge (@erikeldridge)
Yahoo! Developer Network