Adam Leach

We've reduced the prices for Amazon S3 storage again. As is always the case, the cost to store your existing data will go down. This is markedly different than buying a hard drive at a fixed cost per byte and is just one of the many advantages of using cloud-based storage. You can also count on Amazon S3 to deliver essentially infinite scalability, eleven nines of durability (99.999999999%), and your choice of four distinct geographic locations for data storage.

So, starting November 1, 2010, you'll see a reduction of up to19% in your overall storage charges on a monthly basis. We've created a new pricing tier at the 1 TB level, and we have removed the current 50 - 100 TB tier, thereby extending our volume discounts to more Amazon S3 customers.

The new prices for standard storage in the US Standard, EU - Ireland, and APAC - Singapore regions are as follows:

Reduced Redundancy storage will continue to be priced 1/3 lower than standard storage in all regions.

The full price list can be found on the Amazon S3 page. We'll continue to work relentlessly to drive our costs down so that we can pass the savings along to you!

We've got several more announcements related to S3 coming up in the near future, so stay tuned.

The S3 team is hiring Software Development Engineers, a Technical Program Manager, System Engineers, Administrators, and Product Managers. More information and instructions for applying can be found on the Amazon S3 Jobs page.

-- Jeff;

Just in time to demo at the RightScale User Conference, we started a private beta of two new cluster monitoring features: stacked graphs and heat maps. We’ve been using them for a while internally and they’ve been invaluable to quickly determine the cause of issues. The idea behind these types of graphs is not new, but what we’ve been able to do is automate the back-end machinery that lets you create these graphs across any cluster of servers with just a few clicks.

The cluster monitoring we’ve had for a while shows an individual graph per server. Below is an example showing the cpu load on our web front ends. Each graph shows user time in blue and idle time in green over the course of one week. You can see that one of the last two servers has been sitting idle and the other one was launched less than a day ago. These individual graphs are nice for a small number of servers, but once your cluster has more than about a dozen they cease to be practical.

A stacked graph is a great alternative to display many servers on one graph where their activity contributes to a total or sum. In the case of the web front-ends, they all serve HTTP requests and contribute to the total of HTTP requests served by the cluster. This is what the graph below shows: each color band shows the requests/sec for one server, and the color bands are stacked on top of one another such that you can read the requests/sec served by the application at the top. This now gives a nice overview of what’s happening in aggregate. Also, if one server were serving a lot more requests than the others, you’d be able to spot it because its band would be significantly wider. However, something that is actually not easy to notice in the stacked graph is that two of the servers are not serving requests. You’d have to start counting color bands to notice.

A heat map shows a somewhat different view of the same action. In the heat map below each color bar represents the activity of one server and the color of the bar at each point in time shows how “hot” the server is, i.e., the value of the variable being displayed color coded from blue to red. Here you can again see that the top two servers are outliers, one being idle the whole time, the other having just launched. On the other hand, it’s pretty difficult to make out absolute values for how busy a server is.

The bottom line is that none of the cluster graph types is better than the others, it just depends on what you’re looking for, so it’s nice to be able to flip back and forth. To illustrate this further, here is a real-life example of an issue we encountered. On a Wednesday morning we had alerts going off on a small number of our monitoring servers that showed unusually high load on those servers. We first looked at the heat map plotting I/O wait time for the cluster:

You’ll notice that there are a lot of bars! The heat maps are currently limited to showing 100 servers at a time and we have more than 100 monitoring servers, so you’re seeing a sampling (including the longest running, the shortest running, and some of each different ServerTemplate). Also, you’ll notice the color coding alternates between blue-red and green-orange every 10 servers to make it easier to count. The red bars make it pretty clear that something started to affect a small number of servers around 8am. After looking at a number of other variables we saw the following stacked graph of the number of servers monitored.

Here each color bar represents the number of customer servers monitored by one of our monitoring back-ends. This view displays the activity for a whole week and highlights the fact that a slew of additional servers were monitored right when we got the alerts. From there it was easy for us to pinpoint the cause, which turned out to be a limitation in our monitoring back-end assignment algorithm.

I’m really excited about these new monitoring features in part because I’ve built similar graphs manually several times at different companies and being able to get them automatically is amazing. You may have noticed that we produce the graphs using RRDtool, which we’ve extended quite a bit, in particular to draw the heat maps. The way the graphs are rendered is that a monitoring front-end queries the data series from the appropriate back-end servers and then assembles everything into one graph which is sent to the browser. The result is that each of the two graphs above displays 60,000 data points, that’s a lot of data to be able to see at a glance!

The stacked graphs and heat maps are currently in private beta. If you’re running lots of servers using RightScale and would like to try them out, please drop me an email. One of the tasks still ahead of us before we can release this to everyone is improving the parallelism of the data fetching so we can plot more than 100 servers at a time.

Thanks to the around 200 developers who came to Yahoo! recently for our monthly Hadoop User Group meeting. The energy in the packed room was phenomenal, and conversations continued long after the formal sessions.

The event started with Chris Riccomini talking about Pig at LinkedIn. It was great to get a firsthand view of how industry is leveraging the power of Pig to solve their data processing problems. Chris covered how Pig is an integral part of data analytics at LinkedIn. He showed how Pig is used to design, develop, and deliver data products at LinkedIn. He explored a successful example of Pig deployment at LinkedIn, pain points, and integration with Azkaban, Voldemort, Hadoop, and the rest of LinkedIn’s ecosystem. Chris also covered the most frequent gottcha's and learnings, and then concluded with some of his thoughts on the evolution of Pig. The talk generated many interesting questions.

Next we had Dhruba Borthakur and Dmytro Molkov from Facebook talking about HighAvailability of the Hadoop NameNode. Although not many current users face name-node failures, it is certainly the next big hurdle to deliver high availability. Dhruba talked about the specifics Facebook has innovated to make the name node highly available and discussed the design and the advatanges of the proposed design. He described in detail a hot-standby solution called the AvatarNode, then talked about the capabilities of the SecondaryNameNode and the BackupNode. Very insightful presentation.

Finally, we had Ahad Rana from Opencrawl talking about building a scalable Web crawler with Hadoop. Ahad talked about his experience in building an open and accessible Web-scale crawl. He discussed the Hadoop data-processing pipeline, including PageRank implementation. He also described techniques to optimize Hadoop and the design of their URL Metadata service. Finally he concluded with details on how users can leverage the crawl (using Hadoop) today. The discussion generated very detailed questions and folks remained way past the event deadline to understand the internals of search index and how they can leverage Opencrawl to solve their business needs.

We at Yahoo! embrace Hadoop, and are looking for exciting technologies and experiences you want to share. Please contact me via the Hadoop Bay Area User Group Meetup page.

Thanks to the 125 or so developers who came to Yahoo! recently for our monthly Hadoop User Group meeting, in spite of the Giants' World Series baseball game. The conversations continued before and long after the formal sessions.

The event started with a presentation on state-of-the-art productivity tools for developers and analysts by Ben “Shevek” Mankin, from Karmasphere. Shevek talked about the challenges developers face when interacting with Hadoop programs — MapReduce, Hive, and so on. He demonstrated how to debug big data jobs on Hadoop from your desktop using familiar graphical interfaces. The graphical IDE certainly improves engineering productivity and accelerates the Hadoop job development process.

Next we had Marc Limotte talk about how Cascalog fills a need as an internal domain-specific language for map/reduce jobs. Marc talked about about how Cascalog differs from other options for creating map/reduce programs and the advantages of this option.

Finally, we are always looking for Hadoopers who will want to talk about how they are leveraging the power of Hadoop to solve their business needs. If you have any experiences you would like to share, please contact me via the Hadoop Bay Area User Group Meetup page.

See you at the next HUG, November 17, Wednesday, at Yahoo! headquarters.

You probably saw the Twitter storm over Oracle’s pricing changes and InnoDB in the last few days. The fear about Oracle removing InnoDB from the free version of MySQL was baseless — it was just a misunderstanding. Still, in the years since MySQL has been acquired by Sun, and then by Oracle, many MySQL users have been uncertain about the future. We think it’s important for us to make our own position and plans clear and public, so that there is no fear or doubt about what Percona will do.

We want to begin by saying “thank you” to those who built and maintained MySQL over the years. Many of us at Percona came from the original MySQL team. We know that it took a decade-long group effort from Monty, David, Marten and many others to develop it into the world’s leading open-source database. Much of their work is never publicly recognized, but it is the foundation of our own work, and we’re indebted to it. MySQL development suffered some detours and slowdowns over the last few years. This era is over. MySQL has never progressed so rapidly towards better features, better performance, and better quality. We applaud the great work coming from Oracle.

Always business-focused

We know that statements such as these can be filled with corporate double-speak. Let there be no doubt: Percona is in business to make money, not to make flowers grow taller and the sun shine brighter. Likewise, our customers are in business to make money. We write software that contributes to our business goals, and to our customers’ goals. And we think the best way to do that is by adhering to open-source principles.

Always improving

Our flagship database server Percona Server with XtraDB powers critical portions of the U.S. economy, applications you rely on every day, utility services, and governments. Thousands of people use it. We take that very seriously.

Our roadmap is to continue what we are doing: diagnose and solve the root problems our customers experience, and build new features to overcome limitations in the server. We have a dedicated team of engineers making sustained and rapid progress on improving it. We plan to continue releasing new versions frequently.

Always free

Our software is 100% free. We are a pure-play open-source company. We do not practice dual-licensing. We hold nothing back. Our strategy is to be a services company, pure and simple. The value we provide to customers is our depth of knowledge about how to solve their problems. The software we produce is evidence of our abilities.

Always independent and objective

Percona Server with XtraDB is an enhanced drop-in replacement for MySQL, for those whose needs are not met by the official MySQL server from Oracle. It offers more performance, more features, and more diagnostic capabilities. If it meets your needs, we’re happy, but we make our money from services, not software. Our job is to tell our customers what we think is best for them. That’s partly why we are a pure-play open-source company. If we earned money directly from our software, that would compromise our core principle of objectivity, because we’d have a financial incentive to push customers towards our software instead of helping them make the right decisions for themselves.

Always open-source

We will always work to create and improve open-source software. When there’s a need, we respond to closed-source software by providing open-source alternatives. An example of this is Percona XtraBackup. We have also worked to create open-source tools for monitoring, query analysis, replication management, and other capabilities.

If you’re worried about Oracle making MySQL closed-source, stop worrying. It can’t be done. Oracle cannot take back what is already in the community’s hands; at worst, they can only stop updating it with new versions. And we can and will continue to provide an open-source database server that delivers improved performance and functionality to our customers.

Always fairly priced

Percona is the oldest independent support provider for MySQL. We opened in mid-2006. Since then we’ve grown to over 40 full-time staff and over 900 customers. We think this success is partially because we’re fair to our customers. Our pricing model has always been to match value for the customer with Percona’s costs as closely as possible. This is why our consulting is billed by the hour in a pay-as-you-go model: you pay only for what you use. Our support and maintenance offering covers unlimited servers and CPUs, because the cost of providing support is not related to how many servers you’re running. Per-unit pricing would also compromise our objectivity — we’d have an incentive to increase your server count, instead of helping you make your systems more efficient.

Conclusion

We’re dedicated to advancing the state of the art in MySQL performance and functionality, because our customers pay us to do it. You can count on us continuing to provide an outstanding open-source database server and other software tools, with accompanying world-class services and support. We salute everyone who’s helping advance MySQL and open-source software, including Oracle, and look forward to a bright and prosperous future for our businesses.

Sincerely,
The Percona Executive Management Team:

Peter Zaitsev, Chief Executive Officer
Vadim Tkachenko, Chief Technology Officer
Baron Schwartz, Chief Performance Architect
Tom Basil, Chief Operating Officer
Bill Schuler, Vice President of Sales
Espen Braekken, Vice President of Consulting and Support

Entry posted by Baron Schwartz | 10 comments

Add to: | | | |

There are some exciting things happening with the OpenStack efforts. The team over there is getting ready to hold their semi-annual Design Summit this week in San Antonio, TX downtown in the Weston Centre from November 9-12. With an expected crowd of over 300 attendees, this event serves as the official kick-off for the next technology release (code named Bexar) as well as a chance for the general community to discuss business, marketing, and community governance. Drawing from the diverse community, the business track has speakers from Accenture, Anso Labs, NASA, PWC, Bechtel Corporation, Nicira Networks, Cloudscaling, and Cloudkick. The technical sessions are led by developers proposing new features to create a clear roadmap for the development community to execute to for Bexar and future releases. The final two days of the event are more hands on with an InstallFest, Document Sprint, and open developer coding sessions.

All business track and general sessions are being video taped for publication to the OpenStack website after the event. Technical sessions are being live audio-streamed and are open to anyone wishing to participate. Watch the OpenStack blog  or follow OpenStack on twitter (@openstack) for more event details before, during, and after the event.

Interested in getting involved?

You can obtain the latest OpenStack software, code named Austin, at http://openstack.org/projects and for documentation go to http://wiki.openstack.org.  The community also has several mailing lists in place for members to stay informed, the complete list is at http://wiki.openstack.org/MailingLists.

If you have more specific questions on this event or the OpenStack community, please contact Stephen Spector, the OpenStack Community Manager.

I have a 5G mysqldump which takes 30 minutes to restore from backup.  That means that when the database reaches 50G, it should take 30×10=5 hours to restore.  Right?  Wrong.

Mysqldump recovery time is not linear.  Bigger tables, or tables with more indexes will always take more time to restore.

If I restore from a raw backup (LVM snapshot, xtrabackup, innodb hot backup), it is very easy to model how much longer recovery time will take:

Backup is 80G
Copy is at 70MB/s.
10G is already complete.
= ((80-10) * 1024)/70/60 = ~17 minutes


I can tell progress with mysqldump by monitoring the rate at which show global status like 'Handler_write'; increases and compare it to my knowledge of about how many rows are in each table.  But progress != a magic number like “17 minutes”.  Not unless I do a lot of complex modeling.

I am not saying a 5 hour recovery is good or bad.  What I am saying is knowing remaining time is very important during disaster recovery.  Being able to say “we’ll be back at 2PM” is much better than saying “we’ll be back between 1PM and 4PM.. maybe”.

Entry posted by Morgan Tocker | 15 comments

Add to: | | | |

Amazon CloudFront uses an ever-growing network of edge locations to give your users high speed, low latency access to your content, regardless of where they happen to live.

Until now, CloudFront could serve up content from Amazon S3. In content-distribution lingo, S3 was the only supported origin server. You would store your web objects (web pages, style sheets, images, JavaScript, and so forth) in S3, and then create a CloudFront distribution. Here is the basic flow:

Effective today we are opening up CloudFront and giving you the ability to use the origin server of your choice.

You can now create a CloudFront distribution using a custom origin. Each distribution will can point to an S3 or to a custom origin. This could be another storage service, or it could be something more interesting and more dynamic, such as an EC2 instance or even an Elastic Load Balancer:

Putting intelligence (dynamic processing) behind CloudFront gives you the ability to create content on demand without having to worry about storage, caching, or global distribution. You can have content that appears static (based on its URL) but which is actually built on the fly the first time that it is needed.

This can be handy when the content is reusable but somewhat expensive to create and changes infrequently. I believe that map tiles are the classic example. Imagine a system which overlays a base map with one or more custom layers. Generating all possible combinations of tiles would be prohibitively expensive, and also wasteful since most of them would never be seen. Instead, the path component of each tile's URL can include the parameters needed to generate the tile. If the tile is already present in a particular CloudFront edge location then it will be served up directly, otherwise it will be generated, returned to the edge location, and then used to satisfy future requests (time to live and invalidation notwithstanding).

The newest version of the CloudFront Developer Guide contains some guidelines for the use of custom origins. Here are some of the most important ones:

• Respect the HTTP Keep-Alive header to improve performance.
• Ensure that the Date and Last-Modified headers are accurate on the generated content.
• Do not use query string parameters since these are not preserved by CloudFront.
• Use the Cache-Control header to indicate whether responses can be cached.
• CloudFront accepts HTTP 1.1 requests but makes only HTTP 1.0 requests to the origin server.
• Cookies are forwarded only on misses and Set-Cookie headers are removed from cached content.

The following tools, toolkits, and applications already support this new feature:

• CloudBerry Explorer
• Bucket Explorer - Download info, documentation.
• RightScale
• JetS3t
• CloudBuddy
• Boto

Like many of the most popular AWS features, we implemented this one because people like you asked for it! If you have additional ideas or suggestions, post them as comments here or in the CloudFront discussion forum. We're all ears!

If you want to help make CloudFront even cooler, check out these jobs:

• Software Development Engineer - Amazon CloudFront
• Senior System Engineer - Amazon CloudFront
• Product Manager - Amazon CloudFront
• Senior Software Development Manager - Amazon CloudFront

-- Jeff;

I'll be brief. Two quick yet important Amazon CloudFront announcements:

1. First, we've removed the beta tag from CloudFront and it is now in full production. During the beta period we listened to our customers and added a number of important features including Invalidation, a default root object, HTTPS access, private content, streamed content, private streamed content, AWS Management Console support, request logging, and additional edge locations. We've also reduced our prices.
2. There's now an SLA (Service Level Agreement) for CloudFront. If availability of your content drops below 99.9% in any given month, you can apply for a service credit equal to 10% of your monthly bill. If the availability drops below 99% you can apply for a service credit equal to 25% of your monthly bill.

This doesn't mean that we're done. In fact, the team is growing rapidly and we have lots of open positions:

• Software Development Engineer - Amazon CloudFront
• Senior System Engineer - Amazon CloudFront
• Product Manager - Amazon CloudFront
• Senior Software Development Manager - Amazon CloudFront

-- Jeff;

Around a month ago we released the first beta version of Qt Creator 2.1. We have not added any significant features since the first beta, and therefore, the Qt Creator 2.1 beta announcement is still a pretty good overview of what is new in Qt Creator 2.1.

After the release of beta 1, the translations into Russian, Polish, Japanese, French and German have been updated. Those translations are for the most part contributed by the community. Thank you! If you want to help translating, see Qt Localization in the gitorious wiki.

Apart from the translation updates, around 250 commits have been made since beta 1, almost all of them fixing bugs. Focus areas were the QML and JavaScript editor as well as deployment to Maemo devices. The debugger also received a lot of fixes. Furthermore, the documentation has been polished, featuring a brand new tutorial on Qt Quick development with Qt Creator.

We think this beta is already very solid, and we are using it for our development.

We welcome any feedback via:

• Mailing List
• The #qt-creator channel on Freenode
• Our bug tracker

There were countless small improvements to Qt Creator contributed by externals. Thank you!

And last but not least, the actual download link. Have fun.

We have three brand new products for Estate Agents designed to:

• Make moving home more affordable & achievable
• Help you win new business
• Help you to keep existing business
• Generate more income for you

1. Upfront Funding

Times are changing, if you think all of your clients want No-sale No-fee then you need to think again. Give your clients the option to pay upfront fees to benefit from an enhanced marketing package. Click here for more details.

2. Sellers Fee Funding

Whether your customers opt to pay all or part of their fee at completion now you can offer them the chance to spread the cost interest free over 12 months. And what’s more, this service costs you and your client absolutely nothing. Click here for more details.

3. Purchaser Fee Funding

Whether your buyer is struggling with stamp duty, conveyancing or mortgage set up fees you can help them spread some of those costs over 12 months interest free, again, at no cost to you or your client. Click here for more details.

For more information on “Simplicity” follow the product links above.

To arrange an appointment to join the scheme call us today on 0843 221 0611 or email us at l4ap@lms.com.

Look4aProperty.com Money “Simplicity” is a post from: Look4aProperty.com's Blog

Qt 4.7.1 has been released today.

You can download source and binary packages for this release from the Qt Download Page. Alternatively, you can grab the source directly from the public git repository, where the “v4.7.1″ tag matches the content of the released packages. The Qt 4.7 documentation has also been updated.

For information on what’s changed in this release you can check out the Qt 4.7.1 changelog and the Qt 4.7.1 Closed Tasks in the Qt Bugtracker for a high-level view, or you can dive into the detail by comparing the v4.7.0 and v4.7.1 tags in the git repository.

Many thanks to the those users who reported bugs or submitted merge requests after the 4.7.0 release. Some of that feedback is already incorporated in today’s 4.7.1 release and more will be addressed for Qt 4.7.2, which is already in progress. If you would like to provide some feedback on this release, you can do so using the Qt Bugtracker. If you want to contribute code, documentation or autotests to Qt, all the information you need to get started can be found at qt.gitorious.org.

One change I want to highlight for this release is the discontinuation of the Qt SDK product, which shipped both Qt and Qt Creator in one package. The new Nokia Qt SDK product, which debuted a few months ago, also bundles Qt and Qt Creator in one package (along with a number of other useful tools). This makes the old Qt SDK rather redundant, and we have therefore decided to retire it so that we can put a little extra effort into polishing the next version of the Nokia Qt SDK, which we expect to release before Christmas.

For those who want to use the latest Qt Creator version (2.0.1) with Qt 4.7.1, you can tell Qt Creator to use the newly-installed Qt version via the Build Settings Dialog.

Before I start a story about the data recovery case I worked on yesterday, here’s a quick tip – having a database backup does not mean you can restore from it. Always verify your backup can be used to restore the database! If not automatically, do this manually, at least once a month. No, seriously – in most of the recovery cases I worked on, customers did have some sort of backup, but it just wasn’t working, complete and what not. Someone set it up and never bothered to check if it still works after a while.

Anyway, this post is not really about the backups but rather about few interesting things I learned during last recovery case.

First, some facts about the system and how data was lost:

• MySQL had a dedicated partition on XFS file system
• Server was running innodb_file_per_table
• There was a production master and two slaves, all had same setting
• Developer accidentally ran DROP DATABASE X on the wrong machine (production master)
• All slaves followed and dropped their copy of the data
• The important tables were all InnoDB
• Having a backup, customer has first attempted to restore from backup on the production master

Luckily (or rather, unfortunately) backup only had table definitions but not the data so no data was written to file system. Mind however that restoring a backup could have been fatal if it would have written some junk data as that would have overwritten the deleted files. Now, here’s what I learned while working on this case:

Recovering from XFS is possible. Just a month ago we had a team meeting in Mallorca where we went through various data loss scenarios. One of them was deleted files on xfs – we all agreed on few things:

• recovering files from xfs is hard, if at all possible
• we had no recovery cases on xfs, most likely because:
• whoever is using xfs, is smart enough to have backups set up properly

Now I’m not picking on the customer or anything – indeed they did have a backup set up, it’s just that some (most important) tables weren’t backed up. We did not try any of the file recovery tools for xfs – apparently they are all targeting specific file types and sure enough InnoDB is not one of the supported files. What we did is we simply ran page_parser on the (already) unmounted file system treating it as a raw device. I was surprised how amazingly simple and fast it was (did you know that latest version of page_parser identifies pages by infimum and supremum records?) – 10G partition was scanned in like 5 minutes and all 4G of innodb pages were successfully written to a separate partition. That’s the easy part though – you run page parser, wait and see what you get.

If InnoDB Data Dictionary was not overwritten by an attempt to restore from the backup, actually second part would’ve been quite easy too, but it was so I could no longer identify correct PK id for specific tables by just mapping data dictionary table records to index records. Instead I had to grep for specific character sequences against all pages. Note however that only works for text in uncompressed text columns (varchar, char, text) but what if tables don’t have any text columns at all? Then, you read further.

GNU grep won’t match binary strings. This isn’t new, I kind of knew grep couldn’t look for binary “junk”, but I really needed it to. Why? Well, here’s few of the scenarios we’ve gone through yesterday:

1. There was this rather big table with integer and enum columns only, where we knew a rather unique PK, well something like 837492636 so we needed a way to find pages that match it. InnoDB would internally store integers in 4-bytes rather than 10 bytes if it were stored as a sequence of characters, so “grep -r 837492636 /dir” would not have worked.

2. There was another table, a small one with 4 smallint columns where all we could match on was a sequence of numbers from a single record – customer knew that there was at least one row with the following sequence: 7, 3, 7, 8. Matching by any of the numbers would be insane as it would match all of the pages while matching on numbers as a sequence of characters would not work for many reasons.

This is where I found bgrep which was exactly the tool for the task. In the case number one, I have just converted number 837492636 to it’s binary representation 0×31EB1F9C and ran “bgrep 31EB1F9C /dir” – there were only like 10 other matches across the 4 gigabytes of pages, some of them probably from the secondary pages, but when you only have that many pages to check it’s really simple.

Second case seemed somewhat complicated, but it really wasn’t – all of the columns were fixed size – 2bytes each, so the thing we had to look for was this sequence: 0007000300070008. I was expecting a lot of mismatches but in fact I ended up with only one match pointing exactly to the right page and so the right index id.

The other thing I would note about bgrep – it was so much faster than matching text using grep, so if you happen to have a lot of data to scan and you have to choose between matching text and number, matching a number using bgrep may work much better.

We are considering shipping bgrep as part of percona recovery toolset, with some additional converters so we can match against various date/time columns as well.

Entry posted by Aurimas Mikalauskas | 4 comments

Add to: | | | |

We are actively developing a high-performance, on-demand load balancing solution–built in the cloud, for the cloud.

Cloud Load Balancing will allow customers to create more robust, scalable, and high availability configurations in the cloud in a matter of seconds. Imagine being able to launch a load balancing solution with the push of a button or via API. That’s what we’re building! The result is considerable time savings compared to manually configuring a solution and significant cost savings compared to going through our professional services team.

Cloud Load Balancing will include numerous features that help you ensure that your applications are online and are highly responsive:

• Advanced Algorithms – Including support for round robin, weighted round robin, least connections, weighted least connections and random. Customers will have the ability to ensure traffic is being properly routed to the back-end nodes in a way that is optimal for their workload.
• Static IP Addresses – Every load balancer is assigned a dedicated static IP address that can be selectively shared with other load balancers under a single account. IP addresses are persistent and won’t change while a load balancer is active.
• Health Monitor – In the event that a back-end node fails, the load balancer will quickly remove it from rotation. The health monitor is even capable of inspecting your HTTP traffic to ensure your workload is performing how you expect it to be.
• Simplified Maintenance – We will perform rolling application restarts that will have zero customer impact by draining your nodes and taking them out of rotation immediately.
• Advanced Access Control – You will be able to easily manage who can and can’t access the services that are exposed via the Cloud Load Balancing service.
• Session Persistence – If you are load balancing HTTP traffic, the session persistence feature will enable an HTTP cookie to ensure subsequent requests are directed to the same node in your load balancer pool.
• REST-based API – The RESTful API will let you develop custom solutions to automate the entire process of managing and controlling the Cloud Load Balancing service.

Our Cloud Load Balancing service is based on Zeus technology and built on top of our robust network, you’ll have the ability to burst and only pay for the resources used. At launch, we will support load balancing HTTP, HTTPS, LDAP, LDAPS, IMAP, FTP, POP3, POP3S and SMTP protocols on either a public interface or internal interface.

As we work through the beta development phase of this project, we’re looking for customers to evaluate and test our API. The ideal customer will be familiar with consuming RESTful web service APIs, as well as the general concepts of load balancing.

If you would like to help us raise the bar on Cloud Load Balancing, please email beta@rackspace.com. Tell us your current configuration and use case ideas for how you would use this new service.  Invitations for our private beta will start to go out this week.

We hope you are excited by this new development, we certainly are! We are also working on lots of other great additions to our services and can’t wait to share them with you in the very near future. Stay tuned!

For those who don’t know me, I’m a researcher for BBC Research & Development. For the last fifteen months I’ve been on attachment with BBC Scotland learning about how R&D interfaces with production divisions. Prior to my attachment I worked closely with Ian on Backstage events such as the Edinburgh Un-Festival and Over the Air so now I’m back in the fold I’ve been asked to work with a small team on the closedown and migration of BBC Backstage. 

A few weeks have passed since the announcement that Backstage is due to close. It’s taken a while to get up to speed but recently we’ve really started to get a clear idea on what we need to do to close down Backstage in a way that preserves its legacy and continues to support its community. Here’s a brief rundown of what we’ve been up to.

At the moment we’re working our way through the current website to working out which projects are still live. All APIs and feeds will remain available until we establish what will replace the website at which point we’ll migrate everything to its new home. We’re also looking at how to archive the content no longer in use as a resource for the future. 

We’re in discussion with a couple of organisations about the new home for the Backstage developer community. What we want to do is plug the current community into the wider groups at large in the UK looking at open and linked data. Allied to this is a piece of work looking at  how we continue to evangelise the provision of open data internally at the BBC.

To help commemorate Backstage we’ll be producing a retrospective ebook at the end of the year. Created in collaboration with Suw Charman-Anderson and Kevin Anderson it will draw on the whole of Backstage’s five year run, telling the stories of the people, projects and institutions involved in a project that crossed lines and pushed boundaries. We’ll be running a number of extracts from over the coming months. In this first one Kevin tells the story of Backstage’s birth:

“The whole idea was to do it quite slowly and quietly, don’t have a big fanfare, do it under the radar.” James Boardwell.

Out of the ashes of the dotcom crash rose a new, more open web. Instead of simply offering linked pages, sites began offering APIs - application programming interfaces that allowed developers to build additional functionality on top of existing sites.

Google launched its first APIs in April 2002, allowing developers to query its index of more than 2bn web documents. Amazon launched its Web Services three months later. Both companies gave developers a way to build applications with their content and integrate those applications easily with their sites.

The shift from static web pages to APIs and applications was not lost on Matt Locke, at the time the BBC’s head of innovation. Just after the turn of the century Locke was part of a team working on a report looking at the predicted state of broadband in the UK in 2014. The report found that,

“the BBC, which is very focused on control and broadcasting and one-to-many communications, was unlikely to be able to adapt enough to get the full affordance of network connections, social media and so on.”

Locke believed that the BBC should enable open innovation by working with lead users of new technologies to spur development. Tom Loosemore, at the time with BBC Future Media & Technology, was working with developers both inside and outside the BBC. Locke and Loosemore met at Bush House, the headquarters of the BBC World Service, and over pizza sketched out a model for innovation to engage with lead users. This model was to become Backstage.

Backstage was all about enabling the BBC to engage with the external developer community. Image courtesy of Rain Ashford.

The community would be open but self-selecting, attracting people possessing not only unique skills but also a focused passion for digital technology and the future of media. Locke asked a member of his innovation team, James Boardwell, to  manage the project.

The first step was to see what feeds already existed at the BBC. 

“A lot of data was available without anyone actually knowing it, especially around news,” Boardwell said. Many BBC sites already were producing RSS feeds but while people were aware of them, the knowledge and understanding of them was limited.

Due to the abundance of feeds from News, Loosemore went to them to find a developer for the project. Ben Metcalfe had already been doing similar work. Previously he had given a presentation to BBC News Website management about what a ‘BBC News API’ might look like. Nothing came of the meeting but he continued to build small prototypes in his spare time. One of these prototypes caught Loosemore’s eye and he asked Metcalfe to join the Backstage project.

“Use our stuff to make your stuff.” A project using Arduino kit and weather feeds to create ambient info-lighting? We can do that. Image courtesy of Rain Ashford.

In its early days the project had a informality and a daringness to it that was quite uncommon in the BBC. Instead of going to management and asking for permission for the data, Metcalfe would speak directly to other BBC developers and ask them to expose a feed. Often it was possible but just wasn’t being done.

News was their first early win, with Metcalfe working with many of his former colleagues.

“That was what we launched with. The whole idea was to do it quite slowly and quietly, don’t have a big fanfare, do it under the radar,” Boardwell said.

Metcalfe remembers Backstage then having the atmosphere of a start-up.

“In the early days it was just James Boardwell and myself working full time on the project. We both did anything that needed to be done – from working with ops guys to set the server up through to liaising with the legal department on the creation of the license Backstage made the data available under.”

Much of Boardwell’s early work was with legal teams to come up with a licence acceptable to both the BBC and the external developers they hoped to engage.

“At the time, there was a massive thing about alternative use, not allowing reuse, which obviously for developers was a  nightmare,” Boardwell said. They spent a lot of time introducing the legal teams to the idea and eventually got a license that the legal team were happy to “run with for a while.” They agreed on a wording that was sufficiently vague to allow developer use but with one important caveat: the use had to be non-commercial.

With knowledgeable and enthusiastic staff on board, a licence in place and sources for the data feeds acquired, Backstage was ready for launch.

We’ll be posting more extracts from Suw and Kevin over the next month. In the meantime please feel free to share your memories of Backstage with us and let us know your thoughts on our plans for the future.

In Zend Framework 2.0, we're refactoring in a number of areas in order to increase the consistency of the framework. One area we identified early is how plugins are loaded.

The word "plugins" in Zend Framework applies to a number of items:

• Helpers (view helpers, action helpers)
• Application resources
• Filters and validators (particularly when applied to Zend_Filter_Input and Zend_Form)
• Adapters

In practically every case, we use a "short name" to name the plugin, in order to allow loading it dynamically. This allows more concise code, as well as the ability to configure the code in order to allow specifying alternate implementations.

WTF. Why didn't you all tell me this is a fucking disaster ?

First of all, the fucking Blu Rays want to connect to the internet. Umm, no you don't. Second, they want to run fucking Java to make their menus even more fucking annoying than the damn DVD menus.

Okay, friggle frack, I hate it, but I can live with that.

Then I find YOU CAN'T FUCKING RESUME on BD-J discs !? WTF "Resume" is like the most basic feature that you need in a movie player, like I'm not allowed to fucking stop my movie and then start it again !? I love the "recommended fixes" from even the official Sony page, to just "skip ahead to where you stopped watching". Are you fucking kidding me? Somebody should be shot for this, it's like the engineers making a video format without audio. You add a bunch of menu features and shit and oh, by the way it doesn't play audio anymore. You can't break the basic fucking functionality when you add features!

Resume Play feature on Blu-ray titles - Blu-ray Forum
Recommend me a blu-ray player with resume after power off [Archive] - DVD Talk Forum
Hugely disappointed by my Blu-ray player. [Archive] - AnandTech Forums
Blu-ray Resume Playback Guide Useful Information Sony Asia Pacific

Okay, so all Blu Ray players have that problem, but the PS3 is actually even worse than most and won't even Resume from normal blu rays unless you fucking say a hail mary and cover it with garlic and press just the right buttons. And god forbid it do something like remember the location you stopped watching even after you power it down or remove the disc, like you know every fucking DVD player from 10 years ago does. ( PS3 missing resume function ).

(aside : that PS3 link is a pretty funny example of people defending their enemies :
"damn just remember the chapter before shutting down."
"quit being lazy and find the chapter where you left off and then press play on that"
"I just leave mine on 24/7 and if I stop a movie, it resumes where I left off until I eject it."
WTF R u serious? )

And of course every time you do watch a Blu Ray, you get the wonderful fucking joy of sitting through a bunch of mandatory fucking previews and other nonsense because they "forbid that operation". Yeah yeah you can usually just skip chapters to get to the menu, but you can't go direct to top menu on most discs, and you can't jump directly to playing.

How to Skip Movie Trailers [Archive] - Audioholics Home Theater Forums
Any way to skip previews on blu-ray - AVForums.com
Any tips to skip DVDBD trailers, warnings, and ads on PS3 - Home4Film
annoying mandatory previews - Blu-ray Forum

As for HDTV, it's mostly good. The fucking Bezel on my LG is glossier than greased shit, which is pretty damn awful. Also HDMI connectors are the fucking devil, they're like bad old USB connectors, they don't go in far enough and are wobbly. I have problems with the HDMI connector making good stable contact at the TV and at the PS3 and the computer.

Locking HDMI Cables and Connectors � Reviews and News from Audioholics
HDMI Connector - High Def Forum - Your High Definition Community & High Definition Resource
anyone else have problems with loose hdmi connection on tv - AVS Forum

Oh, and of course neither the TV nor the PS3 came with cables even though they're like fifty cents OEM price now. That's just fucking cheap ass annoying shit.

So far the only really huge win is that the Netflix app for the PS3 is way better than the one for the HTPC, and of course Netflix movies fucking start playing directly without a bunch of shit, and you can resume them. I would just use Netflix 100% of the time except for the fact that Brownstripe is not reliable, and there's nothing more annoying than being in the middle of a movie and your net connection decides to flake out (I wish I could fucking locally cache the whole movie before watching it to ensure full playback).

As my uptime reports have shown, and as some of you have reported, my blog's load time has increased steadily over the past few weeks. It turns out that one of my VM's was on a physical machine that had some trouble and I was reaching a point where GlusterFS's replicate functionality couldn't meet my performance needs.

Instead of using GlusterFS as I had before in my redundant cloud hosting guide, I decided to use DRBD in dual-primary mode with OCFS2 as the clustering filesystem on top of it. The performance is quite good so far:

Pingdom Response Time Graph for rackerhacker.com

I switched over the DNS late last night and the response time has fallen from the two to three second range (during times of low load) to right around one second per request. In addition to the reduced load times, I can support higher concurrency without significant performance degradation.

Don't worry - I'll make a detailed post on this topic later along with a guide on how to set it up yourself.

©2010 Racker Hacker. All Rights Reserved.

The property industry is starting to point the finger of blame, but just where to the point the finger?!  Steve Morgan, chairman of Redrow has been widely reported as  pointing his finger at lenders (aka Banks).  He is saying much of what has been said before, namely that the banks attitude to lending in the property industry is creating a mortgage famine. He remembers a life not so long ago of 95% mortgages (yes, its true).

The FSA however say that far from relaxing the rules, new tougher regulations should be put in place to prevent lenders lending to people who may not pay.  Starvation more than famine and one has to ask how many years will first time buyers now spend in the desert?   Unable to get a mortgage will they just give up and rent?  I foretold a story a while back, where a generation of first time buyers would simply rent.

But is this all really necessary? The FSA say that had their proposed measures been in place when the banks were lending to anyone who could spell their name on a form the recession would not have happened (I am ad libbing a little here, but you get the point).  Commentators (the FT I think) suggest that one should actually look at the statistics on repossessions as they don’t really tally with this statement.  So are they being over zealous?!

The Council of Mortgage Lenders (CML) think so and they have urged the FSA to think again. Redrow and the CML have a point, the 95% mortgage has vanished and the 90, 85, 80% mortgages are becoming niche products.  Indeed,  it makes one wonder how any first time buyers could raise the sort of cash required for a deposit.   The ramifications of a dried up first time buyers market are huge of course and this isn’t an issue to be dismissed.

It begs the question, why bother selling and why bother moving? Perhaps this is all a government plan (a conspiracy if you like) grind the property market to a standstill, force the masses to stay where they are then start again in 10/15 years time whenever they think it is safe!   Is the city effectively saying we don’t need a housing market ?  Could this be an element of the Kosovo plan Boris Johnson has been banging on about?  Subject for another blog maybe!

Anyway, I am off to apply for 99% mortgage just for the fun of it.

This post was written and contributed by Major Hayden, Linux Systems Engineer for the Rackspace Cloud.He’s also known as “Racker Hacker.”

One of the most common questions that I see in my favorite IRC channel is: “How can I secure sshd on my server?” There’s no single right answer, but most systems administrators combine multiple techniques to provide as much security as possible with the least inconvenience to the end user.

Here are my favorite techniques listed from most effective to least effective:

SSH key pairs

By disabling password-based authentication and requiring ssh key pairs, you reduce the chances of compromise via a brute force attack. This can also help you protect against weak account passwords since a valid private key is required to gain access to the server. However, a weak account password is still a big problem if you allow your users to use sudo.

If you’re new to using ssh keys, there are many great guides that can walk you through the process.

Firewall

Limiting the source IP addresses that can access your server on port 22 is simple and effective. However, if you travel on vacation often or your home IP address changes frequently, this may not be a convenient way to limit access. Acquiring a server with trusted access through your firewall would make this method easier to use, but you’d need to consider the security of that server as well.

The iptables rules would look something like this:

iptables -A INPUT -j ACCEPT -p tcp --dport 22 -s 10.0.0.20
iptables -A INPUT -j ACCEPT -p tcp --dport 22 -s 10.0.0.25
iptables -A INPUT -j DROP -p tcp --dport 22

Use a non-standard port

I’m not a big fan of security through obscurity and it doesn’t work well for ssh. If someone is simply scanning a subnet to find ssh daemons, you might not be seen the first time. However, if someone is targeting you specifically, changing the ssh port doesn’t help at all. They’ll find your ssh banner quickly and begin their attack.

If you prefer this method, simply adjust the Port configuration parameter in your sshd_config file.

Limit users and groups

If you have only certain users and groups who need ssh access to your server, setting user or group limits can help increase security. Consider a server which needs ssh access for developers and a manager. Adding this to to your sshd_config would allow only those users and groups to access your ssh daemon:

AllowGroups developers
AllowUsers jsmith pjohnson asamuels

Keep in mind that any users or groups not included in the sshd_config won’t be able to access your ssh server.

TCP wrappers

While TCP wrappers are tried and true, I consider them to be a bit old-fashioned. I’ve found that many new systems administrators may not think of TCP wrappers when they diagnose server issues and this could possibly cause delays when adjustments need to be made later.

If you’re ready to use TCP wrappers to limit ssh connections, check out Red Hat’s extensive documentation.

fail2ban and denyhosts

For those systems administrators who want to take a bit more active stance on blocking brute force attacks, there’s always fail2ban or denyhosts. Both fail2ban and denyhosts monitor your authentication logs for repeated failures, but denyhosts can only work with your ssh daemon. You can use fail2ban with other applications like web servers and FTP servers.

The only downside of using these applications is that if a valid user accidentally tries to authenticate unsuccessfully multiple times, they may be locked out for a period of time. This could be a big problem if you’re in the middle of a server emergency.

A quick search on Google will give you instructions on fail2ban configuration as well as denyhosts configuration.

Port knocking

Although port knocking is another tried and true method to prevent unauthorized access, it can be annoying to use unless you have users who are willing to jump through additional hoops. Port knocking involves a “knock” on an arbitrary port that then allows the ssh daemon to be exposed to the user who sent the original knock.

Linux Journal has a great article explaining how port knocking works and it provides some sample configurations as well.

Conclusion

The best way to secure your ssh daemon is to apply more than one of these methods to your servers. Weighing security versus convenience of access isn’t an easy task and it will be different for every environment. Regardless of the method or methods you choose, ensure that the rest of your team is comfortable with the changes and capable of adapting to them efficiently.

Caching is great. CloudFlare makes sites faster in part by taking the static portions of your site and moving them closer to the visitors to your site. CloudFlare is smart about only caching static objects so you can run a dynamic, database-driven site and it will behave as you expect. However, sometimes you want to edit static files like CSS, Javascript, or existing images and it can be frustrating to wait a few hours for the CloudFlare cache to expire. There are two ways to deal with this problem:

Go to Development Mode
Development Mode bypasses CloudFlare's accelerated cache and slows down your site, but is useful if you are making changes to cacheable content (like images, css, or JavaScript) and would like to see those changes right away. You can get to Development Mode by going to:
Settings->CloudFlare Settings->Development Mode->Toggle On.

Purge Cache
We read every email and tweet that we get. Often users write in with feature requests and we aim to build the most common ones quickly. This week, we released a new CloudFlare feature based on user feedback called the "Cache Purge". Some frequently asked questions about the "Cache Purge" can be found below:

Where can I find the purge cache option?
The cache purge option is found by going to Settings->CloudFlare settings.

What does the cache purge option do?
Cache purge will Immediately purge all cached resources for your website. This will force CloudFlare to expire all static resources cached prior to the button click and fetch a new version.

I have more than one domain. Do I have to purge the cache for each site?
The cache purge is done at the domain level (purging will refresh the cache for all your websites that share a domain).

Can I only purge certain file extensions?
No, not at this time. Please contact us if you would like to see us develop additional caching features.

How often should I purge my cache?
You should purge your cache if there is a cached resource that you want to immediately stop serving. If you are doing development and making constant changes, use Development Mode instead.

Keep in mind...
You will end up slowing down your site if you're constantly purging resources from your site. When you purge the cache, that means we have to fetch all requested resources from your server. It will take up to 3 days to rebuild the cache to an optimal level depending on the traffic to your site. What this really means is that your site will not be at optimal performance until the cache rebuilds.

Permalink | Leave a comment  »

I'm sure everyone else already knows this, but I've recently discovered that Zend_Validate_PostCode doesn't work with UK postcodes unless you first remove the space between the first and second parts. This is due to a bug in the underlying regular expresssions that are provided by CLDR.

It's easy enough to add a filter to remove the space, but I'm a little worried that when (and if) it gets fixed, will the fixed version Zend_Validate_PostCode then fail to validate postcodes without the space? In theory it should as the space is part of the spec. I'd hate my code to unexpectedly break due to a valid bug fix.

I can easy work around this worry though by simply creating my own extension of Zend_Validate_PostCode, so that's what I did:

library/App/Validate/PostCode.php:

<?php

/**
 * @see Zend_Validate_PostCode
 */
require_once 'Zend/Validate/PostCode.php';

class App_Validate_PostCode extends Zend_Validate_PostCode
{
    public function isValid($value)
    {
        $this->_setValue($value);
        if (!is_string($value) && !is_int($value)) {
            $this->_error(self::INVALID);
            return false;
        }

        if ($this->getLocale() == 'en_GB') {
            // From PEAR's Validate_UK - http://pear.php.net/package/Validate_UK
            $value = strtoupper(str_replace(' ', '', $value));
            $format = "/^([A-PR-UWYZ]([0-9]([0-9]|[A-HJKSTUW])?|[A-HK-Y][0-9]"
                . "([0-9]|[ABEHMNPRVWXY])?)[0-9][ABD-HJLNP-UW-Z]{2}|GIR0AA)$/";
        } else {
            $format = $this->getFormat();
        }
        
        if (!preg_match($format, $value)) {
            $this->_error(self::NO_MATCH);
            return false;
        }

        return true;
    }
}

And of course, with the benefit of ZF's plugin loader system, once I'm happy that the bug is fixed in Zend_Validate_PostCode, I'll be able to just remove my override and Zend_Form will carry on working.

Another option is to use Paul Court's Pmc_Postcode_Validate which has also been written to the UK spec.

or, alternative title: “The media and your users won’t understand or care you went for an MVP”

Path has finally been revealed to the world via one of those ‘oh lets just put a post up on our Tumblr’ anouncements on a Sunday night that suddenly is joined at the top of TechMeme with half-a-dozen-or-so considered and clearly embargoed puff pieces by each of the usual suspects.

My favorite is probably Caroline McCarthy’s CNet piece because it has a photo of the view my old apartment used to enjoy before they erected the shiny green Infinity Towers in front of it. Path is housed in a condo near the top floor of Infinity II. I guess there is a metaphor in there about the new disrupting the old, but whatev’s.

So this launch was embargoed to the hilt. And we know TechCrunch doesn’t do embargos so it almost certainly didn’t get a briefing. So it was interesting that rather than have an AolTC staffer here on the West Coast write up the buzzy launch, they had co-editor Erick Schonfeld go out of his way to post a negative missive at 3am local time to him in NYC. The TC post does smell of the editor launching a politically-motivated chastisement rather than letting one of his numerous local SF subservient staffers write it up based on the service’s merits alone. TC, after all, has to do what it can to influence against embargoes (or not giving it the exclusive, for that matter).

Biggest Minimum Viable Product so far?

For me, what is so interesting about Path is that its clearly the most visible MVP (Minimum Viable Product) launch to date since the meme/concept came to fruition. If you don’t know what that means, go check out the interview with Eric Ries at Venture Hacks.

You can only use Path on iPhone, and even then you can only upload photos – no commenting for example. No other phones are supported, not even iPod Touches. There is a web-based version but you can’t add anyone as a friend (even if you know their url), they can only add you if they happen to have an iPhone.

It would have been relatively easy for Path to have added obvious features like some kind of friending functionality on the web-based version but they clearly decided they really want you to use the iPhone or not take part yet. A total MVP approach. Build the minimum needed, test the market place and learn from the reaction. Repeat until you have success.

The issue is few people really understand MVP as a strategy, even in the industry let alone the wider public.

In an era of ‘write it up in 15 minutes to earn your $20-per-post’, bloggers and journalists will analyze what is put in front of them rather than take the time to consider where a product is on its lifecycle. That lack of understanding and need to publish fast might also explain another reason why Erick tore Path to shreds and even Mashable’s Ben Parr isn’t sure either.

When you put aside the warm-and-fuzzy puff-pieces that come out of being invited into an embargoed meeting with Path CEO Dave Morin to discuss the merits of the work of Robin Dunbar (anthropologist) and Daniel Kahneman (Nobel-laureate psychologist) while taking in views of the Bay Bridge, you have to wonder whether any of the other folks writing about Path really understand the MVP concept either.

My takeaway is that Path should be communicating its MVP’ness directly, pointing out to users where things will be changing and improving – even if that does give some of the game away to Path’s competitors. Everyone wins in the long-haul as users are more likely to stick around knowing that improvements are in the works.

And MVP proponents – such as Eric Ries, Dave McClure and the startup founders who utilize it – need to do a better job of priming the pump and getting the media on-side. I just spoke to one tech blogger who had never heard of MVP, and said that she didn’t understand the point if it meant that products were being launched that users might have difficulty using.

The Sixth Platform Preview of IE9 includes support for CSS3 2D Transforms. Similar to the use of “ms” in the Web Timing APIs, the 2D Transforms properties begin with –ms: -ms-transform and –ms-transform-origin. This post explains the purpose of vendor prefixes, how we decide when to use vendor prefixes, and how we recommend developers treat these properties.

The purpose of vendor prefixes

Vendor prefixes are a mechanism that several browsers have used for handling timing conflicts between implementations and specifications.

When writing a specification, working groups need feedback from early implementations. However, they also need freedom to change the spec without being locked into the behavior of an early implementation.

Vendor prefixes exist to balance these needs by providing browsers a clear way to tell developers a feature is experimental. Developers should expect these features might work differently across browsers. Behavior could change significantly in future releases.

Vendor prefixes are crucial to achieving the goal of making the same markup, CSS, and script work across browsers. They bridge the gap between early implementations of the early specification and a stable implementation of a stable specification. Without them, browsers might support different versions of the same specification, leaving developers to deal with the inconsistent results.

Meanwhile, everyone involved in the standardization process works together to stabilize the specification so all browsers can remove the prefix as soon as possible.

When we use vendor prefixes

If a property, method, or object’s specification is likely to change enough to make future updates not backward compatible, IE uses the vendor prefix. Convention is to use a prefix until the specification is a Candidate Recommendation (read this post for an outline of the full process). When we added border-radius, we decided not to use –ms as the specification was a Candidate Recommendation at the time. The full module is again a Working Draft to resolve box-shadow issues but border-radius is still at Candidate Recommendation stability. Other browsers either already support the unprefixed version or will in an upcoming release.

The 2D Transforms module, however, is in a different position. The specification is a Working Draft as the group continues to discuss issues. As a result, all browsers support these properties with a vendor prefix.

How developers should use properties with vendor prefixes

Vendor prefixes indicate an experimental implementation. If you’re interested in using an experimental feature in production code, it’s important you:

• Only use properties you can test. This sounds obvious, but in an attempt to “future proof” your site, it can be tempting to add the prefixed version for all browsers or add the unprefixed version before any browser supports it. Doing this puts your site at risk because with experimental features there’s no expectation that the next implementation will work like the current one or that the future unprefixed version will work like the current prefixed one.
• Test your site as browsers update. Again, the behavior is likely to change, so you need to stay up-to-date as browsers change.

Take CSS3 Gradients as an example. When Mozilla and WebKit introduced the feature, each used a different syntax and significant feedback exists on the editor’s draft. As a developer, guessing at future behavior only puts your site at risk of behaving unexpectedly as browsers respond to changes in the specification.

Using Vendor Prefixes in Samples and Tests

Vendor prefixes create a challenge for those trying to test implementations since you can’t have the same test code run in all browsers. As a result, the W3C does not accept test cases that use vendor prefixes.

Developers creating samples with 2D transforms will need to update their CSS to use the appropriate vendor prefixes. As you come across these, some may not work even in the latest IE9 Platform Preview because the author has yet to update the page.

We’re excited to have 2D Transforms in IE9 so web developers can learn, experiment, and provide feedback. We’re even more excited for reaching consensus on the specification and making this another dependable part of the interoperable web platform.

John Hrvatin
Lead Program Manager, Internet Explorer

We noticed some buzz on twitter today about JSLint moving to GitHub. JSLint is a code quality tool for the Javascript language, there's a more detailed explanation available here. The main JSLint repo can be found under Douglas Crockford's account. Here's some facts on the man if you're not familiar with him.

We're always delighted to see established projects move to GitHub to leverage our awesome community of users and the collaborative tools we're building. It's also nice to see how rapidly the Javascript community has caught up with Ruby to tie for the most popular language on the site.

Popularity is determined by the size of the files for a given language and is calculated differently than the way we determine what language a given repo is written in.

Ralph has just posted an excellent article on how many to many relationships work with Zend_Db_Table:

Basically, I’ve created a single class that effectively take the place of Zend_Db_Table_Row::findManyToManyRowset() for the purposes of creating an iterable rowset that allows access to both the target many-to-many rowset as well as the junction rowset. This solution is called a Composite Rowset. In this solution, both rowsets (iterators) are kept in sync with one another. This proves to be an ideal solution in a couple of ways. First, it will produce consistent row objects that are explicitly tied to a row in a database. Second, the cost of creating this composite rowset is at the expense of 2 queries: the original many-to-many query and a similar query to retrieve the junction rowset. This is ideal since previously, to get the junction data, findDependentRowset() would have had to been called on each row within the rowset produced by the Zend_Db_Table_Row::findManyToManyRowset().

He then goes on to show the API for the Composite Rowset class and provides an example of usage.

If you’re using Zend_Db_Table at all, have a read.

My favourite text in the article is however this:

Over the past several years of working on Zend Framework, I’ve noticed the developer population at large is really good at finding undocumented and previously unthought-of use-cases of Zend Framework components. These use-cases, while sometimes “inventive” to say the least- are also sometimes blatant misuses of a component.

That’s one of the pleasures of providing code to the world