Adam Leach

Vikram Vaswani has written an article on Manipulating Configuration Data with Zend_Config.

Zend_Config seemed to meet my needs, so I played with it a little and then deployed it in a couple of projects. It did everything I needed it to, and was easy to integrate with both framework and non-framework PHP projects. It also has a couple of neat features, such as the ability to merge multiple configurations together. Keep reading and I’ll give you a quick crash course in how it works.

He then provides details on how to use Zend_Config to read and write XML and INI config files with plenty of code examples and screenshots as appropriate.

Entrepreneurs crave information about successful startups, and they should. Most of the received wisdom about business and entrepreneurship is simply wrong. Many journalists and conference organizers attempt to fill this demand by giving successful entrepreneurs the opportunity to tell their stories: in magazines, on blogs, and on stage. And yet, most of the time, those opportunities are wasted, because the protagonists tell lies. And while this may sound like a harmless phenomenon – after all, most of these are simple lies of omission – I think the consequences are quite harmful.

I know the word “lie” sounds harsh. But I think our industry has to face up to this unpleasant reality. Luckily, some people have started to collect evidence of misleading behavior on the part of speakers. Paul Graham wrote this just the other day:

I didn't consciously realize how much speakers at more public events censored themselves till I was able to compare the same people speaking off the record at YC dinners and on the record at Startup School. YC dinner talks are much more useful, because the details people omit in more public talks tend to be the most interesting parts of their stories. About half the interesting things I know about famous startups, I learned at YC dinners.

I commend Paul for his honesty, something I have always admired about him. Paul has been – intentionally or unintentionally – running a perfect science experiment to answer the question: do startup speakers tell the whole truth on stage? His results are the same as I’ve observed on many occasions: the simple answer is no. I think everyone who’s been around these events long enough knows this to be true. None of the really successful people I know take what’s said at public events seriously. This is the same issue we see with vanity metrics: companies are giving the appearance of sharing information while actually engaging in spin or outright deception.

I call this the vanity ratio: the amount of apparently interesting information given divided by the amount of useful information contained therein. The higher the vanity ratio, the more effective the PR. Unfortunately – also – the more misleading the story is as a help to others.

And it’s not as if the stories we hear about entrepreneurs are biased in a random way. Paul quotes one of his founders like this: “That's the actual beauty in the off-the-record-ness: you hear just how screwed up most of these successful startups were on the way up.” In my experience, the official stories are always more linear, make the founders and investors look smarter, and dramatically overstate the level of certainty everyone had at every stage of the process. Failures, pivots, and crappy minimum viable products are generally elided. And the kinds of failures that do get airtime are usually failures to adequately plan, anticipate, or design in advance. So, naturally, the kinds of inferences we make from these stories are: we need heroic entrepreneurs, with absolute certainty in a brilliant idea, and we need to plan and execute well.

I call this the mythological-industrial complex, because it serves the interests of many players in preserving the status quo. It sells newspapers and magazines. It helps investors boost their profile and convince entrepreneurs that they offer value-add. It helps companies with PR. It makes successful founders famous. I've certainly been a beneficiary of these forces. Yet I worry that it deters new entrants from disrupting incumbents: if your idea looks a little dubious, your career a little messy, your team a little dysfunctional, if you lack superhuman design skills, maybe you should just give up. You don’t have the Right Stuff to become an entrepreneur.

The reason this is harmful is that it benefits another kind of person even more: the startup advisor. Yup, that’s me. When all of the stories floating around are bogus, there is a huge opportunity for people to claim to have arcane or esoteric knowledge. Paul does it right in his essay: “About half the interesting things I know about famous startups, I learned at YC dinners.” Doesn’t that sound good? Don’t you believe that having that special knowledge would help Paul give you better advice? I emphatically do. Most of my mentors have had this arcane knowledge, and I have benefited from it immensely. They know how the game is really played. They know what really happened in those pivotal moments that shaped famous companies. And I have done my best to pay it forward, by sharing that arcane knowledge with as many startups as I can.

However, the problem is this: how can you tell if someone who claims to have arcane knowledge is telling you the truth? After all, the fact that what they advise you contradicts all public sources of knowledge is part of the pitch. And, of course, it sounds good. All talking heads (including me!) face an overwhelming incentive to say whatever it is that will sound good to their audience, regardless of whether it is true. Believe me, it is hard to resist (as far as I can tell, that’s why cable news is so awful: pretty much everyone on cable news is giving in to this temptation every day). But I have met many startups that are making key decisions based on arcane knowledge that is simply not true. It is extremely hard to help them, because they are following voodoo advice that is nearly impossible to disprove. In fact, in some of my early entrepreneurial ventures, I was taken in by people just like that.

I have tried really hard to hold myself to a high standard in all of my public presentations: to give the unvarnished truth about what it was really like to succeed and fail. I do my best and yet I probably fail sometimes, too. In fact, I rely on readers and attendees to ask me the hard questions that challenge me to root out and eradicate these errors. And this was an absolute requirement of the speakers at sllconf. The reason we had such a non-diverse lineup (much to my chagrin) is that I insisted on hosting speakers whose stories I already knew – because I had been there, as a friend, advisor, or investor. Thus, I knew they were telling the truth and I knew they shared a commitment to speak with integrity even when it’s uncomfortable. This is also why I get so outraged when people treat Lean Startup like a religion. The whole point of transforming advice into the form of a testable theory is to allow people to evaluate whether it works. You can test theories like Lean Startup in your own practice, and discover if they deliver the outcomes they predict – and you can do it in small batches.

That’s why I find lying on stage so upsetting. By misleading future entrepreneurs, we put them in an untenable position. Either take the stuff you read and hear at face value or gain access to a wizard who can guide you to the true path (and hope you don’t get taken in). But this false choice is not the only way forward. As usual, I think there’s an opportunity for our industry to do better.

To be clear, I don’t blame any particular actor in the system for doing what they’re doing. Paul Graham is getting his startups the best advice he can in the best way he knows: by giving them exclusive access to off-the-record conversations that nobody else is privy to. And it’s not reasonable to blame him for their behavior when he gives them the stage at events like Startup School. Similarly, journalists print vanity metrics because that’s all companies will release. And companies crave positive PR and control over their message – all for rational reasons.

And yet you who are reading this right now have tremendous power. You are the intended audience for all of those expenditures of energy. You are the “hits” that websites crave, the followers and the RSS subscribers. Where you put your attention and what you do with it matters.

So I’d like to suggest the following: let’s stop giving lying on stage and vanity metrics a free pass. I think if we can delegitimize this behavior, we can make it stop. We’ll need to ask tougher questions, though: how do you know your company’s success was caused by X? what else was happening around that time that might have caused Y? what did you think was going to happen when you did Z?

Journalists have the highest obligation to ask these kinds of questions, and conferences organized by journalists ought to be the exemplars the rest of us look up to. And yet, today many are not. I am sympathetic, because I have faced this problem, too. If you ask tough questions, are unwilling to help people craft their message, and are skeptical of vanity metrics, you can’t get the high profile guests. That means less attention, less coverage, fewer readers, and lower sponsorship dollars. Assuming, of course, that all of you give your attention disproportionately to famous people.

But if you’re attending a conference or reading a magazine, you aren’t bound by those rules. You can ask any question you want. And you can reward the speakers who tell the whole truth with your support.  Every time you do that, you’re helping make our industry a better place.

(Have a favorite real startup story? Share it in the comments and we can start giving those people some much-deserved attention right away.)

I had a twitter/IRC exchange yesterday with Andries Seutens and Nick Belhomme regarding applications that include widgets within their layout. During the exchange, I told Andriess not to use the action() view helper, and both Andriess and Nick then asked how to implement widgets if they shouldn't use that helper. While I ended up having an IRC exchange with Nick to give him a general idea on how to accomplish the task, I decided a longer writeup was in order.

Continue reading "Using Action Helpers To Implement Re-Usable Widgets"

Matthew Weier O’Phinney has just published an article on how to create reusable widgets with action helpers.

I had a twitter/IRC exchange yesterday with Andries Seutens and Nick Belhomme regarding applications that include widgets within their layout. During the exchange, I told Andriess not to use the action() view helper, and both Andriess and Nick then asked how to implement widgets if they shouldn’t use that helper. While I ended up having an IRC exchange with Nick to give him a general idea on how to accomplish the task, I decided a longer writeup was in order.

He then goes on to explain how to create a module that does not have controllers and actions, but instead uses an action helper to create a widget that allows a user to log in.

This post was written and contributed by Joseph Ruscio, CTO for Librato, a Rackspace Cloud Tools Partner. Prior to Librato, he also worked at Motorola, designing and implementing embedded cellular infrastructure software.

Innovative platforms like Rackspace Cloud are truly changing the game in IT. They enable you to deploy and scale “just in time”, driving up overall efficiencies as workloads ebb and flow. This stands in stark contrast to the norm of just a few years ago (one that persists in legacy data-centers) where capacity is statically provisioned to handle some theoretical peak load. As a rule these servers sit mostly idle during normal periods of activity, and are susceptible to failure if peak demands were underestimated.

As powerful a tool as elastic computing may be however, it’s effectiveness is ultimately determined by your ability to understand and manage the applications you deploy on top of it. Legacy tools focused on server-level activity fail to provide sufficient oversight and command of your applications. You develop/test/deploy applications, not servers and your production tools should reflect that business reality.

Librato’s Silverline provides the capabilities you need to go beyond the server boundary and systematically monitor and control your entire application stack. Our user-level technology seamlessly instruments your applications at the system-call layer without requiring any code changes or recompilation. It both tracks and manages application utilization of CPU, Memory, Disk I/O, and Network I/O. It can handle any complex application ranging from Ruby web applications, J2EE enterprise applications, proprietary databases, and even the venerable sshd. Within minutes you can create workload management policies (e.g. ‘the web servers take precedence over the background workers’), monitoring tags, and set up custom threshold-based triggers all centered around applications, not servers. It does all of this and more with an almost non-existent performance overhead of typically less than 1%.

Designed and implemented for the age of elasticity, Silverline is comprised of a single package to install (easily automated through chef, puppet, etc) and all of your application management policies and monitoring data are securely stored and accessible to you as a service through the Silverline dashboard. Your Silverline dashboard enables you to rapidly pinpoint where you are either under or over-provisioned. We support Linux and Windows distributions to provide a single application monitoring and application management solution for your heterogeneous deployments and bill elastically in increments of a CPU core-hour. You can test the service with a 30-day trial – up to 8 CPU cores are always free.

Want to learn more about Librato Silverline? Sign up for the upcoming webinar.

Date: Thurs, Nov 4th

Time: 1pm CDT

Click here to sign up.

People often ask me if they can buy "Lean Startup in a box" from me. It's a strange thing, to be asked by a potential customer if they can give you copious amounts of money, and then to have to refuse. Startups run into this problem all the time: not every possible way of making money is equally useful. On the other hand, I try to keep in mind the idea that feedback is always about them and never about you.I recognize the need being expressed by this request. People want to get started with Lean Startup but aren't exactly how. If you're in that boat, today my friends at Appsumo are trying a new experiment that just might help you out.

It's called the "The Lean Startup Bundle" and it is available this week only. It is the ultimate product development get-started guide to Lean Startup. We did our best to package as much content (including almsot a dozen books and ebooks) as well as tools and apps, into one low-priced magic offer. Naturally, the bundle includes every essay I have written for this blog, as part of the Startup Lessons Learned ebook series.

But I am even more excited to report that it will include a hardcover copy of my new Lean Startup book. That's right, I am writing an old-fashioned, traditionally-published book full of all new material. Because of the traditional publishing industry's commitment to rapid iteration, the book won't come out until Fall, 2011. But if you buy The Lean Startup Bundle this week, you'll get a pre-order of the book included. That means you will get a hardcover copy of the book in the mail when it is eventually published. (Believe me, you will be hearing a lot more about this book over the course of the next year, so that's all I will say about it right now.)

But that's not all! I'm even more excited to announce two brand-new ebooks from two of my favorite bloggers: Sean Ellis and Andrew Chen. These ebooks were created by my friends at Leanpub especially for this bundle, and include the best essays written by both authors (and a foreword from yours truly). Also included is the incredible Venture Hacks Bible, which is worth the price of admission on its own, and The Entrepreneur's Guide to Customer Development (which I previously reviewed here).

And there's so much more. For two awesome books which have been traditionally published, we have several ebook chapters: Inbound Marketing by Dharmesh Shah, Do More Faster: TechStars Lessons To Accelerate Your Startup by Brad Feld & David Cohen. And also making its debut, you'll get a sample of Ash Maurya's forthcoming ebook Getting Lean.

"But," I hear you say, "so far this sounds like just a bunch of gurus trying to sell me books." Indeed, but - as they say - that is not all. We also reached out to companies that make products useful to accelerating your Build-Measure-Learn feedback loop, from cloud-hosted Selenium provider Sauce Labs (disclaimer: where I have equity) to ScrumPad (where I don't) and many in-between (a sampling is below, but you should really just click through to see the whole smorgasbord). (And even if you don't buy this bundle, one beneficial side-effect of my research for it is this cool list of Lean Startup tools curated by twtpick.in. Because of the short timeline in putting this all together, we couldn't include them all. But, if this bundle is successful, perhaps we'll have a chance to do another.)

Phew, I'm exhausted. "But," you are sure to ask, "how much will all this bundling cost me?" How about $1000? $500? $250? No! All of those prices are much too high. "Ok," you surely say, "how about $100?" Absolutely not. "$75?" No. "$65?" Absurd.

No, all of those prices are incorrect. If you act now, you will pay not one dollar more than $42. Don't panic. Buy now.

Package your favorite software as a ServerTemplate or simply contribute your best ServerTemplate to the cloud community to demonstrate your expertise and win an iPad! Having recently introduced publishing to our full user base we’re now having a contest to recognize the most valuable ServerTemplates published into the RightScale Library. We will pick a “Most Innovative”, a “Most Useful”, and a “Most Popular” ServerTemplate. The three winning authors will be awarded an iPad each at the upcoming RightScale User Conference November 3rd in Santa Clara and their ServerTemplates will be featured in the RightScale Library! Though we’d love to see you at our user conference, you don’t need to be present to win.

ServerTemplates are reusable components for configuring a complete server running a particular software stack in the cloud. It only takes a few minutes to put together the best practices that you’ve been developing for running your favorite software and development environments and publish them for the community to leverage. You’ll get feedback and perhaps improvements in return. We’ll evaluate all published ServerTemplates based on their originality and relevance to the RightScale community to choose the three winners for the ServerTemplate Showdown. Submissions are always welcome from RightScale users, systems integrators, and from software vendors distributing pre-configured application stacks for deployment to the cloud, but to level the playing field, ServerTemplates submitted by software vendors are not being considered for the contest. The final submission date for this contest is 11:59pm PDT Friday, October 29th.

To publish a ServerTemplate, just click on the ‘Publish to Library’ button from a committed version of the ServerTemplate in the Dashboard. You’ll have the opportunity to enter a short and long description with hyperlinks. The short description should summarize what the ServerTemplate contains and the value / functionality it delivers. The long description should contain additional detail as well as any documentation on how to configure and deploy the ServerTemplate. You can also include recommended steps to showcase the server’s functionality and benefits. Keep in mind that you can easily link to other web pages in the descriptions if needed. ServerTemplates with intuitive input labels and clear documentation save everyone time and effort. A guided walk-through of the publishing process is available on the RightScale Support Site.

We’ve also created a new Community ServerTemplate Publishing forum where you can go with questions, requests or discussions related to ServerTemplate creation and publishing. If you would like to see a ServerTemplate and have yet to build it, place a post here to see if someone else has done it or may be able to help.

I’m looking forward to some surprising and exciting ServerTemplates!

Filed under: cloud computing, Thorsten

John Engates, CTO at Rackspace, does a quick interview in the lobby of Moscone Center at VMworld 2010. He talks about OpenStack and what it means for the Cloud.

Since the OpenStack community was founded at OSCON late July, there has been a series of exciting new developments including:

• Hiring of two key employees: Anne Gentle, OpenStack Content Stacker and Stephen Spector, OpenStack Community Manager
• New features in the code base today include:  managing virtual machines, network configuration for those VMs, a fully distributed, replication storage system and the ability to store, retrieve and replicate objects and files up to 5GB in size
• You can also now manage the OpenStack environment via a web control panel, iPad, iPhone and Android device.

Make sure to sign up for the 2nd OpenStack Design Conference coming in November:

Dates: November 9th-12th

Location: The Weston Centre in Downtown San Antonio, TX

Address: 112 E. Pecan St. San Antonio, TX 78205

Click here to read a full update.

Stay up to date with OpenStack:

Follow on Twitter @openstack, join the IRC channel freenode #openstack, read the blog openstack.org/blog. Want to contribute? Click here.

At Yahoo!, we recently implemented a stronger notion of security for the Hadoop platform, based on Kerberos as underlying authentication system. We also successfully enabled this feature within Yahoo! on our internal data processing clusters. I am sure many Hadoop developers and enterprise users are looking forward to get hands-on experience with this enterprise-class Hadoop Security feature.

In the past, we've helped developers and users get started with Hadoop by hosting a comprehensive Hadoop tutorial on YDN, along with a pre-configured single node Hadoop (0.18.0) Virtual Machine appliance.

This time, we decided to upgrade this Hadoop VM with a pre-configured single node Hadoop 0.20.S cluster, along with required Kerberos system components. We have also included PPig (version 0.7.0), a high level SQL-like data processing language used at Yahoo! and Oozie (version 2.2.0), an open-source workflow solution to manage and coordinate jobs running on Hadoop, including HDFS, Pig, and MapReduce.

This blog post describes how to get started with the Hadoop 20.S VM appliance. The basic information about downloading, setting up VM Player, and using the Hadoop VM is same as described in the tutorial module-3 — except the user has to use the following information and links to download the latest VM Player and Hadoop 0.20.S VM Image. You should also review the following information for security-specific commands that need to be performed before running M/R or Pig jobs.

For more details on deploying and configuring Yahoo! Hadoop 0.20.S security distribution, look for continuing announcements and details on Hadoop-YDN.

Installing and Running the Hadoop 0.20.S Virtual Machine

• Virtual Machine and Hadoop environment: See details.
• VMware Player installation: See details. To download latest VMware Player for Windows/Linux, go to VMware site.
• Virtual Environment for Hadoop 0.20.S set up: See the following instructions.

Copy the Yahoo Cloud VM into a location on your hard drive. It is a zipped vmware folder (hadoop-vm-appliance-0-20-S), which includes a few files: a .vmdk file that is a snapshot of the virtual machine's hard drive, and a .vmx file that contains the configuration information to start the virtual machine.

After unzipping the vmware folder zip file, to start the virtual machine you double-click the hadoop-appliance-0.20.S.vmx file. Note: The uncompressed size of the hadoop-vm-appliance-0-20-S folder is ~2GB. (Also, based on the data you upload for testing, VM disk is configured to grow up to 20GB).

When you start the virtual machine for the first time, VMware Player will recognize that the virtual machine image is not in the same location it used to be. You should inform VMware Player that you copied this virtual machine image (choose "I copied it"). VMware Player will then generate new session identifiers for this instance of the virtual machine. If you later move the VM image to a different location on your own hard drive, you should tell VMware Player that you have moved the image.

After you select this option and click OK, the virtual machine should begin booting normally. You will see it perform the standard boot procedure for a Linux system. It will bind itself to an IP address on an unused network segment, and then display a prompt allowing a user to log in.

Note: The IP address displayed on the login screen can be used to connect to VM instance over SSH. The Login screen also displays information about starting/stopping Hadoop daemons, managing users/passwords, and shutting down the VM.

Note: It is much more convenient to access the VM via SSH.

Virtual Machine User Accounts

The virtual machine comes pre-configured with two user accounts: "root" and "hadoop-user". The hadoop-user account has sudo permissions to perform system-management functions, such as shutting down the virtual machine. The vast majority of your interaction with the virtual machine will be as hadoop-user. To log in as hadoop-user, first click inside the virtual machine's display. The virtual machine will take control of your keyboard and mouse. To escape back into Windows at any time, press CTRL+ALT simultaneously. The hadoop-user user's password is hadoop. To log in as root, the password is root.

Hadoop Environment

Linux: Ubuntu 8.04
Java: JRE 6 Update 7 (See License info @ /usr/jre16/)
Hadoop: 0.20.S (installed <@ /usr/local/hadoop, /home/hadoop-user/hadoop is symlink to install directory)
Pig: 0.7.0 (pig jar is installed @ /usr/local/pig, /home/hadoop-user/pig-tutorial/pig.jar is symlink to one in install directory)
Oozie: 2.2.0 (oozie client is installed @ /usr/local/oozie and server is installed at /usr/local/tomcat/webapps/oozie, /usr/local/oozie and /usr/local/tomcat are symlinks)

Login: hadoop-user, Passwd: hadoop (sudo privileges are granted for hadoop-user). The other users are hdfs and mapred (passwd: hadoop).

Hadoop VM starts all the required hadoop and Kerberos daemons during the boot-up process, but in case the user needs to stop/restart:

• To start/stop/restart hadoop: login as hadoop-user and run 'sudo /etc/init.d/hadoop [start | stop | restart]' ('sudo /etc/init.d/hadoop' gives the usage)
• To format the HDFS & clean all state/logs: login as hadoop-user and run 'sudo reinit-hadoop'
• To start/stop/restart Kerberos KDC Server: login as hadoop-user and run 'sudo /etc/init.d/krb5-kdc [start | stop | restart]'
• To start/stop/restart Kerberos ADMIN Server: login as hadoop-user and run 'sudo /etc/init.d/krb5-admin-server [start | stop | restart]'

To shut down the Virtual Machine: login as hadoop-user and run command 'sudo poweroff'

Environment for 'hadoop-user' (set in /home/hadoop-user/.profile):

$HADOOP_HOME=/usr/local/hadoop
$HADOOPCONFDIR=/usr/local/etc/hadoop-conf
$PATH=/usr/local/hadoop/bin:$PATH

Running M/R Jobs

Running M/R jobs in Hadoop 0.20.S is pretty much same as running them in non-secure version of Hadoop. Except before running any Hadoop Jobs or HDFS commands, the hadoop-user needs to get the Kerberos authentication token using the command 'kinit'; the password is hadoopYahoo1234.

For example:

hadoop-user@hadoop-desk:~$ cd hadoop
hadoop-user@hadoop-desk:~$ kinit
Password for hadoop-user@LOCALDOMAIN:  hadoopYahoo1234
hadoop-user@hadoop-desk:~/hadoop$ bin/hadoop jar hadoop-examples-0.20.104.1.1006042001.jar pi 10 1000000

For automated runs of hadoop jobs, a keytab file is created under the hadoop-user's home directory (/home/hadoop-user/hadoop-user.keytab). This will allow user to execute the "kinit" without having to manually enter the password. So for automated runs of hadoop commands or M/R, Pig jobs through the cron daemon, users can invoke the following command to get the Kerberos ticket. Use command 'klist' to view the Kerberos ticket and its validity.

For example:

hadoop-user@hadoop-desk:~$ cd hadoop
hadoop-user@hadoop-desk:~$ kinit -k -t /home/hadoop-user/hadoop-user.keytab hadoop-user/localhost@LOCALDOMAIN
hadoop-user@hadoop-desk:~/hadoop$ bin/hadoop jar hadoop-examples-0.20.104.1.1006042001.jar pi 10 1000000

Running Pig Tutorial

The Pig tutorial is installed at "/home/hadoop-user/pig-tutorial". Example commands to run the Pig script are given in "example.run.cmd.sh". The Data needed for Pig scripts are already copied to HDFS. See more details about the Pig Tutorial at Pig@Apache:

hadoop-user@hadoop-desk:~$ cd pig-tutorial
hadoop-user@hadoop-desk:~$ sh example.run.cmd.sh

Running Oozie Tutorial

The Oozie tutorial is installed at "/home/hadoop-user/oozie-tutorial". The example commands to run Oozie workflow are given in "readme". The Data needed for all Oozie workflow jobs are already copied to HDFS. See more details about the Oozie Tutorial at readme:

hadoop-user@hadoop-desk:~$ cd oozie-tutorial
hadoop-user@hadoop-desk:~$ vi readme

If the HDFS is re-formatted or the data (hdfs://localhost:8020/user/hadoop-user/examples) is removed, please upload the examples directory (/home/hadoop-user/oozie-tutorial) to hdfs manually.

hadoop-user@hadoop-desk:~$ hadoop fs -put examples /user/hadoop-user/

Shutting down the VM

When you are done with the virtual machine, you can turn it off by logging in as the hadoop-user and running the command 'sudo poweroff'. The virtual machine will shut itself down in an orderly fashion and the window it runs in will disappear.

Last but not least, I would like to thank Devaraj Das and Jianyong Dai from the Yahoo! Hadoop & Pig Develoment team for their help in setting up and configuring Hadoop 0.20.S and Pig respectively.

---

Notice: Yahoo! does not offer any support for the Hadoop Virtual Machine. The software includes cryptographic software that is subject to U.S. export control laws and applicable export and import laws of other countries. BEFORE using any software made available from this site, it is your responsibility to understand and comply with these laws. This software is being exported in accordance with the Export Administration Regulations. As of June 2009, you are prohibited from exporting and re-exporting this software to Cuba, Iran, North Korea, Sudan, Syria and any other countries specified by regulatory update to the U.S. export control laws and regulations. Diversion contrary to U.S. law is prohibited.

---

CloudFlare passed a big milestone yesterday: over the last 30 days we've served more than 1 billion requests. The CloudFlare network is architected to handle many, many times this level traffic, but it's still pretty amazing to realize the extent of what already powering.

Permalink | Leave a comment  »

(I am often asked to explain how to apply Lean Startup approaches to domains beyond software. In order to answer, I have taken to drawing a two-axis diagram. 

On one axis we have the degree of market uncertainty for a given industry. For "cure for cancer" type businesses, there is no question about who the customer is and what the customer wants, and therefore there is no market uncertainty. On the other extreme, modern web-based applications face almost no technical risk, and are governed by high market uncertainty.

On the other axis we have the underlying cycle time of the industry in question. Slow-moving cycles, like drug discovery or new automobile models, govern the slow part of the axis. On the extreme opposite end are rapid iteration businesses like software or fashion.

The key to understanding Lean Startup is to recognize two things:

1. Lean Startup techniques confer maximum benefit in the upper-right quadrant, namely high market uncertainty coupled with fast cycle time.
2. Every industry on Earth is currently undergoing a disruption that is causing it to move along both axes: more uncertainty and faster cycle times.

I am aware of no industries that are moving "backwards" on either dimension. Thus, more and more industries are starting to look like the software business. Of course, the underlying root cause of this worldwide disruption is the software and semiconductor revolution. Industries are disrupted as their traditional work process is "infected" by software. And, as a result, more and more companies are able to benefit from Lean Startup practices. 

The following case study looks at one such industry, consumer electronics, where the pace of iteration has taken a marked turn towards high speed. It is written by Ronald Mannak, who is currently the CEO of a startup named Yobble. What follows are solely his opinions. -Eric)

In a bar in Amsterdam in 2005, my two cofounders and I came to the sad conclusion that startup we tried to built for two years was doomed. In 2003 we started developing a martial arts motion sensing toy, a full three years before the Nintendo Wii changed the world of motion sensing. The toy (we called it Ninja Master) consisted of two hardware units, attached to both wrists. When a child would perform a perfect karate move (or better yet: a combo of several karate moves in a row), Bruce Lee-like karate sounds would emerge from a small speaker in the device. We loved the product. Test users loved the product. It was way ahead of its time. We thought we were visionaries and believed the future was motion control. Yet, we failed to sell the toy. We talked to every toy company imaginable, but none wanted to license our toy. " Kids nowadays don't want to move, they play Playstation" was the most often heard reply, even though our user tests suggested otherwise. To make matters worse, we lived in a country (Holland) without a proper functioning startup, VC and angel ecosystem. The company was doomed. My co-founders decided startup life wasn't for them.

However, one new idea emerged at that meeting. What if we could make an air drum? Drum sticks with sensors in them. Now that was an idea. Music is much easier to sell (to toy companies) than the abstract martial arts Ninja Master toy. Besides, we could easily expand the line with with an air guitar and a device to link the air instruments to a PC. How cool. I loved the idea so much that I decided to pursue the idea.

I envisioned the product would be popular with 8 to 12 year old boys. I thought the price couldn't be higher than $40. I already knew how the product would be used. Boy, was I wrong.

Waterfall
I previously worked on a couple of IT projects that used the 'waterfall model' where specifications were written down by one team, thrown over an imaginary wall and implemented by another team. Every single waterfall project I encountered turned out to be a disaster in every way. Specifications turned out to be open to multiple interpretations, usability was the last priority (if a priority at all). It Just Did Not Work. As a beta tester of the first Borland Delphi, I learned the wonders of rapid prototyping and fast iterations. I wondered if we could do the same for hardware development. It turned out we indeed could.

The first hire
The first hire was critical. I wanted somebody who was creative first and technical secondly. I found the perfect person at the department of Industrial Design Engineering of the Delft University of Technology. Joris. Joris was creative and eager to learn. Better yet, he plays drums. Even better than that: he likes to tinker with electronics. Hiring him was a no brainer, and he didn't disappoint.

The internship only lasted six months. That's not much time, considering the scope of the project. I convinced the university that Joris should not be writing specifications and other nonsense first, but start right away building prototypes. And he did.

Joris suggested that before he started working on electronics, we should invite children, give them wooden drum sticks and let them pretend they were playing air drums. It turned out to be an excellent idea. Children are perfect test subjects. To our surprise, every single child did something we didn't anticipate. Without any exception, they all whacked the wooden drum sticks *sideways* and made 'crashing sounds'. I certainly didn't think of sideways movements when I created the first ideas, but apparently it was a good idea to implement.

The prototypes
The next day we started building the first prototype to see if the sensors actually behaved like they were supposed to, and to see if we could measure the sideway movements. The prototype was crude. Joris taped sensors on his arms with duct tape and started drumming in the air with wooden drum sticks (that did not contain any electronics). We connected the sensors to a seven year old pc with an Arduino-like interface that ran a simple drum program we developed. The results were amazing. It actually worked. (A video of the first prototype can be found here.)

We now knew what kids liked and we knew the product was technically feasible. Yet, I still felt we didn't know all we needed to know and wanted to test more. And I'm glad we did.

For the next prototypes we placed the sensors in PVC pipes to optimize sensor angles and added features to the pc software.

We made another discovery we did not anticipate. We found out that parents (who came along with their children for user testing) often liked the prototype as much as their kids. We decided to interview the parents and quickly found out that the parents who like our product were video games players. Of course we liked our product, but we never would have guessed other grown ups would like our product too. Knowing this, we invited test users from 12 to 30. They also loved the prototypes. Our target audience just exploded in size. We decided to make a few changes that would make the product less 'toy' and more 'gadget'.

Over a period of six months, we made eight generations of prototypes, each version adding more features and making the product more reliable. By testing each generation, we learned that a lot of hypotheses were correct, but a large number of hypotheses were incorrect. By testing early and often, we were able to adjust the product. I believe that we demonstrated that it is indeed possible to iterate fast and often with hardware development.

Product Launch
After some financing-related delays, the products went on sale in Europe and Asia in the summer of 2008. The retail selling price was $40, exactly what we targeted. In less than six months, we sold over 90,000 units. All shops sold out our products two months before christmas, all without spending one penny on marketing. The products were voted 'best music gadget' on television program The Gadget Show, became the best selling music toys on Amazon.co.uk and the best selling products on Firebox.com. Best of all, users love the products. On Firebox.com, the average user rating (740 users) is 4.5 out of 5 stars (link). We couldn't have been happier.

Post mortem
We demonstrated it is possible to iterate often and fast. I believe a lot of the product's success can be attributed to the iterative development process. We didn't find every issue though. We didn't test the price and we didn't see the Nintendo Wii or Guitar Hero coming. We chose to enter the market though the low margin toy market, where (in hindsight) we should have positioned the products as video games with higher video games margins.

Another thing we missed: after we launched we received many requests to add double bass drum, as often used in metal. The drums include two drum pedals and a double bass drum could have been added by a simple and minor change to the embedded software. However, updating the embedded software in sold devices isn't possible with the microcontroller we used. We could have included the feature in a 1.1 version of the product, but the toy manufacturer we licensed the toy to, wasn't interested in a new version, as the original version continues to sell well to this day.

Tools to develop hardware get better and cheaper. Open source projects like Arduino and SuperCollider make iterative hardware development cheaper and faster than ever. We learned that connecting the prototypes to PCs and user the PC to run the program is a very good way to test hardware (developing on a PC is still much faster than embedded developing on a standalone hardware device).

In the summer of this year, I moved to San Francisco and founded a new startup that makes music related games and hardware controllers that connect to the iPhone. There are a lot of new opportunities. New cheap flashable micro controllers make firmware updates possible for low cost hardware. With hardware connected to the internet (in our case through the iPhone) it should be possible to use continuous deployment: small and very frequent updates of the firmware instead of less frequent large updates. Bugs in firmware could be fixed within minutes or hours instead of weeks or months.

(Continuous deployment of hardware is an exciting new capability. In addition to continuous deployment of firmware via the Internet, it is also possible to do continuous deployment by taking advantage of a small-batch production process. When the complete cycle time of assembly is low and the design can be specified mostly through software, it's conceivable that each batch rolling off the line could have a different design. -Eric)

As a final thought, I am convinced iterative design depends mostly on the mindset of the team and the company culture and less on the tools. I was lucky to have a great team of A-players that were willing to take responsibility and risk. If the company culture is such that mistakes are punished, I am pretty sure iterative development won't work.

AcceloWeb, a Rackspace Cloud Tools Partner, is an advanced website optimization software service that enables virtually any website to render incredibly fast. AcceloWeb is completely transparent and requires no server side code changes or browser side plug-ins. Its capabilities can be fully experienced as a Rackspace Cloud service in a shared or dedicated configuration option.

Key benefits include:

• Increased user traction/adoption and customer satisfaction– fewer page load delays translate to better user experience and increased website traffic
• Improved sales conversion rates – resulting from customers experiencing less frustration with slow-to-load shopping, e-commerce and media-oriented web pages
• Reduced risk of churn and abandonment– near instantaneous web page load times keep users’ fingers off those “Back” buttons
• Increased revenues – even the smallest of time savings generate significant increases in page views and revenue
• Safe and reliable – optimization requires no website code changes or client-side plug-ins; accelerated websites’ usability, appearance and functionality remain intact

Peter Spiliakos, EVP World Wide Sales for AcceloWeb and Gil Shai, COO & Co-Founder of AcceloWeb will be hosting a webinar that will show you how to take advantage of AcceloWeb on The Rackspace Cloud.

Date: Tues, Oct 26th
Time: 11am PDT, 1pm CDT, 2pm EDT

Click here to sign up.

Lew Moorman, President of Cloud at Rackspace and Chief Strategy Officer, recently did an op-ed for SandHill.com discussing how the cloud is enterprise ready. “Cloud computing is for everyone but not for everything,” Lew is often quoted saying and enterprises are slowly discovering which pieces of their IT infrastructure fit in the cloud and which pieces are optimal to keep in-house or hosted on dedicated gear. This hybrid approach allows businesses, small and large, to run more efficiently and save money where appropriate.

Lew goes on to discuss the various cloud definitions, the open-source movement into the cloud and the opportunities available for software vendors.

Click here to read the full op-ed.

With the launch of OpenStack, enterprises can avoid vendor lock-in when coming to the Rackspace Cloud. It also serves as an opportunity for developers and vendors to offer services to enterprise customers to get their applications ready for the Cloud – from monitoring to integration to scaling and more. Click here to learn more about OpenStack.

(for those not familiar with the back story, check here).

Having become clear Libya has no intention of ever releasing the vb.ly domain back to Violet, we have course-corrected by reopening her url shortening service under the new domain vbly.us.

We had originally pursued a number of other 2-letter options (eg xx.yy) to relaunch the service under. However, given what has happened with Libya and the .ly space we decided to avoid any ccTLD that has regulations that were unclear or open to interpretation. Ultimately, we don’t want what happened with Libya happening again to the service’s users, so for the stability and assurance of the service going forward we elected to register a .us domain.

A .us domain, registered and owned by a US Citizen (Violet) and hosted on servers in US (Newark, New Jersey to be exact) ensures that the service completely resides under US jurisdiction and benefits from full First Amendment Rights.

The domain vbly.us also means that you can easily modify your existing urls to work with the new domain. Thus http://vb.ly/yoururl becomes http://vbly.us/yoururl and will work as before. No urls were lost or forgotten during the downtime.

At a higher level, the “vb.ly saga” has been an interesting experience which I will write a more reflective post about shortly. Issues to be discussed range from the wonders of elastic server instances to meet the worldwide media influx through to dealing with a highly sensitive and emotionally charged issue such as Islam and the legal implications of Sharia law.

In the meantime, I’m sorry to all the users of the service who were let down by what had happened. With the .us registration we do not expect this to happen again.

Please follow the @vb_ly twitter account for further developments! You can also track this story on Hacker News.

We’re happy to announce that 64-bit Ubuntu 10.10 (Maverick Meerkat) images are now available for Cloud Servers users.

In order to take advantage of these images, you can add a new Cloud Server to your account or rebuild an existing one.  Keep in mind that the rebuild process will erase any existing data on the cloud server.  You can also upgrade your server from the command line using Ubuntu’s standard tools.

For additional information, check out these links:
Release notes (including Overview, Installation, Upgrading, and Known Issues):  https://wiki.ubuntu.com/MaverickMeerkat/ReleaseNotes
Download Ubuntu Server Edition here: http://www.ubuntu.com/server/get-ubuntu/download
Upgrading to Ubuntu 10.10:  https://help.ubuntu.com/community/MaverickUpgrades

If you have any questions or run into any issues, please catch us in live chat or give us a call at (877) 934-0407.

As has been discussed recently in the press and various channels online, the BBC has taken the decision to close BBC Backstage in December 2010. Given the report recently in the Guardian Tech blog this no doubt comes as little surprise to most. However, I thought I’d take the opportunity to explain why this decision was made and what it means for the BBC as an open innovator in the future.

BBC Backstage has been a great success. I am very proud to have worked with the team on numerous projects. It was the forerunner to many other emerging, successful initiatives and has made a valuable contribution in driving the BBC towards genuine open innovation. In many ways it has been very much of its time.

Following the official launch of Backstage in 2005 the project has consistently delivered groundbreaking prototypes using BBC data. Personally, the stories of Hackday 2007 at Ally Pally where lightning did indeed strike twice remains one of the numerous high points.  On that remarkable day the creative energies of the developer community prevailed, sheltering under umbrellas at times, with many people working through the night to build outstanding prototypes over the course of the weekend. Lord Reith would have approved of Backstage and its community on that day and many others.

Looking forward, BBC Future Media will shortly launch a more comprehensive repository of data feeds, development tools and APIs which will supersede the current Backstage collection. The BBC is in early discussions with a number of the other UK developer communities about aspirations to create a larger network of independent developers that will absorb and support the current BBC Backstage community.

As many of you know Ian Forrester, the producer of Backstage, fell seriously ill in May of this year. The plan for last ten months was always to implement a systematic winding down of Backstage in early summer but Ian’s illness precluded this. Now that Ian is once again able to take up his role within BBC R&D it feels like the right time to retire Backstage and allow Ian to concentrate on new projects. Innovation and Innovators don’t stand still and with the Backstage model now firmly established elsewhere it’s time to move on.

 Over the remaining months, we will be using the Backstage blog to celebrate the achievements of the project. We’ll be sharing some articles written by Suw Charman-Anderson, commissioned earlier this year to produce a retrospective piece on the initiative. We will discuss in more detail the aspirations around the growth of a richer developer community in UK and Europe, explore the legacy of the project and its place in UK developer history and also share some details on some of the newer projects that the team have been working on. 

Huge credit and recognition needs to go out to so many people involved since the germ of the idea that became BBC Backstage was conceived in 2004. In particular I’d like to give thanks to the following people for their efforts: James Boardwell, Ben Metcalfe, Matt Locke, Tom Loosemore, Jem Stone, Ian Forrester, Matthew Cashmore, Rain Ashford and Ant Miller.

Finally, and most importantly, the BBC recognises that the success of Backstage is down to the whole community of developers, designers, contributors (and even critics) built around the project - a community who pushed it far beyond its original concept.

The BBC wants to support the community that is BBC Backstage and we’d really welcome ideas on how to extract as much learning from the project as possible as well as your thoughts on what its spiritual successor should tackle next. I want to say personally that BBC R&D is committed to retaining the team who have worked tirelessly on Backstage and I am looking forward to working with them on applying the Backstage philosophy to the new challenges faced by BBC. 

Cheers and hope to see you in the next incarnation.

This post is the fifth of a series that started here.

From the previous posts of this series, we now have an instance restart script that can restart the database node in case of failure and automatically reconfigure Pacemaker and the other servers that needs to access the MySQL server. What we will cover in this post is the monitoring script that run on the MySQL node.

At its smallest expression, the instance monitoring script is a simple empty loop that will run forever like:

Although fully functional, this monitoring script is rather weak could be not working and the cluster would unaware of it. Very complex monitoring scripts can be written but let's provide a basic functional one that monitor MySQL with the mysqladmin ping command.

For the ones that knows Pacemaker a bit, they could wonder why I didn't use crm_resource to migrate the MySQL resource to the other node. The problem with this approach is that the migration with crm_resource is achieved by setting the host affinity for the resource to INFINITY for the Monitor node. When the resource is moved, the MySQL node is restarted and its local copy of the cluster configuration is lost. After restart, the copy of the cluster configuration on the Monitor node will be pulled back by the MySQL node and since this copy has INFINITY as affinity of the resource to stay on the monitor host, the resource will stay there. If the resource is not move away quickly (within 5 min) from the Monitor node after the restart of the MySQL node, the instance restart script on the Monitor node will loop and restart again the MySQL node. Stopping the heartbeat service achieve the desired result without polluting the cluster configuration.

The following post in this series will add some details as how the IP of the MySQL server is broadcast to the client nodes.

---

Entry posted by Yves Trudeau | No comment

Add to: | | | |

This is the second in a series on what's seriously limiting MySQL in certain circumstances (links: part 1). In the first part, I wrote about single-threaded replication. Upstream from the replicas is the primary, which enables replication by writing a so-called "binary log" of events that modify data in the server. The binary log is a real limitation in MySQL.

The binary log is necessary not only for replication, but for point-in-time recovery, too. Given a backup and the corresponding binary log position, you can replay the binary log and roll forward the state of your server to a desired point in time.

But enabling the binary log reduces MySQL's performance dramatically. It is not the logging itself that's the problem -- writing the log is usually not much additional work. It's ensuring consistency and durability that is expensive. Flushing it to disk adds an fsync call for every transaction. And the server performs an XA transaction between InnoDB and the binary log. This adds more fsync calls, and causes mutex contention, and prevents group commit, and probably other things that aren't coming to mind now.

The performance reduction can be an order of magnitude or more.

What's the solution? I'm not sure I can summarize it concisely. There is a lot of complexity, and honestly I don't understand some of the server internals fully enough to have a 50-thousand-foot view of it all. The binary logging and replication code, and its interaction with InnoDB, is difficult to understand. Kristian Nielsen has an extensive series of posts on group commit alone.

I think that a full fix might require significant architectural changes to MySQL. This will be hard. Maybe Drizzle is going in a good direction -- time will tell. All of the solutions that I can think of are too simplistic. For example, doing replication through the InnoDB transaction log would work fine if a) all the data were in InnoDB, and b) InnoDB's data didn't have to be synchronized with the .frm files (and Drizzle has gotten rid of the .frm files, hooray), and c) privileges and other changes to the non-InnoDB data in MySQL were handled manually.

It could work if you just made sure that you didn't change privileges or schema, but that's a description of a pretty limited, clunky replication system from the user's point of view. Still, I have considered it. There would need to be a mechanism of transporting the log files, and InnoDB would have to be put into a state of constant "recovery," and it would have to be modified to be available read-only in this state so that it could be used for read queries. This can be done, of course. It's just a matter of how hard it is.

It's worth noting that PBXT does replication through its transaction logs, so there's even precedent for this among MySQL storage engines. And there is Galera's multi-master synchronization technology to look at, too.

---

Entry posted by Baron Schwartz | 11 comments

Add to: | | | |

The parameter sort_buffer_size is one the MySQL parameters that is far from obvious to adjust. It is a per session buffer that is allocated every time it is needed. The problem with the sort buffer comes from the way Linux allocates memory. Monty Taylor (here) have described the underlying issue in detail, but basically above 256kB the behavior changes and becomes slower. After reading a post from Ronald Bradford (here), I decide to verify and benchmark performance while varying the size of the sort_buffer. It is my understanding that the sort_buffer is used when no index are available to help the sorting so I created a MyISAM table with one char column without an index:

and I inserted 100k rows with this simple script:

I know, I could have used the uuid() function of MySQL. For the benchmark, I used an old PII 350 MHz computer, I think for such CPU bound benchmarks, an old computer is better, if there are small differences, they'll be easier to observe. I varied the sort_buffer_size by steps of 32 KB and I recorded the time required to perform 12 queries like 'select * from sorttest order by data limit 78000,1' with, of course, the query cache disabled. I also verified that during the whole process, the computer never swapped and I pre-warmed the file cache before the benchmark by doing "alter table sorttest engine=myisam;". The script used for the benchmark is the following:

which in addition to output the total time, output the number of Sort_merge_passes, which will be useful to interpret the results. The figure below shows a graphical representation of the results.

The first we can notice by looking at the graph is that the expected correspondence between the time for the queries and the number of sort merge passes. For the small values of the sort buffer size, below 440KB, there are many sort merge passes and the time for the queries hover around 18s. Above 440K, as the sort merge passes drops to 1, there is a large drop of the time for the queries below 14s. Then, as the sort buffer size is further risen, the performance gain is negative up to the point, around 6.4MB where no more sort merge pass are required and then, the time for the queries loses all dependency over the sort buffer size. I am still trying to figure out why the number of sort merge passes felt to zero at 6.4MB since the total size of the table is less than 3MB. That seems to be a pretty high overhead per row (~37 bytes) but if the structure has a few pointers, that can add up to such amount of bytes pretty quickly.

The important point here, at least for the Linux, glibc and MySQL versions I used and for the test I did, there doesn't seem to be an observable negative impact of the glibc memory allocation threshold at 256KB. I'll try to find ways to repeat this little experiment with the other per session buffers just to confirm the findings.

OS: Ubuntu 10.04 LTS, Linux test1 2.6.32-21-generic-pae #32-Ubuntu SMP Fri Apr 16 09:39:35 UTC 2010 i686 GNU/Linux
MySQL: 5.1.41-3ubuntu12.6-log

P.S.: Gnumerics for graphs is so much better than OpenOffice Calc.

---

Entry posted by Yves Trudeau | 7 comments

Add to: | | | |

This is the third in a series on what’s seriously limiting MySQL in certain circumstances (links: part 1, 2). This post is about subqueries, which in some cases execute outside-in instead of inside-out as users expect.

It’s easy to pick on subqueries in MySQL, so I’ll try to be gentle. The following query will surprise users unpleasantly:

<!--eb87a9188ded6fc3cac213da26c8cbd9-->

Users expect the inner query to execute first, then the results to be substituted into the IN() list. But what happens instead is usually a full scan or index scan of table a, followed by N queries to table b. This is because MySQL rewrites the query to make the inner query dependent on the outer query, which could be an optimization in some cases, but de-optimizes the query in many other cases. NOT IN(SELECT …) queries execute badly, too. (Note: putting a literal list of items in the IN() clause performs fine. It’s only when there is a SELECT inside it that it works poorly.)

The fix for this has been in progress for a few years, and Sergey Petrunia committed working code to the stalled 6.0 release. But it’s not quite clear whether that code was a complete solution. It has not been in any GA or RC release, so it hasn’t been used widely.

To be fair, many other database servers also have poor subquery performance, or have had it in the past and have fixed it. And many MySQL users have learned to simply write JOINs instead, so it isn’t that much of a limitation. But it would be a big improvement if it were fixed.

See if you can guess what limitation number 4 will be!

---

Entry posted by Baron Schwartz | 17 comments

Add to: | | | |

This post was written and contributed by Dan Di Spaltro, Founder of Cloudkick, a Rackspace Cloud Tools Partner.

We were very happy to participate in a webinar with the Rackspace Cloud talking about managing Windows Servers in the Cloud.

Managing infrastructure on the Cloud, or anywhere for that matter, is a tough proposition.  Windows infrastructure is no exception.  Rackspace has focused on making the provisioning process straightforward so it was a natural step to help the management and monitoring.  Getting alerts upon failures, spotting hot-spots with your data, graphing any metric you want and supporting Windows/Linux hardware side by side with cloud, dedicated and fully managed.

Key points covered:

• Cloudkick tagging is very important in the UI, it semantically defines permissions and monitors.
• Adding credentials to Cloudkick only requires looking up the API keys and then entering them into Cloudkick.
• Getting the agent up and running is simple and straightforward.
  • Getting the OAuth credentials from Cloudkick requires logging in and copying the key and secret and distributing it to every machine you want monitored
• Defining custom monitors allows any technical, application, or business metric graphed/trended in the system.
  • Setting custom alerts allows for the ultimate flexibility in alert infrastructure
• Fault detection with Cloudkick is unified between Windows and Linux operating systems.  You are able to see metrics from either systems through a single pane of glass.
• The full permission system

The Cloudkick delivery model, briefly in the webinar, eliminates the need to update the agent software.   We focused on creating a very flexible model for adding new checks in the Cloudkick  system.  When Cloudkick adds a new check, the code is signed, encrypted and sent to a 3rd party CDN for delivery to each agent.

Again, a special thanks to the Rackspace Cloud team for putting together such a successful event! Click below to watch the full webinar.

A couple of weeks ago I blogged about Sharing an auto_increment value across multiple MySQL tables. In the comments, a few people wrote in to suggest alternative ways of implementing this.  I just got around to benchmarking those alternatives today across two large EC2 machines:

(Measured in transactions/second – higher is better)

What is the conclusion?  With the exception of my original option2, they actually all perform fairly similar.  The Flickr and Option1 tests perform marginally better.  Test “arjen2″ is option2, but with a MyISAM table — it suffers a little because EC2 can be a little high for latency, and there’s one additional round trip.  Test arjen2005 is not too dissimilar from the Flickr solution, but uses a MySQL stored function.

Full Disclosure.

---

Entry posted by Morgan Tocker | 7 comments

Add to: | | | |

Rackspace continues to be dedicated to open source software, including contributing resources to the Apache Cassandra project and founding OpenStack, the open-source cloud platform.

The Apache Cassandra team is in the process of rolling up a release candidate for Cassandra 0.7.  That makes this a good time to highlight some of the new features and changes that will be in this version.

Big Changes

Memory Efficient Compactions (CASSANDRA-16)

Compaction is the process by which Cassandra iterates over its data, removing deleted data forever and combining multiple data files into a single data file.  This results in faster read operations.  One limitation of the implementation was that if a row of data couldn’t fit in memory, then the data file it belonged to could never be compacted.

Since one of the main features of Cassandra is its use of sparse columns, users expect to be able to add columns without penalty until disk runs out.  Cassandra 0.7 solves this by making compactions happen incrementally.

Online Schema Changes (CASSANDRA-44)

Prior to Cassandra 0.7 adding and removing column families and keyspaces required you to first distribute an updated configuration file to each of your nodes and then execute a rolling restart of your cluster.  That was not too bad, but it was manual and required human intervention, so it was possible to make mistakes.

Cassandra 0.7 solves this problem by exposing the ability to create and drop column families and keyspaces from its client API.  Using the same methods there is limited support for updating existing column families and keyspaces (e.g., increasing the replication factor for a particular keyspace).

Additionally, we’ve made it easy to migrate your keyspace definitions from your 0.6 storage-conf.xml configuration file using bin/schematool.  In the future we plan to add the ability to rename column families and keyspaces.  A wiki article contains more detailed information.

Secondary Indexes (CASSANDRA-749)

Cassandra has always been an excellent platform for storing data.  However, it has lacked an expressive, efficient way to query data by any property other than the primary key (you need the answer to “what keys match these particular columns and values?”).  To cope with this, many application developers ended up creating their own secondary indexes that were basically inverted indexesof already existing column families.  This solution worked well, but was tedious to maintain and involved a lot of duplicated effort.

The need for this feature has weighed heavily on our minds for a while.  Work even began on it prior to the release of Cassandra 0.6.  One design decision that required some wrangling over was whether secondary indexes should use node-local or distributed storage.  Each approach had distinct advantages and disadvantages and served different use-cases.  We ended up using node-local storage (the secondary indexes are wholly contained on the same node whose key they index). Distributed secondary indexes may eventually make their way into Cassandra, but not in 0.7.

Using the API for online schema changes, secondary indexes can be added to existing or brand new column families.  An index can be queried on the client side using the get_indexed_slices() API method.

Time to Upgrade?

For these reasons, Cassandra 0.7 is a compelling upgrade from earlier versions.  But wait!  The Cassandra team has been busy fixing bugs and adding more features.  You can get the full details from the changelog, but I’ll highlight a few of them here.

Streaming fixes

The streaming code has been simplified to offer better tracking when files get streamed.  Additionally, the requirement for anti-compaction, the time-consuming process of splitting a data file into multiple parts, has been removed.   Streaming is faster and more predictable now.

Read Performance

Several things have happened to make reads faster overall.  First, the read path has been optimized to take further advantage of the row cache introduced in 0.6 (CASSANDRA-1267 and 1302).  Second, row keys have been switched from strings to byte arrays, picking up some speed by eliminating the overhead required for java string construction.   Third, usage of byte[] has been replaced with java.nio.ByteBuffer. This allowed us to reduce the amount of byte array copying on the read path and eliminated a significant amount of garbage generation (bonus!), easing the pain still felt by stop-the-world garbage collection.

Thrift

We’ve upgraded our Thrift version from 0.2 to 0.5.  This comes with significant performance improvements.  Thrift framed transport is now the default, in contrast to non-framed which was default in 0.6 and earlier.

API

As with every release of Cassandra so far, our client API has fluctuated quite a bit—something we are not necessarily proud of.  A mind-shift is occurring though.  Most Cassandra experts, including myself, have begun recommending that users access Cassandra through one of the available third party libraries (e.g.: Hector, Pelops, Pycassa) instead of directly via Thrift.  The main benefit of this approach is that users will be immune from the sort of API changes that happen from version to version so long as client developers keep up.

There are several things worth noting about the updated API.

1. The keyspace argument has been dropped from nearly every method and has been replaced by a set_keyspace() method that sets the keyspace for the current connection session.
2. Expiring (TTL) columns have been introduced.  This gives you the ability to set the time-to-live for columns so that they will automatically be removed by compaction without any intervention on your behalf.
3. ConsistencyLevel.ZERO is removed.

Configuration
storage-conf.xmlhas been replaced with cassandra.yaml. The intent is to provide a format that is more human-readable.  We have provided a converter that will allow you to convert your XML configuration into YAML prior to upgrading.  Also, more settings are configurable than ever before.

Other Changes

1. We have continued the trend of exposing more operations and metrics via JMX, including compactions, sstable accesses, endpoint states (gossiper) and TCP messages.
2. The authentication framework and built-in Hadoop support have both received a lot of attention.
3. Introduction of native code (JNA) for snapshots and help with memory swapping.
4. Enhanced CLI experience

Conclusion

Cassandra 0.7 offers significant advantages in terms of features and speed than in earlier releases.  We’ve done our best to make the upgrade process simple: old data files are still compatible, but your cluster must be brought down entirely.

As always, if you would like to learn more about Cassandra, good places to start are the website, wiki and #cassandra on IRC (freenode).  Apache also hosts several mailing lists that are excellent resources.  Additionally, NEWS.txt is a good place to go to find out more about some of the high-level changes in Cassandra.

This is the third in a series on what’s seriously limiting MySQL in core use cases (links: part 1, 2, 3). This post is about the way MySQL handles connections, allocating one thread per connection to the server.

MySQL is a single process with multiple threads. Not all databases are architected this way; some have multiple processes that communicate through shared memory or other means. It’s cheap to create a connection to MySQL, because it just requires creating a thread (or taking one from a cache). This is generally so fast that there isn’t really the need for connection pools as there is with other databases, at least not in the same way. Windows in particular has had excellent threading support practically forever; Linux has very good threading now, but that wasn’t always the case.

However, many development environments and programming languages really want a connection pool. They’re just built that way (I’m looking at you, Java). And many others use persistent connections by default, so that a connection isn’t really closed when it’s closed; it’s kind of like a connection pool, except that the connection is persisted from request to request within the same process, rather than being shared with whichever request needs a connection.

Connection pools and persistent connections combined with a large number of application servers can lead to a situation where the database server has a very large number of connections open to it, most of which are doing nothing. It’s not uncommon for me to see a server with 1000 to 5000 connections open, and maybe one to three are actually running queries on average. These connections originate from dozens to hundreds of application server instances. When you have a heavily sharded or otherwise horizontally scaled application, it’s not only easy to get into this pickle, it’s really hard or impossible to avoid it.

And with 5000 connections open, you get 5000 threads in the server. That increases the overhead from thread scheduling, and potentially memory usage as well. I feel like I’m forgetting some reasons that this matters — please fill in whatever’s missing in the comments.

There can be more than one solution to this problem, but the one that’s actually partially implemented is a pool of threads, which was originally coded for MySQL 6.0, but is available now in MariaDB.

Unfortunately it isn’t a full solution, because it can cause undesirable lock-out or waiting, and the specific implementation has a scalability bottleneck on multicore servers. Mark Callaghan has done much more investigation of the pool of threads than I have. There are more details in this blog post by Mark, and two followup blog posts from Tim Cook (1, 2).

Thanks for the great comments on the last post. Some of them were good guesses. Remember that the context for this series isn’t micro-limitations or edge-case badness (even if they are serious in some cases), but rather a focus on shortcomings in the main use cases for the server. There are a lot of things MySQL doesn’t do well, but it doesn’t matter that much, because that’s not what it’s designed for. Wrong tool, wrong use, NotABug. I’m thinking of the lack of sort-merge joins or intra-query parallelism, for example. It would be lovely to have those things, if you’re running a data warehouse on MySQL, and in some cases for other uses too (note that most databases that do have these query plans usually try to use nested-loop joins whenever possible, because of things like the lower startup cost for the query). But MySQL isn’t a data warehouse DBMS first and foremost. It’s a general-purpose OLTP database server that runs well on affordable hardware and is great for Web usage. It’s so good, in fact, that it can be used for tons of other things such as… data warehousing. But it isn’t a Netezza or Paraccel, and if it were, it wouldn’t be a great OLTP web database too.

MySQL replication is one of the core, fundamental features — and it’s single-threaded and relies on the binary log, which are two major limitations. And it has subqueries, which are a core, fundamental part of SQL — but it’s bad at certain kinds of them. That’s why I listed those as major limitations. And because MySQL is a multi-threaded database for Web usage that tends to be used in sharded environments with tons of application servers, which creates a situation with many thousands of connections to the database, and because it doesn’t handle that very well, I list its one-thread-per-connection design as a serious limitation.

---

Entry posted by Baron Schwartz | 13 comments

Add to: | | | |

As AWS technology evangelists, we often meet startups working on cool stuff.  Every so often we discover startups that have done incredible things on AWS.  Recently, I came across Navitas, a Berkeley-based company with development teams in Silicon Valley, Ecuador, and Thailand. Since I am deeply interested in location-based services and geo apps in the AWS, I dived a little deeper to learn more about the company and its architecture.

Navitas is the creator of TAPTIN, a location-based service similar to Foursquare and Gowalla. However, TAPTIN goes beyond mere check-ins.  The TAPTIN platform enables the creation of locally branded apps, such as Berkeley Local, which has events and recommendations for UC Berkeley and the city of Berkeley.  TAPTIN is thus a new form of local media, with built-in Foursquare-style check-in features as well as services for merchants to engage with their customers, such as through coupons, loyalty campaigns, etc. so you can build locally branded apps for every city around the world.  Another example of an app built on the same platform is “We Love Beer” app. This beer app has a beer catalog, and pubs can link to the catalog categories to create their own beer menus.  This app enables you to find what beers are available nearby, to locate a particular kind of beer, and to find your friends at local pubs.

Recently, Navitas abandoned their server farm and moved their entire development and production environments to AWS. It runs 100% in AWS cloud. TAPTIN is scaling on AWS across multiple tiers of servers. The founder of the company, Kevin Leong, was helpful in explaining their architecture in detail below.

What Kevin and his team have done is commendable, especially given that they did it by bootstrapping, which Kevin says would not have been possible without AWS.

Production Environment
The figure below depicts the Navitas production environment, which consists of seven scalable layers, all using open source enterprise technologies. Load balancers are employed in multiple tiers.  Search is based on Solr, a popular open source search platform from the Apache Lucene project. Solr is also used for geospatial search. The search tier uses HAProxy on an Amazon EC2 instance to apply write updates to a Solr master server, and these updates are then distributed to the two Solr read-only slaves.

 
 
The application tiers consist of three layers. Web pages are implemented in PHP and consume REST APIs running on Jetty servers.  Some PHP pages are also calling Solr directly.  The company originally started with Enterprise Java Beans (EJB) running JBoss servers but then decided to use lightweight Java Persistence API (JPA) with Hibernate and Spring Framework HTTP Remoting. The caching layer runs memcached which provides dynamic load balancing cache services. They employ two layers of cache.  First, they employ memcached that is deployed in the web tier.  If an object is not found in memcached, it will be requested from the persistent tier, which for most recently used objects are probably in cache.  This technique gives a higher performance.  Memcached is configured to scale automatically with new servers.

While load balancing and automatic instance deployment ensures high availability as TAPTIN and Grand Poker apps scale, Kevin’s team also implemented a failover strategy, automatic data backup and implemented data recovery steps, as well as recovery of Solr search indexes. Because everything is done behind AWS, there is no bandwidth usage.

Navitas uses PostgreSQLon Amazon EC2 to store structured data with pgpool for load balancing, failover and asynchronous replication.  It's very easy for them to add another instance to Pgpool to replicate to support load balancing and parallel queries. 

Media, such as photos, are transcoded and stored in Amazon S3.

Sandbox and Development Environment
Having sandbox and source code repository (SVN) on AWS was not only cost-effective but also a huge productivity gain for the team as it was easy to launch another instance. With Amazon Machine Images (AMIs), developers create and launch a consistent environment for development, test and production environments. Kevin said that his developer team which is spread out around the world (in California, Latin America and Asia Pacific) can launch the same pre-configured sandboxes in that AWS region within minutes. This saved a lot of time and increased the developer productivity. The company uses spot instances for all development work, whenever available, which is cheaper.

They also create a new sandbox environment on AWS for testing.  With SVN on Amazon EC2, Navitas does their nightly build in the cloud. Source code is checked out to a build directory where it’s compiled, built and deployed. Unit test hornets are also run to ensure no code breakage and to ensure performance of function is maintained. Kevin talked about automated performance testing will be coming later when the company has more resources. 

With having an automated build on AWS, basically they were able to migrate their extreme programming development methodology to AWS by having developers commit their code daily for nightly build.  They commit all development code to SVN trunk for all projects, and can build as required for testing in their sandbox environment. They create SVN branches for all production releases, allowing them to bugfix quickly and efficiently, and immediately release new application binaries, or plan and stage back-end upgrades.

The company maintains a series of build-scripts and uses Maven to manage the build dependencies.  The server configuration is externalized so that the build scripts can pick up the appropriate configuration for sandbox and production.  They create their sandbox which is a mirror of production and the bootstrapped AMI does all the magic. Cloud-powered SDLC clearly has lot of advantages.

Final Thought
What I really liked about Kevin's strategy was the "Think Scale" approach.  Not many startups invest in designing a scalable architecture early on especially because its time-consuming and distracting. Some think it's too expensive. His message to Start-ups was "think scale" from the beginning as it is really not too expensive to do in the cloud. To quote him, "I did it by bootstrapping.  They can also.  Amazon AWS is the way to go."

- Jinesh Varia

Percona Community,

Percona Server version 5.1.51-rel11.5 is now available for download.

The main purpose of this release is to update the current Percona stable release to the latest version of MySQL 5.1.

Functionality Added or Changed

•  Percona Server 5.1.51-rel11.5 is now based on MySQL 5.1.51.
•  New Features Added: None
•  Other Changes: None

Bugs Fixed

•  Bug #661354 – Fixed a problem compiling query_cache_with comments for 5.1.51-rel11.5. (Oleg Tsarev)
•  Bug #661844 – Fixed a problem with server variables failing test for 5.1.51-rel11.5. (Oleg Tsarev)

The Release Notes for this and previous releases can be found in our Wiki. 

The binary packages are available on our website.
The latest source code for Percona Server, including the development branch, can be found on LaunchPAD.

Please report any bugs found at Bugs in Percona Server.
For general questions, use our Pecona-discussions group, and for development questions our Percona-dev group.
For support, commercial, and sponsorship inquiries, contact Percona.

---

Entry posted by Fred Linhoss | No comment

Add to: | | | |

We added SSL support to the Elastic Load Balancer a couple of weeks ago. This met the needs of many of our users, but not all of them. Some of our users wanted to be able to tell if the HTTP requests arriving at their EC2 servers had been transmitted across the Internet using HTTPS. For example:

One question. How will the webservers know the original request came from HTTPS?
We currently add the "X-Forwarded-Proto: https" header when that happens.

As I always do, I passed this feedback along to the proper team for consideration. The "voice of the customer" is highly valued at Amazon, and feedback like this helps us to assign the proper priority to each feature.

I am happy to be able to report that we were able to turn this request around very quickly and I am happy to say that requests processed by newly created Elastic Load Balancers now include two additional headers:

• X-Forwarded-Proto specifies the protocol ("http" or "https") of the original request made to the Elastic Load Balancer.
• X-Forwarded-Port specifies the port of the original request.

These headers are injected when the request arrives at the Elastic Load Balancer. If the headers are present on incoming requests they will be overwritten.

If you would like to be part of a team that can turn on a dime, check out the following jobs:

• Software Development Engineer - Elastic Load Balancing.
• Senior Software Development Engineer - Elastic Load Balancing.

-- Jeff;