Adam Leach

Write-read sequences can be tricky for users of MySQL replication, if what has been written needs to be available for reading immediately. Slaves could be behind. You have to run the sequence on the master. If MySQL semisynchronous replication would tell the client which slave is in sync, a client library could select that slave for load balancing.

The more we look into load balancing for PECL/mysqlnd_ms, the PHP mysqlnd master-slave replication load balancing proof-of-concept plugin, the more restrictions and issues come up. An obvious restriction is that connections have a state.

No solution: binlog monitoring

But not only state is a "trouble maker". Critical write-read sequences can be another one. If the client must be able to "see" the result of a write send to the master immediately, the safest choice is to run the next read on the master. The downside is what the master gets some read load. Alternatively, the client could try to monitor replication binlog or replication lag to choose a slave who has already replicated the write. If it hasn’t been replicated yet, the load balancer could block the client for a wait period until a slave catches up before the load balancer sends the read to the master. Uuups, conceptually that is what semisynchronous replication does. Of course, the server does it in a much more elegant way. And, if the server told PECL/mysqlnd_ms which slave has catched up…

No solution: global transaction ID

Another solution to the critical write-read sequence is using a global transaction ID as proposed by Jan in "MySQL Proxy and a Global Transaction ID". The idea is to maintain some sort of transaction ID or record version which is replicated by the server. The load balancer will pick a slave that, for example, has already replicated the critical write by checking the global transaction ID.

Shall PECL/mysqlnd_ms use a global transaction ID?

PECL/mysqlnd_ms could do the magic of maintaining a global transaction ID in C. Let us know if you want this. To me it does not really sound as it is a bit of a hack, but who cares about hacks, if it solves problems…

This post is from OUR perspective. The next one I'm going to do is from our attendee feedback! 

Last week was our first week delivering a new style of Immersion Event. And, now that the class has finished, we're really happy with our decisions on content and naming. The name was new because we had originally called these Master Immersion Events and the class was somewhat new because we replaced our ONE 5-day SQL Immersion Event (Internals/Design/Performance/Maintenance) with FOUR new 4-5 day classes (19 days total). 

Why were they "master" events and why did we remove that part of the title? First, let me begin with with we did add it! We added "master" because these new courses cover the wide range of topics as required by the exam and practical for those working toward their Microsoft Certified Masters - SQL Servercertification. However, soon after, we ended up dropping the "master" part because it lead to confusion (some folks thought that the class was meant ONLY for those preparing for the exams). And, we changed them from SQL Immersion Events because we significantly extended the content to cover a wider spectrum of topics but also rearranged them to create concise, individual weeks of training. So, while they're still our fun, deep, focused Immersion Events, we have a lot more depth and content (19 days just in this area) covered for those who really want to design, tune and manage SQL Server systems effectively. Paul wrote an excellent post about it here: Leaving the Land of Confusion...

For those who attended, the feedback was fantastic and I have a post coming about that. But for this post, I just wanted to give you some insight into how the class was run and what it's like to attend an "Immersion Event" with us! 

Why do we deliver these at hotels?

We set up to deliver our event at the Marriott Suites Market Center in Dallas, TX and everyone that was not local stayed at the hotel. While being local is definitely nice for folks that don't have to travel, staying at the hotel with attendees is really the icing on the cake for these events. We've chosen Hiltons/Marriotts (in general) because they're comfortable and usually have good food and a nice lounge. And, don't think that I'm talking about drinking heavily every night. No, that's not really conducive to absorbing all of the content that we're going through. In fact, I didn't even have my first drink until Friday (but, I'm also a big wimp and I can't be impaired in any way if I'm going to lecture for 10 hours!!).

What the hotel/lounge/atmosphere allowed us to do is really focus on learning. We didn't have leave to forage for food. We didn't have to think about parking or traffic or having to arrange with a facility that we wanted to run over into the evenings. We were able to retreat to a warm and cozy lounge with a fireplace and relax together, have a drink and just kickback before heading off to bed at a reasonable hour. The days were intense and these events really take a lot out of you... Many commented on how relaxed they were and how comfortable the hotel was for them. Specifically, let me give you a picture of a typical day.

A Day in the Life of an Immersion Event Attendee

• 7:45 am - Breakfast(thanks to SQLSentry's sponsorship we added a Full American breakfast - yum!)
  • In Dallas we had a Full American breakfast with plenty of bacon. And, eggs, pastries, fruits, cereals... high carb or low carb, vegetarian or not - you were covered.
• 8:30 am - Dive right in
  • Content: We don't dance around, we don't waste time. We were ALL about the content and as much of it as we could fit in 5 days...
  • Breaks:We'd take 1-2 breaks in the morning for a total of about 20-30 mins of break time. But, even these we often filled with chats around the coffee/tea or chats with one of us around the whiteboard.
  • Whiteboard:Speaking of the whiteboard... we do A LOT of whiteboarding! So much so that we have our OWN whiteboard shipped to the event (the flash created a bit of glare in the pic but we fixed that with lighting). And, for this one we had two (we had a second smaller whiteboard in the back where we could write notes to the attendees however, this one turned out to have the MANY twitter aliases of attendees).
• 12:30 pm - Lunch(again, sponsored by SQLSentry which was fantastic, we had plenty of food and plenty of choices)
  • Lunch was served at 12:30 pm and what was great here is that no one had to leave the hotel to go forage for food somewhere. We could really take a break and relax. Again, people could network and chat over lunch as well. This really adds to the Immersion experience.
• 1:30 pm - Dive back in
  • The afternoon was much like the morning but we know it's always harder to focus while digesting... We took more breaks to get people moving/stretching but often shorter breaks just to break up the time sitting. We also had dessert served as part of an afternoon break along with a coffee refresh and a host of low-carb snacks as well.
• ~5:30 pm - Wrap-up
  • Officially class would end around 5:30 but most days we'd go a bit longer with questions... 
• 6:30 pm - There was always something going on in the evenings... 
  • Monday night: We all gathered in the bar and did our "meet'n'greet." We bought the first round, chatted with everyone and ended up staying until around 8:30 or so. It was a really nice way to get the week rolling and make sure that no one was nervous to ask questions. However, I don't think anyone was shy even by Monday lunch!
  • Tuesday night: SQL Sentry delivered a session on how to use their FREE Plan Explorer tool. Greg Gonzalez (t|b), President/CEO of SQLSentry flew in for the evening presentation. After the presentation, we all retired to the lounge and spent a bit more time talking about the tools, SQL Server and how fit Greg is! (CrossFit is definitely a powerful workout!) 
  • Wednesday night: Immersion Event attendee and Director of Engineering Services for SQL Sentry, Steven Wright (t)delivered a session on how to use their Performance Advisor tooland the new features of v6. 
  • Thursday night: some attendees offered to give micro sessions on some of their favorite topics and we thought that was a great idea! We ordered a bunch of appetizers, opened a tab at the bar, and away we went! For those that asked, we also provided speaker feedback to help folks improve their speaking skills. Trevor Barkhouse (t|b), Martin Catherall (t|b), Eric Humphrey (t|b), Brian D. Knight (t) and Kendra Little (t|b) all presented 20 min sessions and informed/entertained us all! It was the PERFECT way to host our last evening of the class... And, we think this might become a staple at future Immersion Events!
  • Friday night:Even on Friday... we had to pack a bunch of stuff but many stayed and chatted and then we all headed over to the lounge for dinner and drinks. I finally relaxed and had a nice big glass of wine! What a great way to end the week!!

The Attendees

First off, we had a fantastic group with 29 attendees from as far away as New Zealand. Four attendees were winners from our contest (Martin Catherall (t|b), Eric Humphrey (t|b), Kendra Little (t|b) and Sankar Reddy (t|b)) and most were not local. In fact, one attendee used to work with Kevin Kline (t|b) in Hermitage, TN when I first met them (and a few others) back in the mid-90s. It was a bit of a blast from the past!!

And, many of the attendees were on twitter and have blogs too. All of these things continued to add to the content learned during the week and the connections made. Most folks are now following each other and keeping up with what they're learning and uncovering with SQL Server. And, we're going to see a few of them in Dallas in March for our second Immersion Event on Performance Tunng. In fact, 8 attendees for the class coming up in March are folks that have attended the week 1 (Internals and Performance) Immersion Event either this year or last and a couple are coworkers of those who have attended. So, it will be great to meet up with them again and see what they're up to!

The End Result

We love the new format and we love the feedback we've received so far. We had a great time sharing our experiences and delivering our new style Immersion Event and now we're going to put a few more on the schedule. If you're interested in immersing yourself in SQL Server - check out our Immersion Events.

And, consider becoming a SQLskills Insider as well. For our London class in June, we've already offered a discounted rate and an earlyearly-bird registration site. And, the hotel has a wonderful pub, internet access, and a full English breakfast (Paul's *very* happy about that). We're really looking forward to heading over! And, we're planning to add another US event (Immersion Event on Internals and Performance) to the schedule for early to mid-May.

If you want to get the scoop early - become a SQLskills Insider now!

Thanks for reading!
kt

After installing a clean Debian Lenny Xen system using xen-tools, I received this strange error when trying to connect using ssh to the machine:
PTY allocation request failed on channel 0
stdin: is not a tty

It looks like for some reason, xen-tools didn’t install the udev package. So in order to fix this issue, I had to connect (using the virtual console) to the xen machine:
xen console 1
and install udev:
apt-get install udev
strange enough the /dev/pts mount entry was present in /etc/fstab so all I had to do was to remount it with:
mount -a
(if you don’t have this entry make sure to add it in /etc/fstab:
devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
and if the folder /dev/pts doesn’t exist create it first and after that mount -a).

This should fix the ssh problem and you should now be able to ssh into the xen machine. Next to see if this is fixed in xen-tools in Squeeze, and if not to file a bug for it.

Introducing the best load balancing policy for PECL/mysqlnd_ms: "random_once". A (PHP) web request is short-lived. A web server can handle multiple requests per second. There are many web servers running an application. For each web request choose a random slave to use for the rest of the request. Don’t do load balancing, don’t switch connections during a web request . Result: even load distribution, less connection overhead and less connections overall, cache loves you, less hassle switching stateful slave connections…

Load balancing policies featured by PECL/mysqlnd_ms

As of today, PECL/mysqlnd_ms, our PHP mysqlnd toy plugin for replication load balancing, features four built-in load balancing policies:

1. random: for each read query, choose a random slave
2. round-robin: for each read query, choose the next slaves from the slave list in a round-robin fashion
3. user defined: picking a server is left to the PHP script
4. new random_once: for each web request choose a random slave for use with all read queries for the rest of the request.

The "random_once" is inspired by one of the most basic approaches to support MySQL replication in a PHP script, see the below snippet. At the beginning of the script run you open two connections: a master connection, a slave connection. The slave is picked in a random fashion. You send every write using $master connection handle to the master. And, of course, you detect and direct all reading queries to the slave via the $slave connection handle. In some cases, such as for critical write-read sequences if you must be able to read what you had just written, you skip the R/W split to run your critical write-read sequence on the master. In other words: you can overrule the default R/W split, if need be.

$master = new mysqli("master", "user", "password", "database", "port");
	
$slave_list = array("slave_a", "slave_b", "slave_c", "slave_d");
$slave_idx = mt_rand(0, (count($slave_list) - 1));
$slave = new mysqli($slave_list[$slave_idx], "user", "password", "database", "port");
	
/* For the rest of the script: manual R/W split */
$master->query("INSERT INTO table_a(col_a) VALUES (1)");
$slave->query("SELECT * FROM table_b");
$slave->query("SELECT * FROM table_a");

The plugin counterpart: C for comfortable

PECL/mysqlnd_ms implements R/W split detection in C. Automatic R/W split decisions can be overruled by the application using SQL hints. The new "random_once" load balancing policy implements the above logic for picking a slave. But because its a plugin and (semi-)transparent. You have only one connection handle to deal with in your PHP script. This makes it possible to use the plugin with existing applications without (significant) code changes. Just replace the host name with a config section name, that’s about it (full example).

/* automatic pooling */
$mysqli = new mysqli("mysqlnd_ms_config_section", "user", "password", "port");
	
/* automatic R/W split and balancing */
/* master */
$mysqli->query("INSERT INTO table_a(col_a) VALUES (1)");
/* slave x, x = random pick from S1, S2, S3 */
$mysqli->query("SELECT * FROM table_b");
/* slave x, no balancing, no switch to another slave */
$mysqli->query("SELECT * FROM table_a");

(config, see also full example)

[mysqlnd_ms_config_section]
master[] = master
slave[] = s1
slave[] = s2
slave[] = s3
pick[] = random_once

Less connections

The new load balancing policy significantly improves the plugins use of connections maintaining even load distribution. MySQL is certainly very fast establishing a connection, much faster than other databases. However, there is always a cost: the network connection itself and the server resources occupied by a server connection thread. The less connections a PHP web requests opens, the better.

This is where "random_once" shines. Let’s say your set up consists of one MySQL master and three MySQL slaves. To make HTML table hacking easier for me, let’s further assume you run only two web servers. Let’s take a snapshot of the system when both web servers run two PHP requests and each PHP script running has run one write query and four read queries through PECL/mysqlnd_ms.

Compare the two system snapshot illustrations from above and the beginning of the blog ost. Using round robin policy to load balance queries during script run we see a total of 16 connections (4 master, 12 slave). With random-once it is only 8 connections (4 master, 4 slave). Query distribution is comparable in the example. In both cases two of the three slaves have run four reads and the third slave has run eigth reads. Over time, distribution will become more even with both random once and round robin. However, illustrating that is beyond my rusty HTML table hacking skills .

Chance of better cache usage

If a PHP script runs many similar SELECT during a request, there is a chance that upon subsequent execution the query is served from the database server caches. But that hot cache chance is only given, if the load balancer does not switch connections in between two SELECT queries. The built-in random and round-robin load balancing policies do switch connections for every query. The new random-once policy picks one random slave when the PHP script starts and uses it until the script ends - together with the web request. This increases the chance of better cache usage.

Statefull vs. stateless slave connection

One big pitfall is system immanent to any driver redirecting queries over multiple connections: state. Connections have a state (see also "PHP: Master slave load balancing in mysqlnd - how transparent?") which may "forbid" switching. With "random_once" this is not solved but cured a bit because there is only one slave connection. It is still up to the application to hint the driver/plugin which connection (master or slave) to use for SQL units that rely on connection state. But there is only one slave connection, not many, making it a bit easier.

Comments welcome

Please recall that PECL/mysqlnd_ms is a proof-of-concept. Its far from stable, we are still trying to figure out how to do things. If you have any idea to share, … happy hacking!

A common misunderstanding about innodb_support_xa is that it enables user-initiated XA transactions, that is, transactions that are prepared and then committed on multiple systems, with an external transaction coordinator. This is actually not precisely what this option is for. It enables two-phase commit in InnoDB (prepare, then commit). This is necessary not only for user-initiated XA, but also for internal XA coordination between the InnoDB transaction logs and the MySQL binary logs, to ensure that they are consistent. Consistent is an important word with a special meaning.

We have done some benchmarking and performance research on this option in the past (see also: post 1, post 2). This was motivated by the severe performance hit that occurred when the commit process was changed to an internal XA implementation. This did two things: it disabled group commit, and it added an extra fsync per commit. The extra fsync is required because during the internal ‘prepare’ stage, the transaction can’t be considered prepared (and thus guaranteed to be recoverable) until it is synced to durable storage. The final fsync marks the transaction as committed, not just prepared. The extra fsync can be avoided by disabling the option. (Group commit can’t be re-enabled unless the binary log is also disabled.)

The prepare-then-commit process does two important things: it ensures that the binary log and the engine’s transaction log agree on which transactions are committed; and it ensures that they agree on the order in which the transactions were committed. The first is important for crash recovery, and the second is important for replication.

However, it is not as simple as “if you want higher performance, disable innodb_support_xa.” Our past blog posts might have been too shallow in their discussion on this point. You can disable InnoDB two-phase commit if you wish, but you need to weigh the risk of out-of-sync data on replicas (and after point-in-time recovery) when you do that.

Unfortunately, the manual isn’t that helpful in this instance — it doesn’t clear things up very well:

When the variable is enabled (the default), InnoDB support for two-phase commit in XA transactions is enabled, which causes an extra disk flush for transaction preparation. If you do not wish to use XA transactions, you can disable this variable to reduce the number of disk flushes and get better InnoDB performance. Having innodb_support_xa enabled on a replication master—or on any MySQL server where binary logging is in use—ensures that the binary log does not get out of sync compared to the table data.

That sort of touches on both internal and external XA transactions, but doesn’t distinguish between them. Hopefully this post will help clarify the difference, and help people make good decisions about the setting. Thanks to Kristian Nielsen for review and feedback.

Over the past couple of days I've been fighting with a weird puppet problem , we eventually cracked it , but I promised a bunch of you to fully explain it here ;)

So we were deploying 2 Blade chassis at a pretty remote location with a mix of phyisical and virtual machines, some 48 instances in total. This is a pretty standard rollout, we've got a bunch of similar platforms in our lab , so we knew about a couple of glitches, what to expect etc.

I was just keeping an eye on the deployment, looking at the logs seeing if things were running fine, when suddenly a couple of puppet runs didn't come trough, we had seen such behaviour before, usually it's a matter of running them a gain a couple of times and they will come trough. (Upgrading ruby and putting passenger in front of puppet actually solved those issues,
We'd even had a loop built in the platform that runs puppet a couple of times till it returns with the correct exit code just to make sure. )

We were first scratching the A chain of our setup, so that in the event of failure we could still bring up the B chain of the platform and be up and running again. Actually machines were coming up.. slowly .. some of them took a bit longer . One of the machine's clock was seriously off .. the SSL was barfing on it , so we set the bios clock, and restarted .. it was the machine with 6VM's took a while but everything was back on schedule.. then suddenly things were going down fast more and more puppetruns started failing and .. , at some point in time actually none of our puppet runs were working again .
I'd see the puppetmaster perfectly compile it's catalog

notice: Compiled catalog for ctl-0-a

Then the client .. not wanting to get it ..

Mar  1 11:10:45 ctl-0-a puppet-agent[3674]: Not using expired catalog for ctl-0-a from cache; expired at Tue Mar 01 09:50:06 +0000 2011

We had gone from about 60% of our fresly deployed boxen working fine, to not one
So what do you do .. indeed .. you turn on debugging.
You put both your puppetmaster and client in debug. Nothing, no errors no nothing ..

I asked some collegues, asked on irc .. much ideas but none of them that actually cracked the problem. I did what I knew that solve similar problems before,

I switched our serialization format from yaml back to pson , and back, no luck.
I upgraded ruby to a version from the glei.ch repository. No luck.
I upgraded our Puppet version 2.6 to a version from the TMZ Epel repo , we cleaned out ssl the certificates on all sides multiple times. Cleaned out /var/lib/puppet , We uninstalled puppet and reinstalled it.
It wasn't a DNS Problem

I had started stripping my manifests to empty runs, those worked, then started uncommenting the actual manifests again ... Then in the middle of the debug our VPN connection to the remote location broke down, we'd only be getting it back in the morning ..about 12 hours later not fun. Murphy obviously ..

So the next morning we dived right back in ... making those manifests bigger again, removing all the stages, 1 or 2 successful runs, then with the same config .. back to failure. On and off.. successfull and unscussessful. ... it wasn't in the manifests ..

So we decided to roll the puppetmaster back to it's previous version, that one was known to be stable, there obviously was something really fishy going, so that was the safest bet.

Wrong, the machine came up, but it took longer than expected, and when trying to connect new clients to it .. nothing worked anymore .. same problem as before .. puppetmaster compiles catalog, clients didn't get anything. we started to suspect faulty hardware .. but how could that bee.. the puppetclient looked liked the only malfunctioning thing around .

Then Dim0 suggested me to look at the that one logfile I hadn't looked , /var/log/puppet/masterhttp.log and then we saw it . it was being flooded with ssl errors, ssl errors from clients that shouldn't even be connecting to the puppetmaster at all.

[2011-03-02 13:32:00] ERROR OpenSSL::SSL::SSLError: tlsv1 alert decrypt error

What happened was that 'we' decided to bring of the one backup machines back online, afterall once the slow starting server came trough, it would be the passive node in the cluster , no worries there, right ? Wrong,
This physical machine had 6 virtual machines with old ssl certificates that got stuck in an loop which was put there to sure their puppetrun came trough correctly at boot time.

Those 7 rogue clients which generated little to no relevant traffic on the network were saturating the default webrick, killing them solved the problem and we were back to regular deployment in no time.

The sad part is that our upcoming release already has passenger , a fresher version of ruby etc .. and that most of the above mentioned errors won't occur anymore there.
But in short .. don't use the default webrick .. it will kill you :)

And no , not everything is a freaking dns problem, ssl is a big pain in the B too .. :)

Technorati Tags:devops passenger puppet ruby ssl webrick

Share with Shareomatic!

OK, I know that a lot of you know where I was this week... yes, the MVP Summit. I absolutely love getting together with so many friends that I don't regularly see... This is by far, my favorite part about this annual event. Yes, I know I get access to cool information about upcoming technologies but after 20 years of this (remember, I started when I was 2 :)), I've also learned that a lot changes and so I no longer spend a ton of time learning all about early betas. Having said that, I am pretty excited about SQL Server Denali. I've already started to dive into CTP2 and I'm even more interested in CTP3. But, things can still change and so I'm not betting the farm on what I see.

Having said that, one thing did surprise me this week. A few of the sessions had slide decks where a horrible "event" template was applied. I don't directly blame a lot of these presenters because they're just not regular speakers at events. Many of these folks are developers and program managers on the team and so they haven't all been burned by "slide management" teams (like I have) and so many didn't even think to review their reviewed/tweaked decks. As a result, a few slides ended up with some bad color choices. From what I was told, somewhere a template was applied and somewhere whites turned into blacks (or similar dark colors) and the next thing we know, there are some VERY hard to read slides. And, because the content is the most important to me, well, I'm still happy.

However, I decided to put together a short (and fun) slide deck with a few tips and tricks that I've learned over the years. I even accepted a few tweets about other snafus during presenting (again, mostly related to slide deck no-nos) and added a few of them here. Feel free to comment about other things you've seen, learned and now do differently/better! I'd love to hear more!!!

So, without further ado, here's the deck titled: How to make the audience turn on you... FAST!
(with a subtitle of: kind of like the cougar in Ricky Bobby’s car… but worse!)

For all of the details on bad presentations shown in the deck - here are some VERY helpful links to help you create a better presentation and become a better presenter.

• Paul Randal’s Public Speaking: A Primer
  • http://www.sqlskills.com/BLOGS/PAUL/post/Public-speaking-about-SQL-Server-A-Primer.aspx
• Kimberly Tripp’s Getting started in speaking publicly – clear and concise presentations
  • http://www.sqlskills.com/BLOGS/KIMBERLY/post/Getting-started-in-speaking-publicly-clear-and-concise-presentations.aspx
• Scott Hanselman’s Tips for a Successful MSFT Presentation
  • http://www.hanselman.com/blog/content/radiostories/2003/01/22/scotthanselmanstipsforasuccessfulmsftpresentation.html
• Brent Ozar’s How to Deliver a Killer Technical Presentation
  • http://www.brentozar.com/archive/2009/09/how-to-deliver-a-killer-tech-presentation/
• Greg Low’s series: Presenting at Large Events (Lessons Learned)
  • http://sqlblog.com/blogs/greg_low/archive/2008/10/01/presenting-at-large-events-lessons-learned.aspx
• Kendra Little’s What’s in a Name? Guidelines for Presentation Titles
  • http://www.littlekendra.com/2011/02/11/whats-in-a-name/
• Paul Randal’s Configuring SSMS for presenting
  • http://www.sqlskills.com/BLOGS/PAUL/post/Configuring-SSMS-for-presenting.aspx

Good luck and fun!
kt

We receive many requests for help with server stalls. They come under various names: lockup, freeze, sudden slowdown. When something happens only once or twice a day, it can be difficult to catch it in action. Unfortunately, this often leads to trial-and-error approaches, which can drag on for days (or even months), and cause a lot of harm due to the “error” part of “trial-and-error.” At Percona we have become skilled at diagnosing these types of problems, and we can solve many of them quickly and conclusively with no guesswork. The key is to use a logical approach and good tools.

The process is straightforward:

1. Determine what conditions are observably abnormal when the problem occurs.
2. Gather diagnostic data when the conditions occur.
3. Analyze the diagnostic data. The answer will usually be obvious.

Step 1 is usually pretty simple, but it’s the most important to get right. I estimate that about 80% of the cases I work on show up as abnormally high values of Threads_connected or Threads_running. That’s a good place to start for finding a symptom that can trigger the problem. If you aren’t sure what “normal” looks like, and therefore what’s abnormal, then gather samples of things you can observe (such as SHOW STATUS snapshots) and establish both baseline behavior and deviations from it.

The hard part used to be step 2. However, over time the Percona team members have built a good set of tools for doing this on Linux. It is no harder in principle on other OSes, but we have so many customers on Linux that we end up scripting mostly Linux-centric things, so a little tweaking to the tools might be needed on other platforms. The rest of this blog post is about how to set up and use the tools.

The tools are all part of the Aspersa toolkit. There are three primary tools for troubleshooting intermittent problems: stalk, collect, and sift. As the names suggest, they are used to watch the server, gather diagnostic data when the condition occurs, and help you look through it quickly (there’s a lot of it). You can get the tools easily. The best thing to do is put them into root’s home directory; they will gather the most data possible if they’re run as root. Running under sudo is an option too. Assuming root, here’s how to do it:

cd /root
mkdir -p bin
cd bin
wget http://aspersa.googlecode.com/svn/trunk/{stalk,collect,sift}
chmod +x stalk collect sift
cd /root

Now you need to edit the stalk file and tell it your definition of “abnormal” — by default it has some vanilla settings that aren’t useful. (There is no such thing as a good default, sorry.) Let’s assume that your server normally has only a few Threads_running, and when the problem happens, it spikes up into the range of 20 or higher. A safe trigger might be 15, if your server never reaches 15 under normal conditions. The more carefully you determine normal versus abnormal, the higher quality your results will be. Find and change the following lines:

# This is the max number of <whatever> we want to tolerate.
THRESHOLD=${THRESHOLD:-100}

# This is the thing to check for.
VARIABLE=${VARIABLE:-Threads_connected}

You need to change the threshold to 15, and the variable to Threads_running. Check the other settings, but this is often all you need. Finally, make sure that the script can log into MySQL (if necessary, put a password into .my.cnf, or in the MYSQLOPTIONS variable). Now you can simply start running bin/stalk in a screen session. It shouldn’t do anything but print a timestamp line every 30 seconds. Come back later and see what it found, or set the EMAIL setting to notify you.

What’s so special about the collect tool? Isn’t [insert your favorite OS's awesome tool that Linux doesn't have] better? What’s nice about collect is that it gathers many different types of data, including different angles on the same data. No single tool, no matter how powerful, is an answer for every problem. For example, the collect tool makes it easy to gather snapshots of waiting (stack traces) and CPU consumption (oprofile). That’s important because sometimes the problem is because of blocking and waiting to do work, and sometimes the problem is doing too much work without blocking. Likewise, it gathers lots of I/O data, because sometimes the problem is that the disks are not responding within reasonable and expected tolerances, and sometimes the disks are responding very well but the server is abusing them and overloading them. And sometimes, of course, the CPUs and disks have nothing to do with anything, which is why the collect tool also gathers information about network connections, and so on. If you assume a particular type of trouble in advance, you’ll be wrong sometimes, and it’s back to trial-and-error. You need to gather all the data, and then look at it and see which bits are diagnostic and which bits show nothing relevant.

After you’ve gathered some data, you need to look through it. It is stored in /root/collected/ by default. Use the “sift” tool to examine it. I’ll demo on my laptop:

[baron@ginger]$ sift collected

  2011_02_28_14_51_38  2011_03_01_02_52_52  2011_03_01_14_29_35
  2011_03_01_18_38_09  2011_03_02_08_36_25  2011_03_02_11_33_59
  2011_03_02_15_10_03

Select a timestamp from the list: 2011_03_02_15_10_03

It’s prompting you to choose which sample of data to analyze. There are 7 samples, and it chooses the most recent by default, so you can just hit the Return key and start browsing the files. The tool is keystroke-driven, Vim-style, so you can navigate the samples with the ‘j’ and ‘k’ keys. I won’t repeat the official documentation here. Give it a try and see how it works; press the ‘?’ key for a list of the keystroke commands you can use. If you find that one of the samples looks suspicious, you can drill into it with a few of the other Aspersa tools, or ask sift to start ‘less’ on a sample’s files so you can look at them. You can analyze the files without sift, but it saves a lot of typing, especially when you have dozens or hundreds of samples to examine.

The sift tool isn’t the final step in analysis. By default it just shows you a few of the highest-level summary tidbits of information, and you often really need to drill into a particular sample’s files and analyze it to see what’s going on. Sometimes it’s glaringly obvious even at a high level, but not always. Most of the time the problem is clear upon a closer inspection, though.

If the answer is not obvious, then maybe you have a false positive and didn’t really catch the problem in action; or collect isn’t gathering the information needed to diagnose (pretty unlikely — it gathers a ton of stuff); or you don’t know how to interpret what you have found. Within Percona, the best course of action is often to ask a more senior person to help. I ask people such as Peter, Vadim, and Yasufumi for help all the time. If you’re not a Percona support/consulting staff member, then of course you can hire us.

If I get a few spare hours, I will put together some screencasts illustrating how to use these tools. I think that’ll be an easier and more fun way to learn them. I have some interesting cases that I can use as demos, after I scrub them for sensitive details. When I finish these, I’ll post them on Percona.TV, our MySQL video/screencast blog site.

Another interesting topic is how to examine a system that appears to be fine and determine whether it is experiencing any very short stalls that you are not aware of. There are many things that can cause a stall in MySQL, and they usually begin microscopically and get worse over time, sometimes abruptly worse. I’ll try to write more about this as time permits, too.

ZF2 development is ramping up. We've been at it for some time now, but mostly taking care of infrastructure: converting to namespaces, re-working our exception strategy, improving our test suites, and improving our autoloading and plugin loading strategies to be more performant and flexible. Today, we're actively working on the MVC milestone, which we expect to be one of the last major pieces necessary for developers to start developing on top of ZF2.

A question I receive often is: "How can I contribute to ZF2?"

Consider this your guide.

Distributing your files over a content distribution network (CDN) is all about increasing speed and expanding your global footprint.  The Cloud Files team is dedicated to making sure our CDN is up to date with technologies that can increase that performance whenever possible.

We are pleased to announce the launch of HTTP Compression on the Cloud Files CDN.  HTTP Compression is a simple, cost saving process.  When a browser requests an object, it may also supply an “Accept-Encoding” header that tells the CDN whether or not it will accept a compressed response. The CDN will then serve the object compressed, or not, at the browser’s preference. With this feature, the Cloud Files CDN is smart enough to know how the browser wants the object, convert that object and serve it accordingly.

Compression occurs when you store an object in Cloud Files and serve it over the CDN.  HTTP Compression will only work for uncompressed content in Cloud Files that the browser requests to be served compressed over the CDN.   If an object is stored in Cloud Files compressed and the browser requests to serve it uncompressed, the CDN edge can’t currently convert the object.  We plan to add this functionality (storing compressed files and serving them uncompressed) in upcoming releases.

Why do you want HTTP compression on your CDN?

Compressed objects can be downloaded more quickly than uncompressed objects resulting in faster page load timesfor your users and lower bandwidth costs for you. The Cloud Files CDN supports compression on HTML, JavaScript, and CSS, files ( “text/html*”, “application/x-javascript*”, and “text/css*” content types, respectively).

This feature has been added to Cloud Files functionality and does not require any specific configuration or action on your part.  We’re excited to offer this feature to you and look forward to hearing your responses, questions, and feedback.

Yes, it has been quite some time since I blogged, work has been very busy lately.

Currently we have a number of various backup strategies that our partners may use, one of which has been hot backups via xtrabackup (or innobackup/MySQL Enterprise Backup – with a license fee of course).

At this time we have one person dedicated to maintaining the backups, which includes rewriting innobackupex to handle extras and also write wrapper scripts around the original one.

This morning he contacted me because he was running into problems with xtrabackup 1.5 and streaming. No, not the usual performance issues etc, but rather a few random .MYI files were missing and the xtrabackup_checkpoint and xtrabackup_logfiles were missing.

What was interesting was the the MYI,frm and MYD files missing was random – it was mainly static but would change every now and then when his script ran.
At first glance without reviewing his script, it seemed like something was going on with xtrabackup streaming mode, since the help_*.* files contained the checkpoints info, so it looked like it streamed it to the wrong place..

However, after I was informed about it, I decided to look at the script this person had written, and I realized that he did something like..
innobackupex –backup ……. –stream=tar /path 1>/file.tar 2>/logfile.

After testing, its clear that xtrabackup does not like that. You are not able to separate stdout and stderr like this, since it breaks the application. By changing 1>/file.tar 2>/logfile to just >/file.tar it all worked well.

Why does innobackupex write things to STDERR that should be sent to the tar stream? I do not know, but I hope that someone here can help out.
Either way, this is a reminder for all of you that want to use it – you cannot suppress output, since that will break the stream mode.

Have a great spring!

When I logged into the ‘free‘ airport wifi at Calgary airport a few moths back I was in invited to either give “BOLDstreet Wireless” permission to my Facebook account or otherwise pay a few dollars for an hour’s connection.

At first I thought was a strange choice, until it dawned me on that this WiFi company was essentially putting a marketing $ value on my Facebook account… Hand over access and they’ll monetize my account in some untransparent way in order to cover the cost of the not-so-free-anymore wifi.

It turns out BOLDstreet Wireless has built this out as a product which companies like Calgary Airport Authority can be purchase to track, monetize and analyize public wifi hotspot activity.

In true hacker mentality I logged in with a fake developer account I use for testing purposes – but whatever.

I forgot all about this until today when I was invited to do exactly the same – give permission for an app to access my Facebook account in return for a ‘free’ Häagen-Dazs ice cream.

Now, there is nothing new or usual about companies wanting to get a little information about your for the CRM systems in return for providing a free sample. But there are some stark and concerning differences created with this new approach:

• Unlike a survey which questions you directly, there is no transparency as to what information is being taken
• In addition to my own profile data, limited data about my friends is being handed over too.
• A fresh snapshot of this information can be requested at any time due to the fact permission persists until the user turns it off
• More personal data might be made available in the future as Facebook evolves the data they store about you – eg phone number

Perhaps one of the most concerning aspects of all this is the fact that BOLDStreet and Häagen-Dazs are potentially getting access about me through my friends using their service – data I did not give either company permission to have. In fact, I wouldn’t even know if they had this information.

There’s nothing new per se with the issue of applications having access to this data – this has been the case since day 0 for apps. However, one argument has been that socially orientated apps need this information in order to be able to provide a social experience. But this use case is certainly new and doesn’t warrant these types of companies gaining access about a user’s social graph in addition to the user’s personal details directly.

From my own experience, this is becoming a common trend. Facebook Connect certainly has advantages but it also has disadvantages too. Be careful who you are giving permission to your account to and make sure you regularly review the list of companies and apps with permission to access your profile (ditto for Twitter too).

I've been investigating ways to incorporate third-party repositories and libraries into my Git projects. Subversion's svn:externals capabilities are one compelling feature for that particular VCS, and few, if any, other VCS systems, particularly the DVCS systems, have a truly viable equivalent. Git submodules aren't terrible, but they assume you want the entire repository -- whereas SVN allows you to cherry-pick subdirectories if desired.

Why might I want to link only a subdirectory? Consider a project with this structure:

docs/
    api/
    manual/
        html/
        module_specs/
library/
    Foo/
        ComponentA/
        ComponentB/
tests/
    Foo/
        ComponentA/
        ComponentB/

On another project, I want to use ComponentB. With svn:externals, this is easy:

library/Foo/ComponentB http://repohost/svn/trunk/library/Foo/ComponentB

and now the directory is added and tracked.

With Git, it's a different story. One solution I've found is using git-subtree, an extension to Git. It takes a bit more effort to setup than svn:externals, but offers the benefits of easily freezing on a specific commit, and squashing all changes into a single commit.

Jon Whitcraft recently had some questions about how to use it, and I answered him via email. Evidently what I gave him worked for him, as he then requested if he could post my guide -- which you can find here.

Aren’t those “This site has unsecure content…” popups annoying? I bet your users think so.  Rackspace Cloud Files is pleased to announce the support of SSL delivery over the Akamai content distribution network (CDN).

SSL delivery, or https, encrypts your data in transit as it goes through the CDN.  Your content remains encrypted all the way from the origin servers out to the browser.

You can choose to serve your content over SSL on a per object basis simply by using your SSL URL instead of your traditional CDN URL.

If you are a customer interfacing with our Cloud Files API, when you request your CDN URL, you will now be returned with both your traditional URL as well as your new SSL URL.  Look for the header in the API “X-CDN-SSL-URL” and use this URL to securely deliver your content.

If you are managing Cloud Files through our Cloud Control Panel, you can take advantage of SSL as well by following a few simple steps:

Step 1:  replace the “http” with “https” in your CDN URL.

Step 2:  replace the “r00” portion of your CDN URL with “ssl”

For example -

If my CDN URL looks like this:
http://c186397.r97.cf1.rackcdn.com/CloudFiles Akamai.pdf

My SSL URL for CDN would look like this:
https://c186397.ssl.cf1.rackcdn.com/CloudFiles Akamai.pdf

That’s it!  With these small changes you are ready to begin serving content over SSL.

Not sure if SSL is for you?  For the next six months (March-August 2011), Rackspace is offering delivery over SSL at the same price as traditional CDN delivery so you can try it out at no additional cost!

If you’ve got questions or comments, we’d love to hear them.

This post was written and contributed by Allen MacCannell, VP Sales & Marketing at Web CEO, a Rackspace Cloud Tools Partner.

The importance of having a website optimized for search engines cannot be underestimated. Web CEO has been producing search engine optimization (SEO) software since 2000 and recently gathered fresh data over a two year period that showed 50% of the visitors of 300,000 websites consisted of unpaid (organic) traffic from search engines, mainly Google.

It is wasteful to spend a lot of money on pay-per-click (PPC) advertising via Google and other search engines, without at least spending a fair amount on optimizing a website for organic search engine traffic (conducting organic SEO).

Desktop SEO software has, for a long time, been the means with which webmasters could optimize and promote their sites. For instance, with Web CEO’s desktop addition (~800,000 downloads), webmasters could educate themselves about SEO, audit their sites for errors, build links, research keywords, check rankings, check for backlinks and backlink quality and get plenty of suggestions for optimizing their sites. All of the results could be mailed to oneself and others via configured reports.

However, a big disadvantage of using desktop software for SEO is that the analysis and the configurations for the reports emanate from only one person’s computer.

Let’s say a CEO spends a lot of time in Europe while a Marketing Director is in California and a Webmaster is in Montana. If all three of these people wish to collaborate in configuring tools and conducting SEO analysis, they will need to be able to logon a web-based program rather than duplicate efforts on three separate installations of desktop SEO software. As everyone knows, web-based services allow many people to work with each other remotely.

Web CEO Online was launched in January 2011 as a web-based SEO tool suite that will allow greater collaboration for website marketing and promotion teams. Web CEO Online is a set of web-based tools that will improve a website and then measure and track its performance.  The service can be used by individual website owners with one or two domain names or by SEO agencies that work to optimize thousands of sites.  Web CEO Online includes an easy-to-follow website improvement roadmap (eight different Working Plans including SEO, link building, social media optimization and promotion tasks and recommendations) and tools to complete the tasks such as the Keyword Research tool that uses the Google AdWords API, Search Engine Optimizer, tools to build backlinks and a Site Quality Auditor).

Technologically, Web CEO Online is fully cloud-based, which makes the application very flexible and robust.

The free version of the service will help anyone who has a single website with very few pages and only 5 keywords to optimize for. A $49 per month version will help a larger organization optimize up to 10 domains for 90 keywords. A $99 per month version (+ $2 per active URL per month) will help any SEO agency optimize an unlimited number of websites for an unlimited number of keywords.

Give Web CEO Online a spin for the benefit of your websites with this special one month Rackspace offer (a $50 value).

Mike Mayo is the mobile apps developer for Rackspace. If you’re interested in joining his team, click here.

Today I’m excited to announce the release of Rackspace Cloud 2.0 for for the iPhone, iPad and iPod Touch – something I’ve been working hard on to build.

This app is a complete rewrite of our existing iPhone and iPad apps, and is now universal.  You can run this app on an iPhone, iPad and iPod Touch.  Rather than having separate apps for iPhone and iPad, we can now push out new features (like Chef Bootstrapping!) to all iOS devices at once.

Another important thing to know about this app is that it’s open source and part of the OpenStack project -  the open source software that powers the Rackspace Cloud.  The source code to the OpenStack iOS app is available at http://launchpad.net/openstack-ios.

Here are a few notable features in the app:

Support for Multiple Accounts and API Endpoints

You can now manage as many Rackspace Cloud accounts as you want.  You can connect to the US and UK clouds, as well as any custom OpenStack cloud you may be running.

Need for Speed

In the old apps, you would have to wait for several seconds to log in and load data for your Cloud Servers, images, and sizes.  Now all of that data is stored locally so you can get to it very quickly.  You can literally reboot a server in just a few seconds!

Passcode Lock with Data Wipe

Your cloud resources are extremely important and should be protected.  With the new app, you can turn on a passcode lock to prevent others from tampering with your servers or files.  You can even turn on a data wipe feature that wipes the app of all account data after 10 failed login attempts.

Create Multiple Chef Bootstrapped Cloud Servers at Once

You can provide a Chef run list when you create Cloud Servers and create up to 10 servers at a time.  Need 5 new web servers in your cluster?  You can have it in less than 30 seconds from your iPhone.

IP Addresses

On the new Cloud Server screen, when you tap an IP address, you can choose to copy it to your pasteboard, ping it from 50 locations around the world, or open it with an SSH client installed on your device (such as the excellent iSSH).

Detailed Error Handling

If a call to the Rackspace Cloud API fails, you can see exactly what happened by pressing the Details button.  You’ll be able to see a complete equivalent curl command for what the app was trying to do, as well as the HTTP response from the API.  You can then email that to troubleshoot further.

Fanatical Support in Your Pocket

You now have full access to our Cloud Servers and Cloud Files System Status RSS feeds, as well as buttons to contact us directly via phone or Twitter.

Awesome CDN Support for Cloud Files

You can completely control everything regarding your CDN enabled Cloud Files containers such as change the TTL, logging, ACLs.  If you have the Twitter app installed on your device, you can even tweet links to your files.  You can also share files via email -  a great way to share photos and video while still having complete control of where your files are stored.

View and Print Cloud Files

When you download a file from Cloud Files to the app, it will try its best to open it.  The app can open text files, PDFs, audio, video, images, and office documents.  If the app can’t open it, it will check your device to see if any other app can and if so, it will give you that option.  Also, if your file is printable, you can print to a printer on your network directly from the device.

Other Features

- Accounts can be sorted for easy access to your most important accounts
- Reboot, rename, resize, and rebuild your Cloud Servers
- Change the root password for your Cloud Servers
- Manage backup schedules for your Cloud Servers
- Delete Cloud Servers
- Create and delete Cloud Files containers
- Create folders in Cloud Files containers
- Create media (photos and video) and text files in Cloud Files containers
- Delete Cloud Files
- View your API rate limits

We are very excited to give you this app free in the App Store, and we hope you enjoy it.  If you need any help with it or want to know more, feel free to get in touch with me via email or Twitter.

We’re Hiring!

If you think this app is cool and would like to help make it (and Android) even better, join us in our new San Francisco office! Click to learn more.

It’s no secret that Rackspace loves startups – we were one not too long ago. We are still committed to helping entrepreneurs succeed and drive the next generation of businesses through cloud innovation.  Now, with the Rackspace Startup Program, we’ll be providing one-on-one guidance and Cloud Compute resources to help power these startups to success.  And we’ll be helpful all along the way.

The Incubators

We’ve partnered with leading startup accelerators and incubators such as 500 Startups, TechStars, Y Combinator and General Assembly to providing cloud computing resources to their participating startups.

Dave McClure, Founder of 500 Startups

“A lot of folks don’t understand that Rackspace is actually a startup success story – they grew from a 10 person startup to a leading cloud player, so they understand what it takes to succeed. Fortunately for the tech community, Rackspace continues to invest in entrepreneurs by providing compute power to hundreds of startups.”

The Startups

Many startups are using our Cloud today.  Companies like SwebApps and Freshbooks have grown their business from the ground up and continue to see success. David Stein, founder of Dubset, a General Assembly startup, quotes:

“The Rackspace Cloud allows us to stream music while accurately tracking what gets played—which we believe is the future of music consumption. The Cloud is key to the success of our company: our costs are low and we never have to worry about our Cloud Servers.”

We hope to see more startups like Dubset build on our Cloud.

For more information

Please contact Jeff Lamboy with any questions or comments. Click here to read the official press release and check out our video.

Rackspace Startup Program from Rackspace Hosting on Vimeo.

Find us at SXSW

If you will be attending SXSW and want to learn more, we’d love to catch up with you. Here’s where we will be:

•    11:00 a.m., Saturday, March 12, King of the Apps Showdown, The Lean Startup Track, Room # 204 @ AT&T Conference Center

•    3:30 pm, Sunday, March 13, Kill Your Call Center! Bring Your Support Center Home, Salon D @ Hilton

•    12:30 p.m., Monday, March 14, Live  CloudU Streaming Webinar @ Kung Fu Saloon

•    3:30 p.m., Monday, March 14, OMG My Customer’s Pissed and Uses Twitter , Room # 203 @ AT&T Conference Center

•    5:00 p,m., Monday, March 14, Scalability: Covering Your Rear with a Good Backend, Ballroom C @ Austin Convention Center

•    6:00 pm, Monday, March 14, Rackspace + OpenStack Arcade Bash: Drinks, food and games @ Kung Fu Saloon

I can't seem to find an article here that consolidates my thoughts on Zend Framework's view helper system, so I thought I'd better correct that. Zend Framework's Zend_View component supports helper methods known as view helpers. They are used like this in a view script:

<?php echo $this->myHelper('myParam1'); ?>

Behind the scenes, this is implemented as a method within a class something like this:

<?php

class Zend_View_Helper_MyHelper extends Zend_View_Helper_Abstract
{
    public function myHelper($myParam1)
    {
        $html = '';
        // some logic that fills in $html.
        return $html;
    }
}

Note that by convention, view helpers return the string and then the view script echos it and the Zend_View_Helper_ section of the class name is known as the prefix.

A typical Zend Framework project using Zend_Application, such as that generated using the zf command line tool, will have a folder called helpers within the views folder for each module. There will also be a helpers folder within the layouts folder too. If you place your view helper in one of these helpers folders, then the prefix is Zend_View_Helper_.

We can also tell the view object about our own view helper folder. Typically this will live in the library/App/View/Helper/ folder and so, you probably want a prefix of App_View_Helper_. To do this, we simply add this line to application.ini:

resources.view.helperPath.App_View_Helper_ = "App/View/Helper/"

The view helper now lives in App/View/Helper/MyHelper.php and looks like this:

<?php

class App_View_Helper_MyHelper extends Zend_View_Helper_Abstract
{
    public function myHelper($myParam1)
    {
        $html = '';
        // some logic that fills in $html.
        return $html;
    }
}

Due to the way Zend Framework's plugin loader works, our view script doesn't change at all as the view object's plugin loader will look across all registered paths to find the view helper specified in the view script.

I tend to prefer using the App folder over the helpers folder within layouts as I like to have one place to go to for my "site-wide" classes and so my App folder contains front controller plugins, validators, filters, form elements as well.

As you have extended Zend_View_Helper_Abstract for creating your own helper, you have access to the view object itself too. This is helpful for accessing other view helpers, such as escape. My personal preference is for my view helpers to manage their own escaping as requires as that way I can include html tags within my helper.

For example, suppose we want to create a table from an array that looks like this:

$data = array();
$data[] = array('Name', 'Email');
$data[] = array('Alison', 'alison@example.com');
$data[] = array('Bert', 'bert@example.com');
$data[] = array('Charlie', 'charlie@example.com');

Our view script may look like this:

<?php echo $this->tabulate($this->data, array('class'=>'tabulated')); ?>

(assuming our designer wants a class name attached to the table for styling.)

The view helper itself would then look something like:

<?php

class App_View_Helper_Tabulate extends Zend_View_Helper_Abstract
{
    public function tabulate ($data, $attribs = array())
    {
        $attribString = '';
        foreach ($attribs as $key => $value) {
            $attribString .= ' ' . $key .'="' . $value . '"';
        }

        $header = array_shift($data);
        $html = "<table $attribString>n<tr>n";
        foreach ($header as $cell) {
            $escapedCell = $this->view->escape($cell);
            $html .= "<th>$escapedCell</th>n";
        }
        $html .= "</tr>n";
        foreach ($data as $row) {
            $html .= "<tr>n";
            foreach ($row as $cell) {
                $escapedCell = $this->view->escape($cell);
                $html .= "<td>$escapedCell</td>n";
            }
            $html .= "</tr>n";
        }

        $html .= '</table>';
        return $html;
    }
}

As you can see, we cannot escape the output of the view helper, so the view helper has to do its own escaping via the view property.

If you end up using the same set of view helpers on each project you do, then consider putting then into a vendor library. This is the same concept as using the App folder, but you use a different name (I use Akrabat) and store them in a separate vcs repository. This allows you easy reuse. You add a new line to application.ini to pick them up:

resources.view.helperPath.Akrabat_View_Helper_ = "Akrabat/View/Helper/"

and you're done. Note that the order of the helperPath in application.ini is important. You want the vendor path before the App path so that you can override your vendor ones specifically within a project if you need to.

resources.view.helperPath.Akrabat_View_Helper_ = "Akrabat/View/Helper/"
resources.view.helperPath.App_View_Helper_ = "App/View/Helper/"

Plugin helper path order is one of the more frustrating things about the plugin loaders used within Zend Framework and affects form elements, validators, filters and view helpers to name but a few. So if you discover that a view helper isn't being called when you think it ought to be, then check that it's not being overridden further down the chain. You can of course use this to your advantage and override a default Zend Framework view helper by simply creating an App version with the same name.

The OpenStackManager extension is a web interface for OpenStack, and a manager for a fully integrated test and development network being written primarily for Wikimedia Foundation use.

This is a features and bugfix release of the OpenStackManager extension. There have been extensive changes since the last release. I’ll try my best in the future to do releases more often to avoid having changelogs this long. The following has changed:

• Added support for managing sudo policies. Currently supports adding, modifying and deleting sudo policies with sudouser, sudohost, sudocommand, and sudooptions attributes. Requies wiki admin privileges to use the special page.
• Changed host addition behavior to add project to the host entry’s puppet variables by default; the variable added is:
  • instanceproject
• Added support for adding/modifying/deleting instance information in Nova_Resource: wiki pages when creating/configuring/deleting instances. This can be disabled using the $wgOpenStackManagerCreateResourcePages configuration option.
• Added support for creating/deleting/attaching/detaching volumes.
• Changed the display of all tables to use the sortable CSS class.
• Changed name of the VM namespace to instead be Nova Resources. All Nova resources will be placed in this namespace.
• Various MediaWiki 1.17 compatibility changes.
• Added a default shell config option ($wgOpenStackManagerLDAPDefaultShell) so that a shell can be added to user entries.
• Changed behavior of host addition to add a location attribute on instance DNS entries.
• Code cleanup for localization.
• Fixed a number of spots the extension was throwing warnings.
• Changed Special:NovaDomain to check for cloudadmins membership if roles are intersecting, as otherwise netadmins for specific projects could also manage DNS domains.
• Fixed role check bug in Special:Domain, so that cloudadmins/netadmins can manage domains.
• Fixed various bugs in project/role membership management that would cause addition removal to fail.
• Fixed a bug in public host management when an associated domain was removed that was also named for the dc attribute.
• Changed logic of which images will be shown for image creation: only show images that are public, are available, are of type machine, and have an image name.
• Changed behavior of create action for Special:NovaInstance to show image names instead of image ids.
• Added an upstart that can be used to enable consoles in instances.
• Added support for the create action in Special:NovaInstance to list detailed information about instance types.
• Added a global config option for the Nova EC2 admin API’s endpoint ($wgOpenStackManagerNovaAdminResourcePrefix)
• Added support for project role members to add/remove other project role members.
• Added a utility upstart that can be used to do some puppet configuration changes before it starts on initial boot. Should be removed from the instance during first puppet run.
• Added support for getting console output from instances.
• Made the display of actions on all special pages consistent.
• Added script and localization messages for sending notification emails when instances are fully created. Can be used in conjunction with puppet variables.
• Added default puppet variables for all instances created; adding the following variables, which represent the instance creator’s email address, (wiki) user name, and interface language:
  • instancecreator_email
  • instancecreator_username
  • instancecreator_lang
• Changed scripts and upstarts in clouddata to use named values in the arrays, where the named values are the name of the file that will be created on the instance.
• Changed behavior of instance termination to ensure any associated addresses are disassociated first.
• Added validation checks for nova resource creation and modification.
• Changed creation of host entries to use instanceid instead of hostname for the dn, and changed creation to only use associated domain, not cname, since cnamerecords don’t work when created that way.
• Added a script to automatically sign puppet certificate requests for hosts that exist in LDAP.
• Upgraded AWS SDK for PHP to 1.2.6 “Ifrit”.

Want to help out with the extension? I could use help testing it; if you find a bug, please report it. If you’d like me to add a feature, add a bug report. If you’d like to help by contributing code, squash a bug. If you’d like to work in a pre-configured environment, contact me, and I can give you access to my testing environment.

Share:

Related posts:

1. Announcing OpenStackManager extension for MediaWiki
2. OpenStackManager version 1.1 released
3. Building a test and development infrastructure using OpenStack

Chris Shiflett has asked the question, why don't people blog anymore? In this age of real-time streams and dead-simple status updates, blogs often feel like the uncared-for step-child or a website; indeed, many folks are trading their blogs for pages that simply track their various lifestreams (tweets, facebook status, flickr images, and more).

While this sort of thing is trendy and interesting, it also sucks:

Introduction

The previous post in this series covered the next generation of Apache Hadoop MapReduce in a broad sense, particularly its motivation, high-level architecture, goals, requirements, and aspects of its implementation.

In the second post in a series unpacking details of the implementation, we’d like to present the protocol for resource allocation and scheduling that drives application execution on a Next Generation Apache Hadoop MapReduce cluster.

Background

Apache Hadoop must scale reliably and transparently to handle the load of a modern, production cluster on commodity hardware. One of the most painful bottlenecks in the MapReduce framework has been the JobTracker, the daemon responsible not only for tracking and managing machine resources across the cluster, but also for enforcing the execution semantics for all the queued and running MapReduce jobs. The fundamental shift we hope to effect takes these two complex and interrelated concepts and re-factors them into separate entities.

In the proposed new architecture a global ResourceManager (RM) tracks machine availability and scheduling invariants while a per-application ApplicationMaster (AM) runs inside the cluster and tracks the program semantics for a given job. An application is either a single MapReduce job as the JobTracker supports today, it could be a directed, acyclic graph (DAG) of MapReduce jobs, or it could be a new framework. Each machine in the cluster runs a per-node daemon, the NodeManager (NM), responsible for enforcing and reporting the resource allocations made by the RM and monitoring the lifecycle of processes spawned on behalf of an application. Each process started by the NM is conceptually a container, or a bundle of resources allocated by the RM.

Scheduler

The ResourceManager is the central authority that arbitrates resources among various competing applications in the cluster.

The ResourceManager has two main components:

• Scheduler
• ApplicationsManager

The Scheduler is responsible for allocating resources to the various running applications subject to familiar constraints of capacities, queues etc. The Scheduler is pure scheduler in the sense that it performs no monitoring or tracking of status for the application. Also, it offers no guarantees on restarting failed containers either due to application failure or hardware failures.

The Scheduler performs its scheduling function based the resource requirements of the applications. Applications express their resource requirements using the abstract notion of resource request, which incorporates elements such as memory, cpu, disk, network etc.

The ApplicationsManager is responsible for accepting job-submissions, negotiating the first container for executing the application-specific ApplicationMaster and provides the service for restarting the ApplicationMaster container on failure.

This post focuses on providing details about the Scheduler, the resource-model, the scheduling algorithm etc.

Resource Model

MapReduce NextGen supports a very generic resource model that allows cluster resources such as CPU, memory, disk and network bandwidth, local-storage etc. to be efficiently scheduled among applications.

The result of a resource request is a container. A container is a conceptual entity that grants an application the privilege to use a certain amount of resources on a given machine to run a component task.

The Scheduler, in v1, models every machine in the system as composed of multiple containers (with a lower-limit on the size of memory, say 512MB or 1 GB).

The ApplicationMaster request a container along with the resources (amount of memory, cpu capability, etc) it needs. It can request multiple type of containers, each with its different resource needs. In addition it can optionally specify location preferences such as specific machine or rack; this allows for first class support of the MapReduce model of performing the computation close to the data being processed.

Thus, unlike current implementation of Hadoop Map-Reduce, the cluster is not artificially segregated into map and reduce slots. Every container is fungible, yielding huge benefits for cluster utilization. Currently, a well-known problem with Hadoop MapReduce is that jobs are bottle-necked on reduce slots and the lack of fungible resources is a severe limiting factor.

Resource Request

As described in the previous section, the ApplicationMaster can ask for resources with specific capabilities (memory, cpu etc.) on specific machines or racks. The ApplicationMaster can ask also ask for different kinds of resources for differing needs (e.g. maps and reduces have different resource necessities in the MapReduce paradigm). Among various resource requests, the ApplicationMaster can use request priorities to influence ordering of containers allocated by the Scheduler (e.g. run maps before reduces in MapReduce). All outstanding resource requests are serviced strictly in priority order, as defined by the application.

The ApplicationMaster can ask for any number of containers of a given request type (<priority, capability> ).

Thus, the resource request understood by the Scheduler is of the form:

<priority, (hostname/rackname/*), capability, #containers>

Resource Negotiation

The ApplicationMaster is responsible for computing the resource requirements of the application. In a MR application, the input-splits express its locality preferences. The MR ApplicationMaster translates them into resource requests for the Scheduler.

The expected mode of operation is for applications to present all of their resource requests upfront, on start-up, to the extent feasible. Of course, the protocol allows for applications to present updates to the resource requests via deltas.

As described above the resource request is of the form: <priority, (hostname/rackname/*), capability, #containers>.

Thus, for MapReduce applications, the MapReduce ApplicationMaster takes the input-splits and generates resource requests by inverting the split to host mappings, then annotating each request with priorities (map v/s reduce containers) and number of containers required on each host. The resource requests are also aggregated by racks and then by the special any (*) for all containers. All resource requests are subject to change via the delta protocol.

An illustrative example:

Consider a MapReduce job with 2 maps, each requiring 1G of RAM and 1 reduce requiring 2G of RAM.

The input-splits are:

The corresponding resource requests from the ApplicationMaster to the Scheduler:

Notice how the containers field is aggregated across the different hosts, racks and any (*) for both maps and reduces.

The main advantage of the proposed model is that it is extremely compact in terms of the amount of state necessary, per application, on the ResourceManager for scheduling and the amount of information passed around between the ApplicationMaster & ResourceManager. This is crucial for scaling the ResourceManager. The amount of information, per application, in this model is always O(cluster size), whereas in the current Hadoop Map-Reduce JobTracker it is O(number of tasks) which could run into hundreds of thousands of tasks. For large jobs it is sufficient to ask for containers only on racks and not specific hosts since the ApplicationMaster can use them appropriately since each rack has many appropriate resources (i.e. input splits for MapReduce applications).

Scheduling

The Scheduler will try to match appropriate machines for the application; it can also provide resources on the same rack or a different rack if the specific machine is unavailable. The ApplicationMaster might, at times, due to vagaries of busy-ness of the cluster, receive resources that are not the most appropriate or receive them too late; it can then reject them by returning them to the Scheduler without being charged for them.

The Scheduler is aware of the network topology of the cluster and uses the racks and any (*) fields as gating factors to ensure that ApplicationMasters do not get inappropriate resources without resorting to needing very detailed task-level information.

For e.g. in the above example the Scheduler doesn’t need to know that map0 can run on <r11/h1001, r22/h2121, r31/h3118>.

If a container is available on r22/h2121, the Scheduler allocates it to the application and invalidates requests on h2121, r22 and * by updating the resource request list as follows:

The ApplicationMaster can then, optionally, invalidate requests for h1001, h3118, r11 and r31 after it decides to run map0 on h2121.

Now, if a container is available on r11/h1010, the Scheduler allocates it to the application and invalidates requests on h1010, r11 and * as follows:

The ApplicationMaster can then, optionally, invalidate requests for h2121, h4123, r22 and r45 after it decides to run map1 on h1010.

The important detail about the scheduling policy is that:

• When the number of required containers reaches 0 on a host, the Scheduler will stop granting further containers on that node to a the application at that priority/capability.
• When the number of required containers reaches 0 on a rack, the Scheduler will stop granting further containers to any node on that rack to the application at that priority/capability.
• When * reaches 0 for an application, the Scheduler stops granting further containers on any node for the application at that priority/capability.

The Scheduler aggregates resource requests from all the running applications to build a global plan to allocate resources. The Scheduler then allocates resources based on application-specific constraints such as appropriate machines and global constraints such as capacities of the application, queue, user etc.

The CapacityScheduler plug-in uses familiar notions of Capacity and Capacity Guarantees as the policy for arbitrating resources among competing applications.

The scheduling algorithm is straightforward:

• Pick the most under-served queue in the system.
• Pick the highest priority application in that queue.
• Serve the application's resource asks appropriately.

The FairScheduler plug-in uses fairness the main policy for arbitrating resources among active applications.

Note: A rogue ApplicationMaster could, in any scenario, request more containers than necessary - the Scheduler uses application-limits, user-limits, queue-limits etc. to protect the cluster from abuse.

Scheduler API

There is a single API between the Scheduler and the ApplicationMaster:

(List <Container> newContainers, List <ContainerStatus> containerStatuses) allocate (List <ResourceRequest> ask, List<Container> release)

The AM ask for specific resources via a list of ResourceRequests (ask) and releases unnecessary Containers which were allocated by the Scheduler.

The response contains a list of newly allocated Containers and the statuses of application-specific Containers that completed since the previous interaction between the AM and the RM.

Resource Monitoring

The Scheduler receives periodic information about the resource usages on allocated resources from the NodeManagers. The Scheduler also makes available status of completed Containers to the appropriate ApplicationMaster.

In future, obvious enhancements to the Scheduler include using the per-container resource usage information provided by the NodeManagers to do over-committing of resources etc.

Conclusions

We believe the proposed ideas for the Scheduler in the Next Generation of Hadoop Map-Reduce are both novel and crucial for the fine-grained scheduling required to meet our stated goals for Apache Hadoop MapReduce. They allow the Scheduler to scale to very large clusters and still provide very fine-grained and efficient scheduling for exploiting data-locality, performance and reliability.

We expect and encourage other cluster schedulers/managers to incorporate our ideas and we look forward to learning from their experiences too.

If you look at shell scripts which were written to run on various implementations of the shell, they are much different than shell scripts written just for a specific type of host. Say a modern Linux system.

As such, I am wondering, is there any circumstance where this:

var=`<file`

does not work? My eyes burn every time I see this:

var=`cat file`

But they really burned one time when I saw this:

if [[ "`cat file`" = "" ]]
then
echo file is empty
fi

Image via Wikipedia

A long time ago someone coined the phrase Cross-Site Scripting and it became popularly abbreviated as XSS (the X was suggested to avoid confusion with CSS). XSS is a family of vulnerabilities that allows an attacker to inject arbitrary content, often Javascript, into the output (not necessarily HTML) viewed by users of a web application. These injections tend to do bad things. It is a plague upon web applications and not just those written in PHP.

Defeating Cross-Site Scripting

The solutions which prevent and defend against XSS in HTML are commonly known:

If you inject data into HTML (e.g. a template), and cannot be 110% sure it never crossed paths with a malicious user, you escape it. In PHP this means passing such data through a function like PHP’s htmlspecialchars(), always remembering to pass the character encoding of your output as the third parameter. An alternative exists for cases where you do not determine the HTML markup of output, for example, when aggregating content from RSS or Atom feeds, from web service API responses, from HTML emails, from user comments where HTML is allowed, or even from the output of HTML transformers such as libraries which translate BBCode, Markdown or some other intermediate format into HTML. These alternatives are usually called HTML Sanitisers or XSS Cleaners.

The first case is simple, easy to execute, and very difficult to spoof. Its main problem is that it requires foreknowledge of the character encoding of the output since HTML special characters may differ between encodings. A simple example of this encoding difference is found by comparing UTF-8 and UTF-7. Whereas UTF-8 is US ASCII compatible, UTF-7 is not. Escaping UTF-7 markup as if it were UTF-8 would cause the escaping mechanism to fail in detecting the angular brackets that HTML tags are enclosed by since they occupy different points in UTF-7. Obviously, such a failure is a potential disaster – especially if your output supports a UTF-7 encoding, or if it never specifies a character encoding at all either via a header or a HTML meta tag since this may allow some browsers (cough…Internet Explorer) to guess the wrong encoding to use.

The second case is complex. There are no easy solutions or single paragraph pearls of wisdom you can rely on. Instead of a simple function, you need a library of code capable of parsing HTML and handling character encoding differences. Then you need a friendly API so programmers aren’t buried in the complexity of the task, a whitelist and whitelist limiter to defend against misconfiguration, knowledge of every HTML standard since the dawn of time and up to the minute advice on emerging HTML quirks across all browsers (even the ones you think are no longer used). After that, you are not done. You’re only beginning. You’re going to need a parser and lexer, a character encoding handler, a HTML tidier so you don’t break stuff, a possy of XSS wizards to tell you when you’re failing, and enough unit tests that if Sebastian Bergmann knew what you were doing, even he would jump out the nearest window screaming. Even then – you probably missed something.

The second case is complex. Horribly complex. Yes, I’m repeating myself.

PHP developers have responded to this complex second case by creating HTML Sanitisers/XSS Cleaners. They poured over the theory behind Cross-Site Scripting like ants. Yes, their combined intellect was activated, bringing the force of the PHP language to bear in ensuring all programmers are sufficiently protected. One unusually bright spark, Edward Z. Yang, succeeded and HTMLPurifier was born. An open source library which quashed XSS with the kind of force usually reserved for inter-galactic warfare.

The Truth Is Out There…If You Bother Looking

But HTMLPurifer was not alone. There were others.

People began to realise even before HTMLPurifer turned up that XSS used an interesting tactic. It usually added a string into an even bigger string. It was a realisation of epic proportions because it emerged that PHP had support for removing strings from even bigger strings. This marvellous invention was known as Perl Regular Expressions (or Regex for short). Regex based HTML sanitisers were a boon to the PHP community. They offered a fast means of stripping HTML of naughty XSS injections, leaving behind safe and pristine HTML suitable for output. These solutions thrived and became an essential tool in every PHP programmer’s toolbox. To this day, they are universally popular and can be found in applications, libraries and frameworks used by thousands of PHP programmers every day.

Regex based HTML Sanitisers do have one teeny tiny little wrinkle. They don’t work.

I’ve just spent a year researching these…things…on and off. First, I examined all the prominant examples of standalone HTML sanitisers. They all had vulnerabilities and I previously blogged about several. Next, I tried solutions integrated in applications and a few suggested alternatives. They all had vulnerabilities. Next…well, I can’t disclose those for another five weeks. But, they all had vulnerabilities too. There have been no exceptions.

I feel it’s important to clarify my approach since I simply cannot enumerate all the vulnerabilities that do exist which, I acknowledge, is hardly the basis of a conclusive statement of fact. My research relied on locating 2-5 vulnerabilities per solution based on a set of five common XSS exploit categories (character encoding, CSS parsing, HTML quirks, tag detection and javascript obscuration). Its purpose was not to locate every single vulnerability – that’s effectively an impossible task without a rigorous testing suite and a mountain of time I can’t afford. Instead, I assessed each solution on the basis of its approach to sanitising HTML and used perceived flaws to extrapolate a set of likely vulnerabilities. This rough approach had no trouble yielding successful vulnerabilities which were selected from a publicly available set of documented browser quirks, i.e. widely known vectors for XSS. Based on the rate of success, I was then able to conclude whether a solution was fundamentally flawed, i.e. whether it was highly probable that additional XSS vulnerabilities existed beyond the handful I elected to test for. All of the Regex based HTML Sanitisers I looked at inevitably fell into the category of being fundamentally flawed.

I also feel it’s important to address the behaviour of those who maintain Regex based solutions. Most of those who received my reports were more than happy to fix the reported vulnerabilities. A small number drifted down the path of ignoring my reports, downplaying my reports or being openly hostile. Even though all were informed that I was offering examples and not a comprehensive list of vulnerabilities, nobody took the hint that this would require an in-depth review of their source code. As far as I am aware, all the solutions examined remain publicly available for download. All of them still call themselves HTML Sanitisers or Cleaners.

When Will We Ever Learn?

Right about this time, I would normally sign off and let you go. Not today. After my umpteenth report (this one containing eight examples of vulnerabilities), I’ve decided to end my research. I don’t regret persuing it because, frankly, nobody else has bothered doing something similar. It needed to be done. It should have been done years ago but apparently nobody cared enough to dig around for the truth before now (myself included) or their study/articles never saw widespread attention. What started out as a quick expedition to serve my own curiousity about whether PHP needed yet another HTML sanitiser has ended up uncovering what I now suspect is a massive problem. A problem which has been ignored, sidelined and is subject to a continuing campaign of deliberate or unintentional misinformation which conceals a simple truth.

Regex based HTML sanitisation is an utter failure. A disgrace. A terrifying menace to PHP security which must, at all costs, be eradicated. Now. I don’t care if you swear blind your Regex solution works – it doesn’t and you’re either insane, ignorant or a fucking moron to believe otherwise. Your shit is useless. If you keep offering it for download and pushing its “benefits” on users, you are perpetuating a practice of inflicting security vulnerabilities on those users who trust you. I don’t believe that you are all doing this deliberately but you are going to have start asking yourself some tough questions about what exactly you’re doing to the PHP community because of your ignorance. What is PHP? That joke of a horribly insecure language that sucks compared to everything else or a glowing example of best security practice? Pick one.

If you are a user of such a Regex based HTML Sanitiser, replace it with HTMLPurifier. HTMLPurifier is a library that is reliable, peer reviewed, uses a sound strategy and is phenonomally open about the infrequent vulnerabilities reported against it. There is no such thing as a perfectly secure solution for XSS filtering of HTML but HTMLPurifier gives that statement a run for its money. If HTMLPurifier is honestly too slow for what you need to do, then start hoping you have a dead simple use case which might scrape by with a very restrictive solution. There is no alternative choice.

In approximately five to six weeks (grace period for fixes), I’ll summarise the final set of vulnerabilities I have sent out about HTML Sanitisers.

This post was authored by Andrew Hutchings, Drizzle Developer at Rackspace. Click here to read more about him.

This week, we launched the first GA release of the Drizzle project, a microkernel fork of MySQL aimed at the web and cloud.  I want to go over what Drizzle isn’t, what it is and what some of the new features are in this release.

Drizzle isn’t…

Before I go into what Drizzle is, first I should start by saying what Drizzle isn’t.  Although it started life 3 years ago as a fork of the now defunct MySQL 6.0 codebase, it isn’t a drop-in replacement for MySQL, like many of the other forks are.  The development of Drizzle took the radical step of breaking backwards compatibility so that it can meet the project goals.  That is not to say you cannot migrate from MySQL to Drizzle; there are even tools provided to help you do this and Drizzle speaks the MySQL protocol.  But our data types, syntax and general execution behavior is different to MySQL.

As for what Drizzle is…

Drizzle is a lightweight database designed for the web and cloud.  To achieve this, we have moved much of the code into plugins.  You can pick and choose which plugins you want to fit your needs.  We have also put emphasis on features that the web and cloud will use, such as only supporting UTF8 and catalogs support.  The codebase has been completely re-factored (and a lot of code has been re-written) so that any developer can pick it up and hack on it.  This has helped us get some great contributions from people such as Google Summer of Code students.

If you wish to learn about Drizzle or wish to help contribute docs/code/anything we have several talks and an expo hall stand at the 2011 O’Reilly MySQL Conference and Expo as well as Drizzle Developer Day the day after the conference (Friday 15th April at the Santa Clara Convention Center).  The developers are available on the #drizzle channel on Freenode as well as the Drizzle mailing list.

Drizzle has a very open development model.  There are Drizzle developers employed by several companies as well as general community users who contribute code.  All contributions are treated equally and are often in the trunk within hours.  All code going into trunk must pass a very strict and extensive regression test that tests multiple platforms not just for errors but also for any performance issues a patch could cause.

Features

So what can you expect from Drizzle7 and future releases?  Drizzle7, our first GA, is our stable release and has far too many features to list here, but a few notable ones are:

* Libdrizzle – A BSD licensed, non-blocking, MySQL compatible client library.
* New data types – Including a TIMESTAMP type with microsecond precision and a UUID type.
* Many plugins – Over 80 included which cover storage engines to functions to authentication.
* A new native replication solution – Using Google Protocol Buffers so applications can be written to read our log very easily in almost any language.
* MySQL to Drizzle migration – Our drizzledump tool can migrate from a MySQL server to a Drizzle server on-the-fly.

This also has the framework on which we can build the future on, there are a few things planned that are not in Drizzle yet, primarily catalog support and native sharding.  We plan to have a GA every 6 months from now on and with the framework there, we can build on these very quickly.

A few other notes

Drizzle7 is already compatible with SQLAlchemy used in Openstack Nova.  It can be used with Rackspace Cloud Servers to be installed on anyone’s Linux virtual server. There are also several other cool things planned in the near future so stay tuned for more updates.

The Rackspace Cloud Files team is pleased to announce our support for custom container metadata.  Metadata can be attached to objects and now containers in order to categorize them and mark them with any sort of data.

If you’re new to Cloud Files, let me recap a few basic definitions.   An object is the basic storage entity and any optional metadata that represents the files you store in the Cloud Files system. When you upload data to Cloud Files, the data is stored and consists of a location (container), the object’s name, and any metadata consisting of key/value pairs.

The great news is that you have more options available now that we have launched custom container metadata.  For example, if you are using Cloud Files as the storage target for system backups, you might want to store the time of the backup as the container metadata or the origin of that backup.  You could even store whether that backup was full or incremental, which would help with restores later.

The uses for custom container metadata are simple but often key to implementation strategies for Cloud storage. Container metadata cannot be searched against or listed at this time, but will show up in any HEAD or GET request for a container.

Cloud Files users can set their container metadata by using the Cloud Files API.  Simply do a POST on the container with X-Container-Meta-<user-specified-name> headers.

If you have any questions or comments about this feature, we’re happy to hear them!

Image by bertboerland via Flickr

Dear Gareth Heyes,

I thank you for your response that claims Regex HTML Sanitisation can work.

However, I should clarify that my article, Regex HTML Sanitisation: Off With Its Head!, was written in the context of using Perl regular expressions in PHP to both parse and filter HTML. Your challenge to test HTMLReg was unusual since HTMLReg is written in Javascript, operates as a client side library, and utilises the browser DOM to bypass HTML parsing with regular expressions.

As such, HTMLReg and your article title falls outside the context of my original article. I do, however, applaud the concept of using the browser DOM. While I cannot comment on the efficacy of client side filtering for cross-site scripting (XSS), the use of a DOM is a reliable strategy to bypass parsing problems. A similar approach accounts for the success of HTMLPurifier. Obviously, I do not begrudge some minimal use of regular expressions on pre-parsed normalised input.

This did, however, prompt me to ponder whether such an inapplicable challenge appearing on Planet-PHP undermines my argument anyway by its mere existence and blunt title in a world populated by A.D.D. sufferers. I believed it might and so I found myself determined to crack your Javascript library over a cup of coffee and a biscuit.

The result of this quick examination cannot be publicly reported here as this would be poor reporting practice. Therefore, I will report the resulting security vulnerability by email. You now have six weeks from today’s date in which to release a fixed version of HTMLReg and publicly disclose this vulnerability. I trust you will ensure that all similar or related potential vulnerabilities are also fixed. It would also, optionally, be interesting to see a blog post on the effectiveness of a client side Javascript filter.

A while ago I needed to ask a user for their date of birth on a Zend_Form. The design showed three separate select elements to do this:

A little bit of googling found this site http://codecaine.co.za/posts/compound-elements-with-zend-form which has not unfortunately disappeared, so the code in this article owes a lot of the author of that article.

It turns out to be remarkably simple to create a single Zend Form element that is rendered as multiple form elements. We create an element object and a view helper object and we're done. Usage then looks like:

<?php

class Application_Form_Details extends Zend_Form
{
    public function init()
    {
        $this->addPrefixPath('App_Form', 'App/Form/');

        // other elements before 
        
        $this->addElement('date', 'date_of_birth', array(
            'label' => 'Date of birth:'
        ));
        
        // other elements after

        $this->addElement('submit', 'Go');
    }
}

Obviously, this form lives in application/forms/Detail.php and is rendered as usual in a view script. In our form definition, we have added an element called 'date' and with the addition of the addPrefixPath call have told the form that in addition to using the standard Zend Framework form elements, also look in library/App/Form. (Incidentally, we can also now override any supplied form element by simply dropping a replacement into the libraryApp/Form folder.)

The date form element lives in library/App/Form/Element/Date.php as Zend_Form knows to look in a subfolder for App/Form called Elements for any element objects and will look in the Decorator/ sub folder for decorator objects.

The Date element looks like this:

// Based on code from
// http://codecaine.co.za/posts/compound-elements-with-zend-form
class App_Form_Element_Date extends Zend_Form_Element_Xhtml
{
    public $helper = 'formDate';

    public function isValid ($value, $context = null)
    {
        if (is_array($value)) {
            $value = $value['year'] . '-' .
                $value['month'] . '-' .
                $value['day'];
            
            if($value == '--') {
                $value = null;
            }
        }

        return parent::isValid($value, $context);
    }

    public function getValue()
    {
        if(is_array($this->_value)) {
            $value = $this->_value['year'] . '-' .
                $this->_value['month'] . '-' .
                $this->_value['day'];

            if($value == '--') {
                $value = null;
            }
            $this->setValue($value);
        }

        return parent::getValue();
    }

}

There's quote a lot going on here, but it should be fairly clear. Firstly we specify the name of the view helper to use when rendering this element to be formDate which we will write. We know this element is going to consist of three select boxes and so these will end up being an array of posted data for day, month and year. As a result, we need to override isValid() to turn our array back into a string and then call up to the parent's isValid() in order to do the actual validation required. We also need to override getValue() in the same way to ensure that it is also a string. Again we call up to the parent's getValue() as that does filtering.

That's all there is to the element itself, so now we turn out attention to the view helper that will render the element. This lives in library/App/View/Helpers/FormDate.php and as per my view helpers post, we need to tell the view about that folder via application.ini:

autoloadernamespaces[] = "App_"
resources.view.helperPath.App_View_Helper = "App/View/Helper"

The formDate view helper code looks like this:

<?php

// based on code from
// http://codecaine.co.za/posts/compound-elements-with-zend-form

class App_View_Helper_FormDate extends Zend_View_Helper_FormElement
{
    public function formDate ($name, $value = null, $attribs = null)
    {
        // separate value into day, month and year
        $day = '';
        $month = '';
        $year = '';
        if (is_array($value)) {
            $day = $value['day'];
            $month = $value['month'];
            $year = $value['year'];
        } elseif (strtotime($value)) {
            list($year, $month, $day) = explode('-', date('Y-m-d', strtotime($value)));
        }

        // build select options
        $dayAttribs = isset($attribs['dayAttribs']) ? $attribs['dayAttribs'] : array();
        $monthAttribs = isset($attribs['monthAttribs']) ? $attribs['monthAttribs'] : array();
        $yearAttribs = isset($attribs['yearAttribs']) ? $attribs['yearAttribs'] : array();

        $dayMultiOptions = array('' => '');
        for ($i = 1; $i < 32; $i ++)
        {
            $index = str_pad($i, 2, '0', STR_PAD_LEFT);
            $dayMultiOptions[$index] = str_pad($i, 2, '0', STR_PAD_LEFT);
        }
        $monthMultiOptions = array('' => '');
        for ($i = 1; $i < 13; $i ++)
        {
            $index = str_pad($i, 2, '0', STR_PAD_LEFT);
            $monthMultiOptions[$index] = date('F', mktime(null, null, null, $i, 01));
        }

        $startYear = date('Y');
        if (isset($attribs['startYear'])) {
            $startYear = $attribs['startYear'];
            unset($attribs['startYear']);
        }

        $stopYear = $startYear + 10;
        if (isset($attribs['stopYear'])) {
            $stopYear = $attribs['stopYear'];
            unset($attribs['stopYear']);
        }

        $yearMultiOptions = array('' => '');

        if ($stopYear < $startYear) {
            for ($i = $startYear; $i >= $stopYear; $i--) {
                $yearMultiOptions[$i] = $i;
            }
        } else {
            for ($i = $startYear; $i <= $stopYear; $i++) {
                $yearMultiOptions[$i] = $i;
            }
        }

        // return the 3 selects separated by &nbsp;
        return
            $this->view->formSelect(
                $name . '[day]',
                $day,
                $dayAttribs,
                $dayMultiOptions) . '&nbsp;' .
            $this->view->formSelect(
                $name . '[month]',
                $month,
                $monthAttribs,
                $monthMultiOptions) . '&nbsp;' .
            $this->view->formSelect(
                $name . '[year]',
                $year,
                $yearAttribs,
                $yearMultiOptions
            );
    }
}

Again, there's a fair amount of code, but I expect that it's pretty self-explanatory. Essentially, we have a lot of set up in order to render three select boxes; one for the day, month and year. I decided to use month names but it's easy enough to change to numbers. In terms of configuration, you need to be able to specify the start and stop year numbers. These are then passed in as options when calling addElement.

That's about it. Two separate files is all you need to create a Zend_Form element object that is implemented via compound elements. It also follows that you can do the same for any other conceptual element that you want to be rendered as multiple elements in the page.

Today we’re pleased to announce the Rackspace Cloud Domain Name Service (DNS) API Private Beta to US and UK Cloud Customers by invitation only. This service will allow customers to programmatically create and manage their DNS records.  Currently, the operations available through the API include:

•    List, add, modify, and remove domains
•    List, add, modify, and remove DNS records

Additionally, we’ve exposed several features in the API that previously required a support ticket, including:

•    Management of the default time to live (TTL) and resource record time to live (TTL) values
•    Advanced record types – TXT, NS, AAAA, DKIM, and SRV
•    Ability to configure the priority of all resource records

While this release is limited to the API only, we’re working in parallel to improve the customer DNS experience in the control panel as well.  The control panel will soon feature domain management as a top-level feature and provide a full suite of self-management features that are now being exposed through the API.

As we work through the beta development phase of this project, we’re looking for customers to evaluate and test this service. The ideal customer will have Cloud Servers in their configuration and be familiar with the Rackspace RESTful web service APIs.

If you would like to apply for early access to the Rackspace Cloud DNS API Private Beta program, please email dnsbeta@rackspace.com. Tell us about your current configuration and use case ideas on how you would use this new service.  We’ll start sending out invitations for the Private Beta next week.

We’re excited to work with our customers to finalize this new offering as the feedback received will be crucial to making this product a success.

Be sure to subscribe to our blog for future product announcements.

I've been working on a proposal for including service locators and dependency injection containers in Zend Framework 2.0, and one issue I've had is trying to explain the basic concept to developers unfamiliar with the concepts -- or with pre-conceptions that diverge from the use cases I'm proposing.

In talking with my wife about it a week or two ago, I realized that I needed an analogy she could understand; I was basically using her as my rubber duck. And it turned out to be a great idea, as it gave me some good analogies.

It reminds me that in some of my less successful job interviews, I tried to be honest about my strengths and weaknesses. I said something along the lines of "if you give me interesting technical work, I'm better than almost anyone in the world, but if you give me boring wiring work, I'm very ordinary, or maybe even worse than ordinary". I didn't get those jobs.

Job interviewing is one of those scenarios where honesty is not rewarded. Employers might give lip service to trying to find out what an employee's really like, but the fact is they are much more likely to hire someone who just says "I'm great at everything" and answers "what is your weakness" with one of those answers like "my greatest weakness is I spend too many hours at work".

It's sort of like the early phase of dating. If you are forthcoming and actually confess any of your flaws, the employer/date is like "eww yuck, if they admit that, they must have something really bad actually wrong with them". You might think it's great to get the truth out in the open right away, see if you are compatible, but all the other person sees is "candidate A has confessed no weaknesses and candidate B has said he has a fear of intimacy and might be randomly emotionally cold to me at times, and that was a really weird thing to say at a job interview".

Furthermore, it's sort of just a faux pas. It's like talking about masturbation around your parents. It's too much sharing with someone you aren't close with yet. All the people who understand the social code of how you're supposed to behave just feel really uncomfortable, like "why the fuck is this guy confessing his honest weaknesses? that is not what you're supposed to do in an interview/date". Job interviews/early dates don't really tell you much deep factual information about a person. There's an obvious code of what you're supposed to say and you just say that. It's really a test of "are you sane enough to say the things you are supposed to in this situation?".

This guest post was written by James Tauber, CEO of Eldarion, a Rackspace Cloud Tools Partner.

Eldarion was founded in early 2009 to build and host great websites using Django and to help others do the same. Our overarching mission has been to create tools to help us and our clients get from idea to realization more quickly.

The tools fall into two distinct categories: firstly, as part of the open source Pinax project, we’ve developed a range of reusable components and skeleton projectsso sites can be developed more quickly without having to rewrite the same code over and over again; secondly, we’ve developed infrastructure tools to both accelerate the provisioning of new servers and to make it easy to deploy code to those servers.

When we started the company, all of our hosting was through Cloud Servers. For production sites, we’d fire up two or more Cloud Servers depending on the specific needs for database, background tasks, caching, full-text search, etc. During development we’d use a shared server for showing a running development branch internally or to a client. We internally called our infrastructure “Gondor”.

As we grew, both in the number of sites we were working on and the size of our highly distributed team, we needed a way to provision servers more easily, set them up with the particular configuration needed for a site and then enable frequent deployment with as little effort as possible. Some of our clients had their own in-house developers who also wanted to be able to deploy their changes and we needed a way of allowing them to do so without giving them access to all of our infrastructure.

And so, Gondor began to develop from an internal set of tools into an advanced provisioning and deployment solution for Django, suitable for both short-lived development instances and multi-server production stacks. Now hosted on Rackspace Cloud Servers and Cloud Files, we are looking forward to both supporting and contributing to OpenStack.

We’re very excited about the partnership between Eldarion and Rackspace. With the world-class infrastructure of Rackspace and the Django deployment and hosting expertise of Eldarion, Django will be, even more, the web framework for perfectionists with deadlines.

Gondor is currently in beta and you can sign up at http://gondor.io