March 4, 2012

13:22

@steer2freedom @metofficeEng wasn't lying. Look out the window

Author: addersuk

13:50

My Top 3 #lastfm Artists: deadmau5 (59), ZEDD (10) & deadmau5 & Kaskade (1) #music http://t.co/9PBYJUMP

Author: addersuk | Tags: lastfm, music

March 7, 2012

12:38

Modern platforms: flash, disk, log-structured, update-in-place

Author: Interesting Blog Posts

March 8, 2012

03:30

Module specific bootstrapping in ZF2

Following on from the discussion on modules, we can hook into the event system to do module specific bootstrapping. By this, I mean, if you have some code that you want to run only if the action to be called is within this module, you can hook into the Application's dispatch event to achieve this.

As you already know, your module's init() method enables you to create a hook to the bootstrap event:

module/Simple/Module.php

<?php

namespace Simple;

use ZendModuleManager,
    ZendEventManagerEvent,
    ZendEventManagerStaticEventManager,
    ZendModuleConsumerAutoloaderProvider;

class Module implements AutoloaderProvider
{
    public function init(Manager $moduleManager)
    {
        $events = StaticEventManager::getInstance();
        $events->attach('bootstrap', 'bootstrap', array($this, 'onBootstrap'));
    }

    // Assume that getAutoloaderConfig() & getConfig() are defined here

    public function onBootstrap(Event $e)
    {
        $application = $e->getParam('application');
        $config      = $e->getParam('config');
    }
}

The $application has an Event Manager which you can then use to hook into various stages of the routing and dispatch process. So we can create a callback for the dispatch event which occurs just before the controller action is called:


    public function onBootstrap(Event $e)
    {
        $app = $e->getParam('application');
        $app->events()->attach('dispatch', array($this, 'onDispatch'), -100);
    }

This code will call our module's onDispatch() method when the dispatch event is fired.

Within our onDispatch(), we do:


    public function onDispatch($e)
    {
        $matches    = $e->getRouteMatch();
        $controller = $matches->getParam('controller');
        if (strpos($controller, __NAMESPACE__) !== 0) {
            // not a controller from this module
            return;
        }

        // Do module specific bootstrapping here
    }

This code is called after routing has occurred so we can retrieve the RouteMatch object which contains information about the controller and action that is about to be called. We test if the namespace of the controller matches the namespace of the module and if it does not, we return.

If it does, then we are about to dispatch to a controller action within our module and so we run do any specific code that we may wish to.

As an example we could, for instance, change the layout for all actions within this module by accessing the layout view model:


    public function onDispatch($e)
    {
        $matches    = $e->getRouteMatch();
        $controller = $matches->getParam('controller');
        if (strpos($controller, __NAMESPACE__) !== 0) {
            // not a controller from this module
            return;
        }

        // Do module specific bootstrapping here

        // Set the layout template for every action in this module
        $viewModel = $e->getViewModel();
        $viewModel->setTemplate('layout/simple');
    }

In this case, I have set the layout to layout/simple rather than the default of layout/layout. Of course, you could also set variables into the ViewModel for use in your layout script too.

Author: Interesting Blog Posts

15:51

Devops and Drupal, the Survey, the Results

Author: Interesting Blog Posts

March 9, 2012

15:16

PHP Constructor Best Practices And The Prototype Pattern

<#comment hash="4190b1b9810564229e59173c3c62a9d7" /><#comment hash="46717778bcf59e85eb804bea33c48aa6" />

If your knowledge of constructors ends with “the place where I put my object initialization code,” read on. While this is mostly what a constructor is, the way a developer crafts their class constructor greatly impacts the initial API of a particular class/object; which ultimately affects usability and extensibility. After all, the constructor is the first impression a particular class can make.

Constructors, in their current form, have been in PHP since 5.0.0. Previous to 5.0, PHP loosely followed the style similar to that of C++ where the name of the method matching the name of the class would act as the class constructor. PHP 5 brought us the __construct() “magic method” which greatly formalized the new object initialization routine.

Before jumping into some of the topics covered in this post, there are a few things you might want to be familiar with. First, be familiar with the SOLID principles, particularly the S (single responsibility principle), the L (Liskov substitution principle, commonly referred to as the LSP), and the D (dependency inversion principle). More to the point of the latter, review a previous post on Dependency Injection in PHP for background dependency injection specific to PHP.

The Constructor Signature

In PHP, you create a constructor by adding a method called __construct() to your class. The __construct() method is an instance method and as such, is not marked static. For all intents and purposes, consider the __construct() magic method as a special type of static object factory, one which will always return the type of the object requested via the new keyword.

class Foo {
    public function __construct() {
    }
}

$object = new Foo();

In the above code, PHP will, upon executing new Foo(), internally create a new object from scratch, execute __construct() in the Foo class, and assign this object to the variable $object. Pretty standard stuff. What’s important to know here is that before new Foo(), the object did not exist. It is this fact alone that makes this completely different from any other kind of instance method. That said, without getting into the gritty details, it is this fact alone that excuses the __construct() method from the same rules of the LSP that might apply to other instance methods.

This means that all of the following are legal:

class Foo {
    public function __construct() {
    }
}

class Bar extends Foo {
    public function __construct(ArrayObject $arrayObj, $number = 0) {
        /* do stuff with $arrayObj and $number */
    }
}

class Baz extends Bar {
    public function __construct(Bar $bar) {
        // yes, this is the proxy pattern
    }
}

The above, with E_STRICT enabled, will not produce a warning. Yet, if you renamed all of the __construct methods to anything else, they will produce a E_STRICT warning like:

Strict standards: Declaration of Bar::somemethod() should be compatible with that of Foo::somemethod()

Why is this the case? Simply put, the LSP referrers to sub-types of a particular object, and since before the __construct() method, no type exists (yet). This rules simply cannot apply to something that does not exist. For a more detailed response, go here.

What you should take away from this is that the best-practice is that each concrete object has a constructor with a signature that best represents how a consumer should fully instantiate that particular object. In some cases where inheritance is involved, “borrowing” the parents constructor is acceptable and useful. Furthermore, it is encouraged that when you subclass a particular type, that your new type should, when appropriate, have its own constructor that makes the most sense to the new subtype.

At this point, it should be noted that most other languages do not allow constructors to be marked final, be abstract, or be marked as statics (see above on the static note). Moreover, constructors should not appear in interfaces. In PHP, these rules do not apply, and are all possible. For the reasons listed above, a developer should avoid the practice of marking constructors final, making them abstract, and putting them interfaces, assuming they are trying to utilize PHP’s OO model in a SOLID way. In PHP 5.4, it is also worth knowing that by having constructors in interfaces breaks the common expectation that subtypes are capable of creating their own constructors in favor of enforcing a particular method signature.

Constructor Overloading

PHP does not have method overloading. This also applies to constructors. A class of a specific type can only have one constructor. Since this is the case, PHP developers sometimes loosen a methods signature in order to accommodate multiple use cases. This is done by removing or reducing the types enforced in the constructors signature to allow for more varied types to be passed in by the consumer.

This is an acceptable best practice when done appropriately. What does appropriately mean? What is “appropriate” is, of course, very much subjective. Generally speaking, the differences in the various signatures supported should be minimal at best, yet meaning should still communicated through the name of the parameters. For example, let’s take this constructor:

class Db {
    /**
     * @var string|array|DriverInterface $driver
     */
    public function __construct($driver) {
        if (is_string($driver)) {
            $driver = $this->createDriverFromString($driver);
        } elseif (is_array($driver)) {
            $driver = $this->createDriverFromArray($driver);
        }

        if (!$driver instanceof DriverInterface) {
            throw new Exception();
        }
    }
}

The above signature __construct($driver) technically supports 3 effective signatures:

__construct(/* string */ $driver);
__construct(/* array */ $driver);
__construct(DriverInterface $driver);

The actual signature has not changed, but it is represented all 3 effective signatures that can be further described by the PHP DocBlock.

Constructor Injection

At this point in the PHP community and in PHP-centric developer circles, it is generally accepted that injecting your dependencies is a best-practice. How developers go about injecting these dependencies is still very much debated and, in-part, up to personal and/or team preference.

There are several such methods of dependency injection: interface injection, setter injection and constructor injection to name the primary forms. For the purposes of this post, constructor injection is our primary candidate for discussion.

In short, constructor injection is a pattern of injecting all of your required dependencies into a constructor. These dependencies are usually other objects, often called services. The primary benefit of constructor injection is that after you instantiate the target object, generally, it is in the complete “ready state,” meaning that it is ready to do real work. A typical constructor signature sporting constructor injection looks like this:

class UserMapper {
}

class UserRepository {
    public function __construct(UserMapper $userMapper) {
        $this->userMapper = $userMapper;
    }
}

The above example clearly demonstrates that before a developer can use a UserRepository object, they must first inject it with a UserMapper object.

In PHP, while in recent times we’ve started favoring dependency injection (which can add some complexity to code), we have traditionally gravitated towards code that is easy write and easy to use. Practicing good dependency injection can be tedious at times and, in many cases, dependencies for objects can be stubbed by a sensible default. This practice is also known by the name of Poka-Yoke. It allows us to develop an API that supports explicit injection of dependencies while promoting ease of use in common or majority use cases. Consider the following code:

class UserMapper implements UserMapperInterface {
}

class UserRepository {
    protected $userMapper;
    public function __construct(UserMapperInterface $userMapper = null) {
        $this->userMapper = ($userMapper) ?: new UserMapper;
    }
}

While the UserRepository allows you to inject your dependency of the UserMapper, it will, if one was not provided, instantiate a sensible default UserMapper for you. The benefits are that in the most common use cases, it is a one step usage scenario (just instantiate the UserRepository). But in unit testing scenarios or scenarios where you want to inject an alternate implementation of a UserMapper, that can be achieved through the constructor.

Dynamic Class Instantiation

Generally speaking, the following code, while legal, should be used very seldom, and only when other possible instantiation patterns have been exhausted:

$obj = new $className();
if (!$obj instanceof SomeBaseType) {
    throw new InvalidTypeException();
}

Why is this a bad pattern? First, it makes the assumption up front that the constructor signature is free from any required parameters. While this is good for object types that are already known to this factory, it might not always be true of a consumers subtype of the base object in question. This patten should never be used on objects that have dependencies, or in situations where it is conceivable that a subtype might have dependencies because this takes away the possibility for a subtype to practice constructor injection.

Another problem is that instead of managing an object, or a list of objects, you are now managing a class name, or list of class names in addition to an object or list of objects. Instead, one could simply manage the objects.

If, on the other hand, you know this particular object type is no more than a value object (or similar), with no chance of it needing dependencies in subtypes, you can then cautiously use this instantiation pattern.

Prototype Pattern

So how does one create an unlimited number of objects of a particular type, with dependencies in tact, each with slight variations? Enter the prototype pattern. This is an important pattern to keep handy when you know that you’ll have objects that need to be replicated in some way and they also have service dependencies that need to be injected.

To draw a parallel, this is similar to how Javascript handles its object model. To sum prototyping up in Javascript: functions and properties are defined once per prototype rather than once per object. The new keyword instructs the engine/runtime to create a copy of the prototype and assign to a variable for further specification and interaction.

This is similar to what the Prototype Pattern does in an object-oriented inheritance model. Up front, you create a prototypical instance. This instance will have all its dependencies injected, and any shared configuration and/or values setup. Then, instead of calling new again, a factory (or the consumer) will call clone on the object (a shallow clone will be made), and a new object will be created from the original prototypical object. This newly cloned object can then be further specified, injected with the variations that make this new object unique, thus interacted with as a unique object.

Lets consider the following example involving a database connection and the Row Gateway pattern. We want to iterate a dataset from a database and during iteration, present each row as a RowGateway object. One way of handling this would be to get the array of data from the database, then during iteration, create a new RowObject from scratch injecting the database connection:

class DbAdapter {

    public function fetchAllFromTable($table) {
        return $arrayOfData;
    }

}

class RowGateway {

    public function __construct(DbAdapter $dbAdapter, $tableName, $data) {
        $this->dbAdapter = $dbAdapter;
        $this->tableName = $tableName;
        $this->data = $data;
    }

    /**
     * Both methods require access to the database adapter
     * to fulfill their duties
     */
    public function save() {}
    public function delete() {}
    public function refresh() {}
}

class UserRepository {

    public function __construct(DbAdapter $dbAdapter) {}

    public function getUsers() {
        $rows = array();
        foreach ($this->dbAdapter->fetchAllFromTable('user') as $rowData) {
            $rows[] = new RowGateway($dbAdapter, 'user', $rowData);
        }
        return $rows;
    }
}

A UserRepository will be constructed with a database adapter object. It will then query the database, returning an array of all the rows that satisfied that query. With each row of data, it will create a fresh RowObject from scratch, injecting all the dependencies, configuration and the row data.

At first glance, you might ask “what if I have a specialized version of RowGateway I want to use?” That solution can be easily handled by instead of hard-coding the RowGateway class, but by use the Dynamic Class Instantiation pattern described above:

class UserRepository {

    public function __construct(DbAdapter $dbAdapter, $rowClass = 'RowGateway') {}

    public function getUsers() {
        $rows = array();
        foreach ($this->dbAdapter->fetchAllFromTable('user') as $rowData) {
            $rowClass = $this->rowClass;
            $row = new $rowClass($dbAdapter, 'user', $rowData);
            if (!$instance of RowGateway) {
                throw new InvalidClassType();
            }
            $rows[] = $row;
        }
        return $rows;
    }
}

This partially solves the problem in that now we can now use our own specialized class for the RowGateway implementation, but this too has its own special set of limitations. First, we are incorrectly making the assumption that the constructor signature of a subtype of RowGateway is exactly the same as the base type. This means that if a subtype has additional dependencies, that class will need to do the static dance in order to locate and consume those dependencies that it needs to achieve its specialized functionality. By making this assumption of the classes constructor signature, we’re limiting the consumers ability to practice polymorphism in the subtypes that they might need to have created.

For example, if a consumer wanted to be able to have a RowGateway object that wrote data to one specific database, but refreshed its data from a different database, how might one be able to inject two different DbAdapters into a RowGateway object to achieve this end result?

The answer is to use the Prototype Pattern, and in practice (via pseudo-code), looks like this:

class DbAdapter {
    // same as before
}

class RowGateway {

    public function __construct(DbAdapter $dbAdapter, $tableName) {
        $this->dbAdapter = $dbAdapter;
        $this->tableName = $tableName;
    }

    public function initialize($data) {
        $this->data = $data;
    }

    /**
     * Both methods require access to the database adapter
     * to fulfill their duties
     */
    public function save() {}
    public function delete() {}
    public function refresh() {}

}

class UserRepository {

    public function __construct(DbAdapter $dbAdapter, RowGateway $rowGatewayPrototype = null) {
        $this->dbAdapter = $dbAdapter;
        $this->rowGatewayPrototype = ($rowGatewayPrototype) ? new RowGateway($this->dbAdapter, 'user')
    }

    public function getUsers() {
        $rows = array();
        foreach ($this->dbAdapter->fetchAllFromTable('user') as $rowData) {
            $rows[] = $row = clone $this->rowGatewayPrototype;
            $row->initialize($rowData);
        }
        return $rows;
    }

}

By using a prototypical instance as the base for all future instances, we now allow the consumer the ability to extend this base implementation using sound object-oriented/polymorphic best-practices to achieve their end result. So, assuming our above example of the read/write adapter, a consumer can write:

class ReadWriteRowGateway extends RowGateway {
    public function __construct(DbAdapter $readDbAdapter, DbAdapter $writeDbAdapter, $tableName) {
        $this->readDbAdapter = $readDbAdapter;
        parent::__construct($writeDbAdapter, $tableName);
    }

    public function refresh() {
        // utilize $this->readDbAdapter instead of $this->dbAdapter in RowGateway base implementation
    }
}

// usage:
$userRepository = new UserRepository(
    $dbAdapter,
    new ReadWriteRowGateway($readDbAdapter, $writeDbAdapter, 'user')
);
$users = $userRepository->getUsers();
$user = $users[0]; // instance of ReadWriteRowGateway with a specific row of data from the db

Parting Words

Be nice to people who want to consume and extend your code. A constructor is more than just a place for initialization code. How you craft your constructors, the patterns you use for their signatures, and how you expect to get new instances of objects greatly affects the ability of consumers to extend your code without having to jump through too many hoops in order form them to achieve their specialized use case. It is always better to fall back on SOLID object-oriented practices than to limit someones possibilities by forcing them into coding patterns that require reading in-depth documentation on how the original author expects someone to extend their code.

<#comment hash="1fe368bde34a7e5d2c68b281ea0325b9" /><#comment hash="02f3062f1d1d150258be5f5cc6b2f9e6" />

Author: Interesting Blog Posts

March 10, 2012

16:47

Sponsoring WhereCampEU 2012

Author: Interesting Blog Posts

March 11, 2012

19:24

My Top 3 #lastfm Artists: deadmau5 (52), Knife Party (10) & deadmau5 & Kaskade (1) #music http://t.co/9PBYJUMP

Author: addersuk | Tags: lastfm, music

March 12, 2012

04:55

Some ZendView examples

With the release of Beta 3 of Zend Framework, we now have a significantly refactored the ZendView component.

One of the changes made is that there is a ViewModel object that is returned from a controller which contains the variables to be used within the view script along with meta information such as the view script to render. The really nice thing about ViewModels is that they can be nested and this is how the layout composes the action view script.

However, we can do many more interesting things than this and I've put together a test application with a controller showing some of the things that can be done.

Some examples:

Change the layout in an action:


    public function differentLayoutAction()
    {
        // Use a different layout
        $this->layout('layout/different');
        
        return new ViewModel();
    }

Create another view model at the layout's level:


    public function addAnotherViewModelToLayoutAction()
    {
        // Use an alternative layout
        $layoutViewModel = $this->layout();
        $layoutViewModel->setTemplate('layout/another');

        // add an additional layout to the root view model (layout)
        $sidebar = new ViewModel();
        $sidebar->setTemplate('layout/footer_one');
        $layoutViewModel->addChild($sidebar, 'footer');

        return new ViewModel();
    }

I've created some other examples too, so I recommend that you grab the code from GitHub and have a play! The code also includes a second module, Simple, which shows how to change the layout for an entire module.

You should also read the manual!

Author: Interesting Blog Posts

16:49

A Hitchhiker’s Guide to Cross-Site Scripting (XSS) in PHP (Part 1): How Not To Use Htmlspecialchars() For Output Escaping

Nu wordt het wel heel simpel om XSS zwakheden ...

(Photo credit: bertboerland)

In recent weeks, I consulted with the second most intelligent species on the planet: Dolphins. Dolphins are renowned across the known Universe for their awesome programming skills. After all, it was they who developed such insightful works as “Evolution By Example”, “Dude! We Wrote The Laws Of Physics!”, and “How Many Humans Does It Take To Screw Up A Planet?”. The answer to the last will be published on 01/01/2013 after the experiment is shut down and sent to a landfill site assuming the Supreme Spaghetti Monster signs off on the permit.

Dolphins think we are really dumb and theorise that this level of stupidity has one obvious cause: self-imposed ignorance. We are, after all, only the third most intelligent species on Earth and appear to have aspirations to lower our IQ just a bit more.

While it’s no harm poking fun at ourselves, in PHP we do have a serious problem. Cross-Site Scripting (XSS) remains one of the most significant classes of security problems afflicting PHP applications. Despite years of education, community awareness and the development of frameworks which can offer a huge boost in consistent practices – things are not getting any better.

So, I finally figured out what the core problem is: PHP programmers are completely clueless about XSS. It’s that simple. Instead of going out and studying the topic, we blindly follow some preferred herd of people offering advice with heartfelt conviction despite the fact that they are probably just as ignorant as the rest of us. Does that sound like the behaviour of something which allegedly evolved into an intelligent species? The result is a mix of ignorance and stagnant knowledge that leaves PHP in an unenviable position beset by wrongheaded zealots.

To get the ball rolling, this two-part article series is a tour of how NOT to use the htmlspecialchars() function that is typically pressed ganged into service as PHP’s universal output escaper. By offering an example based guide, I hope it will illustrate just how many ways a prospective attacker using XSS can exploit this function’s misuse to pull off a successful attack. The examples were written for PHP 5.3, so 5.4 users may need to imagine they still have 5.3 installed and/or lodge an official complaint with somebody who looks like they keep a complaints box handy (your local fast food restaurant is a good start).

This example led approach has another motive. Simple examples can be translated into unit tests. Ideally, many of the current crop of frameworks can use this article as a guide to what their unit tests should be looking for. This also makes it far easier for everyday programmers to consume the article and run around the place, drunk with ungodly power, identifying issues in the libraries, frameworks and other projects that they rely on.

To help us on the path of enlightenment before it’s too late (I’d lodge an appeal with the Supreme Spaghetti Monster but apparently the Mayans already tried and failed), I also invite other PHP programmers to blog about a security topic over the next month or two. Give programmers one last chance to get it right before the Planet is demolished by the Vogon destructor fleet. Just pick a topic that drives you up the walls in defiance of gravity and spend an hour writing something useful and (optionally) expletive filled. Every little bit helps.

What Is Htmlspecialchars()?

According to many programmers from Earth, htmlspecialchars() is a function used to escape output to prevent XSS. This is however a completely wrong definition. The function was actually co-opted by programmers to combat XSS because it was either that or create slow userland functions for which the internals developers might get around to creating, when the full moon coincided with the right planetary alignment in another 314 years, a speedier C alternative to. The actual definition (along with a half-hearted self-doubting nod to preventing XSS) is as follows:

Certain characters have special significance in HTML, and should be represented by HTML entities if they are to preserve their meanings. This function returns a string with some of these conversions made; the translations made are those most useful for everyday web programming. If you require all HTML character entities to be translated, use htmlentities() instead. This function is useful in preventing user-supplied text from containing HTML markup, such as in a message board or guest book application.

Note that this hints at, but does not explicitly use, the terms Cross-Site Scripting, XSS or even Security. Then again, it does refer to guest book applications so it was probably written in 1790 by the Dolphin who created PHP v86 and who then got around to backporting version 1.0 for Humans in the late 20th Century out of extreme pity for our reliance on CGI. No, not the let’s take an action movie and turn it into a plotless eyesore with computer generated fake stuff style CGI – though memories of both are comparably bad.

Does this make htmlspecialchars() terrible at preventing XSS? No. As part of a comprehensive well-understood strategy to prevent XSS, the function is very useful. However, in PHP it is frequently overused, misused, abused, confused and…. Darn it, ran out of rhyming words again. Suffice it to say that a good description of htmlspecialchars() is that it’s an unsuitable tool for preventing XSS that has slowly evolved into a better suited tool over the years. I keep telling myself that, at least.

The function, htmlspecialchars(), accepts four parameters. Here is its function prototype as of PHP 5.4:

string htmlspecialchars ( string $string [, int $flags = ENT_COMPAT | ENT_HTML401 [, string $encoding = 'UTF-8' [, bool $double_encode = true ]]] )

The first parameter accepts a string whose special HTML characters will be converted to HTML entities. The second accepts one or more flags which defaults to using ENT_COMPAT (does not convert single quotes to entities) but should be set to use ENT_QUOTES (does convert single quotes to entities). You can include another flag, in PHP 5.4, called ENT_SUBSTITUTE which is not a bad idea for UTF-8, i.e. ENT_QUOTES | ENT_SUBSTITUTE. You can pretend that all the other constants don’t exist. The third parameter accepts a string indicating the character encoding of the string being processed and defaults to ISO-8859-1 for PHP 5.3, and UTF-8 for PHP 5.4. Don’t ever set the fourth parameter to TRUE when escaping unless your filtering logic was written by an Über Dolphin – always keep filtering and escaping separate from each other to avoid confusing the two and then having to pointlessly argue why your way is better in defiance of all logic.

The function, if correctly configured using this super simple article for guidance, will now convert the following characters to entities: <, >, ‘, ” and &. These characters make sense to escape since they are used to construct HTML tags, delineate attribute values or reference HTML entities – none of which we want users to be able to do!

If you want some very good advice before your brain implodes from too much reading, a good way to potentially make yourself vulnerable to XSS is to not explicitly set the first two optional parameters ($flags and $encoding) to an appropriate value. In fact, if you see htmlspecialchars() missing any of those two parameters in someone’s source code, you should request that they fix it or, at the very least, curse their name and pray for the Supreme Spaghetti Monster to label them as biohazardous waste in need of emergency disposal.

Now, let’s get down to overloading your brain with information. I’m told that this part is like being sucked into the Total Perspective Vortex machine on Frogstar World B.

To Quote Or Not To Quote. How Is That A Question?

As it turns out, HTML is not simply a popular markup language, it is a popular markup language designed by a bureaucratic species of transdimensional beings seeking to drive Humanity insane by inventing the most impossible-to-secure markup language known in 172 Universes which is then interpreted by “browsers” written by Mice to test the patience of security professionals and keep the really intelligent Humans distracted from the truth of their soon-to-end existence as they search out ever more ludicrous examples of parsing weirdness. Excuse me, I held my breath writing that and need to fetch my Oxygen tank…

Consider the following example. If you want to see whether they work without copy pasting, you can clone all examples from my ominously titled xss repository on Github into a webroot somewhere to read or execute them.

Single Quoted Attributes

      
        <?php header('Content-Type: text/html; charset=UTF-8'); ?>
        
        <!DOCTYPE html>
        
        <?php
        
        $input = <<<INPUT
        
        ' onmouseover='alert(/Meow!/);
        
        INPUT;
        
        /**
        
         * NOTE: This is equivalent to using htmlspecialchars($input, ENT_COMPAT)
        
         */
        
        $output = htmlspecialchars($input);
        
        ?>
        
        <html>
        
        <head>
        
            <title>Single Quoted Attribute</title>
        
            <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
        
        </head>
        
        <body>
        
            <div>
        
                <span title='<?php echo $output ?>'>
        
                    What's that latin placeholder text again?
        
                </span>
        
            </div>
        
        </body>
        
        </html>

If you run the example from a browser and pass your mouse pointer over the text, you will get a popup saying “/Meow!/”. Granted, this is hardly the most impressive XSS ever but remember that the Javascript executed could be a lot more ingenious and damaging. The reason you see alert() used everywhere in XSS examples is to prove that Javascript was executable – a real attacker will hardly advertise his success like this.

In this case, the htmlspecialchars() function call omits the second parameter which defaults to using the ENT_COMPAT flag. With this setting, the function does not convert single quotes to entities, allowing us to inject an unescaped single quote (to close the title attribute value) and another to start a new attribute and value which will be closed by the final single quote used in the template.

We can fix this problem in one of two ways:

1. Use double quotes which will prevent user input from breaking out of the HTML attribute value context using single quotes; or

2. Set the second parameter to htmlspecialchars() to use the ENT_QUOTES flag which will escape any single quotes a user tries to inject.

The moral of the story can be made even clearer by another example. In this case we use another perfectly validating means of delineating attribute values in HTML5 – we just don’t bother using quotes at all!

Quoteless Attributes

      
        <?php header('Content-Type: text/html; charset=UTF-8'); ?>
        
        <!DOCTYPE html>
        
        <?php
        
        $input = <<<INPUT
        
        faketitle onmouseover=alert(/Meow!/);
        
        INPUT;
        
        $output = htmlspecialchars($input, ENT_QUOTES);
        
        ?>
        
        <html>
        
        <head>
        
            <title>Quoteless Attribute</title>
        
            <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
        
        </head>
        
        <body>
        
            <div>
        
                <span title=<?php echo $output ?>>
        
                    What's that latin placeholder text again?
        
                </span>
        
            </div>
        
        </body>
        
        </html>

Without quotes delineating the attribute value, any space character (including any character a browser might interpret as a space – there are a lot!) allows the user to inject new attributes and values. As from the above, converting all quotes to entities is pointless if there are no quotes to start with! Our escaping doesn’t convert spaces or other space-interpreted characters into entities at all.

By now, you should see the obvious. All HTML attribute values MUST be quoted, and preferably DOUBLE quoted, in any scenario where you suspect untrusted input will be injected into an attribute value, or where htmlspecialchars() calls do not set the second parameter to use ENT_QUOTES. Believe it or not, using single quotes or no quotes remains popular and is perfectly valid under the new HTML5 spec. Some people even celebrate this new insanity. Keep an eye on any designers who look a bit wild eyed or spend too much time smiling while staring into empty space.

Excuse Me, Sir, But Someone Ate My Quotes

One of the great mysteries in escaping output is a common myth known as the Great ASCII Delusion (GAD). Those under the influence of this delusion, besides hearing voices in their head, have arrived at a belief that many character encodings are equivalent for the purposes of escaping those characters which have a special meaning for HTML, e.g ISO-8859-1 and UTF-8. Alas, this is untrue because the Mice created something called Internet Explorer 6 – a thoroughly shameful (but still commonly used) browser which corporations across the Planet continue to insist on using because buying new computers and upgrading operating systems just to use some fancy new Microsoft Office version is seen as a waste of shareholder funds.

Internet Explorer 6 is the bad boy of the XSS world since it’s vulnerable to ridiculous exploits no decent modern browser would dare associate with. Even Netscape would probably spit on it from beyond the grave. For example, have a go with this example using IE6 and PHP 5.3. If you need a testing version of all IE browsers since IE 5.5, you can download IETester from http://www.my-debugbar.com/ietester/index_all.php and use it from Windows. Try hard, I know Windows is bad and the new Tablet makeover for Windows 8 makes you feel ill, but it’s important to see these examples in action.

Source code

      
        <?php header('Content-Type: text/html; charset=UTF-8'); ?>
        
        <!DOCTYPE html>
        
        <?php
        
        /**
        
         * You could also subsititute xC0 or any other impacted character
        
         * above ASCII number 192
        
         */
        
        $input1 = 'fakeimage'.chr(192);
        
        $input2 = <<<INPUT2
        
        onerror=alert(/Meow!/)//
        
        INPUT2;
        
        $output1 = htmlspecialchars($input1, ENT_QUOTES);
        
        $output2 = htmlspecialchars($input2, ENT_QUOTES);
        
        ?>
        
        <html>
        
        <head>
        
            <title>Swallowed Quotes</title>
        
            <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
        
        </head>
        
        <body>
        
            <div>
        
                <img src="http://example.com/images/<?php echo $output1 ?>"
        
                title="<?php echo $output2 ?>">
        
            </div>
        
        </body>
        
        </html>

With the above example, something very weird happens. Using ASCII character number 192 just before a double quote in a document being interpreted as UTF-8 results in the double quote…vanishing in IE6. Seriously, it’s there but not there. Obviously the Mice are behind it – no Human could possibly defy Physics like this!

This allows an attacker to once again break out of the HTML attribute they can inject values into. Using a coincidental opportunity to inject a second free text string nearby which a browser will concatenate to the broken out attribute value of the first, you get an effective XSS combo attack.

This IE6 quirk even bypasses the call to htmlspecialchars() which, as explained above, defaults to the ISO-8859-1 character encoding for PHP 5.3 or less. If the Great ASCII Delusion were not a fabrication of someone’s imaginative wishful thinking, this should not be possible. Not to be too harsh though, this weirdness is due primarily to a bug in IE6′s treatment of the various character encodings where you can trick the browser into thinking something like xC0 (in hex) is the start of a multi-byte character thus swallowing the next ASCII character (the double quote).

To fix the above weirdness, you must make sure that escaping is done using the same character encoding that the document is being served as. The above HTML document is identifying itself as being UTF-8 but the default htmlspecialchars() encoding is ISO-8859-1 in PHP 5.3 – there’s obviously something not agreeing there! This brings us to the absolutely perfect use (well, almost) of htmlspecialchars(), the golden rule, the Word of The Supreme Spaghetti Monster, the bringer of frustration to XSS attackers:

Always set the third parameter to htmlspecialchars(), set it correctly, and make sure your document is never served with a mismatched or invalid character encoding! Don’t expect some theoretically perfect world to magically appear – browsers are filthily efficient at doing weird things you don’t expect.

I suppose I have to mention that most versions of IE have similar issues with other character encodings such as BIG5 and Shift-JIS. You can test your IE versions using http://ha.ckers.org/weird/variable-width-encoding.cgi to see what characters can be used across different character encodings. Believe it or not, these character encodings are actually still being used and, for some strange reason, people from China and Japan do use PHP.

If you want to be completely paranoid, you can either check the input for invalid UTF-8 (Drupal and HTMLPurifier have reusable functions/classes for this), and/or run it through a conversion function which should theoretically filter out the naughty bits:

$input = mb_convert_encoding($input, 'UTF-8', 'UTF-8');

This is probably a good idea for older PHP versions pre 2010 or earlier but recent PHP versions have specifically improved htmlspecialchars() to disallow invalid characters such as the above (if you set the right character encoding!). You should be aware, though, that htmlspecialchars() may still return blank strings on certain malformed input and, since PHP 5.4, will not issue any warnings about this.

I Broke It! I Broke It!

Before you think htmlspecialchars() is getting off lightly, there is one minor quibble. We’ll keep picking on Internet Explorer 6 for the rest of this article since it’s so easy to exploit.

Source code

      
        <?php header('Content-Type: text/html; charset=UTF-8'); ?>
        
        <!DOCTYPE html>
        
        <?php
        
        $input1 = 'fakeimage'."xC0";
        
        $input2 = <<<INPUT2
        
        onerror=alert(/Meow!/)//
        
        INPUT2;
        
        /**
        
         * If you think PHP 5.4 will save you - empty strings make it guess the encoding
        
         * or use the default_charset value from php.ini. You sure everyone on the whole
        
         * planet uses UTF-8? Under 5.3 - empty strings === default encoding.
        
         */
        
        $encoding = ''; // from outside source or unvalidated variable
        
        $output1 = htmlspecialchars($input1, ENT_QUOTES, $encoding);
        
        $output2 = htmlspecialchars($input2, ENT_QUOTES, $encoding);
        
        ?>
        
        <html>
        
        <head>
        
            <title>Swallowed Quotes</title>
        
            <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
        
        </head>
        
        <body>
        
            <div>
        
                <img src="http://example.com/images/<?php echo $output1 ?>"
        
                title="<?php echo $output2 ?>">
        
            </div>
        
        </body>
        
        </html>

Setting the third $encoding parameter of htmlspecialchars() to an empty string in PHP 5.4 will set the encoding to be auto-detected, grabbed from the php.ini value of default_charset, or guessed from the current locale (in that order). Be very careful under PHP 5.4 NEVER to let this happen. Don’t leave your escaping parameters to chance.

Use empty() or strlen(), for example, to spot this issue if accepting encodings from another source or variable that might allow for empty strings. Again, this behaviour is very secure and there’s nothing wrong with it whatsoever. Oh, who am I kidding… This is the dumbest parameter behaviour ever invented. NULL means use the default encoding; blank string means play a guessing game. Even Vogon poetry pales in comparison to such nonsense. One slip and an empty parameter string can rip apart this house of cards because who knows which character encoding will be used.

Oooh, I wonder what this does under PHP 5.3… Yes, er, don’t allow blank encoding parameter strings under PHP 5.3 either. Setting an empty string in PHP 5.3 is interpreted as setting the default character encoding, i.e. ISO-8859-1, instead of triggering the expected warning about an unsupported encoding.

So, be careful kids. When setting the encoding for htmlspecialchars() do a safety check to make sure it’s not an empty string you are passing in. Keep it predictable and consistent.

There’s also one other curious behaviour when using htmlspecialchars().

Source code

      
        <?php header('Content-Type: text/html; charset=UTF-8'); ?>
        
        <!DOCTYPE html>
        
        <?php
        
        error_reporting(E_ALL);
        
        ini_set('display_errors', 1);
        
        $input1 = 'fakeimage'."xC0";
        
        $input2 = <<<INPUT2
        
        onerror=alert(/Meow!/)//
        
        INPUT2;
        
        /**
        
         * Invalid encoding makes htmlspecialchars() throw a warning but it continues
        
         * the current operation anyway using the default encoding even if the default
        
         * is an unsafe choice for the application. Don't allow invalid encodings!
        
         */
        
        $encoding = 'invalid-encoding'; // from outside source or unvalidated variable
        
        $output1 = htmlspecialchars($input1, ENT_QUOTES, $encoding);
        
        $output2 = htmlspecialchars($input2, ENT_QUOTES, $encoding);
        
        ?>

Author: Interesting Blog Posts

March 13, 2012

07:27

Pádraic Brady: A Hitchhiker’s Guide to Cross-Site Scripting (XSS) in PHP (Part 1)

Author: Interesting Blog Posts

09:00

Slides: MySQL 5.6 Global Transaction Identifier and PECL/mysqlnd_ms for failover

MySQL 5.6 Global Transaction Identifier - Use case: Failover

View more presentations on PHP and MySQL

The long lasting MySQL replication failover issue is cured. MySQL 5.6 makes master failover easy, PECL/mysqlnd_ms assists with the client/connection failover. Compared to the past this is a significant step towards improving MySQL replication cluster availability, eleminating the need to use 3rd party tools in many cases. The slides illustrate the basic idea, as blogged about before.

There is not much to say about the feature as such. Slave to master promotion works without hassles, finally. Regardless if you do failover because of an error of the current master or switchover because you want to change the master server, its easy now. Congratulations to the replication team!

Limitations of the current server implementation

The global transaction identifier implementation in MySQL 5.6 has a couple of limitations, though. Its not hard to guess that mixing transactional and non-transactional updates in one transaction can cause problems. Its pretty much the first pitfall I ran into when trying to setup a MySQL 5.6.5-m8 (not 5.6.4-m8…) slave using a mysqldump generated SQL dump. MySQL bailed at me and stopped me from failing.

Let a master run a transaction which first updates an InnoDB table. followed by an update of a MyISAM table, followed by another update: t(U^InnoDB, U^MyISAM, U^X). Let the binary log settings be so that this transaction is written as one to the binary log (binlog_format=statement, binlog_direct_non_transactional_updates=0). It is then copied “as is” to the relay log of a slave. Assume that the slave runs with different binary log settings so that t(U^InnoDB, U^MyISAM, U^X) is split up to t(U^MyISAM), t(U^InnoDB[, …])and logged as distinct transactions in the slaves binary log.

Worst case: conflicting binary log settings
Master		Slave
GTID=M:1	`t(U^InnoDB, U^MyISAM, U^X)`	GTID=M:1	`t(U^MyISAM)`
		GTID=M:1	`t(U^InnoDB[, …])`

Because slaves must preserve global transaction identifiers they got from their master, the two resulting transactions are given the same identifier. The transaction identifier in the slaves binary log is no longer unique, it now refers to two transactions not just one (issue #1). Any slave that would read from the binary log of the above slave may loose the InnoDB transaction because it may refuse to execute a transaction using an id that has been executed already (issue #2).

The workaround? Don’t mix InnoDB and MyISAM updates in one transaction. To me it does not sound that much of an issue in 2012. Please note, I’m describing my experience with MySQL 5.6.5-m8, which is a development version.

The load balancer update

MySQL Replication takes a primary copy approach to replication. A primary/master handles all the updates. Read-only replicas/slaves replicate from the primary. The primary is a single point of failure.

Writes	Primary/master
Reads	Slave		Slave

The failure of a slave is unproblematic. A client usually has plenty of other slaves to start reading from. If no slave is available, reads can even be forwarded to the master.

PHP
Load Balancer, e.g PECL/mysqlnd_ms
Read
~~Slave~~		Slave

All a PECL/mysqlnd_ms user has to do is check for an error after statement execution. If there’s one and the error code hints that the server has gone away, the user reruns the statement. The connection handle remains useable all the time. Upon rerun, PECL/mysqlnd_ms openes a new connection to another slave.

do {
  $res = $mysql->query("SELECT id, title FROM news");
} while (isset($connection_error_codes[$mysql->errno]));
	
if (!$res) {
  bail("SQL error", $mysql->errno, $mysql->error);
}

A master failure is much more problematic. There is no server to send a write to but the master. The master is a single point of failure. The new global transaction identifier help to reduce the time it takes to put a new master in place after a failure.

PHP
Load Balancer, e.g PECL/mysqlnd_ms
Write
~~Master~~

After a master failure some process needs to promote a former slave to the new master and, preferrable atomically, update all other slaves to start replicating from the new master. The below illustration is a bit confusing. It is intentionally. What happens during the simple to say “slave to master promotion” is a complete restructuring of the cluster.

~~Writes~~		~~Primary/master~~ (gone)
~~Read~~	Write	~~Slave~~ Master (promoted former slave)
~~Read~~	Read	Slave (no change)

Don’t forget to update the load balancer

After the cluster has been reorganized, the load balancer configurations must be updated. PECL/mysqlnd_ms happens to be a driver integrated load balancer. However, other than that, it is not different from a classical load balancer. Whatever process restructures the cluster it must take care of deploying the load balancer configurations afterwards.

Global transaction identifiers are a great help for the biggest part of the failover job - the server side. But, they are no swiss army knife. Don’t forget to update your load balancer configuration - the client side. No matter where it is. Whether it is part of the application code, driver integrated or you are using MySQL Proxy. As long as we are talking primary copy, a master failure will always be a major pain.

Happy hacking!

@Ulf_Wendel

Author: Interesting Blog Posts

17:46

Tiernan’s Podcast, Episode 1

Author: Interesting Blog Posts

March 14, 2012