Adam Leach

Most startups fail. But many of those failures are preventable.  The Lean Startup is a new approach being adopted across the globe, changing the way companies are built and new products are launched.

Eric Ries defines a startup as an organization dedicated to creating something new under conditions of extreme uncertainty. This is just as true for one person in a garage or a group of seasoned professionals in a Fortune 500 boardroom. What they have in common is a mission to penetrate that fog of uncertainty to discover a successful path to a sustainable business.

The Lean Startup approach fosters companies that are both more capital efficient and that leverage human creativity more effectively.  Inspired by lessons from lean manufacturing, it relies on “validated learning,” rapid scientific experimentation, as well as a number of counter-intuitive practices that shorten product development cycles, measure actual progress without resorting to vanity metrics, and learn what customers really want. It enables a company to shift directions with agility, altering plans inch by inch, minute by minute.

Rather than wasting time creating elaborate business plans, The Lean Startup offers entrepreneurs - in companies of all sizes - a way to test their vision continuously, to adapt and adjust before it’s too late. Ries provides a scientific approach to creating and managing successful startups in a age when companies need to innovate more than ever.

http://bit.ly/LeanStartupBook

[Update 1:12 AM]I just got home from a night of celebration to find this incredible sight waiting for me:

Amazon Bestsellers Rank: #7 in Books

Unbelievable - Amazon rank #7 in books overall, in just a few hours. Thank you all so much.

[Update 7:21 AM]#3 behind George R. R. Martin.

Image via Wikipedia

When we use the term PHP, we are often silently associating it with the abbreviation LAMP (that’s Linux, Apache, MySQL and PHP just in case you don’t recall). MySQL has been our bread and butter in PHP for over a decade; an old friend, accomplice and partner in crime. This was made possible with the MySQL extension. Indeed, you can scarcely find a basic nuts and bolts PHP tutorial that doesn’t use MySQL. Which is probably why it’s a good idea to give it a huge going away bash (and make sure it finds the exit afterwards and catches a cab to oblivion!). We’ve since seen replacements like the MySQL Improved extension (mysqli) and PHP Data Objects (PDO). These are simply better from the additional features each adds to their integration in higher level libraries such as Doctrine.

But, as with any basic change to a successful formula, there was bound to be some controversy at the mere suggestion of deprecating our old friend (even if preceded by an extended period of educating users on the well established replacements). Manuel Limos and Lucas Darnell have both written blog posts indicating what a bad idea this could be. Their issues are understandable. Once the E_DEPRECATION notices start flying applications that have existed for years (and years) will appear to implode leaving behind a long line of irritated people who may need to hire a PHP programmer to fix stuff. This obviously imposes a cash cost across thousands (probably an underestimation ) of businesses. This may lead to hosting services deferring adoption of the PHP version carrying the deprecation by months if not years. Lucas also raised an interesting point that with so much literature, including books, carrying example after example of (often insecure in my opinion) MySQL extension use, user adoption and education may suffer a great deal.

In a riposte to Manual Lemos, Gregg Thomason perhaps illustrates best why even the feared disadvantages may be worth the cost. MySQL is a historical relic from a past PHP is trying to leave behind. It’s old, doesn’t do a lot to support security and it needs to go. I agree. Gregg says “…this is a forward-thinking business and our job is to invent the future.” Let’s go invent and improve that future – if nothing else it might make Anonymous’ job finding SQL injections at every company they squint at a little harder .

PHP is not a weirdo stagnant programming language used by amateurs who don’t have sufficient brain cells to learn Java, Ruby or Python. That’s the common misconception based largely on two obvious factors: PHP is so amazingly popular and easy to learn that any innocently ignorant person with half a brain cell can write a fabulously insecure application (the examples just keep coming and coming) and, secondly, PHP is a bit on the ugly side and not a “true object oriented language” because it uses functions instead of methods. PHP is actually used by hardcore professionals who build great secure applications and that community has left the original MySQL extension by the wayside in favour of object oriented solutions where MySQL related functions are buried deep behind a wall of classes in their preferred database interaction solution, such as PDO or Doctrine. It’s about time we brought everyone else up to speed with that reality.

While “deprecation” may attract all the attention, let’s remember that pushing the alternatives by any possible means is a great idea. Philip Olson’s proposal on how to encourage users to move away from the original MySQL extension has a lot of merit and is well worth persuing. We need to let go of the past eventually to keep PHP moving into the future.

Within the Zend Framework 2 repository, I recently performed a conversion from DocBook 4 to 5. The latter is a more modern version of the toolchain, and offers a somewhat simpler syntax and workflow. For example, you no longer need to track how many levels deep you are in your chapter sections -- you simply use a <section> tag. One of the nicer changes is that you do not need to specify a doctype any more; you simply declare XML namespaces on your root elements and continue merrily on your way. This actually allows us to remove some actions in our build process, and makes validation much simpler.

Interestingly, for DocBook5 being available in beta since 2005 and an official standard since 2009, there is very little material on migrating from DocBook 4 to 5.

Open data releases (part 1)

In part 1 of this story significant data releases occurred in 2010 via data.gov.uk and the London datastore and a huge number of new web sites and apps have been set up to leverage open data. However, typically these data sets have been the ‘low hanging fruit’ of static reports, maps and statistics. Only a few of the ‘high value’ data sets have been released that contain ‘actionable information’ e.g.

• Tube live departures (see our Tube Radar site)
• Central government spending data (see COINS)
• Live traffic cameras (see Highways Agency)
• Crime maps (see Crime maps)
• Health performance data (e.g. Monthly critical care bed capacity)

If you look at data.gov.uk in detail you will find that it is stuffed with organograms and spending declarations, and that many of the datasets that started publishing have already ceased (e.g. A&E activity. Most government agencies still seem to be operating ‘user pays’ policies when you get beyond headline free data releases e.g. Meteorological Office data services. So while there are some notably early successes, there is clearly much more to do.

The government seems to recognise this and the Prime Minister has just released a letter to cabinet colleagues in which he commits to a further wave of data releases. But hard bitten experience on the GLA Digital Advisory Board suggest that there are still a lot of people to convince. The attitude of many data owners in government is that we like the control that our data monopoly gives us, we keep the (small) amounts of money we make and no one can ask awkward questions about what we do. The challenge now is to show what kind of difference to growth and transparency that open data can make to combat this kind of scepticism.

Critically, there are still only a small number of organisations dedicated to leveraging open data and building businesses or services:

• OpenlyLocal promoting local government transparency
• Placr aggregating public transport information
• CycleStreets open access to cycle routes
• Open Corporates opening company registries
• MySociety campaigning with open data on transport, housing and transparency
• Open Knowledge Foundation creating standards and technologies for data releases
• OpenStreetMap crowd sourcing a map of the world
• The Stationery Office government data publishers
• ITO World managing and visualising transport information

This is pretty much the whole list at present. In my view this is largely because the really important datasets are still not released and the private sector does not yet have the roadmap and the policy stability it needs to invest in the longer term. In short: ‘give us the good stuff’ and we will finish the job.

So it seems timely to re-state the case for open data and to put the argument in economic terms. This allows us to express the value of open data in terms of new taxes and jobs in the UK. If we can’t make this case, then government agencies are going to continue ‘data hugging’ with all the implications for the success of the government’s open data agenda.

Open data releases (part 2)

This is an attempt to work through the case for open data releases from end to end. It is important to look at open data as part of this whole chain… local interests in data often don’t see the bigger picture.

1. Market failure

Why do we have public data in the first place? In simple terms, Government has to act when there is ‘market failure’ to:

• manage infrastructures eg Network Rail, Royal Mail
• run essential services eg schools, health and policing
• act as honest broker eg courts or census

Each of these activities produces data where the government is the necessary first purchaser: you cannot run railways without timetables, or deliver mail without postcodes, or run public services without operational data such as RAISEonline for schools.

2. Free re-use

Datasets produced by government where it is a necessary first purchaser should be released under an open government licence i.e. be free for re-use with the minimum of processing beyond internal integrity checks and creation of metadata. This is because the marginal cost of distribution of data is very low. Whenever this is not done, then any data charges are a stealth tax.

There two limiting cases for government data distribution (and many in-between):

a) Static data… updated daily or less frequently: this should be available through a download link provided by the departmental data creator and published on data.gov.uk. The extra cost of this to government is very low as the download usage is spread randomly over the day. The extra utility in wide distribution of this data is unlimited.

If the government restricts access to this kind of dataset and charges for cost recovery e.g. postcodes (currently £4K p.a. for a full licence) then it is simply taxing end users as it has done in this case for 20 years. This becomes an industry and community cost that places UK PLC at a competitive disadvantage compared to other markets where the data is free, and limits market entrants to those able to bear the cost. The distributing agency also becomes a monopoly without any market pressures on its cost base or pricing. The Ordnance Survey was the best example of this for many years until the open community and the press forced a change in its operating model and release OS Open Data.

b) Real time data from operational systems… generally updated every few minutes or more frequently. This needs scalable distribution outside government as you cannot spread usage like you can with static downloads. In this case the data itself should be freed for re-use by open government licence and government should allow the market to create re-distribution channels from the operational systems. In some cases the market will redistribute for free to all: TfL have done a deal with Microsoft to distribute the tube real-time data in return for the use of the TfL data as a reference case used to advertise its cloud platform Azure. In other cases the market will charge for re-distribution but government may bulk-purchase access to create socially or economically desirable access. Hence Traveline have done a deal with Trapeze to bulk re-distribute bus departures at a fixed price in the Nextbuses API so that end user charges are low and flat for most scales of usage. This kind of distribution allows unprecedented insight into the performance of state-run services and the way the government manages its resources.

If the government want to charge for the redistribution of this kind of data (or fails to put licence arrangements in place to ensure public money achieves access), then they will continue tax end users as it has in this case since rail privatisation in 1996. Note that Association of Train Operating Companies (ATOC) charges £27K p.a. for timetables and fares data from publicly-funded Network Rail. This cost is passed on to consumers… the cheapest rail departures app for smartphone is c. £3 compared to typical transport app prices of £0.59. The costs are also passed onto the UK rail industry, which has the highest unit costs in the EU as the McNulty Report demonstrates. Monopoly distribution by ATOC has led to arbitrary licence refusals now being investigated through an Office of Rail Regulation consultation.

3. Innovation around open data

Data released to the market freely becomes available to drive self-funded community and commercial propositions. Communities without (significant) income streams e.g. commuter groups, school governors and health watchdogs can scrutinise and campaign from a position of knowledge.

Companies can create income streams by adding value to the open data releases for conveniance, immediacy or additional insights. By creating apps or web sites funded by advertising or subscription new business services can be created. As many of the available business models have low margins on low cost bases, they only work for free or marginal cost releases. This is because the open data can only be resold at very small markups as in an app… or because the open data enables a business to acquire users who don’t expect to pay for use of the information. However, once any digital services business has users there are multiple opportunities to monetise, e.g. by selling aggregate anonymised behaviour to other businesses trying to reach these audiences.

4. Open data regulation

Once open data releases have been made the government needs to regulate to ensure that

• access follows public money
• government itself does not compete with open data users
• there is equality of access (net neutrality for data)
• there is stability of distribution
• the distributing agencies do not act like monopolies usually act (to hinder competitors & market entrants)

These principles are needed to ensure that investments can be made in the private sector with confidence in the medium term.

Placr’s business model for open data

Given the availability of free or low cost data, Placr’s business model is to aggregate open data in transport, crime, schools and health and add value to data distributed through our API so it can be monetised in four ways:

a) By ad-funded web distribution… Placr web pages with live departure data are now in the top 5 Google hits for searches like “Baker Street departures”. We are in the process of adding advertising to these pages now they have earned a high search ranking through simple, effective presentation. We are also creating social media feeds for all the bus stops and stations in the country on our new pla.cr transport shortcode service, so that comments and experiences can be shared by users and developers (see placr.mobi for the evolving web app prototype).

b) By revenue-sharing with app developers who innovate with our API feeds at transportAPI.com. We have around 10 developers registered to use our feeds and the number is growing rapidly. We are cutting the time to market for developers and reducing the complexity of the raw data. We use a freemium style ‘click-wrap’ licence so that people can develop business models using our feeds and then pay us a flat 20% when they begin to take revenue. We are now starting to take revenue under this model.

c) By serving the operators and businesses in the market with analytics. We are developing tools to:

• Provide service performance metrics in real time and over the service history
• feed our pla.cr stop/station social media feeds with service updates, and
• mine the conversations for communities/ behaviours that allow the operators insight into their customers

This a B2B service that is dependent on the audiences that use the B2C services… and this audience is built by presenting open data to them for free.

d) By running consultancy and training for public and private sector bodies around open data opportunities, such as our Open Data Release Kit, an open consultancy and seminar offer for organisations looking to see a demo of API and web technology with their own datasets.

The margins on a) and b) are still small… but they enable c) and d). It takes time to erect these ecosystems… and some patience is needed to build services and acquire funding. There are also huge risks for startups in innovating close to a government that has not stabilised policy. We are succeeding with the help of bodies like the TSB (see our LaunchPad application here [not funded, alas]). But we now need decisive action by government to challenge public sector data monopolies so that other organisations have the opportunity to leverage the value in the data.

One of the chief barriers to progress is that we still don’t have free access to the crown jewels of government data, notably rail departures, postcodes/addresses, court judgments and planning applications. Access to these datasets are being restricted by:

• monopoly self-interest in some agencies eg Royal Mail Address Management Unit for postcodes
• IPR contamination by systems integrators handling state-funded data eg rail departures from ATOC, which have the effect or re-copyrighting the data
• pre-existing cost-recovery arrangements that need reform eg Registry Trust handling court judgments on credit records needs harmonisation with other crime data releases

These issues are in the public sector ‘too difficult’ box or will take time to resolve eg by action when licences roll over. But if we want the best open data in the world… and to create companies that can grow to dominate international markets… then the domestic market has to be liberalised across the board, not just where it is easy.

Jonathan Raper

UPDATE: in response to a sceptical comment about the prospects for open data by Steven Feldman, I posted some further arguments on the open data market as a comment on his blog here.

Antidilution Provisions: The conversion price of the Series A Preferred will be subject to a narrow-based weighted average adjustment to reduce dilution in the event that the Company issues additional equity secuities...

I’ve often wondered how things will change in a world where everything is a REST API and how relevant our Unix CLI tool chain will be in the long run. I’ve known we needed CLI ways to interact with data – like JSON data – and have given this a lot of thought.

MS Powershell does some pretty impressive object parsing on their CLI but I was never really sure how close we could get to that in Unix. I’ve wanted to start my journey with the grep utility as that seemed a natural starting point and my most used CLI tool.

I have no idea how to write parsers and matchers but luckily I have a very talented programmer working for me who were able to take my ideas and realize them awesomely. Pieter wrote a json grep and I want to show off a few bits of what it can do.

I’ll work with the document below:

[
  {"name":"R.I.Pienaar",
   "contacts": [
                 {"protocol":"twitter", "address":"ripienaar"},
                 {"protocol":"email", "address":"rip@devco.net"},
                 {"protocol":"msisdn", "address":"1234567890"}
               ]
  },
  {"name":"Pieter Loubser",
   "contacts": [
                 {"protocol":"twitter", "address":"pieterloubser"},
                 {"protocol":"email", "address":"foo@example.com"},
                 {"protocol":"msisdn", "address":"1234567890"}
               ]
  }
]

There are a few interesting things to note about this data:

• The document is an array of hashes, this maps well to the stream of data paradigm we know from lines of text in a file. This is the basic structure jgrep works on.
• Each document has another nested set of documents in an array – the contacts array.

Examples

A simple grep for a single key in the document:

$ cat example.json | jgrep "name='R.I.Pienaar'"
[
  {"name":"R.I.Pienaar",
   "contacts": [
                 {"protocol":"twitter", "address":"ripienaar"},
                 {"protocol":"email", "address":"rip@devco.net"},
                 {"protocol":"msisdn", "address":"1234567890"}
               ]
  }
]

We can extract a single key from the result:

$ cat example.json | jgrep "name='R.I.Pienaar'" -s name
R.I.Pienaar

A simple grep for 2 keys in the document:

% cat example.json | 
    jgrep "name='R.I.Pienaar' and contacts.protocol=twitter" -s name
R.I.Pienaar

The nested document pose a problem though, if we were to search for contacts.protocol=twitter and contacts.address=1234567890 we will get both documents and not none, that’s because in order to effectively search the sub documents we need to ensure that these 2 values exist in the same sub document.

$ cat example.json | 
     jgrep "[contacts.protocol=twitter and contacts.address=1234567890]"

Placing [] around the 2 terms works like () but restricts the search to the specific sub document. In this case there is no sub document in the contacts array that has both twitter and 1234567890.

Of course you can have many search terms:

% cat example.json | 
     jgrep "[contacts.protocol=twitter and contacts.address=1234567890] or name='R.I.Pienaar'" -s name
R.I.Pienaar

We can also construct entirely new documents:

% cat example.json | jgrep "name='R.I.Pienaar'" -s "name contacts.address"
[
  {
    "name": "R.I.Pienaar",
    "contacts.address": [
      "ripienaar",
      "rip@devco.net",
      "1234567890"
    ]
  }
]

Real World

So I am adding JSON output support to MCollective, today I was rolling out a new Nagios check script to my nodes and wanted to be sure they all had it. I used the File Manager agent to fetch the stats for my file from all the machines then printed the ones that didn’t match my expected MD5.

$ mco rpc filemgr status file=/.../check_puppet.rb -j | 
   jgrep 'data.md5!=a4fdf7a8cc756d0455357b37501c24b5' -s sender
box1.example.com

Eventually you will be able to then pipe this output to mco again and call another agent, here I take all the machines that didn’t yet have the right file and cause a puppet run to happen on them, this is very Powershell like and the eventual use case I am building this for:

$ mco rpc filemgr status file=/.../check_puppet.rb -j | 
   jgrep 'data.md5!=a4fdf7a8cc756d0455357b37501c24b5' |
   mco rpc puppetd runonce

I also wanted to know the total size of a logfile across my web servers to be sure I would have enough space to copy them all:

$ mco rpc filemgr status file=/var/log/httpd/access_log -W /apache/ -j |
    jgrep -s "data.size"|
    awk '{ SUM += $1} END { print SUM/1024/1024 " MB"}'
2757.9093 MB

Now how about interacting with a webservice like the GitHub API:

$ curl -s http://github.com/api/v2/json/commits/list/puppetlabs/marionette-collective/master|
   jgrep --start commits "author.name='Pieter Loubser'" -s id
52470fee0b9fe14fb63aeb344099d0c74eaf7513

Here I fetched the most recent commits in the marionette-collective GitHub repository, searched for ones by Pieter and returns the ID of those commits. The –start argument is needed because the top of the JSON returned is not the array we care for. The –start tells jgrep to take the commits key and grep that.

Or since it’s Sysadmin Appreciation Day how about tweets about it:

% curl -s "http://search.twitter.com/search.json?q=sysadminday"|
   jgrep --start results -s "text"
 
RT @RedHat_Training: Did you know that today is Systems Admin Day?  A big THANK YOU to all our system admins!  Here's to you!  http://t.co/ZQk8ifl
RT @SinnerBOFH: #BOFHers RT @linuxfoundation: Happy #SysAdmin Day! You know who you are, rock stars. http://t.co/kR0dhhc #linux
RT @google: Hey, sysadmins - thanks for all you do. May your pagers be silent and your users be clueful today! http://t.co/N2XzFgw
RT @google: Hey, sysadmins - thanks for all you do. May your pagers be silent and your users be clueful today! http://t.co/y9TbCqb #sysadminday
RT @mfujiwara: http://www.sysadminday.com/
RT @mitchjoel: It's SysAdmin Day! Have you hugged your SysAdmin today? Make sure all employees follow the rules: http://bit.ly/17m98z #humor
? @mfujiwara: http://www.sysadminday.com/

Here as before we have to grep the results array that is contained inside the results.

I can also find all the restaurants near my village via SimpleGEO:

curl -x localhost:8001 -s "http://api.simplegeo.com/1.0/places/51.476959,0.006759.json?category=Restaurant"|
   jgrep --start features "properties.distance<2.0" -s "properties.address 
                                      properties.name 
                                      properties.postcode 
                                      properties.phone 
                                      properties.distance"
[
  {
    "properties.address": "15 Stratheden Road",
    "properties.distance": 0.773576114771768,
    "properties.phone": "+44 20 8858 8008",
    "properties.name": "The Lamplight",
    "properties.postcode": "SE3 7TH"
  },
  {
    "properties.address": "9 Stratheden Parade",
    "properties.distance": 0.870622234751732,
    "properties.phone": "+44 20 8858 0728",
    "properties.name": "Sun Ya",
    "properties.postcode": "SE3 7SX"
  }
]

There’s a lot more I didn’t show, it supports all the usual <= etc operators and a fair few other bits.

You can get this utility by installing the jgrep Ruby Gem or grab the code from GitHub. The Gem is a library so you can use these abilities in your ruby programs but also includes the CLI tool shown here.

It’s pretty new code and we’d totally love feedback, bugs and ideas! Follow the author on Twitter at @pieterloubser and send him some appreciation too.