Adam Leach

Recently I started on Maemo development and came across ESBOX the MAEMO IDE. I was glad to see something easy to go kind of thing ( this is what i felt in begining), but soon my opinion changed. First of the Maemo SDK installation itself is a terrible experience, and take me that if you dont have experience on linux you will want to throw your PC out of the window ( not windows ;).

Yes , this is true if you want to dare to develop for Maemo ? Then first you will need to learn linux up to a sufficient level. I believe the Maemo is fun for linux veteran and hard time for linux newbie. Finally by following the GUI installers instruction I installed the Maemo ScartchBox 1, this process took significant time couldnt understand if its my internet speed issue or system itself works slower. Somehow finally I installed the Scratchbox. Now I decided to install ESBOX and started already dreaming about the Maemo apps development. But actually I tried this first with my windows machine already the experience was bitter. I thought may be the virtual image of Maemo SDK is creating trouble so I decided to switch to the actual platform rather than emulating stuff on Windows XP.

On Debian I installed the Scatchbox 1 and again started the ESBOX, believe me , first of all you will find very less documentation about configuring ESBOX with actual Scratchbox ( there is a lot said about Virtual Image SDK installation), so you will have to rely solely on , on screen instructions.

One thing I must say after 24 hours of struggle I compiled my first Maemo program and ran in Scratchbox too, but to reach to this step what exactly I did, how may things I installed I myself dont know. The reason behind this is whenever I went to ESBOX and tried to see the installed targets, it showed none so I went on installing them everytime it showed screens installing many things but there was no success. Finally after lot of trial and error I was able to run Fremantle X86 debug targeted application, I am still not sure about ARMEL target, if it works or not.

All in all this was a painfull journey as compared with Symbian tools and Carbid.c++, Nokia seriously needs to do something about this. Hope to see something revolutionary in comming time, making our lives easy :)

Hy interested ones,

today I integrated two improvements for Zend_Validate_Identical.

Zend_Validate_Identical is a validator which enables you to check if two values are identical.
Values means in this case string, integers, floats, or even objects.

A manual validation would look like this:

$valid = new Zend_Validate_Identical(array('token' => '1234'));
        
if ($valid->isValid($input)) {
    // let's go on
} else {
    // oops... not valid
}

Simple as is…

Now, when $input is the string “1234” we will get true. When it differs we will get false. This means even if we have a integer “1234” the validation will fail. The reason behind this behaviour is that Zend_Validate_Identical does a strict validation including the type of the input.

But sometimes it is wished and necessary to validate only the content regardless of it’s type.
Zend_Validate_Identical supports now also non-strict validation. See the following example:

$valid = new Zend_Validate_Identical(
    array('token' => '1234', 'strict' => false)
);
        
if ($valid->isValid($input)) {
    // let's go on
} else {
    // oops... not valid
}

As you can see the above example is nearly identical to the first example with one difference. We defined the property strict to be false. By using this option we said Zend_Validate_Identical to use non-strict validation.

In this case, when $input is a integer “1234” we will also get true in return and also when it’s a float “1234”.

Now what to do when you want to validate if two form elements are identical. Seems tricky as Zend_Validate_Identical has no connection to Zend_Form, and it would not know which two elements you want to validate.

Easy as is, you can now give the elements name as token which holds the token to validate against. What does this mean? Let’s see a little example:

<?php
error_reporting(E_ALL|E_STRICT);
ini_set('include_path',ini_get('include_path').PATH_SEPARATOR.'../library');
require_once('Zend/Loader/Autoloader.php');
$loader = Zend_Loader_Autoloader::getInstance();
print "<pre>";
        
$request = new Zend_Controller_Request_Http();
        
// setup the form
$form = new Zend_Form();
$form->setMethod(Zend_Form::METHOD_POST)
$form->addElement('password', 'password1');
$form->addElement('password', 'password2', array(
    'validators' => array(
        array('identical', false, array('token' => 'password1'))
    )
));
$form->addElement('submit', 'submit');
        
// check the form
if($request->isPost()) {
    $formData = $request->getPost();
    if($form->isValid($formData)) {
        $form->getValues();
        echo "FORM VALID";
    } else {
        print "nVALIDATION FAILURE:";
        print_r($form->getMessages());
    }
}
print "</pre>";
?>
<html>
<head>
<title>Test</title>
</head>
<body>
<?php echo $form->render(new Zend_View());?>
</body>
</html>

You can run the above example standalone in your browser. It shows you the behaviour of Zend_Validate_Identical and how to combine it efficent within Zend_Form.

For us the important line is:

$form->addElement('password', 'password1');
$form->addElement('password', 'password2', array(
    'validators' => array(
        array('Identical', false, array('token' => 'password1'))
    )
));

As you see we added the Identical validator to the second element by using the first elements name as token.
This way the value of the first element is compared with the value of the second element.

Simple as is :-)

This is handy when you want to validate two user inputs. For example when your user has to enter his email adress two times to be sure he did not mistype it, or he has to enter the same password two times.

Note that these two described features are available within trunk and as with ZF 1.10.5 and NOT BELOW.

I hope you find this two features useful. More to come soon…

Greetings
Thomas Weidner
I18N Team Leader, Zend Framework

Zend Framework Advisory Board Member
Zend Certified Engineer for Zend Framework

Would you like to create professional-looking mobile applications in mere minutes, then distribute them to millions of consumers around the world through Ovi Store? If so, then the new Ovi App Wizard is for you. Designed to be easy and helpful, this tool lets anyone create an app quickly — even those with no design or programming experience. Ovi App Wizard is available now for free as an open beta to all individuals and businesses, and there is no registration or publishing fee for the apps you create.

Get started with the Ovi App Wizard.

If you were at last month's MySQL conference, you might have seen a small flyer we were distributing, titled "A brief introduction to Goal-Driven Performance Optimization." This is a super-compressed technical introduction to one of the methods we use to solve problems as quickly and efficiently as possible. We've just posted this on a new white-paper section of our website. No registration is required; it's just a downloadable PDF.

---

Entry posted by Baron Schwartz | No comment

Add to: | | | |

If you are wondering how to get started with Maemo development then read this article first, you can surely get a quick start. http://wiki.forum.nokia.com/index.php/Developing_with_ESBOX_and_Maemo_SDK_using_Virtual_machine_Image. This article shows how to install the Maemo development environment on Windows and start development with it. Its the easiest way I have found so far on Maemo. Happy maemo programming :)

It is very common for me to look at a customer's database and notice a lot of overhead from checking whether a database connection is active before sending a query to it. This comes from the following design pattern, written in pseudo-code:

Many of the popular development platforms do something similar to this. Two things are wrong with this code: 1) it doesn't actually work and 2) it has a large performance overhead.

It Does Not Work

This code doesn't work because of a race condition. If the connection is alive when checked, there's no guarantee that it is still alive by the time connection.execute(sql) is called. And if it's not alive and gets reconnected, there's still no guarantee.

It's useless to check and then execute. Instead, the code should be rewritten something like this:

Notice that I completely removed the call to is_active(). If the connection is active, the query will succeed; if not, it'll fail, reconnect, and retry.

This code also has the nice property that it also allows you to retry things like lock wait timeouts and deadlocks, if you so desire. These are areas where many applications can benefit a lot, in my experience. Most applications should simply retry these conditions, but instead they don't handle them at all.

Performance Overhead

Checking whether the connection is active typically involves a call to the 'ping' or 'statistics' command at the MySQL protocol level, which will increment Com_admin_commands in SHOW GLOBAL STATUS, or a trivial query such as 'SELECT 1' which is usually harder to diagnose. This has a very high cost in many applications. There are two costs: the cost to the application of the network round-trip plus the query execution time, and the added load on the database server. The added load on the database server can be very high. A few days ago I saw a Ruby on Rails app where the 'Administrator command: statistics' command was taking over 40% of the server's total query time. Eliminating the useless connection checks could have nearly halved the load on this database server. That's not unusual!

When the application's queries are long, the extra query is lost in the noise. But high-traffic apps put tremendous effort into getting query times down, and some highly tuned apps are worried when their queries take longer than a millisecond or so. When you're running 20k queries per second against your database server, an extra 20k queries per second to check whether the connection is alive matters a lot. Those 'statistics' or 'ping' queries are nearly as expensive as the actual queries the application wants to run!

And that's just the load on the database server. On the application side, you essentially see doubled query latency. Every time you want to run a query, your app framework is doing a network round-trip to the database to check the connection, then another network round-trip to run the query. Again, this matters a lot.

The problem is that the bad pseudo-code I showed above penalizes the common case in favor of the uncommon case. Connections are usually alive and don't need to be pinged and reconnected. A better approach is to use the same code that fixes the race condition. Again, if the connection is down, you'll find out when you try to run the query. Until then, assume everything is OK and just run the queries.

I hope that the upstream maintainers of the offending libraries can find and solve these issues, because it'd be a big help to apps when they grow. It's one of those things that works fine in the lab, and even in the field, until performance really starts to matter -- and then it sticks out like a sore thumb.

(Edit)

Here's another example of the impact of these silly queries:

That's right, 73% of the server's load is consumed by checking to see if the connection is still alive

---

Entry posted by Baron Schwartz | 27 comments

Add to: | | | |

The YUI team released YUI 2.8.1 and YUI 3.1.1 today. Each of these minor releases contains a set of targeted bug fixes. Notably, the History component in both libraries was updated to correct an issue that could result in an XSS vulnerability in some implementations.

History Issue

Ryan Grove corrected an issue relating to YUI 2 Browser History Manager and YUI 3 History. Writes Ryan:

In previous YUI versions, HTML markup in history values was not escaped before being written to a hidden iframe in IE6 and IE7, which could result in script execution in certain circumstances. An attacker with the ability to create an unfiltered link on a page that uses YUI 2 Browser History Manager or YUI 3 History (for example, by posting a comment on a blog, where the blog software did not properly filter the comment) could in theory inject arbitrary HTML or script that would be executed when a user then clicked on that link. YUI 2.8.1 and YUI 3.1.1 address this issue. To mitigate this in versions of YUI prior to 2.8.1 and 3.1.1, ensure that HTML characters in user input are always escaped (whether on the server or on the client) before being used to construct links or history values.

These updates are strongly recommended for all users of YUI’s browser history functionality.

YUI 3.1.1

In addition to the History fix, YUI 3.1.1 contains about 20 minor updates throughout the library. Refer to the YUI 3.1.1 ticket list for full details.

YUI 2.8.1

Issues addressed in YUI 2.8.1 include the History vulnerability and minor updates to AutoComplete, DataTable, TreeView and Uploader.

We also updated a number of YUI 2 examples that employ the Flickr API. These examples were previously using a Flickr API key that was beginning to become overused — probably through inadvertent copy-and-paste into third-party applications. If you find that this issue is affecting your application — i.e., your Flickr images are no longer appearing or you’re seeing errors in Flickr-related implementations whose source was derived from a YUI example — you should head over to Flickr and pick up your own API key.

In his fictional story "The Library of Babel", Jorge Luis Borges describes a vast storage universe composed of all possible manuscripts uniformly formatted as 410-page books. Most are random meaningless sequences of symbols. But the rest excitingly forms a complete and an indestructible knowledge system, which stores any text written in the past or to be written in the future, thus providing solutions to all problems in the world. Just find the right book.

The same characteristic fascinates us in modern storage growth: The aggregation of information directly leads to proportional growth of new knowledge discovered out of it. A skeptic may doubt that further reward in knowledge mining will not justify the effort in information aggregation. What if by building sophisticated storage systems we are chasing the 19th century’s horse manure problem, when at the dawn of the automobile era the scientific world was preoccupied with the growth of the horse population that threatened to bury the streets of London nine feet deep in manure?

The historical judgment will turn one way or another. In the meantime, we want to know how far we can go with existing systems even if only out of pure curiosity. In my USENIX ;login: article “HDFS Scalability: the Limits to Growth,” I studied scalability and performance limitations imposed on a distributed file system by the single-node namespace server architecture. The study is based on experience with largest deployments of the Hadoop Distributed File System (HDFS) currently in production at Yahoo!.

The first part of the study focuses on how the single name-server architecture limits the number of namespace objects (files and blocks) and how this translates to the limitation of the physical storage capacity growth.

The second part explores the limits for linear performance growth of HDFS clusters bound by natural performance restrictions of a single namespace server. As the cluster grows, the linear increase in the demand for processing resources puts a higher workload on the single namespace server. When the workload exceeds a certain threshold, it saturates the server, turning it into a single-node bottleneck for linear scaling of the entire cluster.

In 2006, the Hadoop group at Yahoo! formulated long-term target requirements for HDFS and outlined a list of projects intended to bring the requirements to life. Table 1 summarizes the targets and compares them with the current achievements:

	Target	Deployed
Capacity	10 PB	14 PB
Nodes	10,000	4,000
Clients	100,000	15,000
Files	100,000,000	60,000,000

Table 1: Targets for HDFS vs. actual deployed values as of 2009

The bottom line is that we achieved the target in Petabytes and got close to the target in the number of files. But this is done with a smaller number of nodes and the need to support a workload close to 100,000 clients has not yet materialized. The question now is whether the goals are feasible with the current system architecture.

Namespace Limitations

HDFS is based on an architecture where the namespace is decoupled from the data. The namespace forms the file system metadata, which is maintained by a dedicated server called the name-node. The data itself resides on other servers called data-nodes.

The namespace consists of files and directories. Files are divided into large (128 MB) blocks. To provide data reliability, HDFS uses block replication. Each block by default is replicated to three data-nodes. Once the block is created its replication is maintained by the system automatically. The block copies are called replicas.

The name-node keeps the entire namespace in RAM. This architecture has a natural limiting factor: the memory size; that is, the number of namespace objects (files and blocks) the single namespace server can handle.

Estimates show that the name-node uses less than 200 bytes to store a single metadata object (a file inode or a block). According to statistics on Y! clusters, a file on average consists of 1.5 blocks. Which means that it takes 600 bytes (1 file object + 2 block objects) to store an average file in name-node’s RAM

To store 100 million files (referencing 200 million blocks), a name-node should have at least 60 GB of RAM.

Storage Capacity vs. Namespace Size

With 100 million files, each having an average of 1.5 blocks, we will have 200 million blocks in the file system. If the maximal block size is 128 MB and every block is replicated 3 times, then the total disk space required to store these blocks is about 60 PB.

100 million file namespace needs 60 PB of total storage capacity on the cluster

As a rule of thumb, the correlation between the representation of the metadata in RAM and physical storage space required to store data referenced by this namespace is:

1 GB metadata ≈ 1 PB physical storage

The rule should not be treated the same as, say, the Pythagorean Theorem, because the correlation depends on cluster parameters (the block to file ratio and the block size). But it can be used as a practical estimate for configuring cluster resources.

Cluster Size and Node Configuration

Next we can estimate the number of data-nodes the cluster should have to accommodate the namespace of a certain size. On Yahoo’s clusters, data-nodes are usually equipped with four disk drives of size 0.75 – 1 TB, and configured to use 2.5 – 3.5 TB of that space per node. The remaining space is allocated for MapReduce transient data, system logs, and the OS.

If we assume that an average data-node capacity is 3 TB, then we will need on the order of 20,000 nodes to store 60 PB of data. To keep it consistent with the target requirement of 10,000 nodes, each data-node should be configured with eight 1TB hard drives.

To accommodate data referenced by a 100 million file namespace, a HDFS cluster needs 10,000 nodes equipped with 8TB of total hard drive capacity per node

Note that these estimates are true under the assumption that the block per file ratio of 1.5 and the block size remain the same. If the ratio or the block size increases, a gigabyte of RAM will support more petabytes of physical storage and vice versa. Sadly, based on practical observations, the block to file ratio tends to decrease during the lifetime of a file system, meaning that the object count (and therefore the memory footprint) of a single namespace server grows faster than the physical data storage. That makes the object-count problem — which becomes file-count problem when the average file to block ratio is close to 1 — the real bottleneck for cluster scalability.

The Internal Load

Apart from the hierarchical namespace the name-node’s metadata includes a list of registered data-nodes, and a block to data-node mapping, which determines physical block locations.

A data-node identifies block replicas in its possession to the name-node by sending block reports. The first block report is sent at the startup. It reveals the block locations, which otherwise are not known to the name-node at startup time. Subsequently, block reports are sent periodically every 1 hour by default and serve as a sanity check providing that the name-node has an up-to-date view of block replica distribution on the cluster.

During normal operation, data-nodes periodically send heartbeats to the name-node to indicate that the data-node is alive. The default heartbeat interval is 3 seconds. If the name-node does not receive a heartbeat from a data-node in 10 minutes, it pronounces the data-node dead and schedules its blocks for replication on other nodes.

The block reports and heartbeats form the internal load of the cluster. If the internal load is too high, the cluster becomes dysfunctional, able to process only a few, if any, external client operations such as ls, read, or write. The internal load depends on the number of data-nodes. Assuming that the cluster is built of 10,000 data-nodes having 8 hard drives with 6 TB of effective storage capacity each, we estimate that the name-node will need to handle

• 3 block reports per second, each reporting 60,000 replicas
• 10,000 heartbeats per second

Using the standard HDFS benchmark called NNThroughputBenchmark, we measure the actual name-node performance with respect to the two internal operations. Table 2 summarizes the results. Note that the block report throughput is measured in the number of blocks processed by the name-node per second.

	Throughput
Number of blocks processed in block reports per second	639,713
Number of heartbeats per second	300,000

Table 2: Block report and heartbeat throughput

The implication of these results is:

The internal load for block reports and heartbeat processing on a 10,000 node HDFS cluster with the total storage capacity of 60 PB will consume 30% of the total name-node processing capacity.

The internal load is proportional to the number of nodes in the cluster and the average number of blocks on a node. Thus, if a node had only 30,000 blocks — half of the estimated amount — then the name-node will dedicate only 15% of its processing resources to the internal load. Vice versa, if the average number of blocks per node grows, then the internal load will grow proportionally. The latter means that the decrease in block to file ratio (more small files with the same file system size) increases the internal load and therefore negatively affects the performance of the system.

Reasonable Load Expectations

We have learned by now that the name-node can use 70% of its time to process external client requests. Even with a handful of clients one can saturate the name-node performance by letting the clients send requests to the name-node with very high frequency. The name-node most probably would become unresponsive, potentially sending the whole cluster into a tailspin because internal load requests do not have priority over regular client requests.

In practice, the extreme load bursts are uncommon. Regular Hadoop clusters run MapReduce jobs, and jobs perform conventional file reads or writes. To get or put data from or to HDFS, a client first accesses the name-node and receives block locations, and then directly talks to data-nodes to transfer file data. Thus the frequency of name-node requests is bound by the rate of data transfer from data-nodes.

Typically a map task of a MapReduce jobs reads one block of data. We estimate how much time it takes for a client to retrieve a block replica and, based on that, evaluate the expected read load on the name-node — namely, how many “get block location” requests the name-node should expect per second from 100,000 clients. We apply the same technique to evaluate the write load on the cluster.

Performance Limitations

The read and write throughputs have been measured by the DFSIO benchmark and are summarized in Table 3.

	Throughput
Average read throughput	66 MB/s
Average write throughput	40 MB/s

Table 3: HDFS read and write throughput

Another series of throughput results produced by NNThroughputBenchmark (Table 4) measures the number of open (the same as get block location) and create operations processed by the name-node per second:

	Throughput
Get block locations	126,119 ops/s
Create new block	5,600 ops/s

Table 4: Open and create throughput

The read throughput in Table 3 indicates that 100,000 readers are expected to produce 68,750 get-block-location requests to the name-node per second. And Table 4 confirms that the name-node is able to process that many requests even if only 70% of its processing capacity is dedicated to this task.

A 10,000 node HDFS cluster with the internal load at 30% will be able to handle an expected read-only load produced by 100,000 HDFS clients.

The write performance is not as optimistic. According to Table 3, 100,000 writers are expected to provide an average load of 41,667 create block requests per second on the name-node. This is way above 3,920 creates per second — 70% of possible processing capacity of the name-node (see Table 4).

A reasonably expected write-only load produced by 100,000 HDFS clients on a 10,000 node HDFS cluster will exceed the throughput capacity of a single name-node.

We have seen that a 10,000-node HDFS cluster with single name-node is expected to handle a workload of 100,000 readers well. However, even 10,000 writers can produce enough workload to saturate the name-node, making it a bottleneck for linear scaling.

Such a large difference in performance is attributed to get-block-locations (read workload) being a memory-only operation, while creates (write workload) requires journaling, which is bounded by the local hard drive performance.

There are ways to improve the single name-node performance. But any solution intended for single namespace server optimization lacks scalability.

Looking into the future — especially taking into account that the ratio of small files tend to grow — the most promising solutions seem to be based on distributing the namespace server itself both for workload balancing and for reducing the single server memory footprint.

I hope you will benefit from the information provided above. Please refer to the USENIX ;login: article for more details.

Konstantin V. Shvachko
Hadoop Distributed File System (HDFS) Team, Yahoo!

Microsoft and Nokia have launched the first application of the mobile-productivity alliance the two companies formed last August. The new application, Microsoft Communicator Mobile for Nokia, is a unified communications client that connects directly with a company's communications systems to streamline mobile collaboration. Owners of Nokia E72 and Nokia E52 smartphones can download the English version now from Nokia's Ovi Store. Looking ahead, the two companies plan to deliver Communicator Mobile for Nokia pre-installed on select Nokia smartphones; they also plan to support additional devices, including the new Nokia E5.

Learn more about the application’s debut.

As we’re now shipping the beta version of Qt 4.7 - I want to share a bit of our future product plans, including some changes we’re implementing. The Qt Quick journey so far has been pretty amazing, going from being a technology that few understood or embraced, to cautious optimism and more recently a more insatiable pull and momentum building up around the Alpha version we released back in March.

There are already a number of examples of how cool apps you can build with Qt Quick. For instance there is Ryan Paul’s RSS feed wall example:

Digia also built a complete Connected Car demo:

And, the Qt sources come with a huge number of examples and demos in $QTDIR/examples/declarative and $QTDIR/demos/declarative respectively. On the right is a screen-cast to show ze lazy ones, but in the Declarative UI category here on Labs you’ll also find a lot of Treats.

There are some brave souls out there as well, who recognize the value Qt Quick offers; allowing an exceptional level of differentiation in the User Experience - agile turnaround in the UI design and having designers and developers work tightly and effectively together - talking the same language and working in the same environment. We’ll see hands-on examples from these once they are ready to display their wares - but not yet; we’re preparing some awesome show-cases to feast on for the launch party.

As part of developing Qt Quick we have been working with our friends at Intel through MeeGo and with Nokia’s internal design groups. They have found that Qt Quick allows them to have a fluid transition from prototype to production code. But, bringing a new technology to market is always hard. Working with these teams has made it apparent that a visual editor needs to be exceptionally good to be useful to visual designers. Because of this it has been decided to just offer the visual editor as a technical preview till we can get it to reach the high standards we expect from ourselves, something that will happen later this year.

We would love for you to enable the Preview Qt Quick Designer, put it through it’s paces, and provide feedback as to how you would like to see it evolve - preferably as new suggestion tasks at bugreports using component ‘Qml Support’. But that’s not all the news about Qt Quick; we made Qt Creator ready for prime-time with added support for Qt Quick; New Qt Quick Project Wizard, QML Syntax Highlighting, QML Code completion, Documentation integration, Object Inspector and Integration with the qml launcher for ‘running’ QML files. So, if you’re a technical Graphic Designer or UX Engineer, you’ll find ample support in Qt Creator for the best thing since sliced bread!

To enter the Cult of QML, please sign up to the Qt QML mailing list at qt-qml@trolltech.com and join us on #qt-qml at irc.freenode.org!

More than a month has passed since the release of Creator’s 2.0 alpha release. The Berlin trolls were truly busy, not only with fixing reported bugs, but also by improving the overall developer experience. This beta is another milestone in the 2.0 release cycle, as we hope you will test-drive this release, to ensure 2.0 will be rock-stable, and to get to know the new features that were introduced.

Most changes are small but important details, like a shiny new options dialog, but also fundamental changes which aim to help development of a single application on multiple targets: the beta now extends shadow building to support different build directories per target. The code model was also improved, so that defines in the .pro file are respected in the editor as well as for auto-completion. A hard decision however was to disable the Qt Quick visual designer for 2.0 by default. Henrik has shared his thoughts in a blog post.

The alpha blog post was mentioning external contributions, but missed the biggest single contribution so far: The Mercurial plugin, written by Brian McGillion. Kudos Brian, Mercurial support was a frequently requested feature!

The bigger picture

For all those who haven’t been following the development of Qt Creator 2.0 closely, here are most significant changes since 1.3.1: Project parsing was greatly improved, with Creator now using multiple threads in addition to honoring the .pro file defines as mentioned above.

As the version suggests, the new version also means changes in the overall functionality. The GUI is now optimized to handle multiple targets, which conceptually have become aspects of a project. This reflects in a new layout of the projects page, which now allows to set build- and run settings per target, as well as the newly introduced target selector, where you can quickly switch between different targets and projects.

People frequently suggested to separate design and code view, which was implemented for 2.0. This is especially useful when using the QML visual designer, but also works for classical .ui files. To make up for the additional space required for the new mode, the output mode was replaced with the option to enlarge the output window. And of course we introduced support for the new python backend for GDB.

For a more verbose changelog, take a look here.

Finally, Qt Creator has become the heart of the new Nokia Qt SDK, which brings the integrated Qt development experience to Maemo and Symbian developers. For the curious, Maurice answers a lot of question in his blog post. Note that we will also ship the Qt SDKs for Linux, Mac and Windows desktop development, but they are not part of this release.

Get your hands dirty!

You can download the Nokia Qt SDK, which includes the beta, or get a creator-only binary installer from the release page. Note that the Nokia Qt SDK does not contain a desktop version of Qt. If you want to develop for the desktop, why not try the Qt 4.7.0 beta?

I've been hearing about and reading about Gearman for a couple years now, but, due to the nature of my work, it's never really been something I needed to investigate; when you're writing backend code, scalability is something you leave to the end-users, right?

Wrong! But perhaps an explanation is in order.

Continue reading "Writing Gearman Workers in PHP"

Winter is long in Oslo. This was particularly true for the last winter. In fact, we had a little snowstorm on Tuesday! But I am digressing…

We spent most of winter and the season known as spring in other areas in the world on the newest kid on the block - The Qt Developer Network. Now it’s live. Whew.

And what is the Qt Developer Network, please?

Before Christmas 2008 we took a look at the state of our web sites and agreed that it needed some serious work, we made a 3 year plan.
The Qt Developer Network was in there from the beginning but it wasn’t until the fall of 2009 that we got the time and money to make it a reality. Now we finally have an organized place to collect both internal and external Qt knowledge.

But the Qt Developer Network is not all new, we had some of the pieces scattered around from before. You might have seen the forums right here on Qt Labs. They are in a rather sad state and will be shut down once we open up for registration.

Qt Developer Network features a forum, a wiki, a fully fledged blog, a totally awesome Qt FAQ generated and auto-updated from our support database every night and most of the content that used to live in the developer zone on qt.nokia.com. We implemented a nice system of points and rewards - doesn’t Area 51 Engineer sound charming to you? - which was actually quite fun to come up with. And we strongly believe in fun at work!

This, however, is only the beginning.

We have a somewhat solid roadmap sketched out for the coming months. Apart from a tag engine which is still missing behind the dummy tags all over the site, there will be a group module and Ovi authentication coming soonish.

Looking at the content, we still need to move the books section and the eLearning videos. That will hopefully happen over the coming weeks.

And now, we are looking for your feedback. We have placeholders in our task tracker for requested features and lacking functionality to be implemented as we go along. So, don’t wait - have a look at what we’re building and let us know if we are headed in the right direction.

Hey, I can’t sign up! How am I supposed to test and give feedback?!

Currently, we are in a closed beta testing phase. We have sent out invitations to pre-registered users so we can start with baby-steps. We will not do a Gmail and stay beta for years; we should reach open beta soon and 1.0 should be out this summer.

The Qt 4.7 Beta1 was released today, and I’m happy to announce that a set of experimental Qt 4.7 packages is available for Maemo 5 as well: http://chaos.troll.no/~harald/qt4-maemo5-4.7/20100505/ (all packages in one single tarball here).

Note: The packages now strictly require PR 1.2. Please only install them in the latest Scratchbox based SDK, which already ships a pre-release of PR 1.2. Once PR 1.2 is officially released (*), the packages can also be installed on the device. MADDE and Qt Creator support is pending.

Following a request from the Maemo community, the experimental Qt packages were renamed from libqt4-maemo5 to libqt4-experimental. The packages still install to /opt/qt4-maemo5, so they don’t clash with the device’s Qt. This ensures that installing the experimental Qt 4.7 packages won’t have bad side effects on existing apps.

How to install

We suggest to remove the old libqt4-maemo5 packages first:

fakeroot apt-get remove .*qt4-maemo5.*

After downloading the experimental packages, install them with the dpkg utility:

x86 Scratchbox target:
fakeroot dpkg -i *i386.deb

ARM Scratchbox target:
fakeroot dpkg -i *armel.deb

The road ahead

Once PR 1.2 comes with official support for the stable Qt 4.6.2 version, experimental Qt 4.7 snapshots will be pushed to the Maemo extras-devel repository. If everything goes well, the release after PR 1.2 will ship with Qt 4.7.x (where “x” is whatever patch release is available at that time).

Feedback and contact

As always, you can use the qt-maemo-feedback mailing-list (info) or the IRC channel #qt-maemo at irc.freenode.org to get in touch with us.

Happy experimenting!

---

(*) Sorry, we don’t know the PR 1.2 release date either

The amount of memory Innodb will require for its data dictionary depends on amount of tables you have as well as number of fields and indexes. Innodb allocates this memory once table is accessed and keeps until server is shut down. In XtraDB we have an option to restrict that limit.

So how much memory can it really take ? Here is some production stats from real system:

In this case The memory stats from SHOW INNODB STATUS look like:

So considering about 700M were allocated for dictionary cache in this case we have about 15KB per Table or 1.5KB per Index. These numbers will change a lot depending on your table structure - This given schema had many tables with 50+ columns but I believe it can be helpful as ballpark figure.

Also note Dictionary Cache is not included in "Total Memory Allocated" in SHOW INNODB STATUS any more, as that allocations were moved to use malloc() and so Innodb does not track them any more.

---

Entry posted by peter | 2 comments

Add to: | | | |

I've got a cool new Amazon S3 feature to tell you about, but I need to start with a definition!

Let's define durability (with respect to an object stored in S3) as the probability that the object will remain intact and accessible after a period of one year. 100% durability would mean that there's no possible way for the object to be lost, 90% durability would mean that there's a 1-in-10 chance, and so forth.

We've always said that Amazon S3 provides a "highly durable" storage infrastructure and that objects are stored redundantly across multiple facilities within an S3 region. But we've never provided a metric, or explained what level of failure it can withstand without losing any data.

Let's change that!

Using the definition that I stated above, the durability of an object stored in Amazon S3 is 99.999999999%. If you store 10,000 objects with us, on average we may lose one of them every 10 million years or so. This storage is designed in such a way that we can sustain the concurrent loss of data in two separate storage facilities.

If you are using S3 for permanent storage, I'm sure that you need and fully appreciate the need for this level of durability. It is comforting to know that you can simply store your data in S3 without having to worry about backups, scaling, device failures, fires, theft, meteor strikes, earthquakes, or toddlers.

But wait, there's less!

Not every application actually needs this much durability. In some cases, the object stored in S3 is simply a cloud-based copy of an object that actually lives somewhere else. In other cases, the object can be regenerated or re-derived from other information. Our research has shown that a number of interesting applications simply don't need eleven 9's worth of durability.

To accommodate these applications we're introducing a new concept to S3. Each S3 object now has an associated storage class. All of your existing objects have the STANDARD storage class, and are stored with eleven 9's of durability. If you don't need this level of durability, you can use the new REDUCED_REDUNDANCY storage class instead. You can set this on new objects when you store them in S3, or you can copy an object to itself while specifying a different storage class.

The new REDUCED_REDUNDANCY storage class activates a new feature known as Reduced Redundancy Storage, or RRS. Objects stored using RRS have a durability of 99.99%, or four 9's. If you store 10,000 objects with us, on average we may lose one of them every year. RRS is designed to sustain the loss of data in a single facility.

RRS pricing starts at a base tier of $0.10 per Gigabyte per month, 33% cheaper than the more durable storage.

If Amazon S3 detects that an object has been lost any subsequent requests for that object will return the HTTP 405 ("Method Not Allowed") status code. Your application can then handle this error in an appropriate fashion. If the object lives elsewhere you would fetch it, put it back into S3 (using the same key), and then retry the retrieval operation. If the object was designed to be derived from other information, you would do the processing (perhaps it is an image scaling or transcoding task), put the new image back into S3 (again, using the same key), and retry the retrieval operation.

Update (for HTTP protocol geeks only):

I’d like to provide clarification regarding our choice of the HTTP 405 (“Method Not Allowed”) status code. Although 410 (“Gone”) may seem more appropriate, the HTTP 1.1 spec says that “this condition is expected to be permanent” and that clients "SHOULD delete references to the Request-URI". In other words, the 410 status code indicates that the object has intentionally been removed and will not return. That is not necessarily true when data is lost. The object owner may wish to resolve the data loss by reuploading the object, in which case it would have been inappropriate for S3 to return a 410 status code. We believe that 405 is most appropriate because other methods (e.g. PUT, POST, and DELETE) remain valid for the object even if the object’s data has gone missing. The object’s name (its URI) remains valid, but the data for the object is gone. The 422 and 424 status codes are specific to WebDav and don’t apply here.

We expect to see management tools and toolkits add support for RRS in the very near future.

You can use either storage class with Amazon CloudFront, of course.

I anticipate many unanticipated uses for this cool new feature; please feel free to leave me a comment with your ideas.

-- Jeff;

PS - check out Amazon CTO Werner Vogels' take on RRS. His post goes in to a bit more detail on how S3 was designed so that it will never lose data -- "Core to the design of S3 is that we go to great lengths to never, ever lose a single bit. We use several techniques to ensure the durability of the data our customers trust us with..."

Whether your company has tons and tons of small files from image and video libraries or is looking to implement business continuity, disaster recovery plans or data archiving plans, Rackspace Cloud Files(tm) is the perfect solution for you. Storing your data in house is risky and expensive, so why take that risk? We’ve developed a robust utility storage platform that allows users to easily and securely transfer their files into the Cloud. It’s time to utilize the Rackspace Cloud for online secure storage. And for a limited time, we’re offering the first 100GB free for the first 30 days.* Read more to learn why you should sign up.

Unlimited, On-Demand, Pay for What you Use

With our distributed architecture, users can transfer an unlimited number of files onto our platform seamlessly. Once you’ve set up your account, your files can be transferred and living in our Cloud in a matter of minutes. Just like our Cloud Servers offering, you pay only for what you use. If you are running a development project or marketing campaign, once it’s complete, you can remove the documents from Cloud Files and stop paying – you’re are committed to only what you’ve used.

Secure and Easy To Use with open API access

We’ve made it easy to upload files through our control panel. First, it is simple to upload your data to private containers. Then Cloud Files will ensure that traffic generated by application calls to the data utilizes SSL to establish a secure, encrypted channel. This ensures data (usernames, passwords, and content) cannot be intercepted and read by a third party. You can also create public containers where your media files receive a public URL. We also have a robust API allowing developers to build their own unique solutions on top of Rackspace’s infrastructure. Cyberduck is an example of an application that was built using our Cloud Files API. I actually use Cyberduck to transfer images to Cloud Files for this blog. Here’s a quick video – http://screenr.com/gss

Place your Content Close to your Users and Serve it at Blazingly Fast Speeds

Cloud Files automatically includes access to the Limelight Content Delivery Network (CDN) – no long-term contract is required! We’ve partnered with the leading CDN provider to provide users access to over 70 points of presence from all over the globe. More than 9 petabytes of storage and 900+ direct access ensures your content sits at the location that is closest to them, ensuring a faster response time for your audience. But don’t just take our word for it. Chris Meller, Rackspace Cloud customer, performed an analysis comparing Amazon CloudFront vs. Rackspace Cloud Files CDN performance. He quotes: “After 24 hours of monitoring the Cloud Files average response time was one tenth that of Amazon CloudFront’s and its slowest response time was still 40% faster than CloudFront’s average.”

Transcode your Videos with Encoding.com

We have partnered with Encoding.com to bring you a very simple way to manage and serve your videos online to web and iPhone / iPad users. Within 5min, you can create a cost-effective video workflow as powerful and flexible as any of the expensive solutions currently available. And to make it even easier, Encoding.com is offering a free month of encoding (a $50 to $300 value) to all Rackspace customers who sign up now. Please click here to learn how to bring your video online in 5 simple steps and take advantage of this offer.

Service Matters in the Cloud

We’re certainly not the only provider delivering storage. TechCrunch just announced a rumor that Google will be launching a cloud storage solution that will compete directly with Amazon’s S3. We think this will provide great competition to Amazon in the commodity market and welcome the validation that Google will bring to Cloud Storage. One thing remains true, while competitors compete on features and price, we lead on service. What does this mean? You can call us, catch us in chat or email us any time of the day, any day of the year whenever you run into a technical issue. The support we provide is included with the services that you use. You won’t pay an extra $100 or $400 per month to enjoy Cloud Files Fanatical Support.

What Customers Are Saying

Customers from media publishers to social media sites to SaaS providers use Cloud Files to store large amounts of content.

Australian-based Conversant Media is a great example of a forward-looking company ready for the next evolution of online publishing. They made the switch from MediaTemple to Rackspace Cloud Files specifically to utilize our integrated CDN technology. Co-founder Zac Zavos quotes:

“Our sites are very image heavy.  As a result, they tend towards appearing a little sluggish to load for our users. Our focus this past quarter has been on site performance. The first step was to utilize Cloud Files, which offers the Limelight CDN. This was important to us as it has an AU node and a large portion of our traffic is in Australia. It means that although the server is in the US, our users in Australia will experience Australian-like download speeds of the files.”

Tweetphoto is a photo-sharing platform for the real time web allowing users to share photos on Twitter, Facebook and through your favorite desktop and mobile clients, for free. TweetPhoto was looking to bring a new level of functionality and bandwidth-intensive features to photo sharing, while addressing the performance challenges experienced with other photo sharing platforms. Rodney Rumford from TweetPhoto quotes:

“Cloud Files has allowed us to store photos on the Cloud and make them easily accessible throughout the world using the CDN technology.”

As of November 2009, TweetPhoto was transferring around half a million images to Cloud Files each day!

Freshbooks, a leader in online invoicing and bookkeeping for professionals is a long time Rackspace customer, using a variety of our hosting services, including Cloud Files. Mike McDerment, CEO of Freshbooks, quotes:

“By moving to The Rackspace Cloud, we can continue to deliver more value to our customers at lower costs.”

FreshBooks has over 1,000,000 users and growing. A typical customer gets paid an average of 14 days faster using their service, and sixty percent of FreshBooks users are collecting more money through their service due to timelier billing and improved record keeping made available via Cloud Files.

These are just a few examples of how companies are utilizing Cloud Files. If you’re looking for a place to store tons of images, remember to check us out. It’s cheap to get started and easy to set up. Check out our introduction video to see how easy it really is:

*Offer good 5/19/2010 – 5/31/2010.  Usage must occur within 30 days of sign up.  Must use Promo Code:  100Files

Today we are excited to announce a collection of high quality open source web fonts in the Google Font Directory, and the Google Font API to make them available to everybody on the web. For a long time, the web has lagged print and even other electronic media in typographic sophistication. To enjoy the visual richness of diverse fonts, webmasters have resorted to workarounds such as baking text into images. Thanks to browser support for web fonts, this is rapidly changing. Web fonts, enabled by the CSS3 @font-face standard, are hosted in the cloud and sent to browsers as needed.

Google has been working with a number of talented font designers to produce a varied collection of high quality open source fonts for the Google Font Directory. With the Google Font API, using these fonts on your web page is almost as easy as using the standard set of so-called “web-safe” fonts that come installed on most computers.

The Google Font API provides a simple, cross-browser method for using any font in the Google Font Directory on your web page. The fonts have all the advantages of normal text: in addition to being richer visually, text styled in web fonts is still searchable, scales crisply when zoomed, and is accessible to users using screen readers.

Getting started using the Google Font API is easy. Just add a couple lines of HTML:

<link href='http://fonts.googleapis.com/css?family=Tangerine' rel='stylesheet' type='text/css'>



body { font-family: 'Tangerine', serif; }

The Google Font API hides a lot of complexity behind the scenes. Google’s serving infrastructure takes care of converting the font into a format compatible with any modern browser (including Internet Explorer 6 and up), sends just the styles and weights you select, and the font files and CSS are tuned and optimized for web serving. For example, cache headers are set to maximize the likelihood that the fonts will be served from the browser’s cache with no need for a network roundtrip, even when the same font is linked from different websites.

These fonts also work well with CSS3 and HTML5 styling, including drop shadows, rotation, etc. In addition, selecting these fonts in your CSS works just the same as for locally installed fonts, facilitating clean separation of content and presentation.

The fonts in the Google Font Directory come from a diverse array of designers, including open source developers and highly regarded type designers, and also include the highly acclaimed Droid Sans and Droid Serif fonts, designed by Ascender Corporation as a custom font for Android. We invite you to browse through the directory and get to know the fonts and designers better. Since all the fonts are open source, you can use them any way you like. We also have a separate project hosted on Google Code for downloading the original font files. Since they’re open source, they can be used for just about any purpose, including for print.

We’re hoping designers will contribute many more fonts in coming months to the Google Font Directory. If you’re a designer and are interested in contributing your font, get in touch with us by completing this form.

To showcase the Google Font API, Smashing Magazine has relaunched their site using the open source Droid font hosted by Google. We’re excited about the potential for integrating the Google Font API into many types of publications and web applications. For example, the new themes for Google Spreadsheet forms are a great example of a rich visual experience using web fonts.

This is just the beginning for web fonts. Today, we’re only supporting Western European languages (Latin-1), and we expect to support a number of diverse languages shortly.

By Raph Levien & David Kuettel, Google Font API team

At Yahoo!, the first users of Apache Hadoop were researchers developing new algorithms or manually shifting through huge data sets. These users threw away most of their code after a few weeks or months, and the little code they carried forward was not subject to rigorous quality procedures. Thus, these early users cared more about new features and scalability improvements in Hadoop than they did about backward compatibility.

This early focus on bigger-and-better helped Hadoop become the powerful platform it is today. However, over the years, both inside and outside of Yahoo!, Hadoop is increasingly being used to run large, long-lived, enterprise-class applications. Porting these applications to non-compatible upgrades of Hadoop is an arduous, expensive task that distracts teams from finding new and better ways of using Hadoop to bring value to their companies. Today, Hadoop users are demanding backwards compatibility and interface stability; these features are necessary for the next growth phase of Hadoop, as it gains wider enterprise adoption.

Interface Classification

Over the last year, as part of our plan to provide stronger backward compatibility, we have tagged interfaces in Hadoop to denote their compatibility contract for future releases. An interface can be a Java API, a configuration variable, the parameters or output of a command, metrics variables, and so on. Java APIs are tagged using Java Annotations; other types of interfaces, such as configuration options and output formats, are tagged using informal documentation conventions. The upcoming release 0.21 of Hadoop will be the first to expose this classification.

The classification system is derived from OpenSolaris and from our own internal system at Yahoo. The system distinguishes two important aspects of an interface from the perspective of backward compatibility, the audience of the interface, and the stability of the interface:

• The audience (or scope or visibility) of an interface denotes the potential customers of the interface. In addition to the more obvious public and private designations, the audience taxonomy also includes a limited-private category for hooks exposed to peer frameworks or systems.
• The stability of an interface denotes when changes can be made to the interface that break compatibility. Again, a binary choice between stable (guaranteed not to change incompatibly) and unstable (guaranteed to change) is not sufficient. Interfaces may also be marked as evolving, intended for use by early adopters validating their suitability. An evolving interface is marked as public only after it’s been used internally in the system and is close to being stable. These dimensions allow early adopters to gauge the risk in relying on a new interface.

The following chart should help you understand what this tagging system means to you:

• If you are an application developer, stick to public-stable interfaces. If you are early adopter, you may use a public-evolving interface, but be aware that the interface may change slightly in the future, forcing a change to your application.
• If you are a framework developer on Hadoop, you can of course safely use any of the public interfaces, but can also use limited-private interfaces targeted to your framework. For example the Hadoop RPC layer provides limited-private interfaces for HDFS and MapReduce.
• Stay away from the private interfaces unless you are an implementer of the actual sub-component. Pay attention to any private interfaces that are stable. While the vast majority of private interfaces are marked as unstable, a few are marked as stable to warn that breaking their compatibility should be done after only serious consideration (and community discussion). For example the internal HDFS and MapReduce protocols are marked stable to support rolling-upgrades (a feature we would like to add to Hadoop in the near future).

More details are provided in the Apache Hadoop API classification document.

The Larger Plan for Compatibility

This classification system for interfaces is part of a larger, multi-step plan for backwards compatibility, which has been in work for the last two years (HADOOP-5071).

The first step was to clean up the package structure of the Hadoop source base (HADOOP-2884). While the high level structure was fine, finer-grained packages better reflect the abstractions of the underlying system architecture. For example, the HDFS package structure did not distinguish the internal abstraction of block storage and namespace, and there were several cases of layer violations within and across packages. In cleaning-up the package structure, we not only provided a better foundation for future work, but also started the process of identifying and separating public interfaces from the private and limited-private ones. We also changed the documentation to reflect this separation.

At the same time, key Hadoop interfaces were redesigned to allow them to evolve compatibly without limiting innovation in the framework. These key interfaces, which also happened to be where Hadoop has faced its greatest compatibility issues, were the APIs to the HDFS file system (HADOOP-4952) ) and the MapReduce framework (HADOOP-1230).We will discuss the redesign of these interfaces in a future blog.

Finally, we have also started to address the wire compatibility problem, i.e. the compatibility of Hadoop components across RPC boundaries and versions. Hadoop, surprisingly, does not offer wire compatibility. To address this limitation, Yahoo started the Avro project (led by Doug Cutting), which was open-sourced in Apache last year and will be incorporated into Hadoop’s RPC system over time..

Acknowledgments

The source re-factoring was done by Sanjay Radia and Raghu Angadi. The interface classification project was initiated by Sanjay Radia and derived mostly from OpenSolaris’ classification scheme. Jakob Homan designed the use of annotations to tag the interfaces. Suresh Srinivas, Tom White, Arun Murthy, Owen O'Malley, and Sanjay Radia defined the annotations for the various parts of the Hadoop system. Tom White helped drive the labeling effort to its conclusion in 0.21, particularly its integration with Hadoop’s javadoc. Doug Cutting was the lead for the Avro project.

Sanjay Radia
Hadoop Team, Yahoo!

Interest free loan scheme from Caan the Dragon extended to sellers

Dragon Den star James Caan has launched a second major initiative to pump life into the UK housing market.

Last year Caan’s property portal, Look4aProperty.com launched a £1bn fund to offer interest free loans to people desperate to get on to the property ladder.  It is designed to help people afford stamp duty, legal fees and other expenses linked to buying a home.

The idea has proved so successful look4aProperty is now funding a similar scheme to help property sellers.

The Dragon’s Den star is launching the new initiative at Newton Fallowell, a leading estate agency chain based in Mrs Thatcher’s former constituency, Grantham, Lincolnshire.

“We’ve had impressive take-up across the country from people wanting to borrow that extra money to cover the costs of moving or buying their first home. Now we are providing loans – interest free to people wanting to sell,” says Caan.

“It was the obvious next step to give the country’s housing market a boost. We set out to bring liquidity to the market. To do that, we need people buying and selling.

Despite ultra low interest rates the housing market has been frozen by fear since the economic crisis gripped the country in 2007.

Caan, who runs a number of property investment businesses, says people have been holding back because feel they aren’t able to fund many of those unavoidable expenses associated with buying and selling property. “These items represent an expense too far for many people. Look4aProperty|money product is designed to remove that obstacle.

The scheme is available to anyone through a network of estate agents who have signed up to the fund across the UK.

Despite signs that property prices are rising – one report from the Nationwide put the rise in the last year at 10 per cent – the market is not moving. Caan believes the more incentives to inject some liquidity into the market the more the economy will benefit.

• Bookmark on Delicious
• Digg this post
• Recommend on Facebook
• Buzz it up
• Buzz it up
• Tip on Hyves
• Share via MySpace
• Share on Orkut
• share via Reddit
• Share with Stumblers
• Tweet about it

James Caan Opens Property Loan Fund to Sellers is a post from: Look4aProperty.com's Blog

The rise and fall of Reginald Perrin springs to mind (actually the fall and rise, but who is checking).   Anyone under 30 won’t know what I am talking about and even I struggle to remember the series (let alone the books), but the HIPs fiasco reminds me of Reginald Perrin.

Briefly, Reggie was a man trying to make change, but his plans rarely worked the way he had anticipated.  He created bizarre and absurd ideas and plots (round dice and square hoops) which were always met with an even more bizarre reaction from the world at large.   His reaction to this is then more absurd (hiring incompetents to destroy the business for example).  Naturally, all his plans backfire and one is left with an underlying theme of ‘that’s life’.

Anyway, as I wrote a little while ago HIPs started life for all the right reasons, but were then watered down into the dishcloth of a product that will be scrapped today or tomorrow.    The pilot scheme commenced in 1999 would you believe and when Openbook (a company involved in the pilot scheme pulled out of HIPs in September 2008, pundits were all predicting the fall of HIPs.   Indeed, I wrote in September 2008  ‘…One thing is sure, the government will not be coming to the rescue. indeed, the predicted change in government may even go one stage further and scrap them altogether. The HIPS crowd are a powerful lobby, but I for one can not see them persuading the government to rescue the industry….’

So here we are and over on the community we are running a poll to see whether people are happy or sad that HIPs are being terminated.  Not one person has said sad, but I for one am sad.  I am sad for the missed opportunity of the original idea, I am sad that no one has looked upon this as a new opportunity, I am sad for the future government bungling that will inevitably follow and most of all I am sad for those in the HIP industry, who never stood a chance.

The governemnt spin already seems to be that they are scrapping it as it was all about another tax on us (the Telepgraph) tied to increasing council tax.  Aren’t we lucky the Tories and Lib Dems (what do we call this outfit??  the Toribs – you heard it here first) got in when they did!

Anyway, as Reggie might say, ‘that’s life’.

Hi Hadoopers

Thanks to close to 300 developers who came this week to Yahoo! for our monthly Hadoop User Group meeting. The energy in the packed room was phenomenal and conversations continued long after the formal sessions.

Hundreds of Hadoop Fans Flock to Yahoo! for the May Hadoop User Group

A few lucky winners received free tickets to the upcoming Hadoop Summit 2010 (June 29th, at the Hyatt Regency, Santa Clara). Congratulations to those winners – everyone else please register here

The event started with Alan Gates from Yahoo! who described the new features and work done in Pig 0.6 and 0.7 including the Hadoop’s compatibility plan, described in more details in this post.

 

Nathan Marz from BackType presented a cool demo of how easy it is to query existing data stores using Cascalog, a query language for Hadoop. Nathan described how queries can be written as regular Clojure code and combined with Cascading. Be sure to watch the demo as part of the video below.

 

Next was Dmitriy Ryaboy, an engineer at Twitter and a Pig committer. Dmitriy walked us through the extensive use of Hadoop eco-system at Twitter. He explained what are the challenges they face in processing 55 million tweets a day and why they chose to use Hadoop, Pig and HBase. Dmitriy introduced the Elephant Bird libraries and shared interesting tips for dealing with Big Data.

 

We concluded with Tom White from Cloudera who walked us through the release plans for Apache Hadoop 0.21 including the Source Compatibility project described in the Yahoo! hadoop blog

Video is coming soon...Stay Tune!

 

We at Yahoo! are embracing Hadoop – we share the challenges presented by Twitter for processing massive data sets and continue to invest heavily in the technology and the community. We love to hear about the growing ecosystem and solutions like Cascalog.

Please join us at the Hadoop Summit to continue the conversation.

As always, we are looking for exciting technologies and experiences you want to share. Please contact me via the Hadoop Bay Area User Group Meetup page.

Note that we will not have a meetup in June due to the Hadoop Summit . See you all on July 21st, 2010. Registration is available here, agenda will be published soon

 

Dekel Tankel
Director, Product Management
Cloud Computing at Yahoo!

James Caan uses launch of new billion pound property fund as Labour’s HIPs scheme is postponed

Serial entrepreneur and television star James Caan says the Government’s quick thinking in axing Labour’s controversial HIPs scheme is a step in the right direction.

Caan, who was in the East Midlands to launch the second phase of his billion pound interest free loan scheme, welcomed today’s announcement by housing minister Grant Schapps.

“This is good news for people desperate to move up the property ladder. The HIPs scheme was a major barrier to fluidity in the housing market,” says Caan. “I believe this is a wise move by the Government and will bring much needed liquidity back.”

James Caan’s billion pound fund, called Look4aProperty|money, offers interest free loans to people buying or selling property.

Caan’s business partner in the venture, Aaron Turner, the CEO of Look4aProperty.com, says: “HIPs have proved to be prohibitive to property coming onto the market and movements within the markets is what is needed by the UK economy at this time.”

Despite ultra low interest rates the housing market has been frozen by fear since the economic crisis first gripped the country in 2007.

Caan, who runs a number of property investment businesses, says people have been holding back because they feel they aren’t able to fund many of those unavoidable expenses associated with buying and selling property.

“These items represent an expense too far for many people. The Look4aProperty|money product is designed to remove that obstacle,” says Caan.

The scheme is available to anyone through a network of estate agents who have signed up to the fund across the UK.

• Bookmark on Delicious
• Digg this post
• Recommend on Facebook
• Buzz it up
• Buzz it up
• Tip on Hyves
• Share via MySpace
• Share on Orkut
• share via Reddit
• Share with Stumblers
• Tweet about it

Dragon’s Den Star Welcomes Government HIP Replacement is a post from: Look4aProperty.com's Blog