Recently in Metrics and ROI Category

Every so often I decide that I need to skim through my keyword feeds in Bloglines (more to get my number of unread posts down below 4 figures than in the expectation of finding anything really useful). This morning, I was skimming through one of my ROI feeds, following a couple of links, much of the same stuff - yes ROI is probably important, yes we can work out how much paper is being saved, probably, no we can't get into anything more convincing.

And then, like a 40 watt bulb flickering to life I had an epiphany. Given that calculating the ROI of the squishy stuff is really really really hard, and generally it's the squishy stuff that's the most important (more engagement, better communication, accelerated innovation, more reuse of existing information, talent retention, productivity increase, etc), then why not use a tool that seems ideally geared towards arriving at a close approximation of reality without needing or being able to measure directly? I am, of course, talking about crowd wisdom.

Here's how it could work:

• Implement your hard-to-measure-the-ROI-of initiative (for example, redesign your intranet to be more usable, or implement social bookmarking, or an internal blogosphere, or whatever)
• Set up a mechanism to aggregate the wisdom of the crowd - maybe a prediction market (although aren't they usually associated with events that have a measurable outcome such as "Who will be the next President", "Who will win the World Cup, etc" ?). Maybe all that's needed is a simple survey asking people how much they thought had been saved, or how much extra revenue earned.
• Add and divide, and obtain average. I'm sure there are plenty of slightly more rigorous statistical techniques here for processing the numbers (eg throwing away outliers, etc), but essentially, that's it.
• The hard part is now trying to work out whether that number bears any actual resemblance to reality - but maybe that's easier to judge when there's a number in front of you.

I admit that this doesn't let you know the ROI up front (would that be worthwhile, asking employees to predict the ROI before implementing ? Is that too complicated a question to ask ?), but it might give you some justification of impact.

What do you think ? Stupid idea, or just impractical ?

[tags]ROI, measurement, fuzzy, squishy, metrics, aggregation, crowdwisdom, wisdomofcrowds, predictionmarkets[/tags]

A quick roundup of some of recent links to the ROI of blogging. In the main it's external blogging that is under discussion.

• The ROI conundrum
• Signs of a Healthy Blog: Resonation
• Blogging ROI: How Do You Know ?
• Naked Answers (metrics rather than ROI)
• The ROI of blogging , and whether Jonathan Schwartz's blog pays for itself
• Blogging ROI - an empirical approach to measuring effect

[update Constantin Basturea proposes we maintain a wiki page for Blogging ROI - good idea I think]. Technorati Tags: metrics, ROI, blogs, blogging, measurement

Ok - I'm at blogging4business. Come and talk to me if you're interested in internal blogging or blogging full stop - my name is Anu Gupta, and I'm a blogger ! Suw is moderating a panel on blogs and ROI - be interested to see if we get anything beyond what's been out there for a long while. Julian Smith of JupiterKagan - some stuff about tangible and intangibles - conversation blah blah blah, better search engine listings..blah blah.. Anthony Mayfield - gets asks some hard ROI questions..."don't have all the answers..." surprise surprise. changes to means of production and distribution...personal reputations are more important (WHEN WERE THEY NOT ??!!) Showed an SNA slide - talking about "connections..." says that it's a good way to show ROI ??!! yup - ROI blurred over again...no surprise there. Good - let's see what Lee has to say. Ahh - command and control vs fluffy bunny networks...is ROI actually relevant...very hard to measure. New class of software - "the i is small...so the R is easier to feel". Two key concepts - disintermediation and battling against reintermediation..."PR agencies picking up shiny toys...speaking to clients...we will do this for you" . "The Power of We " Large companies are not talking about the ROI...but the transformational potential. "If you start blogging on the external interface without doing anything internal, then you're just marketing..start internally." "The i is people, the R is better people.." - I like that ! Olivier Creche - not a passionate blogger - uses weblogs as a tool for communication. Suw takes us back to the day, and is talking about measurement - a focus on numbers, page impressions, clicks throughs..."it's not the same as blogging - it's about quality and not quantity - how do you get people to stop thinking about big numbers ?" Antony back on the visual tip...Julian "myopic focus on immediate behavioural response...and it's difficult to move people beyond that. But starting to see a shift away from behavioural to attitudinal...in more sophisticated marketers...branding metrics How do you measure culture change ? Lee - "culture change isn't a goal..making money is the goal". Separation of internal and external is key...Hugh's porous membranes." Technorati Tags: blogging4business

David Ferrabee joins the ranks of those searching for a way to rigourously measure communication - at least I hope he wants to get real measures, as opposed to some of the easier to generate but essentially meaningless numbers that can get floated around. The major problem with comms measurement, and what makes it so difficult, is that the thing being measured is really fuzzy - typically impact, or behavioural or attitude change, and is easy to misreport and derive incorrect conclusions, especially when so few companies seem willing to adopt standard experimental techniques to isolate what has caused the effect. Contrast this with epidemiological population studies, or clinical drug trials, in which strenuous efforts are made to ensure that biases are eliminated, variables are unintwined and effects are associated with the correct causes. We could do worse than start from there. Technorati Tags: communication, measurement

I've been following with some interest the lively debate that's been going on between John Wagner and Katie Paine about the value of measuring PR communication.

John Wagner essentially kicked it all off with his post questioning the value of a metric devised by Katie Paine. The response from Katie was robust, and John was labelled as a menace and a dinosaur. Katie finished off her post with the following paragraph.

And to John's claim that PR can't be measured, we offer the Southwest airlines case study that ties millions of dollars in ticket sales straight back to specific press releases.

So, I looked at the case study, and indeed, by using unique URLs in press releases (akin to the technique used by mass mailers)  SouthWest was able to directly relate sales to the PR.

Seems like a powerful case study – communication finally directly linked to revenue – measurement wins !

Unfortunately, I’m not sure that reality is quite that simple – or maybe I’m missing something – cause and effect is rarely that simple. The case study does indeed demonstrate a technique for measuring sales coming from press releases. The problem is that it does not (as far as I can tell) demonstrate anything about the effectiveness of the press releases. And that’s what we really want to measure and understand. Sure, we can see that x dollars worth of sales were generated, but so what ?  Maybe those sales would have happened anyway – or maybe 90% of those sales would have happened without a dollar of extra PR or advertising expense, just by lowering the prices and letting the various search and fare engines find them.

To truly understand effectiveness, we have to improve the rigour of the design and analysis, and start moving to studies that are more statistically valid and provable. The field of epidemiology has much to offer here, as epidemiologists spend a lot of time designing clinical trials and studies, and then analysing them to understand effectiveness.

The ideal strategy is to have a control group – a similar (in terms of population and circumstance) sample used as the basis of comparison and which is not exposed to the intervention (ie the press releases in this instance). Comparing the difference in outcomes between the test and control groups allows isolation of the effects of the intervention. If control groups, or other tried and tested techniques aren’t used, one runs the risk of incorrectly deriving effect from cause.

So, while I think measurement is incredibly important, there doesn’t seem to be much point if we’re just doing it to generate numbers that will look impressive and seem scientific, but would be deemed as hopelessly unreliable when judged by people with a  more rigourous outlook. Unfortunately, doing proper measurement involves more effort, more cost and more time, and is often seen (as someone mentioned in a blog comment somewhere) as consultants trying to add on services that aren’t necessarily needed.

More on this later (and I thank my partner Marie-Claude Boily for giving me a great deal of insight over the years into how large scale studies and trials are designed and tested – and also for “peer-reviewing” this little post)

I should state that all I’ve got to go on is the limited information contained in the case study - and if more extensive design, testing and analysis were performed, then that would change things. However, all we can say is that there is seems to be evidence of association – with the design as stated, the effect of the PR and the other variables, such as the actual reduction in price, are confounded – impossible to disentangle and isolate.

[Update – Shel Holtz also posts some views on measurements]

Technorati Tags: measurement, metrics

Despite outward appearances of fluffiness, I am, in my heart, an unreconstructed believer in measurement. Some might disagree with me, and that's ok !

I'm trying to get my head around some measures for engagement, and am interested in seeing if anyone has some numbers for page views versus comments . Ultimately I think it would be interesting to see if there was some median value, and then to start analysing blogs which "over" or "under" perform compared to this medium, and see why.

Now, I know, really I do, that blogs that differ from my golden average aren't really under or overperforming - so please ignore the judgemental type language. There will also be massive differences depending on the sector, internal versus external, length of time blogging, frequency of blogging, whether comments are turned on or off (!), amount of spam allowed to fester per post, etc etc. But, right now, I have absolutely no sense of any of these numbers beyond the meager numbers of scale|free, where for the record, I'm getting 1 comment per 137 page views, and would like to know more.

Post here or trackback, and I'll summarise later for all to view. Please remember that this isn't a study, and has no significance beyond getting a massively biased gut-feel (if I get a response at all !) of this, very singular, metric.

Ta !

Technorati Tags: metrics, blogs