I've been following with some interest the lively debate that's been going on between John Wagner and Katie Paine about the value of measuring PR communication.
John Wagner essentially kicked it all off with his post questioning the value of a metric devised by Katie Paine. The response from Katie was robust, and John was labelled as a menace and a dinosaur. Katie finished off her post with the following paragraph.
And to John's claim that PR can't be measured, we offer the Southwest airlines case study that ties millions of dollars in ticket sales straight back to specific press releases.
So, I looked at the case study, and indeed, by using unique URLs in press releases (akin to the technique used by mass mailers) SouthWest was able to directly relate sales to the PR.
Seems like a powerful case study – communication finally directly linked to revenue – measurement wins !
Unfortunately, I’m not sure that reality is quite that simple – or maybe I’m missing something – cause and effect is rarely that simple. The case study does indeed demonstrate a technique for measuring sales coming from press releases. The problem is that it does not (as far as I can tell) demonstrate anything about the effectiveness of the press releases. And that’s what we really want to measure and understand. Sure, we can see that x dollars worth of sales were generated, but so what ? Maybe those sales would have happened anyway – or maybe 90% of those sales would have happened without a dollar of extra PR or advertising expense, just by lowering the prices and letting the various search and fare engines find them.
To truly understand effectiveness, we have to improve the rigour of the design and analysis, and start moving to studies that are more statistically valid and provable. The field of epidemiology has much to offer here, as epidemiologists spend a lot of time designing clinical trials and studies, and then analysing them to understand effectiveness.
The ideal strategy is to have a control group – a similar (in terms of population and circumstance) sample used as the basis of comparison and which is not exposed to the intervention (ie the press releases in this instance). Comparing the difference in outcomes between the test and control groups allows isolation of the effects of the intervention. If control groups, or other tried and tested techniques aren’t used, one runs the risk of incorrectly deriving effect from cause.
So, while I think measurement is incredibly important, there doesn’t seem to be much point if we’re just doing it to generate numbers that will look impressive and seem scientific, but would be deemed as hopelessly unreliable when judged by people with a more rigourous outlook. Unfortunately, doing proper measurement involves more effort, more cost and more time, and is often seen (as someone mentioned in a blog comment somewhere) as consultants trying to add on services that aren’t necessarily needed.
More on this later (and I thank my partner Marie-Claude Boily for giving me a great deal of insight over the years into how large scale studies and trials are designed and tested – and also for “peer-reviewing” this little post)
I should state that all I’ve got to go on is the limited information contained in the case study - and if more extensive design, testing and analysis were performed, then that would change things. However, all we can say is that there is seems to be evidence of association – with the design as stated, the effect of the PR and the other variables, such as the actual reduction in price, are confounded – impossible to disentangle and isolate.
[Update – Shel Holtz also posts some views on measurements]
Technorati Tags: measurement, metrics
Anu:
Thanks for your post ... you make the point I was trying -- perhaps unsuccessfully -- to make, and that is it is difficult to track cause and effect with something as varied and complicated as communications.
I don't believe that measurement is impossible, as some have claimed I've said, nor do I think it's unnecessary, as has always been written.
What I do believe is that generating metrics that truly don't mean anything is of NO VALUE to us as communicators. In fact, it can backfire with senior management that is savvy enough to see through our "results."
John,
Yup - I'm still trying to understand how far one needs to go to provide valid metrics, and how much can be safely sensed / intuited from a less rigorous study.
Measuring the impact of comms really isn't easy - otherwise there would be fewer arguments, and more of it - but it surely is possible (although possibly too expensive to be worth it in many cases). There's plenty of reference material from the sociological and epidemiological fields, and I wonder how much of it has really been studied...
I think in many cases it simply depends upon how comfortable the communications team is with less formal methods, and how much data is required by senior management to be satisfied that the PR department is being effective.
That may not be ideal -- especially for many PR pros who long for the vaunted "seat at the table" -- but it is reality.