Measuring evaluation

Posted on: September 21, 2008

I’ve just chaired the judging of the AMEC awards for PR measurement and evaluation. I wouldn’t dare reveal the results as they won’t be announced until November 19. But I’d like to share my thoughts on the state of the PR measurement and evaluation business. Here goes:

– When media evaluation is combined with qualitative research on political and social issues, very powerful and valuable insights can be offered.

– There is growing confidence amongst evaluators with more offering well-based consultancy advice to clients. That shows growing maturity in the business.

– Compared with the 2007 AMEC awards, which I also chaired, online PR and media is now being included in many evaluation programmes. But much needs to be done on valid and reliable analysis.

– Coordinated cross-country analysis is developing, with much greater allowance for national media culture differences. For sophisticated organisations, there is less emphasis on uniform delivery of messages across greatly varying cultures.

– More debate is needed on the use of ‘Return on Investment’ as a measure. In many instances, it is strictly applied as a financial benchmark which ignores the contribution that PR makes to organisational strategy and efficiency of operations. The paper by Fraser Likely, David Rockland & Mark Weiner published on the Institute of PR website in 2006 would be a good basis for discussion.

– It’s disturbing to find organisations having prime PR objectives such as generating a volume of coverage (e.g. “1000 clips a year”) but ignoring the communication of key messages or influence on attitudes or stakeholders.

– There needs to be much greater understanding of statistical terms such as “correlation”. To imply that a distributed message is correlated with behaviour, such as product purchase, is the old problem of the “substitution game” in which outputs are presented as outcomes. If evaluators claim correlation, then they have to show the reliability and validity of their methods.

Overall, the AMEC awards when announced in November will demonstrate the growing maturity of the international PR evaluation business. There are some really interesting and challenging gold award winners, with a few of them being “great leaps forward”. You’ll just have to wait till they are announced!


5 Responses to "Measuring evaluation"

Sounds like things have improved since the last time I judged it. I found very little good news two years ago. Waaay too many measures that have nothing to do with the objectives.It would be interesting to compare the entries between the AMEC awards and the Golden Ruler of Measurement

Hi Tom,

You tease you!

But seriously, a great post giving a nice perspective of how things are changing over time in our industry.

At Metrica we totally agree with your comments about there needing to be a greater understanding of statistical terms. We find that too many of the newer companies in the media evaluation field – I’m thinking particularly of the press clipping agencies who now offer very basic evaluation – are particularly guilty of doing this.

Overall, there is still a long way to go with education in public relations on why measurement is important, and, for that matter, planning too. The two are intrinsically linked – anyone starting out on a new campaign should begin by thinking about what they are trying to achieve – which will then formulate the correct metrics that they need to analyse, which in turn will help them with their planning process.

As media outlets explode and our target audiences fragment ever smaller, these basic components of an effective PR strategy are becoming more and more crucial.

We look forward to November, and hope to see you on the night.

Regards, Richard

I wouldn’t be so quick to dismiss pure quantitative objectives (e.g. 1000 clippings/year). It really depends on the market you are dealing with.

For example, in the UAE, 90%+ of articles in newspapers are copy/paste press releases. PR executives are tasked with fighting for column inches because, quite simply, if their client is not there, their competitor is.

When doing surveys for several clients, we have found that brand recognition is strongly correlated with amount of coverage, regardless of messages. This may not be true in a mature market, but I simply want to point out that not all markets are created equally.

With that being said, at Mediastow, we do NOT encourage clients to set that as an objective. We think that is a ‘means’ to achieve an actual objective (e.g. increase sales, etc.).

On another note, looking forward to the announcement of the awards!

Mohammed: Apologies for the delay in responding to your post, which got hooked up in the WordPress spam system.

Quantitative “objectives” aren’t objectives in terms of outcomes. They are a volume measure, usually used by clients or employers to beat the PR advisers. When considering outpout, as in media analysis, PR evaluation should be measuring the message that is communicated, as interpreted by the media, its tone and the potential (but not actual) audience that might receive it. Maybe the PR people in UAE are getting the clients/employers that they deserve if they accept the volume imperative above all. Whether it is a mature market or not, developed or developing country, is irrelevant. The role of media analysis to is interpret coverage of messages and that doesn’t need to have opt-outs on a national or regional basis.

I’d be interested to see the statistical basis on which Mediastow (or any other analysis firm) can say that “brand recognition is strongly correlated to amount of coverage”. Correlation (and sometime causation) is often claimed but I’ve never ever seen any rigorously developed data to support these claims. It’s a big difference between imputing a linkage or guessing it and giving statistical proof.

Please post any information and we can continue the discussion. (BTW, I ran a PR consultancy for 18 years and chaired the UK’s Public Relations Consultants Association for two year, so I know all about the reality of dealing with clients on the delivery and measurement of value from PR programmes. We never resorted to AVEs or volume measures of media coverage and kept our clients.)

[…] Public Relations, Uncategorized | One observation in my recent blog on the AMEC awards, Measuring Evaluation, was that the statistical term, correlation, was both misunderstood and […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 17 other followers



Subscribe in NewsGator Online


Tom Watson

Error: Twitter did not respond. Please wait a few minutes and refresh this page.

%d bloggers like this: