Editor’s Note: Since it’s Measurement Month, we asked a quartet of senior measurement analysts to assess the state of communications measurement, including what’s being done well or badly, the ramifications of AI for measurement, how to push for better measurement and what a company’s budget priorities should be for communications measurement.
In general the four were upbeat about communications measurement and its future, though they believe there’s much work to be done. In short, there’s plenty of very good measurement going on, particularly of the integrated ecosystem that is modern communications. Still, a lot of things need improvement, particularly downgrading vanity metrics, getting brands to measure what is really important to them and connecting what’s being measured to business objectives.
Our roundtable is composed of veteran PR News columnist and Paine Publications chief Katie Paine; global managing director of the International Association for the Measurement and Evaluation of Communications (AMEC) Johna Burke; Mark Weiner, chairman of the Institute of PR’s (IPR) Measurement Commission and chief insights officer at Cision; and consultant Allyson Hugley, a member of the IPR Measurement Commission and former president for measurement and analytics at Weber Shandwick. An edited version of their remarks follows.
PR News: Are you optimistic about the state of measurement as Measurement Month approaches?
Paine Publishing
Katie Paine: I just finished a summit with 17 of the smartest people in marketing, communications and measurement and they made me feel extremely optimistic because finally, finally, finally the world is being forced to change the way it measures. The old, bad, stupid metrics we’ve been using for years are no longer fine.
The most interesting factoid of the summit, but one which everybody agreed with, was that the more sophisticated measurement programs, things like marketing mixed modeling…they’re not being adopted in PR departments, they don’t have the budgets to adopt them. Finance departments and people who run risk assessments are adopting them.
PRN: What does that mean for communicators?
Paine: If finance is paying attention to communications, that’s a good thing. It means there’s a perception of value in communications.
Accounting and senior leadership are saying, ‘Hey, we do need to put a value on [communications] and we can put a value on it. What we’re going to do is see how communications helps mitigate risk, how it provides lift to our sales force, the extent to which communications is helping us keep the people we want and attract new people and make us the employer of choice, the neighbor of choice and the vendor of choice.’
What’s making me hopeful is that people are looking at communications as an integrated, effective function that has impact on the business. That the finance department thinks communications can be measured as part of a business impact makes a huge difference to our profession.
Johna Burke: I’m optimistic because measurement continues to grow. There’s a growing audience of communicators who realize that measurement isn’t an intimating factor, but can actually stimulate success and retention among their teams when they’re more thoroughly engaged. That’s very exciting.
A lot of communicators are seeing management become more receptive because they’re providing meaningful data as opposed to counts and amounts, and clicks and likes. When communicators are able to demonstrate through data their contributions to an organization’s business objectives it gives them more credibility and an expansive role to influence greater change and overall better communications.
Sure, there’s a faction of people who say measurement, or analytics or evaluation, call it what you will, is too quantitative.
But there are practitioners who are either hiring data scientists or who are dipping their toes into becoming more accountable in their communications work by showing the validity of how things truly effect the outcome of their organization through measurement.
PRN: Should communicators be concerned when they hear communications departments are hiring data scientists? Is that communications’ future?
Burke: [Data] is definitely an element of communications. But I think communicators should be encouraged. Someone who really understands data will be able to extract insights from data that will really help communicators make a bigger impact on their organization. So, bottom line, we should all be encouraged by data, we should all be skeptical of data and we should all be empowered by data.
PRN: Allyson, are you optimistic about measurement?
IPR Measurement Commission
Allyson Hugley: I’m highly optimistic. Organizations like AMEC and the IPR Measurement Commission have made significant strides in recent years to advance the discussion around media tracking and intelligence methodologies.
The 2018 AMEC global summit focused on insights, innovation and impact and included sessions on advanced analytics, machine learning and artificial intelligence – all topics significantly influencing the future of media analysis and measurement.
The IPR Measurement Commission has called attention to timely issues such as the rise of disinformation, which has implications for how and what we measure. Another positive sign are the increased discussions around data integration.
IPR Measurement Commission
Mark Weiner: The state of PR measurement today is one in which there’s good news and not-so-good news. The good news is more and more people are measuring: With the proliferation of inexpensive (even free) technologies, more PR people have access to measurement than ever.
Standards also are rising and everyone now has a tool and data.
PRN: Is that good?
Weiner: Yes and no. Everyone has data, but there’s too much data. Smart communicators recognize that tools are a good way to begin, but they cannot evolve through tools alone. Beyond technology, the human element is essential in the form of subject matter expertise, statistical acumen and critical thinking. The new standard for PR research and evaluation is actionable insights (see diagram).
PRN: What are some issues with measurement?
Hugley: Access to data and silos remain challenges on the agency and the client side. Advancing evaluation methods requires harnessing the power of data across disciplines, agency networks and the client-side enterprise. The vast majority of data within organizations is still largely un-utilized or underutilized.
Weiner: The diminished standard of what’s good enough. Free and low-cost tools are better than nothing, but suffer for a lack of content relevancy and data accuracy, which leads to false conclusions and irrelevant findings.
There’s also confusion in the marketplace about what automation can and cannot do, false claims about the presence of AI, a purposely-blurry distinction between proving PR’s value and generating PR ROI.
On a more grassroots level, measurement still is a matter of unwillingness rather than inability. An agency president put it perfectly: “I don’t measure because I’d rather forgo being a proven success in exchange for never being a proven failure.” In general, there’s too much fear in PR and it undermines our professionalism.
PRN: Same question, Katie. What are some lingering issues for measurement?
Paine: Vanity metrics and activity metrics without a point. For example, say you’re an insurance company. Why are you measuring impressions overseas if you’re not allowed to sell outside the United States? The bad stuff is people who count things that don’t matter and who in pursuit of big numbers lose sight of their target audience.
Burke: There’s also a problem with definitions and the amount of black box behavior within the industry. It makes it difficult for people to establish a good baseline. When people are talking about impressions and reach they aren’t looking at some of the elements that are behind that data.
For example, is engagement rated differently on a certain blog? Are a lot of other bloggers responding to a peer’s blog to sort of rig the algorithm of engagement? So I would always challenge people to understand how data is validated. And if you work with someone who can’t tell you about their data sourcing, you could be leading down a path to multipliers and God only knows what else.
PRN: So one of the takeaways is to take pains to use accurate data.
Burke: Yes, especially if you’re going to be putting data in some of your management pieces, make sure it’s good data that you’re basing your summation, your theories, your context on.
The bigger takeaway is to always have that data validated, whether you’re using comprehensive data or a statistically accurate sampling of data. I’m not saying everyone needs to be a mathematician, but they do need to understand what those statistical samples can reveal or conceal.
They have to understand that so if they’re asked any of those questions by their senior team they can express how this data can help.
PRN: What’s being done well?
Paine: I think the effort to look at communications as an integrated whole is something being done well. This is forcing communications professionals to define business value in what they do; and for nonprofits and government agencies it’s mission value. People are being asked to define their value in terms of the business as a whole.
That’s the best thing that can possibly happen to measurement because that means things like AVEs, hits and impressions go down in value unless people can show their contribution to the bottom line.
So you have to have that conversation where someone asks what the business value is of 5 billion impressions when your target audience has only about 5,000 people in it.
It’s good that those conversations are happening, and not just at the biggest companies. They’re happening all over. People are stepping back and looking at the communications function and how it contributes to the bottom line.
PRN: What would you like to see the industry doing differently by this time next year?
Burke: The biggest opportunity is for communicators to speak the language of their organization. That would be a big culture change. It’s up to the communicator to take data and insights and translate them into meaningful business outcomes for their organization.
It’s one thing to take the counts and amounts that you’re given and put them in a pie chart and deliver those to your executive team. It’s another thing to give that contextual layover and tell people what the data means for the business. Really what you want to do is talk about how paid and earned work together, or if they don’t, why not?
PRN: Katie, what about awards and the metrics used in so many of them?
Paine: That’s a huge missed opportunity. In 1995, the IPR Measurement Commission wrote a manifesto and sent it to everyone who had an awards program. It said if you do not tie your results to business objectives your application will be thrown away. And if you use AVEs it will be tossed away.
Today many programs throw out applications that use AVEs. The sad thing is people still use some sort of earned media value on awards applications without questioning it. The thing is, if we award companies for presenting their successes with really terrible metrics we perpetuate the use of poor metrics.
PRN: How should companies spend measurement budgets?
Hugley: They should invest in forward-looking analytics, not measurement. Understanding performance patterns is critical, but it’s valuable only if the analytic purpose is optimizing work going forward.
Measurement efforts that are too rooted in past work without an eye for enhancing the communications quality, efficiency and impact should be scaled down to free resources–human and financial–to support more prescriptive and predictive work.
Paine: The first thing they need to do is spend their money on cookies, Starbucks, alcohol and caffeine, whatever it takes, so they can buy time with senior leadership. This is because they’ll need to get clarity up front about what leadership’s goals and objectives are for communications.
Next buy equal amounts of cookies and Starbucks for the folks who manage your Google Analytics, web analytics, CRM, finance and accounting. You’ll need these people to talk to you and give you the information you require.
After that spend on surveying your target audience. It won’t cost much. Find out from sales and marketing about your target audience and what’s keeping it up at night. Then find out what’s keeping your sales and marketing people up at night. What’s keeping the CEO up at night?
Then start thinking about measurement as a way to solve those problems. If you done research up front to learn about what the problems are, your solutions are much more likely to work.
What happens too often is people get jazzed about a concept or an idea or an initiative, whatever you want to call it. And it becomes all about execution, about how many social media channels you can put it in. But if your audience, for example, isn’t on a particular platform, why invest a lot of money to monitor that platform? Why spend huge amounts of money on things that don’t matter? Invest a little bit of money upfront to find out what matters.
PRN: So the takeaway is first to test assumptions?
Paine: The most powerful words in a communicator’s arsenal when they’re asked to do something are: ‘Are you sure you don’t want to test those assumptions?’ Assumption testing is absolutely the first thing I would tell people to do.
For example, say you have a company whose leaders say, ‘Our product is best known for being top quality.’ When you survey the target audience you might find that no, the audience believes you’re fourth best in quality vs your competition. Or the assumption at a company is, ‘The best way for us to sell product is at trade shows.’ Test and you may discover that’s not true.
So to do measurement, first test assumptions and now your company knows what it has to improve and therefore what it has to measure, instead of wasting money to measure things that don’t matter.
PRN: What if you don’t have much of a measurement budget?
Paine:You don’t need a big budget to run Google Analytics, which will tell you 90% of what you need to know. OK, you might need a budget to get a Google Analytics expert to set up some conversions for you…but really, you don’t need a huge budget to measure.
PRN: Allyson, if you had the budget and the ability, what two aspects of PR measurement would you like to see improved by this time next year?
Hugley: I continue to champion the need for better intelligence about the audiences consuming news and earned media performance at the content (article), not publication, level. Although advancements have been made by companies like Cision and Hitwise, there is more work that can and should be done to help communications professionals drill down into content level, article level, and earned media performance.
In an increasingly cluttered and complex news cycle, better intelligence is needed about the stories that are actually breaking through and with which segments of the public.
Reputation and risk-mitigation tools would be another area of focus. In the current, highly charged political environment, where more brands and corporate leaders are being pushed to own and pick sides on issues, communications leaders need better – more refined and more efficient – real-time tools to inform decisions around the issues where it makes sense for companies to act, the best courses of action, and the cost-benefit analyses around those decisions.
Weiner: I’d like to see attribution analysis become the standard complement to traditional media analysis to unlock our ability to connect media relations performance with business outcomes and to inform media targeting based on who clicks on what, when and with what result.
PRN: What can a brand communicator do to hold an agency’s or vendor’s feet to the fire on measurement?
Hugley: Brand communicators have to own the evaluation process – not necessarily aggregating the data – but defining the research parameters and key performance indicators and forcing agencies to adhere to a core set of consistent standards.
There is still too much customization and variance in the reporting. Good evaluation requires consistency in methods and units of measure. There is still too much inconsistency across systems and approaches to feel confident when evaluating work over time, across tools and across agency partners. Creation and adoption of standards is needed and brand communicators should exercise the leverage they have with data providers and agencies to drive that.
Weiner: We must all remember that we have a responsibility to generate a positive return on our (employers’ and clients’) investment in PR. Those who choose not to quantify their contribution should no longer hold the right to be engaged. To achieve this, corporate and brand communicators must require more from their agencies; agencies will do what their clients direct them to do. If measurement is required, then it’s required. If not performing measurement is unacceptable, then it’s unacceptable. It’s a binary equation.
Paine: If you’re not getting insights from reports, which often is the case when there are complaints, I’d sit down with the vendor or consultant and go through every single chart and page in a report and say, “So what?”
You need to gain insight from data and direction as to how to make a situation better. You can get a program to churn out charts and graphs, but you’re paying somebody to interpret it and tell you what it means.
The other thing is a basic accuracy test. Is the vendor or agency measuring what’s important to you and is the data accurate, valid and complete?
PRN: What are your thoughts about AI and measurement? Down the road will we need human analysts or can well-prepared machines interpret and extract insight from data with little or no human intervention? Can they do that now?
Hugley: Regarding AI, the space and specific application to communications are still relatively nascent and the application and outputs of the technology have not caught up with the promise. We are still a ways to go, I think, before the balance shifts toward machines.
Humans have a significant role to play in the development, testing and refinement of machine-learning algorithms. Humans also are critical to the interpretation of outcomes. They also are needed to effectively apply lessons, which can be surfaced more efficiently through the use of machine learning, to communications and business decision-making.
Burke: My counsel is that people understand how a partner or provider is using the term AI and really question what that includes. So as Mark said earlier, you need to distinguish between automation and AI.
Yes, there are AI systems that can write summaries, but is it something you want to put in front of your board? I don’t think so.
If communicators totally surrender...and give up applying deep insights into what the data means for their organization and how it relates to the overall objectives, I think they will be marginalized and can be replaced by automation. Like anything else, if we leave everything up to an algorithm, we are limiting the potential of what we’re able to see.
Paine: There’s absolutely no reason why a modern system, equipped with a well-trained AI machine, would not be able to take well-coded content and look at it and find messages in it. That technology exists today. Companies are investing in this stuff. They are telling me Boolean is dead and AI will replace it in a year or two.
Then the question becomes who is the first to operationalize it and make it affordable enough for the 99 percent to glom onto it and use it?
PRN: So humans are history in measurement?
Paine: Well, no, but a well-trained machine can replace much of what humans currently do, with about 80 percent accuracy. But you need to generate a large-enough volume of data to make it worthwhile [to invest in a machine].
So, yes, in a year or two you’ll have AI-based systems that will learn from existing stuff. They’re here now but they’re not operationalized and commercialized. If AI machines can replace reporters and sportswriters, why not measurement analysts?
PRN: So if a young person tells you she wants to be like you, a measurement analyst, what do you say?
Paine: Go get an MBA. Understand business. Understand how companies make profits, why they lose money. Yes, you need the PR and communications background, but you have to understand how companies work. Too many people in communications don’t. I know when I graduated, I didn’t.
NOTE: This content appeared originally in PR News, November 2018. For subscription information, please visit: http://www.prnewsonline.com/about/info
CONTACT: [email protected] [email protected] [email protected] [email protected]