With the explosion of consumer-generated media, citizen journalists and consumer empowerment, opinion research is becoming an increasingly important building block of
communications strategies. It is often the best way to take the pulse of your target audience and then shape messaging accordingly; and, with the congruent growth of online
survey technology, it is also cost-effective. However, with all the pros of Web-based surveys comes a fair share of cons, thus making it essential for communications executives
to understand how to make their surveys - and the subsequent results - credible.
Quantifying Quality
Merril Shugoll, president of Marketing Research Association (MRA), made the following remark in MRA's Alert Magazine in April 2007:
"Unfortunately, the burgeoning growth of online research has resulted in anyone thinking they can conduct marketing and opinion research, regardless of their training,
experience or academic credentials...Quality must be added to the list of beautiful features associated with online research."
To ensure quality, Alice Irvan (president, AIRvan Consulting) and Terri Johnson (assistant professor, Eastern Illinois University) outlined a seven-step plan for developing and
implementing Web-based surveys effectively at the 2007 PRSA International Conference.
*Step 1: Plan the process. Irvan and Johnson recommend getting buy-in from management up from to make sure that leadership isn't disappointed with the end result. This avoids
"confusion and misunderstanding about access to individual responses." It seems like a given, but overlooking it could result in research that has to be thrown out the window.
Then, plan to communicate the findings to multiple specific audiences. Knowing these audiences ahead of time will help shape questions and messaging.
In terms of prioritizing, first determine the objectives (preferably no more than three) and decisions to be made based on the research, as well as the analytical techniques
needed to get that data (for a how-to guide for determining these analytical techniques, see page 2).
Secondly, Irvan and Johnson advise establishing the following:
-
A timetable;
-
A policy on anonymity;
-
Whether you will offer incentives; and,
-
A policy on ethical guidelines.
*Step 2: Determine the population and sample. In making this determination, Irvan and Johnson explain two options: using an existing sample, and purchasing a sample. If you
plan to use an existing sample, you have the following options:
-
Self-administration via a kiosk;
-
An invitation link on a site; or
-
An invitation link in an e-mail (remember: if you opt for this method, you must have permission to e-mail).
For purchasing a sample, you can use a vendor that will send out invitations to participate and then monitor the responses; or, you can use online panels for broad national or
international audiences.
*Step 3: Select the survey tool. You have various options:
-
For software: Survey software that is either PC-based, Web-based or stand-alone;
-
For hosting: A hosting service, or your own server;
-
For analysis: A package that's part of a survey tool (Zoomerang, SurveyMonkey, etc.), or one that is separate (SPSS, SAS, MarketSight, etc); and,
-
For reporting: Using what's ready-made from the software, or importing the results into a Powerpoint presentation or Word document.
*Step 4: Implement the survey. When implementing the survey, you simply input the questions into the existing template, but Irvan and Thompson recommend being careful not to
overlook qualifying questions that rule out groups of respondents; poorly structured questions; and questions whose answers can be easily analyzed. Then, they offer this
roadmap:
-
Do a pre-test and revise;
-
Launch;
-
Monitor undeliverables;
-
Send out reminders;
-
Close the survey; examine data frequencies and percentages prior to analysis;
-
Delete "suspicious" surveys; and,
-
Decide whether to include "partials" in the analysis.
*Step 5: Analyze the data. This can be done via cross-tabulation, banners, multivariate analysis and statistical testing. Always look for meaningful differences between sub-
segments, look for patterns and keep in mind the margin of error. Then, manage open-ended questions (of which there should be few) by either coding, including excerpts or listing
responses verbatim.
*Step 6: Report the findings. Refer back to the planning phase and address the target audiences you initially set out to reach, and communicate the findings to senior
management. Best practices include:
-
Making the findings visual by converting numbers into charts and graphs;
-
Keep verbal analyses succinct; and,
-
Customize reports for various audiences to highlight the points most relevant to them.
*Step 7: Evaluate the process. Irvan and Thompson recommend asking yourself the following questions to rate the success of your research:
-
How did the entire process work, from planning to reporting?
-
What was successful, and what could be improved?
-
Are the results being used as expected?
-
What other information was revealed that may require follow-up research? PRN
CONTACTS:
Alice Irvan, [email protected]; Terri Johnson, [email protected]
Numerically Speaking ...
Based on the 2005 data compiled by Cambiar and GMI, the 2007 forecast is as follows:
-
90% of major corporations will be doing online research;
-
61% of small corporations will be doing online research;
-
86% of research budgets will be devoted to online research; and,
-
The global online research market will equal $3.8 billion.
Source: MRA's Alert Magazine, May 2007
Glossary of Selected Research Terms
Statistics: To many people, this means "numerical descriptions." This field has two general areas of applications: 1) describing large masses of data and 2) drawing
conclusions about some set of data based on sampling.
Population: A set of data that characterizes some phenomenon.
Sample: A subset of data selected from a population.
Statistical Inference: A decision, estimate, prediction, or generalization about the population based on information contained in a sample.
Reliability: An estimation of the error of a measure.
Central Limit Theorem: A theory of probability that the means of repeated samples of the same size form a bell-shaped (normal) distribution around the true population mean. The
mean of means converges on the true population mean.
Confidence Interval: A range of numbers above and below the statistic generated from a sample that includes the true population value (parameter) at a known level of
confidence. Sometimes called a margin of error.
Confidence Level: The probability that a confidence interval will include the population parameter.
Standard Error: The standard deviation of the sampling distribution of the mean or proportion.
Census: A complete enumeration of the elements of a population.
Validity: Whether the measurement actually measures what is intended.
Source: Alice Irvan, President, Airvan Consulting