Hard to predict	May 2008

A variety of research techniques exist to help increase the level of certainty that a campaign will achieve its goals and objectives. Virtually all the research techniques available have value provided their underlying limitations and assumptions are fully understood. Mark Thompson, VP, senior brand planner, Palio (palio.com), an inVentiv Health company, explores the challenges of two popular techniques.

Adjusting to uncertainty is more than a little difficult when you've got millions of marketing dollars riding on your next advertising campaign. To help remedy this situation, a variety of research techniques exist that, in theory, should increase the level of certainty that a campaign will achieve the goals and objectives asked of it. Such techniques range from the humble focus group to quantitative copy-testing to expensive in-market area tests. But just how useful are these techniques? With DTC media spending now at more than $5 billion annually, it’s a critically important question to ask.

Before launching into a discussion of techniques, it’s useful to start with what is arguably the most basic of questions: What is the role of advertising in our overall marketing mix? For the launch of a new brand, the primary goal may be increasing awareness. For a well-known but declining brand, modifying key attitudes and perceptions may be the most efficient way to increase sales. For a market leader, the main goal may be to maintain existing (and motivating) brand perceptions in light of increased competitive activity. Having such a single-minded — and often varying — role for advertising may sound heretical. After all, what’s the point of raising awareness if we don’t also see increased interest or desire in a new brand. And surely increased interest is meaningless if we don’t also see an increase in claimed intent to buy or prescribe? And doesn’t the goal of merely maintaining existing perceptions sound a little bit too defensive?

The argument that, in order to be effective, advertising should be shown to perform outstandingly well on all key measures deemed influential (let’s go with Awareness, Interest, Desire and Action, or AIDA, for the sake of simplicity) is a very seductive but dangerous and specious one to make. Over the last 30 years or so, a large number of empirical studies have discredited the so-called “transportation” models of advertising (AIDA being one of the most familiar). Sometimes behavior change precedes attitudinal change, sometimes an increase in awareness alone can spur increased buying behavior without a corresponding increase in positive brand interest or desire, and so on. Finally, claimed intent to purchase (or prescribe) is rarely a great predictor of actual purchasing behavior. A landmark study by The Advertising Research Foundation some years ago found that the “likeability” of an ad was the most reliable predictor of effectiveness. While it’s possible to question the applicability of the foundation’s study to all categories, it nevertheless highlights the problems of choosing between “intermediate” measures such as awareness/recall or action/“intent to purchase” as predictors of effectiveness.

Expert opinion currently seems to favor a pluralistic model of how advertising works. More specifically, it suggests that advertising can — and does — work in a number of ways, depending on category, brand, and consumer dynamics. Every technique that purports to predict advertising effectiveness makes assumptions about how advertising works. And, to make matters worse, those assumptions are usually implicit and rarely articulated. So we have to tread warily when we decide on which tool to use — and how to interpret the results.

Virtually all the research techniques available have value provided their underlying limitations and assumptions are fully understood. Two of the more popular techniques are quantitative copy-tests and qualitative “red-flag” focus groups.

Perhaps the greatest source of research friction between an advertising agency and its client, the copy-test is a quantitative technique that exposes one or more advertisements to a representative sample of the target consumer. Usually the ad is embedded among other ads to better replicate real-world exposure. Though each copy-test has its own set of “measures,” respondents are typically asked for their recall of the brand, the advertisement, the main message being communicated, and a variety of other measures such as “intent to buy,” “likeability,” and other more effective brand responses. The resulting measures are then compared between ads (if more than one campaign or execution is under consideration) and/or against norms that the research company has, based on previous ads in the category.

On the surface, this technique has great intuitive appeal, not least because it allows for a large, robust sample that can yield statistically significant differences in response. And the availability of “norms” would appear to answer the typical question of, “Is an X% unaided recall of our ad good or bad?” But the acid test for any copy-test is whether it actually predicts advertising effectiveness with any degree of certainty. My own personal opinion is that results can vary widely in terms of gauging likely effectiveness. Rather than open up a can of worms and subjectively attempt to separate the good from the bad, here are the questions I always ask of a copy-test vendor: What assumptions does your test make about how advertising works? How applicable are your norms to my brand? Given our advertising objectives, what sample size would you recommend to allow adequate disaggregation of the data?

With a "red-flag" focus group, after all the weeks, if not months, of working on strategy, revisions, creative briefs, tissue sessions, revisions, presentations to the client, revisions, and more presentations, we need to be sure our consumers share our belief in the strength of the campaign(s). Over the past 25 years or so, I’ve attended more of these “red flag” focus groups than I care to remember. But I think they can be very useful and insightful — especially if it’s the first real research being done on the strategy and ad ideas.

By useful, I should add the caveat if there’s enough time in the development process to react to the results of the groups. Ideally, the core creative concept has already been researched, and the “red flag” group serves primarily as a means of identifying any executional nuances that may be negatively affecting the likely effectiveness of a given campaign or execution. And therein lies a critical difference between good and bad groups; the skill of the moderator to differentiate between the advertising idea and the execution. All too often, a great campaign can die a quick death at the eleventh hour because the difference between the two could not be determined.

As with quantitative testing, one of the critical decisions for a successful group concerns what responses to elicit from respondents. It’s a purely personal, subjective opinion, but I don’t think that the oft-practiced ranking or voting for ad concepts has any legitimate role within a focus group. The fact that the sample size of a few groups never supports reliable, projectable findings is the obvious criticism of such a quasiquantitative approach (remember that ubiquitous disclaimer at the beginning of each research report). My concern goes much deeper than that. When asked to quickly make a choice, we naturally enter into a rational mode of thinking, especially when anticipating being asked to justify our choice among our peers. Unless one subscribes to the view that physicians and/or consumers are all-knowing, fully rational economic men and women who make buying decisions purely on the basis of considered, rational thought, then this style of questioning can introduce an artificiality that can bias the results — and conclusions — of the groups. Many a physician has told me that they respond best to ads that just tell them the facts. But I think they’re deceiving themselves some or all of the time. Indeed, recent developments in neuroscience support the view that emotions guide rational thought and perceptions, not vice versa, as was the conventional wisdom.

Similarly, the use of direct versus indirect questioning can be a critical determinant of how predictive the focus group is of real-world effectiveness. When asked why they prescribe brand X, the word “efficacy” will be uttered nine times out of 10 during a discussion with physicians. While efficacy perceptions may indeed be the key driver of brand choice in some cases, often it's merely the socially acceptable, rational reason that disguises less acceptable, more emotional drivers — such as inertia, low category involvement, or brand familiarity; direct questioning rarely uncovers these motives and can accentuate the artificiality of the responses.
Finally, focus groups cannot adequately assess the likely cut-through or attention-getting power of an ad. With the myriad communications bombarding consumers nowadays, getting below their radar is harder than ever. It may take more than the communication of a relevant attribute or benefit to get their attention.