Once, as a brand manager, I was involved in qualitative testing of four advertising directions for an upcoming campaign to identify the strongest one for full-up production. Unfortunately, no clear winner emerged. After the final focus group, we agonized about what to do. Finally, the moderator piped up: “You people are crazy. What did you expect? Did you really think such a small number of consumers could provide that kind of guidance?” He went on to point out that one ad was clearly a dog and one was mediocre, leaving us with two that delivered on all aspects of the creative strategy. He told us, “You’ve got two good options. Just make a decision. As a group, you’re fully qualified to make the call.”
This was a good lesson for me, one that I have taken to heart ever since. Instead of trying to crown a champion, eliminate the losers and select a ‘winner’ from what remains.
I conduct a lot of qualitative research that involves showing creative stimuli. These include things like package designs, advertisements, and new product concepts. It’s common for clients to set identifying the strongest execution as a goal for this type of research. This is an understandable objective, but often not realistic.
Picking the single best performer among several possibilities requires both precision and accuracy to have confidence in your results. Qualitative research, much as I love it, isn’t great for either of those things. Not only are the samples too small, but the research approach doesn’t lend itself well to these goals. The sample is not designed to be balanced and projectable. And the data gathering methods, which are informal and exploratory, don’t produce the kind of consistent data needed for accuracy and precision.
As we all know, qualitative research shines when your goal is to explore foundational issues and develop hypotheses for further testing. It is that additional testing, particularly if it is properly designed quantitative research, that will provide accurate, precise, statistically reliable data.
While qualitative research usually isn’t up to the task of identifying the single strongest option, it’s more than adequate for eliminating the weaker ones. In addition, it can provide detailed and nuanced insights about each execution to inform the client’s process for making a final decision. That way, clients should feel confident in their ability to choose their winner based on the research.
If clients feel they must select a winner, include such evaluative criteria in the discussion guide as relatability, sense of urgency and main message playback. Also, have clients identify any non-research-based criteria that can also be considered. These could include things like strategic considerations, competitive environment, consistency with past marketing activities, etc. You might also want to consider a hybrid approach in which the qual informs more in-depth quantitative insights.
Regardless of your approach, it’s a good idea to agree – beforehand – on what the tiebreaker criteria will be for final selection. And be sure everybody understands the limitations of qualitative research in picking winners when testing marketing concepts and creative stimuli.
