Category Archives: Market research

My pet peeve with surveys

· · Leave a comment

I don’t know how many opinion researchers are themselves members of online survey panels. I’m a member of a couple, mainly because I like to see what other researchers are doing in terms of survey and questionnaire design; to keep up with the current state of the art. Of course, that means seeing what not to do, as well as what I should adapt to my own surveys.

I read lots of blogs from market researchers describing problems with questionnaire design (things like poor scales, biased questions, etc.) but I don’t often see the problem that I find is my pet peeve with surveys. This problem makes it difficult to do that most basic thing, the thing that I try to do with every survey I take: answer the questions as honestly as I can. It’s perhaps easiest to illustrate this problem by way of example.

Recently, I was completing a survey in which the screening questions were related to car ownership. The survey asked if I owned a car (I do) and then asked whether I purchased or leased the vehicle (the former). The next question asked whether I planned to purchase a vehicle in the next 3 years. Again, the answer was yes. The survey then went on to ask a series of questions about what I was looking for in a car dealer. The problem? I do not intend to buy my next vehicle from a dealer. When my current car is paid off (in about a year), I plan to drive it for another year or so, and then sell it and buy a used car for cash. I have no plans to go to a dealer at all. The designer of the questionnaire I was completing clearly never considered this possibility. The questions assumed I would be shopping for a new vehicle at a dealership, and there was no way to skip the dealership-related questions.

Another example: I am a member of a survey panel for a gas station chain. I purchase gas from this chain occasionally, but always by paying at the pump. In a couple of cases when pump payment was not working, I would go inside the store to pay, but I never buy (or even look at) anything inside the store; I just pay for my gas and go. I have completed a number of surveys for this chain that ask a bunch of questions about my preferences in terms of what the stores should sell: should they sell sandwiches (and what type?), coffee (what price?) and so on. Again, I never have the opportunity to say that I don’t shop at the convenience store and have no interest in what they are selling, so I either have to answer all the questions “don’t know” or answer as though I was a shopper. But, if I do the latter, are the answers I give meaningful?

I believe that a lot of what guides this type of questionnaire design is an effort to broaden the response base for the key questions to ensure they have as many respondents as possible and thus reduce the margin of error for those questions. The problem is that many of the respondents they end up with are people like me -very occasional or never-users of the products/services they are asking about. To take another example I have experienced first-hand, if I visited a fast-food restaurant once seven months ago, because I was out on my motorcycle and needed a (aptly named, in my case) “butt break”, does that mean that I should be answering a long series of questions about what I want to see in fast-food restaurants just because I screening in by virtue of having visited a fast-food restaurant in the past year?

It’s not so bad if you have an “other” write-in option where you can describe your situation and then the researcher can decide whether they still want to keep you, but it’s more common to just have to go through question after question where there just isn’t a response choice that really reflects your situation.

One thing I’ve learned from this is to always include an open-ended “comment” question at the end of my surveys so that, if I have inadvertently made a similar error, the respondent can tell me that the questions I have included simply are not relevant to their particular situation.

Have you had similar experiences completing surveys? Other survey pet peeves? Let me know in the comments!

Marketing research: On the gap between traditional and next-gen

· · 1 Comment

Last Thursday afternoon, I sat in a room with some (actually far too few) other people from around the Government of Canada to watch a webinar organized by MRIA (the Marketing Research and Intelligence Association, my professional association), on the subject of mobile research. I really never intended to write too much about my professional life on this blog, but this webinar really got me thinking. Actually, it was one comment from one viewer that got me thinking.

The webinar featured six speakers from five companies active in the mobile research area (i.e., research conducted using mobile phones, apps, etc.) The presentations ranged from the blah (the presentation from Qualvu’s John Williamson was basically just a sales pitch) to the extremely interesting (Jason Cyr of Tiipz [who looks about 16 in his photo] gave a fascinating presentation on the use of mobile research to gather real-time data at events  like music festivals).

But, as I mentioned, it was one question that really got me thinking. That question was, in part, about validity and the fact that none of the presenters mentioned it. Well, when this question was read out by the webinar host, you could almost hear the groans from the presenters. That’s because validity, more than perhaps any other issue, divides practitioners of what is called “next-gen” marketing research from traditional market researchers. Much has been written and said about this issue, much of it in blogs and on social media platforms. In a nutshell, traditional market researchers believe that next-gen research does not have a “scientific” foundation and, thus, its findings cannot be tied to the population as a whole with any precision. On the other hand, next-gen researchers believe traditional market research is mired in “old thinking” and just doesn’t reach large groups of  people (and those people, mostly the young, are of greatest interest to many of the companies that are buying market research services).

Of course, the validity issue strikes to the heart of this divide. Proponents of traditional market research talk about “representativeness” and “probability samples” as the theoretical underpinnings of the whole market research field (at least the quantitative side of it). These underpinnings allow the results of marketing research to be generalized to the population as a whole with a known degree of confidence (see my recent post on “margin of error”). For traditional researchers, if you don’t have this, you have nothing.

On the other hand, next-gen market researchers are often dismissive of these traditional sacred cows. In response to the validity question at the MRIA webinar, one of the presenters said “just get over” not using probability samples. (This echoes the view of another of the younger generation of market researchers, who said of representative samples “forget it, we never had it.”) “Gamification”, social media “listening” and “scraping” are the some of the buzzwords here.

Seeing the next-gen mindset come up against traditionalist thinking, as it did in the MRIA webinar, made me think about where I stand on these issues. As 2012 will be my 25th year in the marketing and public opinion research field, one might think that I fall naturally into the traditionalist camp. However, I try to keep an open mind and to keep abreast of the latest developments in my field. And it’s those last two words – “my field” – that are the key, not only to where I stand, but also to the divide between traditional and next-gen research.

What do I mean? Well, I have always thought of myself as an “opinion researcher.” What I sold to my clients over the years was the ability to tell them what the public’s opinion was about, well, whatever the subject of the research was. If you wanted to know what the general public thought about your potential new product, or what your customers thought about your service, or what Canadian citizens thought about your government department’s proposed policy, I could tell you that. And I put my money where my mouth is. I didn’t just say people liked your new product. I said 41% would buy it. And I sold the confidence that number was pretty accurate. What set me apart from my competitors? Well, I could tell you what people’s opinions really were, through the solid design of my study and the research instruments, and the artful analysis of the data collected.

But when I read about and see presentations of next-gen research, I see something different. They are not selling “opinion”, they are selling insight. By listening to social media (for example), they can give their clients insights into their brand that they might otherwise not get. Social media listening is more akin to observational research than it is to traditional market research. It’s a step removed from merely asking people what they think (even if it’s not quite the same as seeing what they actually do).

The other thing that I see in next-gen research that is fundamentally different from the traditional mindset is engagement. Whereas traditional research tries mightily to remain unobtrusive (stemming from its roots in scientific experimentation), next-gen research is in your face. Companies today are much more likely to see the act of research itself as a means of engaging the research participants, as well as learning from them.

Is there a middle ground? I’m not sure there is, or that there should be. What is important, I think, is that proponents of both camps realize (and communicate to their clients) what their research can AND can’t do. As long as clients have no illusions as to the strengths and limitations of the research they are buying, I think both traditional and next-gen techniques can provide valuable information.

What do you think? Comments, as always, are welcome. I’m particularly interested in hearing from next-gen researchers, as I may have misrepresented their position. I post all comments that are not spam.