In our latest podcast episode, Market Research & Prototype Testing, our host is joined by our Research Director, Lang McGilp, as co-host to speak with episode guest, Ben Jacobson.
Ben is a partner at Conifer Research, a design research consultancy in Chicago that has worked in prototype testing with brands that include Samsung, Pepsi, Nestle Purina and BP.
In the episode, Duncan, Ben and Lang look into just what the practice to prototype testing is all about, Ben tells some stories and relates some of his experiences when it comes to the do’s and don’ts of prototype testing and looks forward to giving his ideas of where the practice of prototype testing, and market research in general, is going in the future.
If you’d like to learn more about Conifer Research and the work they do, check out their website. There, you’ll find a link to the Research RX tool that was described in the episode.
Insightrix Research releases new content regularly!
Check out the Insightrix blog roll for informative articles about everything and anything market research related.
You can also access Stories of Market Research: The Insightrix Podcast on your favourite podcast player app, including Apple Podcasts, Stitcher and YouTube. Make sure to subscribe to be notified the moment a new episode is published!
Episode Transcript[EPISODE INTRODUCTION] [DUNCAN] Prototype testing is a valuable step in the product or service development process.
Whether it’s through co-ideation in development, or to align a product or service with the needs of their users, or to discover and define business opportunities for existing services or products, prototype testing can help companies learn a lot about how real users interact with products and services in real-world situations.
Hello, my name is Duncan McGregor, Marketing and Communications Coordinator at Insightrix Research in Saskatoon, Canada, and your podcast host.
On today’s episode, we’re joined by Ben Jacobson, partner at Conifer Research – a design research consultancy in Chicago, USA.
Conifer Research is a company that works heavily in prototype testing – having worked across a wide swath of industries with the design and innovation teams from companies like BP, McDonald’s, Nestle Purina, Pepsi and Samsung.
In the episode, Ben shares some of his experiences in prototype testing, how it has benefitted Conifer Research’s clients and the ways it has helped them – and – Ben relates some of the techniques his company has employed in the past to get their clients the insight they need.
Ben also talks about where he thinks the field of prototype testing, and market research in general, is heading and offers some suggestions for anyone looking to dive into the field.
And for this episode of the podcast, Insightrix own Research Director, Lang McGilp, sits in as co-host, lending his experience and research acumen to the discussion.
Do’s and don’ts of prototype research, observational research vs. big data and humanizing top box scores – all that and more, in this episode of Stories of Market Research: The Insightrix Podcast.[THEME MUSIC] [INTERVIEW PT.1 BEGINNING] [DUNCAN] Hello. We’ve got Ben Jacobson, from Conifer Research from out of Chicago on the phone. How’s it going Ben? [BEN] I’m doing great, thanks. How are you? [DUNCAN] I’m doing great. Thanks for coming on. I’ve also got our own… [BEN] Thanks for having me. [DUNCAN] Oh no, it’s a pleasure. I’ve also got our own Research Director here at Insightrix, Lang McGilp, on the phone to co-host. So, thanks for going us.
So, Ben is a partner at Conifer Research, a research agency that does a lot of work in prototype testing. This isn’t something we usually do a lot of here at Insightrix, so we wanted to have him on to talk about his company and the work you guys do. Is there anything you could tell us about Conifer Research and the types of prototype testing you do?[BEN] Sure. My first question would be why don’t you do prototype testing up there? But besides that, my little company got started in 2001, and we really got started in all of this – I’m just going to give a shout out to one of our first patron clients, who back in the day was Steelcase Incorporated, run by a visionary CEO whose name is Jim Hackett. And Jim Hackett, of course, is now the CEO of Ford. And they have long sponsored observational research and ethnographic research both for new concept development but also testing all the iterations of the designs that they do. And so, we specialize in observational research, in-context interviewing, those sorts of things, which has a wide variety of uses. One of which is, certainly, prototype testing. And that’s really the first way that we got into it. [DUNCAN] Right on. How do you find that prototype testing benefits your clients? I know it’s a pretty broad question, but what are the most common or key areas that you help your clients with? [BEN] The first thing to kind of wrap our brains around as we talk about this is that testing, the word prototype can mean almost anything, right? [DUNCAN] Yeah. [BEN] It can literally mean some rubber bands and paper clips and pipe-cleaners and things that you’ve made in an ideation session. It can mean sketches and concepts. It could be a fully engineered and absolutely functioning piece of machinery, right? So, the range of what falls into that is quite large – and we love to get involved in all of those aspects of it. We don’t just specialize in one teeny, tiny bit.
And, I guess the key thing to think about is it’s just dramatically cheaper, more interesting and insightful to identify the joys and pains that you’re about to set loose in the world at a small scale rather than at market scale after launch. So, that is probably the number one thing that people out there who are thinking about this should keep in mind, is that this is, sort of, something that you absolutely should be doing.
And I guess you said, the other one was – how does it help my clients? Well, so, I mentioned, we save them valuable time and money by getting things as close to right as possible before they sink a lot of money into engineering and development. And I can tell you some stories about that later. But, sometimes, it even becomes obvious that an idea should be radically transformed, repositioned or even killed outright based on what you can learn in a prototype test. And if you talk to people in business, this is probably the thing that doesn’t get accounted for very often but should be. Which is, how many times did you save yourself from embarrassment in the market? Or building something at scale and launching it, and then having to write it all off two quarters later? Things like that – and when you think about those epic fails, usually at the bottom of that is some kind of vacuum when it really came to understanding how this was going to function in the real world. And I guess the third thing that I would add in that sort of basket of topics is it’s really important to have a different set of eyes on the design and development as you move things from concepts into offerings out there in the world. If you let an internal design team or an external design firm who created the concept do the testing, it often, I think, introduces a whole lot more bias into the process than I personally think is healthy.[LANG] Yeah, we do quite a bit of advertising/concept kind of testing, and I find a similar approach, where sometimes just being that extra party at the table, that’s not involved in creating or taking an idea and putting it into something for consumers to absorb, gives you that chance to be a little more objective and that kind of thing and ensure that what you’re showing people aligns with the intended outcome. [BEN] Oh yeah, Lang – I mean, that gets multiplied, right? I mean, the multiplier effects on this kind of this are so interesting, right? So, it’s at the table with your client and their stakeholders, etc., but even in the conduct of the research. If you’re, in any way shape or form, the people that are engaged in a test or a concept test, something like that, you’ll have an idea about who this is for, and those people are sitting in the room, you immediately introduce this whole sense of how you please the people who are paying you and feeding you M&Ms, right? And it’s really difficult to get out of that unless it’s really clear that you’re an unbiased third party. [LANG] Interesting. Do you, and I wonder if the barriers might be the same with some of the work that we do, but what are some of the challenges that you find with clients who maybe want to just take the phrase “down and dirty” – just something simple. Give me the basic. Or, maybe I’m struggling to convince people to even bother with this and just move things down – move the chains down the field a little bit more on our own. What are some of those barriers that you find that you need to convince your client or they need to convince their internal client this is worth taking the time and a little bit of money to invest and figure things out? [BEN] Well, when you have the formula for that, will you let me know? Because I think that would unlock a lot, right? So, I guess that there’s a couple of things. One is I like to think about it as really honest conversation with people. And it’s hard to do when you don’t even know them, right? When people send you an RFP through the email and they’re asking for something, and you don’t know them. And it’s hard, as well, when you know someone and you have multi-year relationships with people and they’ll say we want to do X, Y and Z, and we’ll say that seems like a good idea or a bad idea. But it’s hard to speak truth to the people who would write you a cheque. And I think we all have to keep that in mind and try to check our egos at the door and be as faithful as we can to what’s the good, right and true thing that should be happening. I like to remind people or even just ask them in the form of questions, how much money are you are you spending to take this from me, let’s say they use a staged gate or some crazy scrum model or whatever, right? Now how much money are you spending to go to move the chains down the field? And how many times have you already done that without checking your assumptions or without exposing this to the people who are going to use it? Because the further you go, the more expensive that really is. And there’s a ton of cost involved in that that people think isn’t real. And people will tell me all the time. They are like, Oh well, this is an innovation project and it has to come up with a new idea, a concept, a service, whatever – that has at least 100 million dollars a year in incremental income to the firm. And then I’m thinking, OK, 100 million dollars is your goalpost and you don’t want to spend $100,000 to figure out whether or not your moving in the right direction. And I think that they have their math. They went to business school, I didn’t. But in my mind, there’s definitely times and places along the way when you need to check things out and make sure that you’re not making some goofy, horrible mistake. And the other part of it is not just the mistakes, but the opportunity to make it better. I mean, not just not bad, right? You can learn things along the way by iterative processes that really amp up the quality and the experience of the final offering at the end. And if you think that you’ve got it all right, from the Post-It note you stuck on the conference room wall all the way through to launching something into the world, that’s arrogance. And that kind of arrogance usually ends badly. [DUNCAN] So, could you give us an example or two of how you’ve used your methodologies to assist your clients? [BEN] Oh God, yeah. We could talk all day about that. [DUNCAN] Awesome! [BEN] I know you don’t have that much time, but we’ll talk about a few things – and shut me up if I’m not making sense or if I’m causing you pain… [DUNCAN] Okee-dokie. [LANG] Oh no, we’re in. This is very interesting. [BEN] …or anything like that. So, here’s a fun one and it’s an oldie but a goodie. Right? So those are kind of fun. So, we helped a large, not-to-be-named department store brand do a test of some price-check machines. You guys know what I’m talking about? [DUNCAN] Yup, you bet. [LANG] Yeah. [BEN] Yeah, those little scanners. They’re all through the store and you know, you slide the barcode under there and this costs X, Y, Z, right? So, first you have to realize that price-check machines were not even close to a new thing in the world when we were doing this. All their competitors had them. And so, it was an internal, emotional problem they were having with making the decision to roll these out to all of their stores. And so, on the good side, it was really easy for them to put their hands on some functional prototypes and put them in a store, which they chose to do literally in a small town in Oklahoma so nobody would find out. And so we put these into five different formats of their stores, which at the time was like overkill, right? Why do you need to go to that length? But they were insistent that they had to do this. And they were worried that the machines would disrupt their shoppers and change their usual purchase behaviour or even cause confusion. So, we set up the machines in real store environments, using fixed video cameras to record behaviour and interaction with the machines. We hung around in the various departments. After people used the machines, we would approach them and interview them about the experience of using the machine, etc., etc. So, that was fairly straightforward, but it got really, really interesting when things started going down the road. So, the thing is, on these machines, they had, of course, a little, small screen that would display the price and some information. They had the scanner with the funky, little laser. And, oddly enough, they had a series of simple, little black buttons on the front – and the buttons had no function, but if you pushed them, they went beep. [LANG] No way. [DUNCAN] So classic. [BEN] I wish I could get the right beep, right? But it was a charming little beep and etc. So, but nothing would happen, right? It just went beep. So, when you present people with a button, many, many of them will push it. [LANG] My kids do that to me all the time. [BEN] Of course, right? That’s how they learn the power button on every device in your home before they’re one year old. They’re going to push the button. They’ve seen you push buttons; they want to push the buttons. You can imagine what the kids in the store – they were going crazy with the buttons. But, either way, when someone pushes a button, they have ideas in their head. They have expectations and assumptions that are connected to some interesting mental model about how they think things work – and even what they think might be possible. And if it beeps, then those expectations are heightened because they already got some kind of response. And so, you know, what we started to discover was that as we were watching the video, we’d see people push the buttons. We’d see them hear the beep. They would look at the screen kind of funny – you’d see little changes in their body language, etc. And then, they would walk off. Then we would interview them. And so, one of the things that we discovered was that what they asked us for, what our clients asked us for, which was people like price-check scanners, especially in stores where there’s always this rolling markdown of prices and that’s part of the brand of the store – so, it’s an unmitigated positive. That was the easy answer. But the really interesting part of this came with the ideas that people talked about when we started to dig around a bit in what they thought might happen when they pushed those buttons. And we wound up with a very forward-looking set of ideas that came out of this work. Literally, just what shoppers thought might be possible through this digital interaction that they were having in the middle of a retail environment. And so, that’s a bit of a goldmine – an unanticipated goldmine. And that’s why I like the story because it really shows you can get a variety of very tactical answers to questions you might have about a prototype, but also introducing something like a button that goes beep or does nothing or what have you, is an interesting way to tap into these future-oriented insights about what your customers are ready for, and maybe already expecting from your brand. [LANG] Interesting. I’ve got a quick question for you here. So, knowing that you’re primarily focused on this prototype testing, have you – like, what are some of the benefits that you often explain to your potential clients? Is this is the benefit of having something physical, hold in the hand or in the store or in the whatever kind of real environment as opposed to perhaps something more conceptual, maybe written on paper, maybe part of a survey. So, people aren’t really – it’s not confronted and it’s not really right in front of them and it’s… What kind of the thing that you often mention to tell people that this is an added value? [BEN] Well, I’ve two things. I think that anything that goes beyond a word or a set of words is incrementally better, and so, I would never pooh-pooh fun sketches on a napkin. Or even non-designer-type sketches that end users or consumers would do in a co-creation or an ideation session. There can be great things to learn from that, but you wouldn’t want to necessarily think about the big processes – thumbs up/thumbs down, right? This is not, “Do we go forward or not?” It’s, “What did we learn from this and how would we incorporate this into the next iteration?” So, that’s my caveat to what my answer is. On the other hand, leaving things only in words is a failure mode. And I do this with my clients all the time. I’ll say, “Everybody take out a Post-It note and write this word down on it.” And it’ll be a word like – here’s my favourite one is collaboration. OK, so, we put the word “collaboration” up on a whiteboard because that’s where all good ideas land – on a whiteboard with a Post-It note that says “collaboration”. And then, what’s fascinating is, what does that mean, right? And very often, people will go around and they distill and they distill and they come down with this vision of a thing and it’s going to be a collaboration zone or a collaboration technology, or whatever it is. It’s the new standard bearer for what your new innovation project is going to be all about, but everybody has a completely different idea of what that means. And they could all be right, but if it was your job to build that thing, you would be way off – probably most of the time. And when you ask people to write down what it means, for some people, it means, well, we shared writing a document. We passed that document back and forth through email until it was finished and then we collaborated and that was the result of our collaboration. For other people, it’s literally we need a bunch of Toobers and Zots and fun playthings on a table and lots of whiteboard space, and we’ll roll up our sleeves and have some fun music playing in the background – that’s collaboration. For other people, it’s a conference call. So, what are you making? And if you leave it in the word, they’re just certain words and people love those words because they’re kind of sexy and fun and interesting, but they hide more than they reveal. And so, the minute you go from a word that’s kind of fuzzy and fun to talk about to something tangible, people go, “Oh, that’s not what I thought you meant.” And it’s horrifying – I’ve seen teams walk out of a meeting and come back a month later and they’ve all been working on different things because they had one word in their head and they were taking it in different directions. So, visualization is key to success, and the more tangible something gets as you take it from a Post-It note to a napkin sketch to a concept story or a storyboard, all the way through to something that humans are going to interact with in some capacity, there are multiple opportunities – and they don’t have to all be big and expensive. You can do things that are quick and not dirty, as long as your expectations are equally adjusted to what you need to get out of it and what you might get out of it if you’re lucky. Anyway, probably a long answer to your very good, short question. [LANG] No, it’s really interesting. [DUNCAN] It’s a great answer. [LANG] You’d mentioned co-creation there a little bit and that’s something that, I think, there’s been some speaking of as well in our industry. [BEN] Yes. [LANG] Can you fill us in on co-creation. Like, how you actually harness people and get them engaged on something which may not necessarily be sexy or an interesting topic to the consumer, but it’s really obviously very important to your client, so that you actually – I guess my ultimate question is through co-creation, can you unlock the door to greater brilliance than more of a traditional approach? [BEN] I would say the answer is probably yes. And I think that what falls under the bucket of co-creation is also large. So, what I’ve found to be pretty successful in this regard is to get people together in a setting that is, if not real, then is at least making a solid, playful effort at creating something of an immersion for them to get them into the right cognitive and emotional state. And so, we often do co-creation activities with people towards the end of projects where we may have done a bunch of initial fairly open-ended research, we’ve come up with some general opportunity spaces for people, for our clients to innovate in. And then we might bring end users or consumers in, and what we’re probably going to do is – here, I’ll give you an example. We did one on cooking and mixing and measuring and like really basic stuff. Like, when you think about cooking and baking, mixing and measuring has not been innovated in – in what 75 years? [DUNCAN] If that… [BEN] Whatever – it’s been a long time, right? It’s been a very basic kind of a thing. And so, getting people to engage with that is an interesting challenge. But we have tremendous visual information because we were in people’s homes and we watched them cook. And so, we created a fairly life-size kitchen environment to bring people into, and we had video stories of people just like them who were talking about how they cooked. And being able to use those research materials as a way to get a new batch of participants into the topic and having some connection to the challenges or that being able to and then share their own stories and, therefore, be in a projective state of mind can be really very effective. I think that the hang-up that a lot of clients have is that they think that that’s going to produce the idea – the one idea. Or that there will be some sort of consensus at the end of these things that yields the brilliance, and I think that’s a mistaken expectation. We often follow co-creation events with co-design events. So, what you learn in co-creation is often how far your users or consumers are already able to imagine and push beyond. You can learn sometimes what they feel like are barriers in the sense of, “Oh my God, I would never want to go there,” kind of a thing. You can learn a lot in the way that they attempt to solve problems and tell stories and imagine things working and not working in their own kitchens or wherever that is. So, the ideas that they come up with are actually a set of interesting things that you should tear apart. Like, “Oh, so this is really telling us the story about we have permission to live on the countertop, but we don’t have permission to be plugged in. We have permission…”. So, you’re pulling out of a session like that what I would often call consumer or user criteria for something like success. And also, like drop-dead, bozo no-noes for failure. And so, what that does is that sets up a professional ideation session where you have designers, engineers, market strategy people, etc. who roll up their sleeves and say, “OK, we learned this from the primary research. We learned this from co-creation. No what can we do with that?” Because that is a tremendous place to start from, as opposed to starting from the Post-It note on the wall. [LANG] So, when you do those co-design sessions, is that also with the consumer or with the target audience, or is it that like what you said – with the engineers and the other folks… [BEN] It’s more internal, yeah exactly. And because you’re really working on more fine tuning of ideas. You’re mixing and matching ideas. You’re trying to hybridize things and take them in different directions without… You don’t want, in that scenario, a person who’s judging the outcome, right? At that point, you’re really in a moment of trying to push the boundaries in different directions. And then, that’s often best handled by people who are really comfortable with doing that and with the ambiguity involved and looking to drive that. And that can often unfold over weeks. You do one session, pull things together, crystalize them, come back again after you’ve let your brain marinate for a week and you do it again and you often you wind up with some much more refined concepts that have all different kinds of nuance baked into them because you gave it time. You took some time in the field to really learn primary things from end users and consumers. You took time with consumers to hear how they would project their own solutions into it, and then you’re combining that with, of course, with some of the company’s own criteria involved. It needs to make the magical hundred-million. It needs to keep a certain factory functioning. It needs buy supplies from wherever, right? So, there are all sorts of interesting business criteria – you don’t want to spray those all over your consumers and users, but there is an important moment for them to play that roll. [LANG] This might be slightly off topic, but I can sort of almost see the world of cars, with infotainment systems as a really fast-emerging that’s happened over the last 10 years or whatever. [BEN] Yeah. [LANG] Where even with like with where people put their smartphone when they’re driving – you can see the redesign of the consoles and dashboards. And the other interesting thing I find is it’s very fractured right now. Each company’s approach to the infotainment system and functionality and touch screen or a knob in the centre of the console – they’re quite different from each other so that we haven’t really got a standardized kind of thing. And I wonder if some of the techniques you talked about would have helped auto manufacturers to go along to where they’re at now. [BEN] I can only imagine that it has. There’s – I haven’t done any work for big car companies. Happy to do that if they ever want to reach out. But I do think that you’re spot on with that. People are innovating in different directions. I think that there’s an underlying notion in that industry that drivers are going away. And so, there’s this – so much of this technology is moving so quickly that the sensibility for making really big investments is probably small. [LANG] Yeah. [BEN] So, here’s a funny example of that. [DUNCAN] Awesome. [BEN] Right. We did a project with a fast food company. This is years ago. You remember when key fobs were all the rage? Where you would put your loyalty card from the supermarket that goes on your keychain – it’s got a barcode or what have you on it, right? So anyway, this fast food company, they do tons of drive-through business, etc., etc., and they wanted people to scan their loyalty card at the drive-through because that way they get their discount, they get their favourite meals already pre-loaded in there, etc., etc. – speed everything up, right? So, we do a lot of work with this company over the years and there was this brilliant moment where we’ve got the key fob. We’re giving it to people. We’re taking it through the drive-through. And it becomes obvious what a classic epic fail this would be because where is the key fob? [DUNCAN] On your keys. [BEN] Where are you keys? [DUNCAN] In your ignition. [BEN] Yeah, right? So, a massive investment in key fobs and all that kind of stuff would’ve been a horrible idea. And they were able to find that out pretty quickly. But you can push a good idea really, really far in your head without sometimes hitting that reality moment, right? Where you’re like, “God, that doesn’t make any sense.” And sometimes it can go much further than you think. It’s hard to invest in technology that changes that quickly because that went to RFID, RFID goes to smartphones. So, how much are you going to invest in the console of a car and an entire infotainment system if in five years, half the cars are going to be driverless. [LANG] Yeah. [INTERVIEW PT.1 END] [INTERSTIAL MUSIC] [ADVERTISEMENT] [DUNCAN] Before investing in a new advertising campaign, you need to know it will work for your brand.
At Insightrix Research, we developed our ad testing solution to be the most comprehensive tool for assessing, predicting and optimizing advertising performance.
Insightrix Ad Check is a pre-testing research tool to assess advertising before it is launched via consumer testing to measure and maximize ad effectiveness and purchase intent.
Insightrix AdCheck will optimize and enhance your copy for the greatest impact and identify the most effective creative to produce and run, allowing you to get the most of your media budget by putting your spending behind proven, strong advertisements.
Using standardized copy testing measures based on a battery of questions, we can help to quickly execute your research project, meet tight timelines and build norms.
Insightrix AdCheck survey designs are flexible! And that means we can work with you to accommodate your individual and specific campaign requirements.
It costs the same to produce a bad ad as a good one. Early stage pre-testing can ensure your final production costs are invested in the very best creative.www.coniferresearch.com. On their website, you can also check out the Research RX tool that we spoke about in this episode. If you didn’t get the URL, we’ve posted the information on our website at insightrix.com/podcast. And of course, we’d like to thank you – our awesome and loyal listeners. Without you, none of this would be possible. If you enjoy Stories of Market Research: The Insightrix Podcast, why not take a minute to rate and review it on Apple Podcasts, Stitcher or anywhere else you access your podcasts. It would mean a lot to us, and it helps get the show in front of more people like you – folks with an interest in research. And if you’d like to know more about Insightrix Research, you can check out our website at Insightrix dot com, or follow us on social media on Facebook, Twitter or Instagram. We’ll be back in another few weeks with another episode of Stories of Market Research: The Insightrix Podcast.