Following on our article (“Numbers and Stories”, 23 November 2009), our friends at Retailwire.com thought it would be interesting to run a poll to ask the Retailwire community what they thought about retailers using research. The original discussion is here on Retailwire, but we’ve reproduced the comments and the poll results as they stand today (16 December 2009).

As evident from the graph below, the short answer is “no, companies don’t use research well”; only 15% of the respondents felt that companies are “good” in using research, at their best. Should we blame the companies or the researchers? The comments seem to suggest that the blame needs to be shared equally.

Comments below:

 

This sounds a lot like a chapter we wrote for ESOMAR’s Best Practices book. We have devalued research in favor of insights, which can rely much more on a good narrative and much less on good data. A management team that expects insights from research all the time is asking for trouble down the road. A research team that doesn’t focus on quality first and insights second is doomed to failure when management makes the wrong moves. Research needs to give management the best information possible in a way that management can understand it. Management needs to understand that research is providing the best information it can within budget constraints. The two need to work together. [Stephen Needel, Managing Partner, Advanced Simulations]

One of the concerns I have at present is how SKU rationalization research is viewed, so quickly judged, and acted upon. Many retailers are looking only through a narrow interpretation based on shear numbers and not taking into consideration other more visionary factors about specialty brands, niche items, and growth brands. If this keeps up, consumers will have very few choices and most of the stores will all look the same with exact assortments. Only price will differentiate one from the other. The results will be rather ironic. [David Biernbaum, Senior Marketing and Business Development Consultant, David Biernbaum Associates]

I agree that there is a lot of bad “research” out there in the world. Any analytical study has to be right, applicable, and actionable. If a study doesn’t meet these criteria, it is worse than useless–it can actually pollute the minds of decision makers by letting them think they know something they don’t. Before spending any valuable share of mind on numbers, executives should ensure:

1) Is it right? If I only had a buck for every time I’ve seen a big name consulting firm presentation with numerical “findings” using a flawed methodology or with no statistical significance…. I’d be retired in Paris right now.

2) Is it applicable? So, some other retailer says their TV spend has a 150% return (or some consultant claims that). So what? Your business is different. Consumers react differently to every retail concept. You do need to know for you.

3) Is it actionable? Oh, we all know about the study designed to validate the CEOs hunch. Want to guess what the consultant’s findings will say? If you are going to do research, you’d better be prepared to act on the findings–either way.

This is why in-market testing is such a powerful technique in retail. While it does take commitment to do it right, it is one of few techniques that almost always meets these criteria. [Jonathan Marek, Senior Vice President, APT]

It is absolutely true that marketers do not often take the time to understand the basis of the research that they are presented with. Understanding how the research is developed and the analysis approach used to develop recommendations is an important, if misunderstood, part of the job description for a data-driven marketer.

I do find, however, that marketers more often do not carry out sufficient research to draw conclusions, even when that research is relatively easy and low-cost to execute. If you have access to email addresses, you can execute basic research surveys to customers and gain valuable insights in less than a week, at a very low cost. Those opportunities to “fill in the gaps” are often overlooked. Sometimes those insights can make the difference between success and an indifferent failure of a key initiative.

Marketers must understand the opportunities that research affords them, even when timelines are tight. Obtaining the Voice of the Customer, particularly the Best Customer, is a practice that should be followed religiously. Only then will marketers be able to gain insight and make truly data-driven decisions.  [Mark Price, Managing Partner, M Squared Group, Inc.]

We have to keep in mind that corporations are run by human beings that often make decisions on emotion rather than logic. I have many clients who have been very successful making decisions by shooting from the hip, yet they always prefer to see my research, just to be sure. Most of my clients are very bright people and my research generally confirms their own instinctive thoughts. There have been times when I have been brought in to do an autopsy on a project to find out why a store failed. Typical reasons are:

Researchers did not want to offend management so they candy-coated the results.

Key decision makers are suffering from some kind of physical or emotional impairment which affects their ability.

Corrupt middle managers that change the research results.

Researchers leaving out a key piece of data (i.e. not telling management that the Mexican format store they have planned is in a Puerto Rican neighborhood).

Overall, I don’t think retailers have a narrow view of research. Researchers can do a better job in communication by simplifying the results, being blunt, and putting their integrity ahead of their paycheck. [David Livingston, Principal, DJL Research]

I think there are two elements at play here:

First and perhaps foremost, retail is an emotional business. We can have reams of data and still use the words “It feels like….” and make significant decisions based on those gut feel moments. Certainly this has long been true in the world of merchandising. Something “feels like” it’s going to be a home run or a dog, and it “feels like” we’d better take a markdown or run a promotion to goose traffic. And actions are taken accordingly.

Now, can I tell a retailer in all honesty to ignore those gut feels? I really can’t. I can encourage them to use data to support actions taken based on those feelings and obviously that’s what I do…every day. But I can’t ask them to ignore their gut completely.

This brings me to the second issue: we don’t always present data in an easily distilled and understandable format. Our retail survey respondents repeatedly ask to have their Business Intelligence delivered in simpler ways. While “red light, yellow light, green light” might be a little too simple for some decisions, the data just has to be usable and quickly actionable.

Finally…if a retailer (or any company really) is going to make such a dramatic shift, it has to be driven from the top. And the C-level exec driving the initiative also has to LISTEN to what he/she is being told in response. Otherwise you get a company similar to Home Depot under Nardelli.  [Paula Rosenblum, Managing Partner, RSR Research]

Decisions are always made without perfect information to support them. But sometimes, decisions are made that ignore the available information or decline the implications.

Research is clearly most valuable when it can be turned into actionable recommendations. We are all too aware that research can be used as a fishing expedition without a clear objective. However, there is also a danger in using research designed just to prove a point rather than develop real, new learning.

On balance, I believe that research, properly done, interpreted, and acted upon, can vastly improve the decision making process. [Ray Jones, Managing Director, Dechert-Hampe & Co.]

I’m reminded of the Samuel Taylor Coleridge quote “Water, water, everywhere, nor any drop to drink.” Research/knowledge is the raison d’ tre for The Luxury Marketing Council www.floridaluxurycouncil.com; it’s what I do. The highest level executives or those who will someday sit in the corner office recognize the value of research. They’re able to sift through the myriad of information to find what’s relevant and actionable.

There are a couple trends re: research. Some executives don’t want facts to get in the way of their vision. Historically, these individuals’ careers stall. Also, there are many companies that have pared down their employees to the point that executives don’t have time for the facts–they’re too busy keeping the ball moving. The third group often doesn’t understand how to read the research, how it affects them and/or how they can use it for their benefit.

Relevant research is an imperative for retailers. Having a 360 on your targeted customer and understanding their collective experiences is the key to personal and business success. [Chris Ramey, President, Affluent Insights]

Devangshu Dutta mentions many of my personal concerns regarding research. But before I go any further I have to add, my name is Joan and I am a researcher. I’ve spent many years working in the industry to help alleviate some of the barriers Mr. Dutta listed. There is still much to be done of course.

In this world of easy access to numbers/statistics, upper management demands and gets “stuff” by which they make decisions. Trying to explain the difference between good research and everything else often falls on deaf ears. Management expects those who supply the data, whether they are staff or outside consultants, to bring the quality, validity and relevance required. And in turn they do not question the underlying premise of what they are buying, or what they are buying into.

And when forecasts don’t work out as expected, new products fail and marketing strategies are ineffective, it’s all about how research failed to deliver. Researchers preach to the choir when they have meetings about quality standards.

CASRO (Council of American Survey Research Organizations) has a Code of Standards and Ethics. Companies who belong to CASRO must adhere to them. And this is one way in which clients can insure research results are reliable.

CASRO is initiating an ISO certification program. I believe more client companies and executives will relate to and understand ISO (because their own companies go through similar ISO certification processes), perhaps choosing vendors and staff accordingly. In my opinion, this could be a turning point for the acceptance and recognition of true quality marketing research. I hope so.

As for the story telling aspect of research…well that’s another “story” entirely. I’ve mentored many researchers and advised them to always answer three basic questions, What? So what? And Now what? These questions get to the heart of why research is conducted in the first place. Good reporting and presentation requires training. If you believe that anyone can be a researcher and choose vendors or staff based on that assumption, you get what you get and it may fall below the standards Mr. Dutta advocates. [Joan Treistman, President, The Treistman Group LLC]

During my 11 years with Kenosia I used a phrase with clients “One truth.” By combining disparate data sources, a retailer or a brand manager can get to the “one truth” and then make a business decision regarding direction. Far too often, decisions are made using one source of data which can lead to less than effective results.

Example, if a retailer only views their loyalty data to make business decisions about advertising, are they understanding all the trends happening in their market? Probably not. Combining their loyalty data with demographic data makes it better and adding additional information from a syndicated data provider makes it even better yet.

The great news is, there are a dozen technology solutions to help both retailers and brand managers combine data and the data to combine is available and affordable. It boils down to first understanding the questions and then going out and combining all the best data sets to create the answers. [John Boccuzzi, Jr., Managing Partner, Boccuzzi, LLC]

The use and misuse of research, data and “insights” varies widely across retailers and brands. Typically, the larger the company, the more primary research they have and the more reliant on primary research they are.

Unfortunately, great research and insights are, as many here are illustrating above, not nearly as commonplace as they should be. There are many reasons for this:

– Flawed and biased methodologies (e.g., let’s “Focus Group” this”);

– Vendors who specialize in one research/data collection area over another;

– The research goals and objectives themselves: pure answers to hard marketing questions rarely come directly from research but rather from what is done with it, i.e., what is the data needed to create information to support or disallow a hypothesis?

Budgets, which have mostly seen cuts for the past two years; though the cost of collecting data has come way down in many cases.

All these challenges with research underscore the importance of knowing who your customers are, having an ongoing dialogue and relationship with them, and gleaning insights from them. There is nothing better than customer (transactional) data to gain an objective perspective and insights for your business.

Unfortunately, many companies are data rich and insight poor. Even worse, many companies, retailers included, don’t know their customers or how they behave. This is continuing to change for the better, however, and those who are focused in this area are the ones who will make better decisions and be more successful in the long run. [Phil Rubin, CEO, rDialogue]

One simple question expresses the confusion around statistics: “Why do we have Democratic and Republican Pollsters?” I think it was Harry Truman who when confronted by economists telling him “Well on one hand the statistics are saying this, but on the other they could mean this” said “Someone get me a one armed economist.”

The thing with retail is that we don’t need answers to thousands of different questions. We ask the same questions a thousand times: how does this product sell, what is its net profit, how important is it to my customers, does it fit my brand objective, how does it relate to other products, are there viable substitutes, etc? Instead of poring over tons of numbers, the POS data should be used to construct answers to questions.

So the fundamental reason retailers (and anyone, really) make bad decisions from raw data is because they don’t know what questions they’re trying to answer. Start there.  [Bill Bittner, President, BWH Consulting]

The fact that information is available and is being used effectively are two separate things! Usually a company uses too much information, regardless of correct or incorrect, or it does not use information. Very few companies strike and maintain a balance between insight, gut feeling/intuition and relevant, timely information in decision making. [Pradip Mehta, Principal, Mehta Consulting, LLC]

I’ve found that the overall views of research within companies goes in cycles. Of course, leadership changes come into play with the research points of view as well. Some executives understand how to use it better. Some have used it so much they can go by gut.

In any case, a research cycle may start when a huge mistake is made by using strictly gut instinct (Tropicana?), then more and more insights will come from research and less from gut until, one day, it is determined that there is too much science and not enough art, and the cycle starts all over again.

I do believe though, that research is one of the faults with our nation’s fashion business today. Too much science, not enough art. The talent and guts it takes to take a big chance on new, fun fashion seems to have been relegated to trend reports, focus groups, ethnography and best seller lists. The rare exception to that rule is Forever21…instincts still survive there. Perhaps they can teach the industry a lesson. [Lee Peterson, EVP Creative Services, WD Partners]

Where would Disney be without fairy tales? SMWeiss’

The problem, dear Brutus, is in ourselves. Retailers and brands have created vast action machines with thought paradigms behind. Anyone doing research is likely to be looking to fit information into the existing machine. And for each individual, when they formulate a research query, they bring their own current thinking into it–obviously! This means that the results they get back reflect, in far too great a way, their own predispositions. This, coupled with the fact that lots of research is “ask” type research–interviews–guarantees a sluggish and distorted view of reality, what I call the Picasso business view; shared distortions, unperceived as such by both researchers and respondents.

The partial antidote to this is “observational” type research, and the hierarchy of truth. And observation here must be of the real world, not some laboratory simulation, which typically just further cements the existing distorted paradigm. The hierarchy of truth means distinguishing between what is most rock solid and least likely to be distorted, and that which may be as changeable as the weather. (That’s right, Maude. It used to be just common sense that the weather changes. ; )

One example near and dear to my heart is OBSERVING how many items people buy in a store. The most common number, whether in a convenience store or a supercenter is ONE. But the “world” absolutely refuses to believe this, because it does not match their own conceptions/perceptions. An observation like this is at the pinnacle of retail truth, and must be allowed to shatter any part of the paradigm that does not conform to it. THAT’S “revolutionary,” but it is also the route to racing past the competition that is spending their time dancing on peanut butter.

No one should reproach themselves for participating in a social Picasso view of their business. (Although I may insult you from time to time. ; ) All of us are afflicted with the phenomenon noticed many years ago, about historians and their “research” of the past. Looking into the past, for a historian, is much like looking at a reflecting pool at the bottom of a very deep well. The image the historian tends to come away with is a reflection of himself. Business people are no different. [Herb Sorensen, ScientificAdvisor, TNS Global Retail & Shopper Practice]

There are plenty of great tools to make sense of all of the information. The challenge is balancing it with the human side. Take, for example, the stores. Are all corporate employees required to walk the stores and report what they see? View the stores in the evening when store management has left for the day. These age-old problems are still far too common, yet they are the source of some of the most valuable information available anywhere. Sales will be increased, labor decreased and earnings improved if this information is acted upon. [Ralph Jacobson, Global Consumer Products Industry Marketing Executive, IBM]

Just after noon, on a weekday, I bumped into a family acquaintance at one of the more successful shopping malls in the city.

The question, “What are you doing here?” was underlined by a mildly accusatory look and the subtext, “Why are you spending a week-day shopping?”

My response that I was “working” wasn’t enough; the further explanation that I was doing “research” received a dismissive smirk and ended the conversation. The fact is that I was repeating the time-honoured ritual of RBWA (research by walking around), with its seemingly aimless strolling, sidelong glances, and possibly turning over a hundred items in a dozen shops without reaching for the wallet even once. This is a ritual that is not taught in our temples of management learning. In fact, it is one of the many tens of methodologies that seem to be missed out during the course of our formal education. And very often, what we do get taught is so remote and opaque to most people that they will promptly forget it the moment they walk out of the examination hall.

I was reminded of this walk-about incident during a conversation with two members of the faculty of a professional institute on the subject of research. Most of their students, I had observed, had a narrow interpretation of research – focussed only on consumers being interrogated through a questionnaire. The students were working from the guidance they had received during the previous semesters at the institute.

Unfortunately, the students are not alone – this is also how too many people identify research, including many executives in decision-making positions. I have been frequently puzzled by the confident (brash?) statement I have heard many times: “We don’t need research.” It is only when I probe further do I, and they, discover that while they perhaps don’t need consumer surveys, there are large gaps in their decision making toolkit which can only be filled by inputs from various other kinds of research.

Sometimes the roots of that statement lie in the perception of research as an impenetrable jungle in which it is easy to get lost but difficult to find something immediately useful. Researchers, like all other vocations, have their own professional shorthand (also known as “jargon”) which they sometimes use to identify their own kind, and perhaps sometimes to exclude people who are not from the trade. Very often this jungle is created by “research-as-a-foreign-language”, which many executives are just too apprehensive or too busy to tackle.

But before you pick up the axe and start cutting away at the creepers of bi-variate analysis, quota samples, correlation and projective techniques, let me give you my very simple definition of research which I like to keep in mind when I am asked the question: “Do we need research?”

To me, research is the discovery and collation of diverse pieces of information from various sources, so that it can be analysed using multiple tools, to discover relationships, patterns and directions that can be used to draw conclusions and take decisions.

There is a purpose for which we would discover or collate that information. There may be a set of questions that we need to answer. We need to understand what are the various places where that information may lie, the different forms it might take or the different ways in which we might need to look at the information before anything useful emerges.

And, in the business context (as in many other situations), research is meant to come up with something that is applicable and directly beneficial to the business. So once we’ve got most of the answers we were looking for, it is certainly useful to stop and apply the newly gained knowledge rather than try to refine and perfect it to the infinite degree.

If this definition of research frames the context well enough for you, then you’re on the way to doing and using research well.

Despite the wealth of information available today, far too many bad business decisions are being made in the absence of good information, either because the executives have not bothered to carry out research, or have not had the capability or the time to question the research which is being presented to them.

Worse – perhaps because of the abundant data and the ease of access to it – today many business decisions that turn bad are taken on the basis of information that is presented by someone else (“secondary research” in research language), without questioning the validity of the conclusions, the structure of the study, the context in which the data was analysed. It’s almost as if we couldn’t be bothered to think, because someone has apparently already done the thinking for us – especially if it comes from a “reputable source”. (Ok, that might be smart sometimes. So let me give you a more graphic analogy – could you think of an adult bird regurgitating pre-digested food to feed the chicks? Hmm, not so pleasant an image after all, is it?)

Also, research (especially the number-oriented kind) seems too dry for most people to take in. And I think that is one place market researchers could do themselves a huge benefit if they could tell the story – especially a story with a moral at the end. That is, create the picture for the user as to what all of that information means in simple language, and also show the user how to use the information in the context of his situation or problem. Bedtime stories during childhood and good movies in adulthood work well because there is a coherent narrative, a start, a middle that is interesting and an ending that stays in the mind. You can see the relationships between the characters, and the consequences of those relationships. A good research project report could be seen as something very similar.

Having said that, of course, there are also some researchers go far beyond, who would never let boring facts get in the way of a good story! Apparently a letter to the editor of the National Observer (London) as far back as 1891 complained: “there are three kinds of falsehood: the first is a ‘fib,’ the second is a downright lie, and the third and most aggravated is statistics.” (Mark Twain famously paraphrased this in his autobiography as “lies, damned lies and statistics”.)

How many stores can you think of which are located at sites where their chances of success are exactly the same as that of a snowball in hell? How many products or brand launches come to mind, where you wondered, “what is this company thinking?!” Of course, there would have been pre-launch studies which would have showed just how successful these would be, where the stories were possibly based more on imagination than on facts.

For a decision-maker, the only way to tell the difference between bad statistics (lies) and the true story of the market is to make sure that he or she is equipped with multiple sources of information, and various tools with which to analyse them. Also, if you recall my earlier definition of research, the starting point was the definition of the objectives which a research is supposed to fulfil – if the objectives are vague or undefined, so will the research outputs be.

Numbers (quantitative research) and narrative (qualitative research) can tell us many wonderful stories about the market. Some of those stories are highly imaginative “fairy tales” because of a bad study – that shouldn’t lead us to ignore all the others which can direct us to our objectives.

BOOK REVIEW: HIDDEN IN PLAIN SIGHT: Erich Joachimsthaler

In the midst of extensive or frequent civil works, fluorescent high-visibility clothing contributes to the invisibility of the individual, and can serve as a superb disguise. Similarly, in the midst of extensive research and in-depth analyses, basic insights can go unnoticed.

Erich Joachimsthaler has plenty of examples in his book Hidden in Plain Sight to drive home the point that attention to stuff that is not so obvious to competition can lead to brilliant success such as Sony’s growth through innovative products (the WalkmanT, for one) that met unexpressed consumer needs. Conversely, an inability to spot this can bring even the leaders down, illustrated once again by Sony’s loss of leadership in mobile personal entertainment to Apple’s iPod.

The challenge for companies is to uncover the hidden opportunities by looking into their business from the outside rather than the usual inside-outwards view, and by accurately defining the ecosystem of demand. For most management professionals, this will be harder than it seems.

The exercise begins with the question, “Why didn’t we think of that?” This is intended to remind the reader of how the obvious escapes attention as we sink deeper and deeper into complex analysis and in developing ever more complicated scenarios. And Joachimsthaler sets out a framework that he believes can help larger companies to innovate in a structured way.

Of course, the reader may feel differently, and quote George Bernard Shaw who divided the world into two kinds of people, the reasonable and the unreasonable, and credited innovation to the latter. Or one may agree with Henry Ford who, apparently, felt that customers did not really know what they wanted. He is reported to have quipped: “If I had asked my customers what they wanted, they would have said, ‘A faster horse'”

Yes, at the cutting edge, innovation may seem to be more about the innovator’s creative desire to do something different, and less about “meeting customer needs”. Yet, it is the unmet and, more importantly, unexpressed customer needs, that offer the greatest source of competitive advantage.

This is why innovation seems to spring more from small companies, or companies that are started up around a specific idea that is unique or new. In such a small company or a start-up, typically the founder/innovator/inventor is drawn from the same pool as the target customer. Therefore, while they may be addressing a need they feel acutely, the innovators are unconsciously plugged into their customer’s unmet/unexpressed needs. There are seldom any silos; the whole team is generally focussed on the one problem to be solved.

However, as companies grow larger, functional specialisation emerges — division of labour based on skill-set is deemed to be a more efficient way of doing things. The design folk design based on “trends”, the marketing folk market as they know best, and the manufacturing folk produce to specification and the “demand” generated.

With this speciality of skills taking over, there is a growing disconnect between their efforts to dig for insight and the gold that is “hidden in plain sight”. While data is available in abundance, real knowledge is scarce, and insight just gets buried in well-structured processes and hand-offs between functional silos.

This trend has only accelerated in the past 15-20 years with pervasive information technology that enables the mundane operational process to the most strategic. Never before have management teams been so focussed on information and analyses. As businesses grow, data warehousing and data mining are defined as the competitive cutting edge, pushed along by interested parties (including IT solution providers, but that is another book!).

However, in reality, excessive information is increasingly passed off as knowledge. An inward focus on the management team”s own objectives is often disguised as insight gained on the customer or the market. Functional specialists analyse the market, the latent needs and the gaps in their own way, and if the company is lucky to have some generalists, some of those dots get joined to form a more complete picture.

It is in reminding management of this reality that Joachimsthaler’s book provides a tremendous service. It presents a well thought out model named, curiously enough, DIG – short for Demand-First Innovation and Growth. The three elements laid out sequentially begin with a framework for defining the demand landscape, identifying the opportunity space within it, and then creating a strategic blueprint for action.

Joachimsthaler’s process to define the demand landscape requires managers to put themselves in the customer’ shoes – a process demonstrated with examples from Proctor and Gamble and Pepsi”s Frito Lay. Using the customer’s goals, actions, priorities (there’s the “GAP”), needs and frustrations, demand clusters can be developed and filled out with additional research. The strategic fit between these demand clusters and the brand can then feed into the next steps of identifying the opportunity space.

The filters, or lenses, as the author calls them, are the “eye of the customer”, the “eye of the market”; and the “eye of the industry”. At every step, assumptions and presumptions need to be challenged. Using these lenses, the sweet spot or spots and the growth platforms can be identified, and extrapolated into the strategy. On the downside, the book is clearly about a framework, which may have been best detailed in an article, rather than being stretched over a book.

The author does stress at one point that it is not about “brainstorming”, but about structured thinking. However, he seems to do this in a tone that suggests brainstorming as something vaguely distasteful due to the lack of directional structure.

While examples from the companies studied keep the text alive, yet in places one struggles to correlate the examples with the framework. Indeed, there may well be too much structure to this book, and not enough examples of how inter-disciplinary thinking and functioning can actually produce sustained innovation.

Understanding the model itself can be a fairly involved process. The best way to tackle it may be to approach it as a project, and use the DIG framework as a how-to guide for a real problem. If you are a structured, methodical, sequential kind of manager and possibly work in a large company, the book could provide tools to put that thinking to work for innovation in a team. On the other hand, if you are more of a “people person”, you may want to leave this book alone. [For more, here’s the book on Amazon.]