Quant before Qual makes no sense. But it does.

As I continue to explore how designers can make better informed decisions by leveraging information, the issue with number aversion is still #1. I talked about this already in my Interaction 10 presentation, but I’ve been digging deeper and have some other thoughts (check my presentation for some base assumptions).

If we agree that quantifiable data, specifically the ever popular web analytics, provide you with rich detail to tell you WHAT is happening, it is comforting to realize that it is the type of data gathering that we already do – design research – that provides the qualitative color to answer WHY said things are happening.

What I am finding, however, is that it is more valuable to START with the quantitative work and get to the WHATs and ask WHYs based on those findings, rather than trying to figure out WHYs in exploratory mode (even if the WHAT’s are going to emerge at one point or another in this quest).

My point is that it’s not sustainable as an approach. It’s inneficient to start digging deeper to answer the WHY questions if you don’t have a baseline of WHATs identified.

The problem is that it is not intuitive for designers to start where they are uncomfortable. We are super comfortable with qualitative approaches – they are our go-to tools because that’s what makes sense for design research. However, quantitative research instruments really help narrow stuff down, but they do require you to understand those pesky numbers in order to a) dig in and get to concrete answers and b) understand what it’s saying so you can ask “why”.

In short, WHATs before WHYs are more efficient than WHYs before WHATs, but that requires designers to start with unfamiliar tools to then apply familiar tools. If it was the other way around I think it would be much easier for designers to bridge both approaches and come out the other end with more useful insights.

In other words, since we don’t particularly feel an attraction to numbers (to put it lightly), why would we start there? It’s such a leap from how we think about problems that it is counter intuitive. I don’t believe designers reject the notion of starting with Quant approaches (WHATS) to expand with Qual approaches (WHYs), but it’s inherently counter-intuitive to think that way.

How can I help designers do this when it goes against their nature? That’s what I’m working on right now. More on this later.

11 thoughts on “Quant before Qual makes no sense. But it does.

  1. Steve Baty

    Liv,

    The question of whether it makes sense to tackle ‘what’ before ‘why’ is an interesting one, but I wonder whether it is dependent on the nature of the design problem. It strikes me that quantitative methods as a starting point are really only useful when you have something concrete to measure beforehand.

    This is an issue I touched on last year in a presentation to a group of design students. There is a class of problem wherein identifying the ‘what’ and then the ‘why’ makes a great deal of sense. But there is another class of problem when you need to be asking ‘why not?’ instead, and a metrics-driven approach doesn’t lend itself to this mode of thought.

    Cheers
    Steve
    (lover of numbers)

  2. Livia Post author

    Hey Steve, I’m sure those circumstances are there (though I’d love examples!). I am really thinking of it in terms of efficiency; if you just take your average design research exercise, the most challenging part is defining what to study. That’s really what I mean by defining WHATs before WHYs. I feel that quant can help get us there – possibly faster, but that’s me speculating.

    I am saying this in part because not every designer is well versed in all research methods. The most common methods rise to the top and quickly become the go-to tools that people use over and over. These are *generally* research instruments that yield qualitative data. I do think some early quant data can help regardless of the problem to be solve, if nothing else to help scope it.

  3. Steve Baty

    Examples: what metrics would I look to before designing the “Belly Bra” for pregnant women? What metrics would have made sense to the designers of the Kindle? or the Walkman?

  4. Sam Ladner

    Liv, I think you’re onto something here but I’m not sure I agree it’s a qual/quant divide.

    You are right; design researchers often reinvent the wheel when embarking on a research project. They repeatedly, incorrigibly ignore mountains of previous insight gathered on their topics. They fail completely in covering the basic findings already published — sometimes for decades! — and instead start from scratch. Again.

    Now does that mean they ignore the WHAT? Yes, but they also often ignore the WHY. If there are Web analytics, they may ignore those data. If there are previous ethnographic studies on, say, mobile use among teenagers in public spaces, they ignore that too. Both the WHAT of how many mobile devices ping the current site and WHY teenagers use mobiles differently with their parents than their friends — both of these already established answers are ignored.

    Interaction designers may ignore the WHAT more readily than the WHY because numbers are icky. But I would argue that they ignore WHYs found by other researchers as well.

    Why does this “start from scratch” phenomenon happen? I believe it because interaction designers are not taught how to do research. Sure, they’re taught a bunch of research methods — usability studies, ethnographic interviews — but they do not take “methodology,” which is the study of methods.

    Social scientists must take these courses. They must learn where a particular method comes from, what assumptions it carries with it, and when it is appropriately used. This includes the dreaded “literature review.”
    And btw, yes, teenagers use their mobiles differently with their parents, and how do I know that? Because I read a paper about it.

    Steve’s point is spot on as well — I think many people dismiss research that isn’t EXACTLY FOCUSED on, say, belly bras (whether it’s qual or quant). Methodology teaches you to find relevance in all sorts of data that may be only tangentially related to your topic. So perhaps you found some metrics on number of bras sold before, during, and after pregnancy. Then you found some qual research that found pregnant women “feel ugly” in most pregnancy bras. Voila! A triangulated insight!

  5. Livia Post author

    Brilliant Sam! You put it way more eloquently than I ever could. I absolutely get what you’re saying; I feel that myself as someone who has not had much of the type of training you describe.

    I was definitely focusing more on the lack of attention to existing WHATs, but I certainly agree that ignoring previous WHYs is also common occurrence. I guess my main hypothesis here is: if all things are equal (which they are not) and designers have the opportunity to start whatever they are investigating by looking at existing WHATs or existing WHYs, they will absolutely go to the WHYs first and not the WHATs. (yeah, most times they are just not going to either, but you get my point).

    In short, yes, it’s not about quant or qual necessarily, but whats and whys. And as you put it, it’s not about research methods but methodology. Hmmm.. how do we tackle that problem???

    Thank you both for helping me untangle the issue!

  6. Adrian Chan

    Great discussion. I’m going to complicate it a bit by suggesting that it’s not a matter of objective Whats and Whys, because there really are no such things. Rather, it’s a matter of Which Whats and Whys.

    Design research ought to help in assessing, by means of reflecting on our own premises and subjective understanding of the design problem, Which matter most. And of using data to surface gaps in our knowledge as well as to verify and validate the relevance of the Whys Which we have chosen to address.

    The reason quant/qual is always a squirmy problem is that it has two worlds — the subjective world of design and process of designing for solutions to subjectively defined problems. And the objective world in which object problems, processes, and solutions exist.

    Resolving the quant/qual matter then is about choosing the Which Whats and Whys that matter. Of constraining the problem area to a specific set of definable design goals against which success can be measured.

    So it seems to me — thanks for bringing this up. I struggle with this all the time in social, where our notions of what users like are based entirely it seems on what they’re doing, which is of course limited to what they can do. So Which one matters is a big issue for me — and should be used to reflect on our own assumptions as much as to guide us in what we see happening in the world of user/consumer behaviors.

  7. Jared M. Spool

    It feels like there’s an implication that everything we can measure is good for everything we want to measure. By saying we need to think about WHAT first, then WHY, or vice versa, we imply that every measure works in every context.

    Take conversion rate. It’s often used to measure an overall site’s e-commerce performance. Unfortunately, because it’s a ratio of purchases to visitors, that’s a stupid way to measure performance, you can increase conversion by decreasing visitors. The business doesn’t benefit, but the metric improves.

    However, when looking at specific marketing campaigns, say ads on particular sites, conversion rate makes a lot of sense, since the ratio normalizes the traffic abnormalities.

    So, I’m not sure that you can separate the WHATs and the WHYs. I think good methodology, as Sam points out, is both.

    Jared

  8. Livia Post author

    Adrian: Well put! The WHICH part (which is what Jared is pointing out too), is definitely a hairy problem. If people don’t even know or care to look at WHATs in addition to WHYs, makes me think they are likely not excellent at determining WHICH Whats and Whys are the best to draw from.

    I realize now that equating or closely associating Quant with WHATs and Qual with WHYs is not a very useful way to dig deeper into the issue (thanks to all of your excellent input!); but going back to what was originally bothering me, I think there is so much good WHAT in the Quant stuff that is available to us today that it’s a shame that design folk don’t go there first – for fear of numbers, not being aware or just not considering the richness that can be found in this kind of data.

    Carrie: Oh wouldn’t it be nice? That is not the reality I see. That’s why I started looking at this from “what is efficient” and hopefully what could yield most value upfront.

    Jared: I implied no such a thing sir! ;) I am certain a lot of metrics are NOT going to be helpful. My point was simply that they may be overlooked entirely. And that is a shame. Now, if people did look at it, there would certainly be the problem of UNDERSTANDING it and being able to differentiate the stuff that is useful and the stuff that is not (as your examples clearly illustrate). I would say though, these metrics are still answering some WHAT questions. WHY you are using those metrics are indeed a separate problem – good catch!

  9. Pingback: Statistiche web e ricerca sugli utenti: in quale ordine? - Alberto Mucignat

Comments are closed.