Category Archives: Measure

Video and Slides from Interaction 10’s KPI talk

Here is the video of my presentation from Interaction 10 on Key Performance Indicators. The video was very nicely made, with the slides being presented just as the right time (so you don’t have to stare at me much). I do recommend you take a look at my annotated slides (below) where I captured some of the things I did not talk about. I wish the Q&A was included, it was such a great conversation!

Your choice of words matters

Yet another reason why designers and business folk talk past each other: people who are purposefully misleading to get attention.

I came to this presentation from Google on their Quality Score measure because someone referred to it by saying “Quality Score is a measure of user experience”. It obviously peaked my interest because it is precisely the qualitative characteristic of user experience that makes it hard to measure.

When you get to slide 4 you realize that Google knows better and defines Quality Score as “an automated measure of how relevant each of your keywords is to your ad text and to a user’s search query.”

It has nothing to do with measuring users’ experiences with anything whatsoever. I realize it sounds naive to be cranky about attention-grabbing people but it baffles me that people do this: misuse the notion of user experience to mean anything at all that they want. It is such a coward move. Be bold, say what you want to say!

More than that, I worry that people just have no clue what they are talking about. Because it that is the case, it is even more worrisome. If people engaged at this level of discussion (i.e.: what measures to use) don’t understand a basic thing such as what user experience means (at its most basic what PEOPLE experience when they INTERACT with something), then we’re all very far from being able to have progress in advancing the conversation about measuring success in the context of user experience.

Quant before Qual makes no sense. But it does.

As I continue to explore how designers can make better informed decisions by leveraging information, the issue with number aversion is still #1. I talked about this already in my Interaction 10 presentation, but I’ve been digging deeper and have some other thoughts (check my presentation for some base assumptions).

If we agree that quantifiable data, specifically the ever popular web analytics, provide you with rich detail to tell you WHAT is happening, it is comforting to realize that it is the type of data gathering that we already do – design research – that provides the qualitative color to answer WHY said things are happening.

What I am finding, however, is that it is more valuable to START with the quantitative work and get to the WHATs and ask WHYs based on those findings, rather than trying to figure out WHYs in exploratory mode (even if the WHAT’s are going to emerge at one point or another in this quest).

My point is that it’s not sustainable as an approach. It’s inneficient to start digging deeper to answer the WHY questions if you don’t have a baseline of WHATs identified.

The problem is that it is not intuitive for designers to start where they are uncomfortable. We are super comfortable with qualitative approaches – they are our go-to tools because that’s what makes sense for design research. However, quantitative research instruments really help narrow stuff down, but they do require you to understand those pesky numbers in order to a) dig in and get to concrete answers and b) understand what it’s saying so you can ask “why”.

In short, WHATs before WHYs are more efficient than WHYs before WHATs, but that requires designers to start with unfamiliar tools to then apply familiar tools. If it was the other way around I think it would be much easier for designers to bridge both approaches and come out the other end with more useful insights.

In other words, since we don’t particularly feel an attraction to numbers (to put it lightly), why would we start there? It’s such a leap from how we think about problems that it is counter intuitive. I don’t believe designers reject the notion of starting with Quant approaches (WHATS) to expand with Qual approaches (WHYs), but it’s inherently counter-intuitive to think that way.

How can I help designers do this when it goes against their nature? That’s what I’m working on right now. More on this later.