User Testing
In my industry, user testing has been the hotness for a while and it's one of the first discussions in interviews.
For me, user testing falls into one of three camps.
- It's a data science performed by professional researchers who have an education in statistics, psychology, and UX.
- It's a gut check for designers who are on the fence.
- It's a comfortable fiction.
1: Science
I've worked with really good user researchers. They know how to recruit people for interviews. They know what self-report data is and how to avoid it (or try). They know how many people to test with and what mechanisms are best used for any given test. They know a lot more than all of those things.
Sometimes user testing is more efficient or effective than a/b testing and I love working with professional user researchers.
2: Testing Intuition
There are times when I don't understand the user well enough to know what they prefer or where they'll go on a certain design. I have a hunch, but I'd like to test it. For this I can either use a dedicated tool to test people (usertesting.com or something) or—and this is my preferred method—I can just grab some people unaffiliated with the project who are the target market.
To me though, this is just design. Call it 'user testing' if you want, but we've been doing this long before user testing was an official thing. "Hey Jane, did that fancy new 'wheel' invention I gave you help you move your cart faster?"
3: Comfortable Fiction
This is the majority of user testing that I've seen in my career. Most of the time, leadership doesn't know good design. They want to know that they aren't spending money on development or losing money from their site/app without the design being tested. It may help you sleep better at night, but user testing is not fortune telling. Even the best user tested theories are still a gamble. But that's ok, if you look at some extremely famous success stories, they were born from courage and conviction around the design, not an abundance of testing.
I've also seen user testing as a way to get consensus. If someone disagrees with you on a design, you user test it and now they can't argue because you can't argue against the data. I'm coming up on 30 years in this industry, do you know how many times I've seen a PM or designer do user testing and come back to say: "Our hypothesis was wrong, let's go a different direction."? Exactly zero times. The odds are staggering, 100% of the time people have user tested, their assumptions have been right?
To be fair, I've seen people come back with some surprises and they've adjusted their course, but that sort of learning can easily be found in camp two.
I've seen some pretty insane stuff go live because it was user tested. Facebook and Instagram are notorious for how much faith they put in user testing, and look at some of the awful products they've released. Products where any good PM or designer would never have allowed to leave the door. I worked at Amazon for five years, and I watched products go through user testing and then fail miserably in a/b tests—often times at a catastrophically embarrassing level.
Most designers I know think they're good at user testing. I used to say that too. And then I worked with a few professional researchers. I also used to think I was a good writer until I worked with a professional writer. User testing is time-consuming, complex, and nuanced and even the best researchers wouldn't put 100% faith in their results until they were tested in the real world.
Inevitably, after I've explained my feeling on this to a fan of user testing, they ask if I just go by my subjective feeling. I'm INTJ, I do actually listen to my intuition quite a bit. But I also love a good a/b test and analytics analysis. The web is a lovely medium to try new things and it's ok to test things for real and then change your mind.
I also am a big fan of the phrase "the devil is in the details" and I think sometimes things need to be built to really be tested.
Do I think you should stop user testing? No, I think you should do what you feel is best. The lovely part of working on a team is that we all have different strengths and different ways of working. We don't all have to have the same process or opinions. The best teams are ones where people are allowed to bring their individual strengths to the table.