The Case Against UX Testing

The UX designer has devised a series of tests to determine if a green button is better than a red button. One of them involves tipping a tortoise onto its back. He looks the website over carefully and says, “Describe in single words, only the good things that come to mind about your mother.”

The website pauses, sweating under pressure, then replies, “Let me tell you about my mother…”

BLAM! The website pulls the trigger of an unseen gun, and the UX designer collapses, leaving the project to be rebuilt from scratch in Material by Harrison Ford, with overuse of Post-its delegated to Edward James Olmos.

Who Does UX Testing Actually Serve?

In the past’s bleak dystopian future (1982’s Blade Runner was set in 2019) no one benefitted from asking the wrong questions. And little has changed.

Designing any test to verify UX is fraught with as many complications as administering the test. Questions are skewed by bias, conscious or otherwise, and competing agendas. Even with something as apparently simple as a split test, the potential for distortion is immense.

When planned by a designer, a UX test offers little benefit to a client; the benefit is to the designer, who can then say their ideas are validated (or not).

Imagine hiring a developer to code a website, only to discover that the developer didn’t know CSS and expected to be paid to learn it before completing the work. You would hire someone else because that developer isn’t qualified.

From a client’s perspective, a UX designer should know, through experience, whether a green button is better than a red button. Designing an elaborate test to split-test the button color serves little purpose other than indemnifying the designer against mistakes.

The ROI of UX Testing

It’s widely accepted that there is substantial ROI (Return On Investment) from UX testing. We’ve all heard apocryphal stories about sites that split-tested their checkout and improved retention by 5%.

I’m going to go out on a limb and say that without user testing, that site could have improved its checkout retention by 4.9% simply by hiring a competent, experienced designer. But what about the remaining 0.1%? Well, for most sites, 0.1% represents very little profit. And the cost of recovering it via testing far exceeds the benefits.

When a company the size of Amazon, Netflix, Spotify, or Google split tests a website, it can afford to allocate $25k for user testing because it stands to gain 0.1%, and that represents far more than $25k. To meet the same 0.1% improvement, a small business has to design and run the same tests, incurring the same costs. But in the case of a small business, $25k could eat up all of its profits.

UX testing almost always works. But it is only profitable at scale.

If a good UI designer with a grounding in UX can improve checkout retention by 4.9%, tripling the project budget for just 0.1% more is a tough sell. Bluntly, that $25k is better spent on advertising.

What UX Designers Can Learn From Psychiatry

We all have the tendency to think we’re unique. It’s a survival trait attributed to our prehistoric brain. That belief in uniqueness is particularly strong in highly competitive people. We all think our site, our side-project, our approach are original. And we’re all wrong.

When a psychiatrist sits down with a patient, they have two immediate goals: categorize that patient into an established diagnosis, and assess the severity of the condition. It may be that the patient is depressed or anxious or even suffering from a potentially more debilitating condition like schizophrenia. What the psychiatrist is not trying to do, is define a new illness.

Occasionally — perhaps once per decade — a genuinely unusual patient will present themselves, and a new form of illness is considered. New treatments are found and tested. These treatments are rarely developed on behalf of individual patients; doctors work with grants from governments, medical schools, or the pharmaceutical industry and publish their results.

The vast majority of websites face similar problems. They deal with similar demographics, work within a similar culture, and deal with similar technology. As such, they can be categorized in the same manner a psychiatrist categorizes patients.

The key to delivering successful UX solutions is not UX testing in individual cases, but rather UX research, examining similar projects, and cribbing their solutions. If you categorize a project accurately, you’ll find a solution readily available.

Replacing User Testing With UX Best Practices

Your client doesn’t need to pay for UX testing to benefit from it. Enterprise sites, government sites, and even personal projects will test UX patterns. Sites like Shopify or Stripe will user-test their checkout processes at scale and enable companies to benefit from the results by adopting their platforms.

If you’re currently testing designs for small business, one of two things is true: either you’re wasting your client’s money investigating a problem someone else has already solved, or you’re designing something so original that it has no precedent (and you probably shouldn’t be).

Designers should be opinionated. Designers should know UX best practices and how they apply to a range of scenarios. Designers should be capable of making an educated guess. Designers should be self-validating.

Once or twice in your career, you may find a legitimate need to test something. However, the vast majority of the time, the correct answer is to tip the tortoise back onto its feet and choose whichever color button has the higher contrast.

Ben Moss

Ben Moss has designed and coded work for award-winning startups, and global names including IBM, UBS, and the FBI. When he’s not in front of a screen he’s probably out trail-running.