A/B testing or Usability testing?
By Richard Whitehand, August 2012
These days there are a variety of tools for conducting A/B testing of web designs online. They facilitate the process of giving different customers different versions (A & B) of a web site - e.g. different product presentations, or different purchase flows. Then, after a number of customers have used the designs, the relative success of the two versions can be compared.
However, can such tools replace the need for usability testing? Not really, though both types of testing are of value. Lets take some typical design questions and look at their strengths and weaknesses...
Which banner placement results in the most click-throughs by users? Which product presentation layout design results in the most sales?
A/B testing of two different versions will give a clear answer here. Usability testing doesn't typically include large numbers of users and thus is inappropriate for raw statistics, though with eye-tracking it may give insights into why one placement or design gets more attention than another.
How can I improve the success rate of users in finding and buying a product? Why do users abort at certain points in an order process?
Usability testing will give you the answers to these types of questions as interaction can be studied in detail and users can be asked about their interaction as they proceed through a process. A/B testing will tell you which version is more successful, but not why it is successful or how to make it even better.
Design B performs better than design A, but how much better could the design be?
Usability testing provides qualitative insights together with user performance metrics whereas A/B testing focuses purely on quantitative data. In some cases qualitative insights and understanding of user interaction can be unnecessary, though in many cases they are invaluable. If you don't know the reason why one design works better than another, or why neither design is great, how do you improve it?
A/B testing can be a quick and efficient way of testing out simple design changes to an existing site design (e.g. changing the text on a button or product teaser). Unlike usability testing, which can be done with simple prototypes, A/B testing requires fully implemented designs and thus often isn't practical for testing two completely different concepts. However, minor changes to a design can be important too (even small percentual increases in sales can be valuable) and usability testing is not a quick or cheap way of exploring these individual adjustments. Indeed, if the changes are very small they will be difficult to measure in usability tests.
In conclusion, if your goal is to make large-scale design improvements you must understand why users are more or less successful in a design, and how to make improvements. Start with usability testing and then consider complementing with A/B tests to polish key parts of the design's performance - once you know you have something that is fundamentally sound.
Related articles:
- Pairing up usability testing with A/B testing (UXmatters)
- A/B split testing vs usability testing - Which should you use? (above the fold)
Did you find this editorial interesting?
You might like to read some of our other editorials.
Please send us an email if you have any comments or suggestions!