Usability Study Part 3: Results and Conclusions

Welcome to the final installment of our three-part blog on conducting a usability study. To recap (aka copy and paste) in the first part, we explained what this is all about, our intentions, process and what we hope to achieve. In the second part, we conducted the usability test, observing the users on the website while they perform the tasks that we have set them in our scenario.

In this part we assess the results, look for trends and see what practical, actionable conclusions we can draw from the test.

One of the things I did with each user after they completed the tasks was to ask them to write down the top three usability issues that they encountered. Sharing this is probably a good starting point, as it does a bit of the work for me!

First user:

1. "When searching for events, choosing schools in the drop down would quite often return a blank, even when 'All ages' or '12+' worked. I found this quite confusing, although got there eventually be simplifying my search. Also think it might be helpful to differentiate between primary and secondary schools ion the search."
2. "It would have been good to select multiple event types. I had to run two searches, first for music, then for performances, to complete one of tasks."
3. "I didn't spot the search box at first, possibly because there was so much going on underneath it."

Second user:

1. "Trying to find the addresses for the venues was very difficult for me."
2. "I wanted to look for a list of exhibitors / performers to see who was taking part, but there isn't one. This is something I often default to."
3. "I found the business of the site distracting, and affected my ability to find things in the page."

Third user:

1. "I prefer a new window if leaving a website" (a real, unbiased, user who is not a web professional said this - now there's a bit of controversy for all the standards pundits!)
2. "Had a couple of problems with the captcha code."
3. "No other issues. Overall the site was clear and easy to use."

So quite a range of results, even from just three users.

Drawing actionable conclusions

While assessing the responses above in conjunction with their videos, I noticed a couple of things that were consistent across all three, even the happiest. I have produced a list of amendments from these to implement on the site, which will be done shortly:

• When using the event search facility, the results returned were not always exhaustive for what was actually on offer, and all of the users missed one event in particular, which was listed as a performance, but was primarily music-based.
  Action: We will add a multiple event type search, and allow tagging of multiple types for where there are events that cross over.
• None of the users immediately found the address details for the events or exhibitions, despite many links to them throughout the listing and detail pages.
  Action: We will add the full venue address, and maybe a small map or link to one on every event detail page.
• The next most consistent issue raised (2/3 users) was that of information overload. They both found it hard to find the search box, citing it was due to the tag cloud underneath it. In fact, none of the users really knew what the tag cloud was. As our sample range was very small, it is hard to draw any wide-reaching conclusions from this particular observation, however Jakob Nielsen appears to support our findings, as mentioned here.
  Action: We will remove the tag cloud in order to reduce visual noise.

Of course this doesn't mean you should try and cater for absolutely everyone and implement every change suggested. There are times when it is appropriate to stick to your guns and way of doing things, but consistent trends should be taken seriously and dealt with. Ultimately, your users will be happier, and your website more successful for it.

Update Thursday 17th June 2010

Illustrating the complexity of usability testing, for various reasons the client has decided to not implement the recommendations put forward.