War Stories - part 4
One of the companies I used to work for ran a lot of unmoderated remote usability tests. This meant I would write a ‘script’ add it to some software along with a link to the page we were testing and then let it run. The company who ran the tests would recruit for it and I would be notified when all the tests I had asked for had been completed.
If I had the choice I would rather run a moderated test than an unmoderated remote test, but they are better than no testing at all, even if you cannot dig into why someone has just done something, like you can when you are there with them.
Because these tests are unmoderated you have to be careful how you word your questions. Once I had written my ‘script’ I would normally get a colleague to have a read through and make sure it made sense and I was not leading the participant. And if I had the time I would put out a pilot test - a single test to make sure it made sense to participants too.
In a rush one day and with what I thought was quite a simple test, I got my colleague to read the questions then pushed the test out to 10 participants. The system we were using did not make it very easy to preview the test, so I just had to hope it made sense, then rushed off to whatever urgent meeting I had been called to.
A couple of days later and all my tests were back in and I started watching them back.
Watching the first chap work though the tasks everything was going well until the second question. For some reason the link was not in the same box with the questions and the way I had written the question made it sound like it should be there. He stumbled and dithered and had no idea what to do.
Finally he gave up to move to the next question and came across the link. Of course he had forgotten what the question was and could not go back to have a look. He tried his best but I think we were both disappointed. He did at least complete the test and I got some useful bits but question two was a washout.
Oh well, on to test two. Who had the exact same problem. In fact they all did. I had messed up the whole set of tests by rushing my question writing and then not checking it with a pilot test.
Now when I set up an unmoderated remote test (and to be honest with all tests) I always make sure I have the time to put out a pilot test before hand. It might mean I need a little more time and planning, but it means that I can be reasonably confident the rest of my tests will come back without too many issues.