Usability test report requirements

I'm in a discussion with some colleagues about the requirements of a good usability test report. I don't get too wrapped up in the details of reports, since I'm usually the only one who reads the reports. I'm more concerned about the content of the presentation and discussion. I often don't send the report out in advance, since I've sometimes had project managers take the report and decline the meeting. It depends on who I'm dealing with. The content of the presentations contain the basic findings of a study, but the conclusions are written specifically for the audience. I try to connect the work to their own concerns, which are different by area: development, marketing, operations, finance, and so on. Sometimes the most valuable part of the exercise is the discussion following the presentation of results. It tends to surface a lot of the assumptions, attitudes, and opinions of the people receiving the report, and gives me a chance to respond to a lot of concerns.

The funniest requirement for a usability report was given to me by a previous manager, thankfully long departed. I finished a test and report and told the manager that I was going to set up a meeting with the project team to deliver the results. She said the project team didn't have time and that I should just send the report to them and skip the meeting. This team wasn't familiar with usability testing or usability reports so I said, "What if the project team has questions?" The manager replied, "Just write the report with enough detail so they won't have questions."

Lessons Learned: Don't ask for the moon

Listening to recorded answers to IVR prompts is a necessary part of tuning a speech IVR application for the purpose of improving recognition performance. It also sometimes reveals what people think about the IVR. Here's a verbatim quote taken from a tuning exercise.

"Oh you gotta be kidding me how the hell am I supposed to know that?"

The IVR had prompted the caller for the last four digits of the credit card number that was used to set up and pay for a service. The service may have been set up as long as a year previously, and many customers have more than one credit card. The question of how the hell the customer was supposed to know that had at least occurred to the IVR designer of this application. There were extra steps placed in the IVR to mitigate this scenario, but none were very effective. Lesson learned: ask for things that you know are easily available to the caller, or prepare them in advance for unusual requests that take time to track down.

Lessons learned: Educate the management, pt. 2

In an earlier post, I mentioned several things that business managers need to know before you dive in and develop a speech IVR for them.

• Strengths and weaknesses of speech rec technology. Pity the poor VUI designer who starts a project whose stakeholders have been educated about speech only by slick sales presentations and glossy brochures. I've been there - it's no fun. Part of the challenge of design is to work within the constraints of current recognition technology, like the ability to discriminate between letters in the "e" group and between the letters s and f.
• Skills required to implement a good speech IVR. Speech IVRs aren't just another IT shop application. The learning curve for speech rec development and testing is very steep. The learning curve for effective dialog design is very steep. Companies that hand complex speech projects to an internal group need to appreciate that and build in a lot of time for training and development of people before the first speech project even begins.
• Persona isn't just about the system's voice. This one is a little controversial, but I can say from experience that managers tend to spend too much time selecting the perfect "voice" for the IVR in the belief that this "branding" exercise makes or breaks customer satisfaction with the system. The sound of the voice - the way the prompts are delivered, in particular - is important, but the design and proper functioning of the application is equally important.

If a management group "doesn't have time" to participate in a level setting meeting I'd recommend pushing back on that very hard. Spending a little time before a project kickoff talking to management about speech projects in general can save a lot of thrashing later on.

Lessons learned: Educate the management, pt. 1

One of the first things I learned about speech projects is how important it is to educate management about speech recognition IVRs. It's not a pretty scene when you roll out a speech application after a year-long effort and the business managers don't understand what they've been given.

• Tuning. Speech IVRs aren't fully tested during user acceptance testing. They can't be. Full testing depends on having large numbers of real customers hitting the IVR, evaluating the data, and making tweaks to the prompting and grammars. That's tuning. Business managers who don't know about tuning are shocked when they learn that testing isn't complete until several weeks after roll out.
• Roll out strategy. Deploying a speech IVR isn't simply a matter of flipping a switch and exposing your customers to the IVR, then walking away. Do you really want everyone to hear version 1.0 (see tuning, above)? Can your CSRs answer the inevitable questions from customers about the new system? Who is going to be responsible for ongoing observation and maintenance (see next point, below)? There are a lot of issues that management needs to settle long before the speech IVR is rolled out.
• Ongoing observation and maintenance. Believe it or not, some business managers think that once the IVR is deployed the business can just walk away from it. That may be true for some limited-functionality DTMF IVRs (and we all know how much customers love those systems) but it certainly is not true for speech IVRs. There's work to do even after a successful tuning cycle, and management needs to plan for that in advance.

I've described three aspects of speech projects that business managers need to know and be fully engaged in to ensure a successful speech rec project. I'll have several others in upcoming posts.

Speech projects are special

The projects that I've worked on since 2002 have nearly all been implementations of speech recognition systems, or applied research on listener perceptions of speech reco systems. I've learned that speech reco projects have a lot of characteristics that distinguish them from typical software development projects. I've made note of those characteristics, and started to develop a lessons learned file to capture the kinds of things that must occur in order to ensure a successful project. I'll start posting those lessons over the next few months. Of course, if you have any favorite lesson learned from speech reco projects, please leave a comment.

(The title of the blog post is a take off on the "speech is special" argument that some linguists make based on phenomena like categorical perception that were first observed in speech experiments.)