Usability test report requirements

I'm in a discussion with some colleagues about the requirements of a good usability test report. I don't get too wrapped up in the details of reports, since I'm usually the only one who reads the reports. I'm more concerned about the content of the presentation and discussion. I often don't send the report out in advance, since I've sometimes had project managers take the report and decline the meeting. It depends on who I'm dealing with. The content of the presentations contain the basic findings of a study, but the conclusions are written specifically for the audience. I try to connect the work to their own concerns, which are different by area: development, marketing, operations, finance, and so on. Sometimes the most valuable part of the exercise is the discussion following the presentation of results. It tends to surface a lot of the assumptions, attitudes, and opinions of the people receiving the report, and gives me a chance to respond to a lot of concerns.

The funniest requirement for a usability report was given to me by a previous manager, thankfully long departed. I finished a test and report and told the manager that I was going to set up a meeting with the project team to deliver the results. She said the project team didn't have time and that I should just send the report to them and skip the meeting. This team wasn't familiar with usability testing or usability reports so I said, "What if the project team has questions?" The manager replied, "Just write the report with enough detail so they won't have questions."

Official Google Blog: Goodbye to an old friend: 1-800-GOOG-411

Google will shut down its speech reco application GOOG 411. I had written about this service before. I don't have a smartphone, and this was a handy service when you were away from your computer. Google rarely hesitates to pull the plug on a service that isn't meeting expectations, so I guess this is no exception. Thanks to Phillip Hunter for alerting me to this story.

Summary of HFES conference in SF

I went to the excellent Human Factors and Ergonomics Society conference in San Francisco last week. The keynote address was delivered by pilot Chesley “Sully” Sullenberger. Some of the talk was motivational, and some was specific to the human factors of the design of airline cockpit alerts, crew training, and performance under stress.

The emergency landing in the Hudson River in January 2009 was illustrated by a simulation showing a map of the area and an animated path of the aircraft from takeoff to landing. Overlaid on the screen was a clock, an altimeter, and some of the voice recordings between pilot and tower. Some interesting details that the speaker mentioned:

• The entire incident until landing was less than four minutes. He formulated his plan in less than a minute.
• There is an SOP for a situation in which both engines are lost, but it’s three printed pages long, obviously written for situations where the aircraft is at a high altitude. Some SOPs are printed, some are electronic, and pilots have to know where to look to get the right SOP.
• He had never specifically trained to ditch a commercial aircraft in water. The only training he received was a theoretical classroom discussion about it years earlier.
• He had never experienced a catastrophic equipment failure in 29 years of commercial flying.
• He feels that he was relying on his experience, including his fighter pilot experience 29 years earlier, to land the aircraft.
• He didn’t realize that he’d done everything correctly until the investigation into the crash concluded two months later.
• He was unable to deploy the flaps as he wanted to due to a software misfeature that was known only to a few software engineers. This caused the plane to come in faster and at a steeper angle than he wanted. This caused more damage to the underside of the aircraft than was necessary, and contributed to an injury.
• Airlines are reducing training to keep costs down. They train only to minimum FAA requirements and no more. He predicts that training shortfalls will become obvious only in the future and in exceptional circumstances. In other words, disasters will have to occur and data collected before changes will be made to training requirements.
• Commercial flight simulators still do not train pilots in the scenario he experienced in Jan. 2009.

The conference was heavy on NextGen research for airline industry and the health care industry. I chaired a session on "Management perspectives on building UX departments," which generated a lot of good discussion. I took a tour of the Autodesk office. I even ran into some people from the rail industry from Australia. And, of course, San Francisco is an excellent city to visit.

HFES in September

I'm going to the Annual Meeting of the Human Factors and Ergonomics Society in fabulous San Francisco. I'm chairing a panel on How to Build a User Experience Group within your company. We'll have an All Star lineup and lots of good discussion.

I haven't been to HFES since 2006, so I'm really looking forward to getting back and seeing all my human factors colleagues again. Drop me a note if you're going to be there.

Strategic business metrics for UX designers

I'll be presenting a session on the topic of strategic business metrics for UX designers at the TriUPA meeting on October 6, 6:30 - 8:30pm at Railinc Corp. in Cary. You can register on the TriUPA web site.

Computing ROI for usability activities

Nearly everyone in the usability field is challenged at one time or another to justify their contribution to a project. UX practitioners hear this statement as a request for some sort of ROI study that presents data showing the unique contribution of usability to, for example, money saved or revenue generated. Since it's pretty hard to separate out the contributions of various players on a project, this usually is very hard to do.

Nevertheless, I was able, one time, to directly quantify UX contribution to cost savings for a call center I did some consulting work for. Callers were unable to correctly enter a policy number in the IVR to obtain automated information on an insurance policy. The problem was that the policy number had four fields, three of which were variable length, and some fields contained letters the A - F. Nearly all callers errored out and wound up in the call center. I and a developer wrote an algorithm that correctly parsed the callers' keypad input, I re-wrote the prompting and usability tested it, and we implemented it. Correct policy number input went from 10% to 90% immediately. Since call centers know their average handle time and the amount they spend per call, it was easy to show how much money was saved by our fix. We knew how much time we spent on the effort. We estimated savings to be $300,000/year due to our one-time fix. We even wrote a paper showing our calculations, and recommended that we be allowed to make similar fixes to the company's 40 other call centers.

This is a well and good, and should have led to an enormous cost savings across the entire company. Unfortunately, there was simply no one to execute this excellent recommendation. The 40 call centers all operated independently, so there was no single person, or even committee, to push for improvements to all call centers. The opportunity was lost because the developer and I simply didn't have enough standing within the company to push for changes.

Bottom line, ROI studies are great, but even if you have them they don't automatically lead to changes. Managers with a mandate to lower costs by some means other than layoffs are the ones who drive change. Sometimes they're harder to find than you might expect.

Carolina Student Design Competition

How do aspiring product designers in the Carolinas win acclaim as innovative designers, not to mention prize money? By winning the first Carolinas Student Design Competition, sponsored by Carolinas PDMA. More details coming soon, but the contest winners will present their designs at the Innovate Carolina 2011 conference in Charlotte on April 15.

The Innovate Carolinas 2010 in April of this year at the UNC Kenan Flager School of Business was attended by over 120 professional product designers, product managers, executives, and university faculty. That's a pretty good audience for a winning college design team.