Heuristic Evaluation for Web Usability: Methods and Findings

Introduction to Usability Evaluation

As part of the web development process, web developers are confronted with evaluating the usability of web interfaces — that is, web sites and applications. Typically, a combination of manual methods and automatic tools is used for effective web site evaluation. For example, manual inspection is needed to supplement automatic validation tool results (Rowan 2000). However, web projects are highly affected by fast-paced life cycles, leaving little room for full evaluations. Other major contributing factors include low budgets assigned for testing and the limited availability of usability experts.

Web developers need effective and affordable approaches to web usability evaluation. Available automatic web usability evaluation tools such as LIFT Online and LIFT Onsite (UsableNet 2002) and WebXACT (WatchFire 2007) have proven useful in finding syntactic problems. These include problems of consistency, verification of broken links, whether pages contain links to the home page, and alternative descriptions of images (using the ALT tag in HTML), among others (Brajnik 2000). Other problems of a semantic and pragmatic nature are not handled by automatic evaluation tools (Farenc 1996). Farenc and collaborators (Farenc et al. 1996) explored the limitations of automatic usability evaluation tools. In analyzing 230 rules for their ERGOVAL automatic usability evaluation tool for Windows systems, they found that a maximum of 78% of the rules could be automated "whatever the implemented methods are." The other 22% require human input to provide information and resolve semantic and pragmatic conflicts.

Usability problems not handled by automatic evaluation tools can be addressed with semi-automatic and manual approaches. In semi-automatic approaches, the identification of usability problems begins with the analysis of source files and is completed with human intervention to provide information, make decisions, or confirm problems. There are three manual methods typically used to find usability problems in user interfaces (Preece 2002): (a) usability testing, where testers observe users performing tasks and report usability problems based on their observations; (b) questionnaires and interviews, where users are asked about their experience using a system, missing features, and overall satisfaction; and (c) inspection methods, where experts examine user interfaces and report usability problems based on their judgment and expertise. The present paper reports a usability evaluation conducted by the author.

The first step was to characterize the inspection process in heuristic evaluation in order to understand it better and develop different ways to support it. A user study in the laboratory was conducted to understand how evaluators apply heuristic evaluation on web interfaces. The output of this step is a rough characterization of the process and tool requirements. Tool requirements were identified from the literature, study findings, and experience. Evaluators in the study were found spending time observing, annotating, and navigating the interface, as well as elaborating usability problems. Tools for inspection are proposed based on these activities.

Concept of Usability and Heuristic Method

The concept of usability was defined in the field of human-computer interaction (HCI) as the relationship between humans and computers. The International Organization for Standardization (ISO) proposed two definitions of usability: ISO 9241 and ISO 9126. ISO 9241 defines usability as "the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use" (ISO 9241-11, 1998). In ISO 9126, usability compliance is one of five product quality categories, alongside understandability, learnability, operability, and attractiveness (ISO/IEC 9126, 2001). Usability depends on the interaction between user and task in a defined environment (Abran, Khelifi, Suryn, & Saffah, 2003; Bennett, 1984). Therefore, ISO 9126 defines usability as "the capability of the software product to be understood, learned, used and attractive to the user, when used under specified conditions" (ISO/IEC 9126, 2001). While this definition focuses on ease of use, ISO 9241 uses the term "quality in use" to describe usability more broadly (Abran et al., 2003; Bevan, 2001). "Quality in use" is defined as "the capability of the software product to enable specified users to achieve specified goals with effectiveness, productivity, safety, and satisfaction in specified contexts of use" (ISO/IEC 9126, 2001).

This term later became "quality of use" due to weaknesses in ISO 9126, such as unclear architecture at the detail level of the measures, overlapping concepts, lack of a quality requirement standard, lack of guidance in assessing the results of measurements, and ambiguous choice of measures (Abran et al., 2003).

The usability of a technology is determined not only by its user-computer interactions, but also by the degree to which it can be successfully integrated to perform tasks in the intended work environment. Thus, usability is evaluated through the interaction of user, system, and task in a specified setting (Bennett, 1984). The socio-technical perspective also indicates that the technical features of health IT interact with the social features of a healthcare work environment (Ash et al., 2007; Reddy, Pratt, Dourish, & Shabot, 2003). The meaning of usability should therefore comprise four major components: user, tool, task, and environment (Bennett, 1984).

Heuristic evaluation is an inspection method proposed by Nielsen and Molich (1990). It follows the "discount" philosophy, in which simplified versions of traditional methods are employed — for example, discount usability testing that does not require elaborate laboratory setups. It consists of having a small number of evaluators independently examine a user interface in search of usability problems. Evaluators then collaborate to aggregate all usability problems. During interface inspection, evaluators use a set of usability principles as a guide — known as "heuristics" — to focus on common problem areas in user interfaces. An example of such a heuristic is "Help users recognize, diagnose, and recover from errors" (Nielsen 2005b). Interface features that violate the heuristics are reported as usability problems.

Problem aggregation has been supported by existing tools (Cox 1998). The intent was not to automate the aggregation process but rather to support evaluators in manual activities such as identifying unique problems, discarding duplicates, and merging descriptions using affinity diagrams (Snyder 2003). There has been some effort in semi-automating problem identification in heuristic evaluation, but this represents a formal, application-dependent approach. Loer and Harrison (2000) developed a system for querying a model checker to search for potential usability problems in user interfaces.

A typical heuristic evaluation session lasts two hours. The evaluation can begin with two passes of the user interface: a first pass to get a general idea of the interface design and overall interaction, and a second pass where evaluators focus on particular elements. Heuristics are meant to help identify usability problems; with heuristics in mind, evaluators carefully examine an interface and report features that appear to violate them.

The output of a heuristic evaluation is a list of potential usability problems. Lists generated by all evaluators are aggregated. Evaluators meet to identify duplicates, combine problem descriptions, suggest solutions, and rate problem severity so that issues can be prioritized. Nielsen recommends using a 0–4 severity rating scale (Nielsen 1995b):

0 = "I don't agree that this is a usability problem at all"
1 = "Cosmetic problem only: need not be fixed unless extra time is available on project"
2 = "Minor usability problem: fixing this should be given low priority"
3 = "Major usability problem: important to fix, so should be given high priority"
4 = "Usability catastrophe: imperative to fix this before product can be released"

Heuristic Evaluation Dimensions

Several heuristic evaluation dimensions can be identified: the heuristics used to guide the inspection, the evaluators performing the inspection, the user interface being evaluated, and the process followed. Heuristics are general usability principles that "seem to describe common properties of usable interfaces" (Nielsen 2005a). Nielsen and Molich (1990) initially proposed nine heuristics, defined based on their experience with common problem areas in interfaces and consideration of guidelines. A factor analysis of 249 usability problems (Nielsen 1994b) led to the widely used set of 10 heuristics:

1. Visibility of system status; 2. Match between system and the real world; 3. User control and freedom; 4. Consistency and standards; 5. Error prevention; 6. Recognition rather than recall; 7. Flexibility and efficiency of use; 8. Aesthetic and minimalist design; 9. Help users recognize, diagnose, and recover from errors; 10. Help and documentation.

Some alternatives have been proposed for specific domains to provide evaluators with relevant domain knowledge. For instance, Dykstra (1993) developed calendar-specific heuristics based on results of user testing different commercial calendar systems. Evaluators performed better when using calendar-specific heuristics — more usability problems were found, and more were severe, compared with a standard heuristic evaluation. However, Dykstra's 9 heuristics had an average of 6.6 sub-headings describing each high-level heuristic, including one heuristic with 19 sub-headings. This begins to resemble a guideline review with 60 guidelines rather than a heuristic evaluation with 9 high-level principles.

Nielsen recommends keeping the list short (about 10) for easy recall (Nielsen and Molich 1990, p. 249), although domain-specific heuristics may be added (Nielsen 2005a). Muller et al. (1998) reformatted the list and added four more heuristics for a participatory approach to heuristic evaluation, calling for the participation of "work-domain experts" (i.e., users) and adding heuristics about human goals and experience.

The role of heuristics is not entirely established. Although heuristics are meant to help evaluators identify usability problems, it is not clear that they effectively support the discovery and analysis of problems (Cockton and Woolrych 2001; Cockton et al. 2003). In usability problem analysis, heuristics as an analysis resource have not proven effective in eliminating false alarms or confirming actual usability problems (Cockton and Woolrych 2001).

Evaluators should not only report likes and dislikes; they should explain problems with reference to violated heuristics or other usability principles or guidelines (Nielsen 2005a). Cockton and Woolrych's (2001) extended usability problem format requires evaluators to "hypothesize likely difficulties in context, rather than to just focus on problem features." This extended format encouraged evaluators to be more "reflective and less likely to propose problems with little justification" (Cockton and Woolrych 2001, p. 175). In an updated version of the form (Cockton et al. 2003), an entry for providing evidence of heuristic non-conformance was added, encouraging evaluators to reflect on their choice of violated heuristics.

Solutions to fix problems can be suggested based on violated heuristics (Nielsen 2005a) or on some other taxonomy, such as the User Action Framework (Andre 2000) for classifying usability problems based on Norman's seven-stage theory of action (Norman 2002, pp. 45–53).

Regarding evaluators, typically 5 (Nielsen 1992; Bevan et al. 2003) to 8 (Nielsen and Landauer 1993) evaluators are used in heuristic evaluation, though the number remains debated (Bevan 2003). Novice evaluators tend to perform poorly (Nielsen 1992; Jeffries et al. 1991; Desurvire et al. 1992). Nielsen (1992) classifies evaluators as "novices," "regular specialists" (those with usability expertise), and "double specialists" (those with both usability and application domain expertise). In his study, regular specialists found 75% of problems when aggregating individual lists; achieving the same success rate required fourteen novice evaluators. Users can also become part of the evaluation force — Muller et al. (1998) incorporated users to capture work-domain expertise.

The user interface format (paper vs. computer-based) and interactivity (simulated or live) may influence how interfaces are evaluated. Nielsen (1990) found that evaluating paper and computer mockups may affect the types of usability problems discovered. When evaluating interactive interfaces, evaluators interact directly with the interface, entering information, navigating screens, and testing functionality, which enables them to experience problems firsthand. Interface complexity may also affect evaluation; Slavkovic and Cross (1999) found that novice evaluators tend to focus on certain parts of a complex interface (the Palm Pilot) rather than evaluating it comprehensively.

Evaluator performance may also be impacted by usability problem formats used to capture problem details. Cockton et al. (2003) designed an extended form and found unexpected improvement in evaluator performance compared with a previous study (Woolrych 2001; Cockton and Woolrych 2001). Results showed a 19% reduction in false alarms and a 26% increase in the appropriateness of heuristic application when using the extended form. Heuristic evaluation is known to produce not only a large number of problems (Jeffries 1991; Bailey 1992; Tan 2009) but also a large number of false alarms (Bailey 1992) — identified problems that are not actual problems in the interface. Minimizing false alarms is important to avoid making interface changes based on inaccurate findings.