Abstract

[Article link]

Keywords

Website quality, Website, Website evaluation, Expert Analysis, Heuristic evalutation, User experience, Usability.

Introduction

Over the last three decades, websites have become one of the most important platforms on the Internet for disseminating information and providing services to society. Shortly after their first appearance, the need to evaluate website quality became evident. The earliest analyses were developed by experts in human-computer interaction and comprised usability heuristics (Nielsen, 2020) and design principles (Tognazzini, 2014) and rules (Shneiderman, 2016), aimed at improving interfaces. In parallel, inspection of the technical specifications of websites and the verification of standards for application development emerged (W3C, 2016). Subsequently, interest has grown in designing for an optimal user experience (Garrett, 2011) and quantifying user experience perceptions (Sauro and Lewis, 2016).

This evolution highlights the fact that, from its very outset, website quality evaluation has taken different approaches, analysing a range of different characteristics and employing a variety of methodologies. This may well be an indication that the discipline of website quality has yet to be fully consolidated and that its field of study is not readily delimited. This conclusion is further strengthened by the fact that the field has yet to agree on a formal definition for itself (Law et al., 2010; Semerádová and Weinlich, 2020).

Over the last twenty years, a number of different authors have offered their definitions. Leavitt and Shneiderman (2006), in one of the earliest attempts, define website evaluation as the act of establishing a correct and exhaustive set of user requirements, ensuring a site provides useful content that meets user expectations, while setting usability goals. For Aladwani and Palvia (2002), website quality is determined primarily by the degree to which a website's features are perceived by users to meet users' needs and to reflect the site's overall excellence; while for Gregg and Walczak (2010) website quality constitutes the attributes that contribute to its usefulness to consumers. Most recently, “the ability of a website to meet the expectations of its users and owners, as determined by a set of measurable attributes” is the definition proposed by Morales-Vargas et al. (2020, p. 3).

While these definitions coincide in the need for websites to satisfy user expectations, they differ in terms of the factors that should come under examination. Indeed, as an emerging research area, website quality in the literature has yet to achieve a common operationalization and each study tends to highlight different measures that are relevant to its own particular context (Law, 2019). When evaluating the quality of a website, it is important to know what can be measured and how to measure it (Akincilar and Dagdeviren, 2014). On the other hand, the evaluation of website quality is not the same as undertaking a traditional quality evaluation since it involves multi-criteria decision-making (Ecer, 2014), making it a particularly complex activity.

Thus, it is critical to identify the factors and characteristics that should be evaluated. In this regard, we can identify a first traditional focus to the question that might be defined as functional. Here, the focus is on the inspection of a website's inherent characteristics, including its content, information architecture and visual design, as well as its technical and operational features, linked to technology and security (ISO, 2008; Leung et al., 2016). The second approach, which we can define as experiential, focuses on user experience and perceptions and examines such factors as usability, accessibility, satisfaction and interaction (Maia and Furtado, 2016; Tullis and Albert, 2013). A third approach is more strategic in nature, focusing as it does on meeting the site owner's objectives, and on the use of performance, visibility and positioning metrics, among others (Kaur and Gupta, 2014; Sanabre et al., 2020).

All evaluation instruments of website quality, regardless of their particular focus, have in common the fact that they seek to conceptualise and delimit the target they seek to measure using some type of unit. The literature employs different names for these units, be it attributes, characteristics, variables (ISO, n.d.), factors or criteria (Chiou et al., 2010). Their use is largely synonymous, being terms that allude to the distinctive features of a certain property of the analysed entity, that is, websites. Codina and Pedraza-Jiménez (2016) propose addressing these properties, from the most general to the most specific, as dimensions, parameters and indicators, a terminology that we employ in this article. Dimensions constitute the generic properties of a website that we might want to evaluate. These can be divided, in turn, into more specific units, referred to as parameters. Finally, the indicators are the core elements of analysis that make it possible to operationalize and assess the parameters. Thus, for example, the dimension of “information architecture” includes “labelling” as one of its parameters and this, in turn, includes, among others, “conciseness”, “syntactic agreement”, “univocity” and “universality” as its indicators.

To evaluate these indicators, website quality studies employ different methodologies, experimental and quasi-experimental as well as descriptive and observational, typical of the associative or correlational paradigm. Likewise, such evaluations might adopt either qualitative or quantitative perspectives, undertaking both subjective and objective assessments. Similarly, they might employ either participatory and direct methods – as they record user opinions – or non-participatory or indirect methods – such as inspection or web analytics.

In the case of participatory methods, user experience (UX) studies have focused on user preferences, perceptions, emotions and physical and psychological responses that can occur before, during and after the use of a website (Bevan et al., 2015). The most frequently employed techniques are testing – which resorts to the use of such instruments as usability tests, A/B tests and task analyses; observation – centred on ethnographic, think-aloud and diary studies–; questionnaires – including surveys, interviews and focus groups; and biometrics – which uses eye tracking, psychometric and physiological reaction tests, to name just a few (Rosala and Krause, 2020).

Among the most common methods of inspection, we find expert analysis, a procedure for examining the quality of a site or a group of sites employing guidelines, heuristic principles or sets of good practices (Codina and Pedraza-Jiménez, 2016). The most common instrument is that of heuristic evaluation, in which a group of specialists judge whether each element of a user interface adheres to principles of usability, known as heuristics (Paz et al., 2015; Jainari et al., 2022).

Other instruments employed in undertaking inspections include checklists, in which each indicator usually takes the form of a question, and whose answer – typically binary – shows whether or not the quality factor under analysis is met; scales, where each indicator is assigned a relative weight based on the importance established or calculated by the experts for each parameter under evaluation (Fernández-Cavia et al., 2014); indices, metrics that not only evaluate a website's quality, but also how good it is in comparison with similar sites (Xanthidis et al., 2009); and analytical systems, typically qualitative instruments of either a general or specialized nature, which are mainly aimed at evaluating individual websites, conducting benchmarking studies, or for use as web design guides. These systems of analysis vary depending on the factors that their creators consider key to determine the quality of a website (Sanabre et al., 2020). In this study, in order to standardise their name, we refer to them as “evaluation instruments”.

These instruments can be applied manually, that is, by experts in website quality or those with an understanding of the discipline; in a semi-automated fashion, with the help of software and specialised validators (Ismailova and Inal, 2017); or in a fully automated manner (Adepoju and Shehu, 2014), using techniques of artificial intelligence (Jayanthi and Krishnakumari, 2016) or natural language processing (Nikolić et al., 2020). Thus, content analysis – a major technique in website quality inspection – can be applied in one of three ways.

Finally, we also find techniques aimed at the strategic analysis of performance (Król and Zdonek, 2020), including return on investment; search engine positioning (Lopezosa et al., 2019); competitiveness, including web analytics (Kaushik, 2010) and webmetrics (Orduña-Malea and Aguillo, 2014). Additionally, within this group we find mathematical models for decision making with multiple, hybrid, intuitive or fuzzy criteria (Anusha, 2014). By employing criteria at different, unconnected, levels, these models establish a hierarchy of evaluable factors (Rekik et al., 2015). They are used, among other applications, to weight user responses and generate indices of satisfaction or purchase intention.

Thus, this review of the literature highlights that the study of website quality is multidimensional. Moreover, such evaluations can employ a range of different focuses and employ multiple techniques and instruments. With this as our working hypothesis, we seek here to determine the properties that characterise the main website quality evaluation instruments, as well as to identify the dimensions, parameters and indicators that they analyse in each case. Based on these outcomes, we develop a comprehensive evaluation framework (Rocha, 2012). This, in addition to unifying the different concepts examined and helping to clarify the broad panorama comprised by website quality publications, should serve both as a guide and model for the development of new instruments that can be employed by professionals and researchers alike in this field.

Citation

Morales-Vargas, A., Pedraza-Jimenez, R. and Codina, L. (2023), "Website quality evaluation: a model for developing comprehensive assessment instruments based on key quality factors", Journal of Documentation, Vol. 79 No. 7, pp. 95-114. https://doi.org/10.1108/JD-11-2022-0246

Useful links

• Article page with metadata

References

• ​​​​​Adepoju, S.A. and Shehu, I.S. (2014), “Usability evaluation of academic websites using automated tools”, 3rd International Conference on User Science and Engineering (i-USEr), IEEE, pp. 186-191.