Hostname: page-component-78c5997874-v9fdk Total loading time: 0 Render date: 2024-11-15T06:53:27.742Z Has data issue: false hasContentIssue false
  • English
  • Français

User-centered evaluation of adaptive and adaptable systems: a literature review

Published online by Cambridge University Press:  01 September 2008

LEX VAN VELSEN ,
THEA VAN DER GEEST ,
ROB KLAASSEN  and
MICHAËL STEEHOUDER
LEX VAN VELSEN
Affiliation:
Faculty of Behavioral Sciences, Institute for Behavioral Research, Department of Technical and Professional Communication, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlandsl.s.vanvelsen@utwente.nl, t.m.vandergeest@utwente.nl, r.klaassen@minaz.nl, m.f.steehouder@utwente.nl
THEA VAN DER GEEST
Affiliation:
Faculty of Behavioral Sciences, Institute for Behavioral Research, Department of Technical and Professional Communication, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlandsl.s.vanvelsen@utwente.nl, t.m.vandergeest@utwente.nl, r.klaassen@minaz.nl, m.f.steehouder@utwente.nl
ROB KLAASSEN
Affiliation:
Faculty of Behavioral Sciences, Institute for Behavioral Research, Department of Technical and Professional Communication, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlandsl.s.vanvelsen@utwente.nl, t.m.vandergeest@utwente.nl, r.klaassen@minaz.nl, m.f.steehouder@utwente.nl
MICHAËL STEEHOUDER
Affiliation:
Faculty of Behavioral Sciences, Institute for Behavioral Research, Department of Technical and Professional Communication, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlandsl.s.vanvelsen@utwente.nl, t.m.vandergeest@utwente.nl, r.klaassen@minaz.nl, m.f.steehouder@utwente.nl
Get access Rights & Permissions [Opens in a new window]

Abstract

This literature review focuses on user-centered evaluation (UCE) studies of adaptive and adaptable systems. Usability, perceived usefulness and appropriateness of adaptation are the three most commonly assessed variables. Questionnaires appeared to be the most popular method, followed by interviews and data log analysis. The quality of most questionnaires was questionable, and the reporting of interviews and think-aloud protocols was found to be shallow. Furthermore, data logs need triangulation in order to be useful. The reports encountered lack empirical value. The article models the iterative design process for adaptive and adaptable systems, linked to the goals of UCE: supporting decisions, detecting problems and verifying quality. This model summarizes the variables to be assessed at each stage of the process and the relevant methods to assess them.

Type
Articles
Information
The Knowledge Engineering Review , Volume 23 , Issue 3 , September 2008 , pp. 261 - 281
Copyright
Copyright © Cambridge University Press 2008

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Akoumianakis, D., Grammenos, D. & Stephanidis, C. 2001. User interface adaptation: Evaluation perspectives. In User interfaces for All, Stephanidis, C.(ed.). Erlbaum, 339–352.Google Scholar
Alpert, S. R. & Vergo, J. G. 2007. User-centered evaluation of personalized web sites: What’s unique? In Human Computer Interaction Research in Web Design and Evaluation, Zaphiris, P. & Kurniawan, S.(eds). Idea Group, 257–272.Google Scholar
Ammenwerth, E., Iller, C. & Mansmann, U. 2003. Can evaluation studies benefit from triangulation? A case study. International Journal of Medical Informatics 70(2/3),237–248.CrossRefGoogle ScholarPubMed
Benbunan-Fich, R. 2001. Using protocol analysis to evaluate the usability of a commercial web site. Information & Management 39(2), 151–163.CrossRefGoogle Scholar
Benyon, D. & Murray, D. 1993. Adaptive systems: From intelligent tutoring to autonomous agents. Knowledge-Based Systems 6(4), 197–219.CrossRefGoogle Scholar
Bohnenberger, T., Jameson, A., Krüger, A. & Butz, A. 2002. Location-aware shopping assistance: Evaluation of a decision-theoretic approach. Lecture Notes in Computer Science 2411, 155–169.CrossRefGoogle Scholar
BSI. 1998. ISO 9241-11: Ergonomic Requirements for Office Work with Visual Display Terminals. British Standards Institution.Google Scholar
Buchauer, A., Pohl, U., Kurzel, N. & Haux, R. 1999. Mobilizing a health professional’s Workstation: Results of an evaluation study. International Journal of Medical Informatics 54, 105–114.CrossRefGoogle ScholarPubMed
Cawsey, A. J., Jones, R. B. & Pearson, J. 2000. The evaluation of a personalised health information system for patients with cancer. User Modeling and User-Adapted Interaction 10, 47–72.CrossRefGoogle Scholar
Chesnais, P. R., Mucklo, M. J. & Sheena, J. A. 1995. The Fishwrap personalized news system. Paper Presented at the Second International Workshop on Community Networking ‘Integrated Multimedia Services to the Home’, Princeton, USA.Google Scholar
Cheverst, K., Byun, H. E., Fitton, D., Sas, C., Kray, C. & Villar, N. 2005. Exploring issues of user model transparency and proactive behaviour in an office environment control system. User Modeling and User-Adapted Interaction 15, 235–273.CrossRefGoogle Scholar
Chin, D. N. 2001. Empirical evaluation of user models and user-adapted systems. User Modeling and User-Adapted Interaction 11, 181–194.CrossRefGoogle Scholar
Cronbach, L. J. 1951. Coefficient alpha and the internal structure of tests. Psychometrika 16(3),297–334.CrossRefGoogle Scholar
Davis, F. D., Bagozzi, R. P. & Warshaw, P. R. 1989. User acceptance of computer technology: A comparison of two theoretical models. Management Science 35, 982–1003.CrossRefGoogle Scholar
De Jong, M. D. T. & Schellens, P. J. 1997. Reader-focused text evaluation. An overview of goals and methods. Journal of Business and Technical Communication 11(4), 402–432.CrossRefGoogle Scholar
Dicks, R. S. 2002. Mis-usability: On the uses and misuses of usability testing. Paper Presented at the 20th Annual International Conference on Computer Documentation, Toronto, Canada.Google Scholar
Ericsson, K. A. & Simon, H. A. 1993. Protocol Analysis. MIT Press.CrossRefGoogle Scholar
Field, A., Hartel, P. & Mooij, W. 2001. Personal DJ, an open architecture for personalised content delivery. Paper Presented at the Tenth International Conference World Wide Web, Hong Kong.Google Scholar
Gates, K. F., Lawhead, P. B. & Wilkins, D. E. 1998. Toward an adaptive www: A case study in customised hypermedia. New Review of Hypermedia and Multimedia 4, 89–113.CrossRefGoogle Scholar
Gena, C. & Torre, I. 2004. The importance of adaptivity to provide onboard services: A preliminary evaluation of an adaptive tourist information service onboard vehicles. Applied Artificial Intelligence 18(6),549–580.CrossRefGoogle Scholar
Gena, C. 2005. Methods and techniques for the evaluation of user-adaptive systems. Knowledge Engineering Review 20(1), 1–37.CrossRefGoogle Scholar
Gena, C. & Weibelzahl, S. 2007. Usability engineering for the adaptive web. In The Adaptive Web, Brusilovsky, P., Kobsa, A. & Nejdl, W. (eds). Springer-Verlag, 720–762.Google Scholar
Goren-Bar, D., Graziola, I., Kuflik, T., Pianesi, F., Rocchi, C., Stock, O. & Zancanaro, M. 2005. I like it: An affective interface for a multimodal museum guide. Retrieved on 20 April 2006 from http://peach.itc.it/papers/gorenbar2005.pdfGoogle Scholar
Gregor, P., Dickinson, A., Macaffer, A. & Andreasen, P. 2003. Seeword: A personal word processing environment for dyslexic computer users. British Journal of Educational Technology 34(3),341–355.CrossRefGoogle Scholar
Hartson, H. R., Andre, T. S. & Williges, R. C. 2003. Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction 15(1), 145–181.CrossRefGoogle Scholar
Henderson, R., Rickwood, D. & Roberts, P. 1998. The beta test of an electronic supermarket. Interacting with Computers 10, 385–399.CrossRefGoogle Scholar
Herder, E. 2006. Forward, Back and Home Again. Analyzing User Behaviour on the Web. University of Twente.Google Scholar
Hertzum, M. & Jacobsen, N. E. 2003. The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction 15(1), 183–204.CrossRefGoogle Scholar
Höök, K. 1997. Evaluating the utility and usability of an adaptive hypermedia system. Paper Presented at IUI '97, Orlando, USA.Google Scholar
Höök, K. 2000. Steps to take before intelligent user interfaces become real. Interacting with Computers 12(4), 409–426.CrossRefGoogle Scholar
Hvannberg, E. T., Law, E. L., Lárusdóttir, M. K. 2007. Heuristic evaluation: Comparing ways of finding and reporting usability problems. Interacting with Computers 19, 225–240.CrossRefGoogle Scholar
Hyldegaard, J. & Seiden, P. 2004. My E-journal: Exploring the usefulness of personalized access to scholarly articles and services. Information Research 9(3), paper 181. Available at http://InformationR.net/ir/9–3/paper181.htmlGoogle Scholar
ISO. 1999. ISO 13407: Human-Centered Design Processes for Interactive Systems, International Standard Organizations.Google Scholar
Jameson, A. 2003. Adaptive interfaces and agents. In Human-Computer Interaction Handbook, Jacko, J. A. & Sears, A. (eds). Erlbaum, 305–330.Google Scholar
Jameson, A. 2006. Adaptive interfaces and agents. In Human-Computer Interaction Handbook (2nd edn), Jacko, J. A. & Sears, A. (eds). Erlbaum, 433–458.Google Scholar
Jensen, A. L., Boll, P. S., Thysen, I. & Pathak, B. K. 2000. Pl@nteInfo®: A web-based system for personalised decision support in crop management. Computers and Electronics in Agriculture 25, 271–293.CrossRefGoogle Scholar
Kaasinen, E. 2003. User needs for location aware mobile services. Personal Ubiquitous Computing 7(1), 70–79.CrossRefGoogle Scholar
Karat, C. M., Brodie, C., Karat, J., Vergo, J. & Alpert, S. R. 2003. Personalizing the user experience on ibm.Com. IBM Systems Journal 42(4), 686–701.CrossRefGoogle Scholar
Ketamo, H. 2003. Xtask: An adaptable learning environment. Journal of Computer Assisted Learning 19(3), 360–370.CrossRefGoogle Scholar
Kitchenham, B. A., Pfleeger, S. L., Pickard, L. M., Jones, P. W., Hoaglin, D. C., El Emam, K. & Rosenberg, J. 2002. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 28(8), 721–734.CrossRefGoogle Scholar
Kjeldskov, J., Graham, C., Pedell, S., Vetere, F., Howard, S., Balbo, S. & Davies, J. 2005. Evaluating the usability of a mobile guide: The influence of location, participants and resources. Behaviour & Information Technology 24(1), 51–65.CrossRefGoogle Scholar
Kolari, J., Laakko, T., Hiltunen, T., Ikonen, V., Kulju, M., Suihkonen, R., Toivonen, S., & Virtanen, T. 2004. Context-Aware Services for Mobile Users (Technology and User Experiences), VTT publications 539.Google Scholar
Kramer, J., Noronha, S. & Vergo, J. 2000. A user-centered design approach to personalization. Communications of the ACM 43(8), 45–48.CrossRefGoogle Scholar
Lentz, L. & De Jong, M. D. T. 1997. The evaluation of text quality: Expert-focused and reader-focused methods compared. IEEE Transactions on Professional Communication 40(3), 224–234.CrossRefGoogle Scholar
Magoulas, G. D., Chen, S. Y. & Papanikolaou, K. A. 2003. Integrating layered and heuristic evaluation for adaptive learning environments. Paper Presented at the Second Workshop on Empirical Evaluation of Adaptive Systems Held in Conjunction with User Modelling 2003, Pittsburg, USA.Google Scholar
Maguire, M. 2001. Methods to support human-centred design. International Journal of Human-Computer Studies 55(4), 587–634.CrossRefGoogle Scholar
Miles, M. B. & Huberman, A. M. 1994. Qualitative Data Analysis (2nd edn), Sage.Google Scholar
Morgan, D. L. 1996. Focus groups. Annual Review of Sociology, 22, 129–152.CrossRefGoogle Scholar
Muntean, C. H. & McManis, J. 2006. The value of QoE-based adaptation approach in educational hypermedia: Empirical evaluation. Lecture Notes in Computer Science 4018, 121–130.CrossRefGoogle Scholar
Nahl, D. 1998. Ethnography of novices’ first use of web search engines: Affective control in cognitive processing. Internet Reference Services Quarterly 3(2), 51–72.CrossRefGoogle Scholar
NHS Centre for Reviews and Dissemination. 2001. Undertaking Systematic Reviews of Research on Effectiveness, University of York.Google Scholar
Pateli, A. G., Giaglis, G. M. & Spinellis, D. D. 2005. Trial evaluation of wireless info-communication and indoor location-based services in exhibition shows. Advances in Informatics 3746, 199–210.CrossRefGoogle Scholar
Patton, M. Q. 2002. Qualitative Research & Evaluation Methods, (3rd edn), Sage.Google Scholar
Savage, P. 1996. User interface evaluation in an iterative design process: A comparison of three techniques. Paper presented at CHI'96, Vancouver, Canada.Google Scholar
Schmidt-Belz, B. & Poslad, S. 2003. User validation of a mobile tourism service. Paper Presented at the Workshop on HCI in Mobile Guides, Udine, Italy.Google Scholar
Smith, H., Fitzpatrick, G. & Rogers, Y. 2004. Eliciting reactive and reflective feedback for a social communication tool: A multi-session approach. Paper Presented at Designing Interactive Systems: Across the Spectrum, Cambridge, USA.Google Scholar
Sodergard, C., Aaltonen, M., Hagman, S., Hiirsalmi, M., Jarvinen, T., Kaasinen, E., Kinnunen, T., Kolari, J., Kunnas, J. & Tammela, A. 1999. Integrated multimedia publishing: Combining TV and newspaper content on personal channels. Computer Networks: the International Journal of Computer and Telecommunications Networking 31, 1111–1128.CrossRefGoogle Scholar
Spector, P. E. 1992. Summated Rating Scale Construction, Sage.CrossRefGoogle Scholar
Stary, C. & Totter, A. 2003. Measuring the adaptability of universal accessible systems. Behaviour & Information Technology 22(2), 101–116.CrossRefGoogle Scholar
Stein, A. 1997. Usability and assessments of multimodal interaction in the SPEAK! System: An experimental case study. The New Review of Hypermedia and Multimedia 3, 159–180.CrossRefGoogle Scholar
Su, L. T. 1992. Evaluation measures for interactive information retrieval. Information Processing and Management 28(4), 503–516.CrossRefGoogle Scholar
Van den Haak, M. J., De Jong, M. D. T. & Schellens, P. J. 2003. Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue. Behaviour & Information Technology 22(5), 339–351.CrossRefGoogle Scholar
Van den Haak, M. J., De Jong, M. D. T. & Schellens, P. J. 2004. Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: A methodological comparison. Interacting with Computers 16, 1153–1170.CrossRefGoogle Scholar
Van der Geest, T. 2004. Beyond accessibility: Comparing three website usability test methods for people with impairments. Paper Presented at HCI. 2004: Design for Life, Leeds, England.Google Scholar
Van Velsen, L., Van der Geest, T. & Klaassen, R. 2007. Testing the usability of a personalized system: Comparing the use of interviews, questionnaires and thinking-aloud. Paper Presented at the IEEE Professional Communication Conference, Seattle, USA.Google Scholar
Venkatesh, V., Morris, M. G., Davis, G. B. & Davis, F. D. 2003. User acceptance of information technology: Towards a unified view. MIS Quarterly 27(3), 425–478.CrossRefGoogle Scholar
Virzi, R. A., Sokolov, J. L. & Karis, D. 1996. Usability problem identification using both low- and high-fidelity prototypes. Paper Presented at the SIGHI Conference on Human Factors in Computing Systems: Common Ground, Vancouver, Canada.Google Scholar
Vredenburg, K., Mao, J., Smith, P. W. & Carey, T. 2002. A survey of user-centered design practice. CHI Letters 4(1), 471–478.Google Scholar
Walker, M., Takayama, L. & Landay, J. A. 2002. High-fidelity or low-fidelity, paper or computer? Choosing attributes when testing web prototypes. Paper Presented at the 46th Annual Meeting of the Human Factors and Ergonomics Society, Baltimore, USA.Google Scholar
Weibelzahl, S., Lippitsch, S. & Weber, G. 2002. Advantages, opportunities, and limits of empirical evaluations: Evaluating adaptive systems. Künstliche Intelligenz 3(2), 17–20.Google Scholar
Weibelzahl, S. 2003. Evaluation of Adaptive Systems. PhD thesis, University of Trier.Google Scholar
Weibelzahl, S. 2005. Problems and pitfalls in the evaluation of adaptive systems. In Adaptable and Adaptive Hypermedia Systems, Chen, S. Y. & Magoulas, G. D. (eds). IRM Press, 285–299.CrossRefGoogle Scholar
Weibelzahl, S., Jedlitschka, A. & Ayari, B. 2006. Eliciting requirements for a adaptive decision support system through structured user interviews. Paper Presented at the User Centered Design and Evaluation Workshop Held in Conjunction with Adaptive Hypermedia '06, Dublin, Ireland.Google Scholar