Reviewing Evaluation Methods
In order to delve into the research on appropriate evaluation methods for Full-Body Interactions Learning Environments, we realized a systematic review of the studies published in the last 14 years (2002-2016) related to the evaluation of learning experiences based on Full-Body Interaction.
In order to delve into the research on appropriate evaluation methods for Full-Body Interactions Learning Environments, we realized a systematic review of the studies published in the last 14 years (2002-2016) related to the evaluation of learning experiences based on Full-Body Interaction. The review was focused towards providing a critical and reflexive analysis of the employed evaluation instruments. In particular, it focused on spotting out current approaches, identifying their limits and potential, and analyzing their consistency with the intended learning goals and with the embodied cognition framework.
The full review has been formalized as a scientific paper (currently submitted). Here the main highlights:
|
Data gathering instrument |
Approaches in the design of the data gathering instrument |
Research goals |
Affordances for evaluating learning |
Affordances to understand embodiment |
Affordances to inform design |
|
Questionnaires |
Multiple-choice questions Open-ended questions |
Tracking learning gains through the comparison of pre and post-test scores Comparing different interfaces / instructional methods |
Conceptualize learning as knowledge acquisition: appropriate to evaluate factual knowledge (e.g. recall of information) Unable to grasp the processual nature of learning and meaning construction Poor sensitivity to evaluate learning goals related to conceptual knowledge |
Focus only in cognitive outcomes Poor sensitivity to understand embodiment as a resource to construct meaning Risk of mirroring the dualistic bias of privileging formal/symbolic knowledge over practical one |
Do not provide robust knowledge to inform design refinements |
|
Interviews |
Structuring of the interview: Open-ended or close questions Scheduling of the interview: concurrent to the task or posterior Methodological triangulation |
Analyzing users’ understanding and subjective interpretations of the experience |
Sensitive to observe reasoning processes, interpretations and indicators of constructive learning
|
Sensitive to research reflection-on-action (how embodied experience is elaborated and interpreted) Risk of focusing only on cognitive outcomes / verbal resources and neglect embodied resources for meaning making |
Ground design refinements on users’ feedback |
|
Analysis of in situ interaction |
Video analysis (different level of granularity) Multimodal approaches / Microethnographic approaches Field observation |
Understand in situ meaning making Assess procedural knowledge |
Sensitive to assess procedural knowledge Sensitive to identify indicators of conceptual learning and strategic reasoning |
Acknowledge and take into account the role of embodied, spatial and social resources Sensitive to understand how users construct meaning in embodied experiences |
Ground design refinements on observation of users interaction Identify usability issues |