OECD Skills Studies Beyond Proficiency Using Log Files to Understand Respondent Behaviour in the Survey of Adult Skills

Computer-based administration of large-scale assessments makes it possible to collect a rich set of information on test takers, through analysis of the log files recording interactions between the computer interface and the server. This report examines timing and engagement indicators from the Survey of Adult Skills, a product of the Programme for the International Assessment of Adult Competencies (PIAAC), both of which indicate large differences across countries and socio-demographic groups, in the amount of time spent by respondents and their levels of disengagement, which reduce the probability of giving a correct answer and consequently reduces measured performance. Such insights can help policy makers, researchers and educators to better understand respondents' cognitive strategies and the underlying causes of low and high performance. This, in turn, can help improve the design of assessments and lead to more effective training and learning programmes. Foreword Acknowledgements Executive summary Chapter 1. Overview The value of log files... ... and their limitations Information available from PIAAC log files Differences in timing indicators across countries and respondents How respondents allocate time to different items Log files can be used to capture respondents’ disengagement Disengagement varies across countries and socio-demographic groups Moving forward Note References Chapter 2. What log files are and why they are useful What are log files? Why are log files valuable? Policy makers, researchers and test developers Survey design and management Content and characteristics of PIAAC log files Other features of test design relevant to analysis of log-file data Routing of respondents Adaptive nature of assessment design Conclusions Note References Chapter 3. Timing indicators in the Survey of Adult Skills Basic facts on timing indicators Interpreting timing indicators Cross-country differences in timing indicators Timing indicators, missing answers and performance Socio-demographic distribution of timing indicators and missing answers Gender differences Age differences Education differences Differences related to familiarity with ICT Differences related to proficiency Conclusions Note References Chapter 4. Allocation of time to different items in the Survey of Adult Skills Introduction Timing indicators and item characteristics Time-allocation strategies Conclusions References Chapter 5. Measuring disengagement in the Survey of Adult Skills Introduction What is disengagement Why disengagement matters Measuring disengagement at the item level Rapid item skipping T-disengagement T-disengagement across countries T-disengagement and background characteristics Further analysis of T-disengagement across countries Comparisons of T-disengagement and other indicators Conclusions Notes References Annex A. The PIAAC LogDataAnalyzer Note Reference Annex A. The PIAAC LogDataAnalyzer