Researchers teach algorithm to predict success in effortful tasks

Researchers from HSE University and Skolkovo Institute of Science and Technology have developed machine learning models that can predict success in visual tasks of mental attention using reaction time and eye movement. The paper ‘A machine learning investigation of factors that contribute to predicting cognitive performance: Difficulty level, reaction time and eye-movements’ was published in Decision Support Systems.

Mental attention corresponds to our ability to effortfully focus on a task.  It is a limited cognitive resource that we continuously exhaust during conscious activity. Success in solving complicated intellectual tasks depends on our overall cognitive abilities and available attention resources.

Attention is particularly important for tasks related to retaining visual information, such as in the work of air traffic controllers or drivers on the road. Mental attention is essential for learning and education because students need to mobilize their attention and focus on thinking and processing available information. That is why it is important to learn to measure the impact of attention on the successful completion of visual tasks.

Measures of mental attention have been extensively studied in developmental psychology and education. Mental attention scores correlate highly with general intelligence and academic achievement. The authors wanted to apply machine learning to understand whether accuracy on the task could be predicted by other variables. This has not been done before and had the potential to provide targets for that can explain what is needed to predict someone’s correct choice.  Researchers use quantitative scores such as the reaction time and eye movement metrics to predict accuracy.

Participants of the experiment completed a colour-matching task with two versions: one with balloons and one with clowns. In each of them, the player sees images with different colours for a short time. The participant has to compare the image with the preceding one and say whether the colours match. This game imitates the tasks facing radiologists, drivers, air traffic controllers and other professionals who have to mentally retain visual information and consider changes quickly.

Both tasks have six levels of complexity depending on the number of colours that have to be processed. The clown game is always slightly more difficult than the balloon game since the image of the clown has more variation in detail.

Eye movements were registered with an eye-tracking device. Data from 57 healthy adults with an average age of 23 years were analysed using machine learning models. Such models help us understand what kinds of data best predict success in task completion. In this experiment, the measure of success was precision (the percentage of correct answers from each participant at each level of difficulty).

The XGBoost regressor model demonstrated the best result. This model predicted whether the participant would give the correct answer with 82.8% accuracy. Response speed proved to be the most effective parameter to predict a participant’s success. The more varied it was, the less correct answers were given by the participant. This may be due to the fact that some participants gave a quick, random answer if the difficulty level exceeded their attention capacity.

Eye movement also impacted the result, though to a lesser extent. It was possible to partly predict the success of a participant by such parameters as the average number of fixations on each image and their length, the number of saccades, the number of blinks and the size of the pupils. Eye movements can reflect the effort made by the participant to retain information.

‘The approach we developed can be used to further study the parameters that predict success in solving mental attentional capacity tasks. It also has practical applications: to directly predict in real time the cognitive abilities of professionals, which can change under the influence of such factors as physical condition and fatigue,’ believes Valentina Bachurina, first author of the paper, PhD student at the HSE Laboratory for the Neurobiological Foundations of Cognitive Development.