Tufts University researchers are developing techniques that could allow computers to respond to users’ thoughts of frustration — too much work — or boredom—too little work. Applying non-invasive and easily portable imaging technology in new ways, they hope to gain real-time insight into the brain’s more subtle emotional cues and help provide a more efficient way to get work done.

“New evaluation techniques that monitor user experiences while working with computers are increasingly necessary,” said Robert Jacob, computer science professor and researcher. “One moment a user may be bored, and the next moment, the same user may be overwhelmed. Measuring mental workload, frustration and distraction is typically limited to qualitatively observing computer users or to administering surveys after completion of a task, potentially missing valuable insight into the users’ changing experiences.”

Sergio Fantini, biomedical engineering professor, in conjunction with Jacob’s human-computer interaction (HCI) group, is studying functional near-infrared spectroscopy (fNIRS) technology that uses light to monitor brain blood flow as a proxy for workload stress a user may experience when performing an increasingly difficult task. A $445,000 grant from the National Science Foundation will allow the interdisciplinary team to incorporate real-time biomedical data with machine learning to produce a more in-tune computer user experience.

Lighting Up the Brain

“fNIRS is an emerging non-invasive, lightweight imaging tool which can measure blood oxygenation levels in the brain,” said Fantini, also an associate dean for graduate education at Tufts’ School of Engineering.

The fNIRS device, which looks like a futuristic headband, uses laser diodes to send near-infrared light through the forehead at a relatively shallow depth—only two to three centimeters—to interact with the brain’s frontal lobe. Light usually passes through the body’s tissues, except when it encounters oxygenated or deoxygenated hemoglobin in the blood. Light waves are absorbed by the active, blood-filled areas of the brain and any remaining light is diffusely reflected to the fNIRS detectors.

“fNIRS, like MRI, uses the idea that blood flow changes to compensate for the increased metabolic demands of the area of the brain that’s being used,” said Erin Solovey, a graduate researcher at the School of Engineering.

“We don’t know how specific we can be about identifying users’ different emotional states,” said Fantini. “However, the particular area of the brain where the blood flow change occurs should provide indications of the brain metabolic changes and by extension workload, which could be a proxy for emotions like frustration.”

In the initial experiments, Jacob and Fantini’s groups determined how accurately fNIRS could register users’ workload. While wearing the fNIRS device, test subjects viewed a multicolored cube consisting of eight smaller cubes with two, three or four different colors. As the cube rotated onscreen, subjects counted the number of colored squares in a series of 30 tasks. The fNIRS device and subsequent user surveys reflected greater difficulty as users kept track of increasing numbers of colors. The fNIRS data agreed with user surveys up to 83 percent of the time.

The Tufts group will present its initial results on using fNIRS to detect the user workload experience at the Association for Computing Machinery (ACM) symposium on user interface software and technology, to be held Oct. 7 through 10 in Newport, R.I.

“It seems that we can predict, with relatively high confidence, whether the subject was experiencing no workload, low workload, or high workload,” said Leanne Hirshfield, a graduate researcher and lead author on the poster paper to be presented at the ACM symposium.

- Tufts University