Microphones as sensors: Teaching old microphones new tricks

Sensor technology: Microphones are designed to capture sound. But they turn out to be able to capture other sorts of information, too

MICROPHONES exist in many shapes and sizes, and work in many different ways. In the late 19th century, early telephones relied on carbon microphones, pioneered by Thomas Edison; today’s smartphones contain tiny microphones based on micro-electro-mechanical systems, commonly called MEMS. Specialist microphones abound in recording studios; others are used by spies. But whatever the technology, these microphones all do the same thing: they convert sound waves into an electrical signal.

It turns out, however, that with the addition of suitable software, microphones can detect more than mere audio signals. They can act as versatile sensors, capable of tuning into signals from inside the body, assessing the social environment and even tracking people’s posture and gestures. Researchers have reimagined microphones as multi-talented collectors of information. And because they are built into smartphones that can be taken anywhere, and can acquire new abilities simply by downloading an app, they are being put to a range of unusual and beneficial uses.

That natural microphone, the human ear, is finely attuned to picking up certain characteristics in a person’s voice. It is not hard, for instance, to infer from a slight change in pitch when a friend might be under stress. Tanzeem Choudhury of Cornell University and her research team are building mobile-phone software that can be trained to do the same thing. That stress results in subtle changes in pitch, amplitude and frequency, as well as speaking rate, has been known for decades. But humans respond to stress in different ways and have different coping styles, says Dr Choudhury. So a one-size-fits-all smartphone app which analyses speech will not provide an accurate assessment of whether someone is stressed or not.

The sound of stress

Dr Choudhury’s solution is an app called StressSense. Running on a standard Android-based smartphone, it listens for certain universal indicators of stress but is able, over time, to learn the specifics of a particular user’s voice. It is unobtrusive yet is also robust enough to work in noisy environments, which is crucial if it is to be of practical diagnostic use. StressSense does not actually record speech, Dr Choudhury emphasises, but simply captures and analyses characteristics such as amplitude and frequency. In a paper published last year, the researchers concluded that “it is feasible to implement a computationally demanding stress-classification system on off-the-shelf smartphones”. Their ultimate goal is to develop an app that can help someone determine the links between irritating situations and subsequent responses. Your phone might realise before you do, for example, that your 8am meetings are the cause of your headaches.

StressSense is still in development. In the meantime, Dr Choudhury’s team has launched an Android app called BeWell that focuses more on overall health by looking at three metrics: sleep, physical activity and social interaction. These three metrics, Dr Choudhury believes, are important yet easily measured indicators of someone’s health. BeWell’s sleep-tracking feature guesses whether the phone’s user is awake or not by analysing usage, light and sound levels, and charging habits. Physical activity is monitored using built-in accelerometers for motion-detection. And social activity is measured chiefly by collecting snippets of sound that indicate that the user is talking to someone, either in person or over the phone.

Again, no actual words are stored, simply features of human speech, which the app can distinguish from background noise such as music or traffic. Certain changes in a person’s social interactions—a sudden drop-off, for instance—can be indicative of health problems such as depression, says Dr Choudhury. The idea is that the app could tip someone off to a change in their behaviour that might otherwise have gone unnoticed.

Microphones need not limit themselves to listening to the human voice, however. John Stankovic of the University of Virginia in Charlottesville is using microphones to capture heartbeats. Researchers in his group are using earphones modified with accelerometers and additional microphones that detect the pulse in arteries in the wearer’s ear. This makes it possible to collect information about the wearer’s physical state, including heart rate and activity level, which is transmitted to the smartphone via the audio jack. The researchers even created an app, called MusicalHeart, that analyses the wearer’s heart rate and recommends songs from a music library based on a heart-rate goal—faster to encourage a runner, or slower to calm someone who is feeling nervous.

Read more . . .