An interface technology that controls a robot with artificial skin that can hear sounds has been developed, and it is expected to be applied to virtual reality, augmented reality, the Internet of Things, and avatar robots in the future.

UNIST (President Yong-Hoon Lee) announced on the 27th that a research team led by Professor Hyun-Hyeop Koh (Department of Energy and Chemical Engineering) and Professor Jae-Jun Kim (Department of Electrical and Electronic Engineering) has developed a human-machine interface that can recognize human movements, touch, and sounds and transmit them to machines.

Regarding this study, the research team said, “It is a human-machine interface that intuitively transmits information to machines instead of pressing buttons or keyboards,” and “Since the sensor is thin and attachable, it can be used in various virtual reality (VR), augmented reality (AR), and Internet of Things (IoT) technologies.”

The interface is based on an artificial skin sensor that mimics the structure of the cochlea of the ear.

The cochlear membrane has different thickness, width, and hardness depending on the area, and this principle allows sound to be received and distinguished by frequency.

These characteristics of the sensor allow it to detect not only slow, repetitive low-frequency signals, such as human motion, but also Not only that, but also high-frequency signals such as fast-vibrating sounds and touch can be transmitted to the machine with a low signal-to-noise ratio.

The research team presented application technologies such as avatar robot hand control technology and smart haptic gloves using this sensor.

In a demonstration of controlling an avatar robot hand with sound, the robot hand's movements could be controlled by changing the frequency.

Additionally, when the user wore the smart haptic gloves and moved, the avatar robot hand followed the user's hand movements exactly, and it also recognized the texture of eight different materials, including glass, paper, and silk, with 93% accuracy.

The developed sensor is in the form of multiple unit frictional electric sensors with different thicknesses, porosities, and areas, like the basilar membrane of the cochlea, attached in series.

In addition, the sensor's internal structure has been specially designed to improve pressure sensitivity by up to eight times compared to existing flat sensors.

The recognition frequency bandwidth is also 45–9,000 Hz (Hertz), so it can recognize all biosignals such as human electrocardiogram signals (0.5–300 Hz), electromyogram signals (50–3,000 Hz), phonocardiogram signals (20–20,000 Hz), and voice (100–400 Hz).

Even in noisy environments, it can recognize human voices with 95% accuracy through machine learning, so it can also be used as a microphone with a noise-cancelling function.

This study was published on March 25 in 'Science Advances', a sister journal of Science, a world-renowned journal published by the American Association for the Advancement of Science (AAAS). It was conducted with the support of the Ministry of Science, ICT and Future Planning's Mid-career Researcher Support Project, Emtek, and POSCO Science Fellowship.