The Bonn team had already fully convinced the international jury at the semifinals in Miami last September, where it scored the most points. The finals now took place in Long Beach, California. The aim of the ANA Avatar XPRIZE competition was to develop robotic systems with which people can virtually put themselves to another place. This works via an operator station and an avatar robot that are connected via the Internet. The avatar robot's sensors capture the environment, which is displayed in the operator station in such a way that the user has the feeling of being in the remote location. The user’s movements are captured and transferred to the avatar robot. This enables the user to move around, handle objects, use tools and communicate intuitively through speech, facial expressions and gestures.
Intuitive usability
In the competition, the avatar systems were not controlled by the developers, but by members of an international jury. They had only a few minutes to familiarize themselves with the system. "Therefore, intuitive usability was an important objective in system design," says Prof. Dr. Sven Behnke of Team NimbRo, who heads the Autonomous Intelligent Systems group at the University of Bonn.
At the finals, there was one qualification run and two competition runs on three consecutive days. In all three runs, the jury member was able to solve all ten tasks, which formed a mission that could be set on a distant planet. Tasks included communicating with a human, operating a switch, estimating the weight of objects, using a power drill, and blindly identifying a stone by the roughness of its surface. In addition to task performance, judges also evaluated the quality of the telepresence experience.
In the two competition runs, NimbRo received full points here as well. In case of identical scores, execution time was the deciding factor. With just under six minutes, NimbRo was almost twice as fast here as the second-best team "Pollen Robotics" from Bordeaux (France). For the best competition performance, NimbRo received the grand prize of five million US dollars. "My fantastic team developed a powerful avatar system for the complex tasks of the competition and successfully solved all problems that arose," said Prof. Behnke.
The NimbRo avatar
The NimbRo team's avatar system was developed over three years at the Autonomous Intelligent Systems group of the Institute of Computer Science VI – Intelligent Systems and Robotics at the University of Bonn, led by Prof. Dr. Sven Behnke. The avatar robot has a human-like upper body with two arms and five-finger hands. Attached to the head are a wide-angle stereo camera, a stereo microphone, and a display on which the user's face is animated. The remote scene is visualized to the user through 3D data glasses, with the user's head movements captured and transferred to the avatar head. This allows the user to look around freely, see around obstacles and view objects from different angles, which contributes significantly to the feeling of immersion in the remote environment. Stereo audio is also transmitted alongside the live video signals.
The user feels the interaction forces of the avatar hands via force-moment sensors in the wrist. Robotic arms sense the user's hand movements and relay them to the avatar arms. Hand exoskeletons mediate the finger movements. Gripping forces are sensed by finger motor currents and transmitted to the user's fingers. Haptic sensor measurements are transferred to the user's fingertips. The avatar robot can move in all directions and turn on the spot, with direction and speed controlled by a foot controller. Orientation is provided by cameras which create an all-round view from above. The data glasses are equipped with three cameras to capture eye movements and facial expressions of the user for a live facial animation displayed on the avatar robot's head display.
Beyond this competition, avatar systems have numerous applications. These include, for example, helping people in need of assistance to manage their daily lives, telemedicine, and use in inaccessible or hazardous environments. "Avatar systems have great potential to overcome spatial separation between people and could be as widespread in a few years as video conferencing systems are today," says Prof. Behnke.
Videos of the final competition runs by Team NimbRo from the University of Bonn:
https://youtu.be/pxblVcN606E
https://youtu.be/8AwgGSpcAe8
Information about the avatar system: https://www.ais.uni-bonn.de/nimbro/AVATAR
Information about the competition: https://www.xprize.org/prizes/avatar
