38 teams from 16 countries had qualified for the semifinals. Among them was NimbRo from the University of Bonn, which was ranked first with 99 out of 100 points. "I am proud of my team, which developed an avatar system well suited for the complex competition tasks and made it fit to the point," said Prof. Dr. Sven Behnke, head of the Institute for Computer Science VI – Intelligent Systems and Robotics at the University of Bonn, after the semifinals. The finals are scheduled to take place next year in the USA. The 133,333 US dollars in prize money that NimbRo will receive for qualifying will be used to prepare for the finals. In the final round, a further eight million U.S. dollars will then be distributed among the top three teams.
The Bonn-based team NimbRo has been developing the Avatar system, which consists of an operator station and a mobile avatar robot, for two years. The avatar has a human-like upper body with two arms and five-finger hands. Attached to the head is a wide-angle stereo camera, a stereo microphone, and a display on which the operator's face is animated. The remote scene is visualized to the operator through a 3D head-mounted display, with the operator's head movements captured and transferred to the avatar head. This allows the operator to look around freely, see past obstacles, and view objects from different angles.
Exoskeletons mediate the hand movements
The three-dimensional perception of the scene is created in the operator's head by the different viewing angles and camera movements. Stereo audio also contributes to immersion. The operator feels the interaction forces of the avatar hands via force-torque sensors in the wrist. Robotic arms sense the operator's hand movements and relay them to the avatar arms. Hand exoskeletons mediate the finger movements. Gripping forces are sensed by finger motor currents and transmitted to the operator's fingers. The avatar's base drive can be controlled by a foot controller, with cameras creating a bird’s eye all-around view for orientation. The head-mounted display is equipped with three cameras that capture the operator's eye movements and facial expressions for facial animation.
In the run-up to the semifinals, the Bonn team used its avatar system to perform numerous tasks in everyday environments. These included, for example, making coffee, playing chess, but also measuring blood pressure, temperature and oxygen saturation like a nurse. Like a gentleman, the avatar helped a person in need of assistance into a jacket. In the semifinal tasks in Miami, the operator played jigsaw puzzles together with a test person through the avatar system, shone at the celebration of a business deal and explored an object with his senses. The system was not operated by the development team, but by members of the international jury. Therefore, intuitive operation was one of the requirements. The Bonn system worked perfectly in the semifinals and impressed the jury members.
That such avatars have a practical significance became clear not least by the contact restrictions of the Corona pandemic. With such systems, people can overcome distances and be present in another location, for example, when working in dangerous or inaccessible environments, when supporting emergency forces to cope with disasters, or in space. "Avatar systems have the potential to overcome spatial separation between people and, through mass production – and the associated reduction in costs – may be as widespread in a few years as video conferencing systems are today," says Prof. Behnke.
Video of semifinals run by Team NimbRo from the University of Bonn: https://youtu.be/UA7vpQO5oZQ
Information about the avatar system: https://www.ais.uni-bonn.de/nimbro/AVATAR
Information about the competition: https://www.xprize.org/prizes/avatar
Contact:
Prof. Dr. Sven Behnke
University of Bonn
Institute of Computer Science VI - Intelligent Systems and Robotics
Tel. +49-228-73 4116
E-mail: behnke@cs.uni-bonn.de
