Durable Interaction with Socially Intelligent Robots (iSocioBot)

Research project supported by The Danish Council for Independent Research | Technology and Production Sciences, Ministry of Science, Innovation and Higher Education (in Danish: Det Frie ForskningsrĆd | Teknologi og Produktion, Uddannelses- og Forskningsministeriet)

Grant DFF – 1335-00162

1 August 2013 – 31 December 2017

A mirror project homepage: http://socialrobot.dk


Project leader:

Zheng-Hua Tan

Department of Electronic Systems, Aalborg University, Denmark

E-mail: zt@es.aau.dk


Project description:

This project develops potentially groundbreaking methods that make service robots socially intelligent and capable of establishing durable relationship with their users. This relies on developing the capabilities to sense and express, which faces grand challenges: the low-quality signals and the poor context-awareness. We first propose a new paradigm called reinforcement fusion, which combines sensor signals in an interactive way: e.g. when a robot detects a sound direction, it turns towards the direction to see better and moves towards it to hear better. Reinforcement fusion is analogous to reinforcement learning, a known term in machine learning. It will dramatically improve robot's sensibility to the context including social behaviours. Secondly we propose a concept of social behaviour entrainment to adapt behaviours. Entrainment is the phenomenon that dialogue partners tend to adapt their speaking style of each other. Our hypothesis is that through reinforcement fusion based tracking and social information extraction, machine learning based context and user modelling, and social behaviour entrainment, durable social interaction between human and robot is achievable. The reinforcement fusion paradigm is applicable to all sorts of systems with steerable sensors.

    Project members: Zheng-Hua Tan, SŅren Holdt Jensen, BŅrge Lindberg, Nicolai Bĺk Thomsen, Xiaodong Duan, Evgenios Vlachos, Ibrahim Hameed (former member).

    Robots: iSocioBot (or SocioBot), Nao, Pioneer 3-DX, TurtleBot, and HOVIS Genie.
  1. Our robotic research with healthy elders and elders suffering from dementia at Fremtidens Plejehjem facility was picked up by Aalborg Kommune LinkedIn page, July 15, 2016.
  2. International Workshop on Sensing, Processing and Learning for Intelligent Machines (SPLINE 2016), jointly organised by iSocioBot project, was held at Aalborg University, Denmark, July 6-8, 2016.
  3. News articles in BiTE (part of BT): Danskudviklet robot er fremtidens sociale hjaelper: Se hvad den kan and Om fĆ Ćr kan du fĆ en robot, der kender dig sĆ godt som dine venner, August 2015.
  4. iSocioBot team offered 3 weeks summer school training course in Programming Social Robots for Human Interaction at Aalborg University, July 20-August 7, 2015.
  5. iSocioBots and the team were present at the People's Meeting (FolkemŅdet) in Bornholm, June 2015. The People's Meeting, an annual event in Denmark, featured with 2700 events and attacked ca. 100 000 guests this year.
  6. The second iSocioBot was built on the basis of Pioneer 3-DX, June 2015.
  7. Visit and talk by Research Scientist Najim Dehak, Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology (MIT), U.S.A. Talk title: “I-vector representation based on GMM and DNN for audio classification", May 19, 2015 at 13.00.
  8. Nao and Pioneer 3-DX both joined the robot team in January 2015.
  9. iSocioBot was present at the Culture Night 2014 in Copenhagen at the Ministry of Higher Education and Science (with about 4500 visitors) and interacted with Sofie Carsten Nielsen, Minister for Higher Education and Science, October 10, 2014.
  10. iSocioBot inaugurated 'Safe 7' in Nibe, September 6, 2014.
  11. Visit and talk by Prof. Jen-Tzung Chien, National Chiao Tung University. Talk title: "Bayesian Nonparametric Information Processing", August 26, 2014 at 13:30.
  12. Our iSocialBot and Presenter Ulla Essendrop presided over the official opening of the Day of Research 2014 in Denmark (Forskningens DŅgn 2014), which Crown Princess Mary attended with 250 invited guests on 24 April 2014. Et dŅgn der varer tre dŅgn | Nordjyske.dk. A video clip of the opening.
  13. Visit and talk by Prof. John H.L. Hansen, The University of Texas at Dallas, U.S.A. Talk title: “Speaker & Noise Variability – Making Speech Systems Robust”, Jan. 29, 2014 at 13:00.
  14. News article in IngeniŅren: Aalborg-forskere: SĆdan bygger vi den menneskelige robot, 1 November 2013. Google English translation.

  1. Elizabeth Ann Jochum, Evgenios Vlachos, Sally Grindsted Nielsen, Anja Christoffersen, Ibrahim Hameed and Zheng-Hua Tan, “”Using Theatre to Study Interaction with Care Robots”,”accepted by International Journal of Social Robotics. (Springer).
  2. Nicolai B. Thomsen, Xiaodong Duan, Zheng-Hua Tan, BŅrge Lindberg, and SŅren Holdt Jensen, “Improving the Convergence of CO-training for Audio-Visual Person Identification,” The International Workshop on Sensing, Processing and Learning for Intelligent Machines (SPLINE2016), July 6-8, 2016, Aalborg, Denmark.
  3. Ibrahim A. Hameed, Zheng-Hua Tan, Nicolai B. Thomsen and Xiaodong Duan, “User Acceptance of Social Robots,” The 9th International Conference on Advances in Computer-Human Interactions (ACHI 2016), Venice, Italy, April 24-28, 2016.
  4. Xiaodong Duan and Zheng-Hua Tan, “Neighbors Based Discriminative Feature Difference Learning for Kinship Verification,” The 11th International Symposium on Visual Computing, December 14-16, 2015, Las Vegas, Nevada, USA.
  5. Nicolai Thomsen, Zheng-Hua Tan, BŅrge Lindberg and SŅren Holdt Jensen, “A Heuristic Approach for a Social Robot to Navigate to a Person Based on Audio and Range Information,” 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (iROS), Hamburg, Germany, September 28 – October 02, 2015.
  6. Sally Grindsted Nielsen, Anja Christoffersen, Elizabeth Jochum and Zheng-Hua Tan, “Robot Future: Using Theatre to Influence Acceptance of Care Robots,” The New Friend 2015 Conference, Almere, The Netherlands, October 22-23, 2015.
  7. Xiaodong Duan and Zheng-Hua Tan, “Local Feature Learning for Face Recognition under Varying Poses,” IEEE International Conference on Image Processing (ICIP 2015), 27-30 September 2015, Quebec City, Canada.
  8. Xiaodong Duan and Zheng-Hua Tan, “A Feature Subtraction Method for Image Based Kinship Verification under Uncontrolled Environments,” IEEE International Conference on Image Processing (ICIP 2015), 27-30 September 2015, Quebec City, Canada.
  9. Rasmus Lyngby Kristensen, Zheng-Hua Tan, Zhanyu Ma and Jun Guo, “Binary Pattern Flavored Feature Extractors for Facial Expression Recognition: An Overview,” CIS-MIPRO 2015, 25-29 May 2015, Opatija, Croatia.
  10. Zheng-Hua Tan, Nicolai Bĺk Thomsen and Xiaodong Duan, “Designing and Implementing an Interactive Social Robot from Off-the-shelf Components,”The 3rd IFToMM Symposium on Mechanism Design for Robotics (MEDER2015), June 2-4, 2015, Aalborg, Denmark.
  11. Nicolai Bĺk Thomsen, Zheng-Hua Tan, BŅrge Lindberg and SŅren Holdt Jensen, “Learning Direction of Attention for a Social Robot in Noisy Environments,” The 3rd AAU Workshop on Robotics (AAUROB2014), Aalborg, Denmark.
  12. Nicolai B. Thomsen, Zheng-Hua Tan, BŅrge Lindberg and SŅren Holdt Jensen, “Improving Robustness against Environmental Sounds for Directing Attention of Social Robots,” The 2nd Workshop on Multimodal Analyses Enabling Artificial Agents in Human-Machine Interaction, September 14, 2014, Singapore.