Paper Blogs 02: Towards Visual and Vocal Mimicry Recognition in Human-Human Interactions

Reference Paper

Towards Visual and Vocal Mimicry Recognition in Human-Human Interactions

Xiaofan Sun, Khiet P. Truong, Maja Pantic, Anton Nijholt

(Digital Object Identifier 10.1109/ICSMC.2011.6083693)

(PDF Download) 

Overview of the Paper

Mimicry occurs in face-to-face conversation both when we agree and when we do not agree. It is found that there is more mimicry when people agree than when they don’t. People try to display shared opinion by displaying similar behavior of his/her counterpart. In this paper, the authors present a method to detect and measure behavioral mimicry in face-to-face conversation by analyzing human actions and human vocal behaviors.

They developed an audiovisual corpus specifically for mimicry research. Their data is drawn from a study session of face-to-face discussion and conversation, of 43 different subjects. The experiment divided into two sessions, in the first session, the participants were asked to present their own stance on a specific topic, and this is the presentation episode. Then they need to discuss with their partner about the topic, and this is the discussion episode. In the second session, they participants were asked to talk about non-task oriented topic among themselves, and this is the conversation episode. For visual recording 7 cameras for each person and 1 camera for both persons were used. Their voices were also recorded. The corpus is later annotated by human behavior science specialist annotators.

For visual mimicry detection, they first extract motion features to identify visual mimicry. They use accumulated motion images (AMI) as feature to represent the motions. In AMI, higher intensity value represents higher degree of complex motions. Then they use hand gesture mimicry behavior of the conversation and present the cross-correlation of movements between two persons. From the similar cross-relation of body movements, they assume that behavioral mimicry probably occurs in that time periods.

For detection of non-verbal vocal mimicry, the authors use the speech rate divided by the length of the signal as the feature. They calculate correlations between the speech patterns in different episodes and compare the correlation to each other. ‘Participant in presentation’ is used as the participant’s base line behavior. Correlations will decrease if the participant adapts the speech behavior of the partner, and will increase if the partner’s speech behavior becomes more similar to the participant’s.   

Evaluation

To validate their result, the correlation curve is presented in the following figure.  The solid line represents the correlation between participants performance in presentation and discussion episode (curve A). The dashed line represents the correlation between participants’ and partners’ performance in discussion episode (curve B).

Three phases are observed in the result. Up to window number 8, both correlations increase. Between window number 8 and 17, curve A decreases and B increases. And, after that, curve A increases and B decreases. In phase 1, correlation A is increasing g because the participant begin with the similar speech style in both presentation and discussion episode. And, curve B is also increasing because the confederate in the discussion phase starts the discussion with more similar speech behavior of the participants. In phase 2, correlation A decreases because the participant starts mimicking the confederate. Correlation B still increases because the participant and confederate are mimicking each other. In phase 3, participant and confederate both know that end of discussion is approaching, the they start express their own opinion in their own style, so correlation A increases and B decreases.   

   

Validity of the Paper

In this paper, authors show that the behavioral information can be extracted from audiovisual data and can be used to measure mimicry. The correlation of visual behavior presented in this paper is not reliable enough to detect visual mimicry. From the result it only can be said that the participants show a similar behavior to a certain degree. They also present some future works for approaching this research problem.

 

Improvement Scopes

I think the main challenge faced by the researchers in this field is the annoted data. They authors generate a face-to-face meeting corpus which is very helpful for the researchers in this field, although the data are collected in highly constrained environment. Some improvement scope exists in making the visual mimicry prediction more reliable. In my opinion, the combination method of results achieved from multiple modalities will be the main improvement scope of this research area.  

Further Reading

One of the interesting articles, which are cited by this paper, is “Histograms of oriented gradients for human detection”, by N. Dalal and B. Triggs (Digital Object Identifier 10.1109/CVPR.2005.177)(PDF Download), which is cited by around 4000 articles. In the cited article, the authors use Histograms of Oriented Gradients (HOG) descriptor for human detection. Their result shows that HOG outperforms most of other existing feature sets in this purpose.

 

References

[1] X. Sun, K. Truong, M. Pantic, and A. Nijholt, “Towards visual and vocal mimicry recognition in human-human interactions,” in Systems, Man, and Cybernetics (SMC), 2011 IEEE International Conference on. IEEE, 2011, pp. 367–373.

[2] N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, vol. 1. IEEE, 2005, pp. 886–893.

Online Courses

Steven Leckart shows his excitements about the online Artificial Intelligence course offered by Stanford in his article on Wired. The class was taken by Professor Sebastian Thrun of Stanford University, one of the leading people in the field of Artificial Intelligence.

Many students do not get the chance to be admitted in a university. Even if some of them get the chance, may be the quality of the offered courses are not up to the mark. Online courses now make it possible for everyone all over the world to register for courses offered by the best universities, mostly for free. In my opinion the first major advantage of online courses is that the high quality courses are now accessible by anyone anywhere, and free. It is very convenient for someone who misses the chance of pursuing higher education due to the financial weakness. One can now register for a course of his/her choice. For most of the courses, the class hours are flexible. This flexible timing is very helpful for those who work. Also the online assessment quizzes help the student to evaluate himself.   

Lack of interaction between the instruction and the student is one of the main limitations of the online courses. Sometimes students need to submit his/her questions online and then need to wait for the instructor to respond to the questions. Also the chances of class discussions are very limited here. Although there are some scopes of improvements, online courses offered by the Universities are a good opportunity for peoples who want to learn.    

Paper Blogs 01

Reference Paper

Challenges and Opportunities in Building Socially Intelligent Machines

Laurel D. Riek and Peter Robinson

(Digital Object Identifier 10.1109/MSP.2011.940412)

(PDF download)

  

Overview of the Paper

Understanding the social context is one of the main challenges which are faced by the researchers working in the field of social computing, social signal processing, human-machine interaction, robotics, computer vision and any other field that is related to automatic human behavior analysis. In this paper, the authors discuss some of the challenges to build a real-time system that can process the contextual information and also and respond to it.

Social context is defined here by the environment where a person is situated. There are some factors which may influence the behavior of that person. These factors are situational context, social role, cultural conventions and social norms. Situational context and social roles may change very rapidly, even in between the same group, and very hard to detect by the machine. Again, people of different cultures show deferent social behaviors. A system trained with some specific culture may not work well while dealing with a different one. Social norms also vary widely by different cultures and locations. As a result, it is very hard to build an intelligent machine which can handle these factors.  

Though some applications have adopted some techniques to realize certain situational contexts, it is a very difficult and broader problem to make the machine learn the cultural context on the fly. Even after solving all these problems by the researchers, the large practical problem still remains, that is unpredictability of the human. Some progress can be made to tackle the challenges of this field by using the context in clever ways.    

 

Evaluation

The overall social context is defined here in the first part. After the definition, different practical problems associated with different factors are presented with examples. Then in the last part, the practical challenges associated with exploiting the contextual information to build an autonomous system are described.  

Validity of the Paper

In this paper, the practical challenges associated with building an intelligent system which can exploit the contextual information, are introduced and discussed. This creates an opportunity for future researchers in this field to use these observations. In conclusion, a valid assumption is made that some progresses can be made on tackling the challenges faced by the researchers in this field, by using the context in clever ways.

Improvement Scopes

I think in this paper the main challenges faced by the researchers in this field to build an autonomous system which can exploit context, are presented in a very nice and compact manner. Some improvements achieved by the research community by this time can be added in the next version.

Further Reading

One of the interesting articles, which are cited by this paper, is “Capturing order in social interactions [Social Sciences]”, by Alessandro Vinciarelli (Digital Object Identifier 10.1109/MSP.2009.933382) (PDF Download). In the cited article, the author presents some of the most promising research directions aimed by the artificial social intelligence community and social signal processing community.

What Every Computer Science Major Should Know

In his article “What every computer science major should know”, author Matt Might discusses some CS subjects and topics, and proposes that every computer science major should know these concepts. I mostly agree with the view of the author.

I completely agree with Matt’s proposal about the programming languages. In my opinion, every CS-major should know at least two major paradigms of programming languages, structured programming language (C) and object oriented programming language (C++ and Java) paradigm. They may need to work on a different programming language or a different paradigm in their latter life; but it should not be hard for them to learn a new one after learning these two. I also agree with author’s view about ‘The UNIX Philosophy’. Being comfortable with UNIX systems will not only help them in debugging there program, but also help them to design and develop operating system level codes.

Discrete Math, Data Structures, Algorithm and Computer Architectures should be in the core of the undergrad CS syllabus. These courses provide the necessary concepts and tools to approach a problem efficiently, which every CS student must know. Author’s suggestions about these subjects are very clear and I completely agree with it.

However, I do not fully agree with author’s view about Parallel Computing and Formal Methods. These concepts are also important, but in my opinion these courses are more on the side of specialized knowledge; rather than on the side of general CS knowledge that every CS major should know. These courses should be in the list of elective subjects. I want to add one more topic in the list, which is ‘Probability Models’ concept. This subject may be covered partially in ‘Statistics’, but a clear knowledge about probability models will help a student to approach and formulate a problem more efficiently, mostly in the field of artificial intelligence and computer networking.