Imitation from Observation (IfO) prompts the robot to imitate tasks from unlabeled videos via reinforcement learning (RL). The performance of the IfO algorithm depends on its ability to extract task-relevant representations since images are informative. Existing IfO algorithms extract image representations by using a simple encoding network or pre-trained network. Due to the lack of action labels, it is challenging to design a supervised task-relevant proxy task to train the simple encoding network. Representations extracted by a pre-trained network such as Resnet are often task-irrelevant. In this article, we propose a new approach for robot IfO via multimodal observations. Different modalities describe the same information from different sides, which can be used to design an unsupervised proxy task. Our approach contains two modules: the unsupervised cross-modal representation (UCMR) module and a self-behavioral cloning (self-BC)-based RL module. The UCMR module learns to extract task-relevant representations via a multimodal unsupervised proxy task. The Self-BC for further offline policy optimization collects successful experiences during the RL training. We evaluate our approach on the real robot pouring water task, quantitative pouring task, and pouring sand task. The robot achieves state-of-the-art performance.

Robots with multi-sensors always have a problem of weak pairing among different modals of the collected information produced by multi-sensors, which leads to a bad perception performance during robot interaction. To solve this problem, this paper proposes a Force Vision Sight (FVSight) sensor, which utilizes a distributed flexible tactile sensing array integrated with a vision unit. This innovative approach aims to enhance the overall perceptual capabilities for object recognition. The core idea is using one perceptual layer to trigger both tactile images and force-tactile arrays. It allows the two heterogeneous tactile modal information to be consistent in the temporal and spatial dimensions, thus solving the problem of weak pairing between visual and tactile data. Two experiments are specially designed, namely object classification and slip detection. A dataset containing 27 objects with deep presses and shallow presses is collected for classification, and then 20 slip experiments on three objects are conducted. The determination of slip and stationary state is accurately obtained by covariance operation on the tactile data. The experimental results show the reliability of generated multimodal data and the effectiveness of our proposed FVSight sensor.

In spoken conversations, speakers and their addressees constantly seek and provide different forms of audiovisual feedback, also known as backchannels, which include nodding, vocalizations and facial expressions. It has previously been shown that addressees backchannel at specific points during an interaction, namely after a speaker provided a cue to elicit feedback from the addressee. However, addressees may differ in the frequency and type of feedback that they provide, and likewise, speakers may vary the type of cues they generate to signal the backchannel opportunity points (BOPs). Research on the extent to which backchanneling is idiosyncratic is scant. In this article, we quantify and analyze the variability in feedback behavior of 14 addressees who all interacted with the same speaker stimulus. We conducted this research by means of a previously developed experimental paradigm that generates spontaneous interactions in a controlled manner. Our results show that (1) backchanneling behavior varies between listeners (some addressees are more active than others) and (2) backchanneling behavior varies between BOPs (some points trigger more responses than others). We discuss the relevance of these results for models of human–human and human–machine interactions.

Flight crews’ capacity to conduct take-off and landing in near zero visibility conditions has been partially addressed by advanced surveillance and cockpit display technology. This capability is yet to be realised within the context of manoeuvring aircraft within airport terminal areas. In this paper the performance and workload benefits of user-centre designed visual and haptic taxi navigational cues, presented via a head-up display (HUD) and active sidestick, respectively, were evaluated in simulated taxiing trials by 12 professional pilots. In addition, the trials sought to examine pilot acceptance of side stick nose wheel steering. The HUD navigational cues demonstrated a significant task-specific benefit by reducing centreline deviation during turns and the frequency of major taxiway deviations. In parallel, the visual cues reduced self-report workload. Pilot’s appraisal of nose wheel steering by sidestick was positive, and active sidestick cues increased confidence in the multimodal guidance construct. The study presents the first examination of how a multimodal display, combining visual and haptic cues, could support the safety and efficiency in which pilots are able to conduct a taxi navigation task in low-visibility conditions.

Piglets on commercial pig farms are often tail-docked to reduce the incidence of tail-biting. While this is a painful procedure, piglets are often not provided analgesia or anaesthesia for pain relief. The objectives of this study were to assess a multimodal approach to managing tail-docking pain in piglets, using 0.4 mg kg-1 meloxicam (MEL), 0.04 mg kg-1 buprenorphine (BUP), and Maxilene® (MAX), a topical anaesthetic. The effectiveness of each drug and drug combination was evaluated using behavioural indicators, vocalisation, and facial grimace analysis. This study also assessed whether male and female piglets responded differently to pain or pain treatments. Piglets were randomly assigned to one of six possible treatments: MEL, BUP, MEL + BUP, MEL + BUP + MAX, no treatment (tail-docked control), or sham (non-tail-docked control). Vocalisations were recorded at initial handling, injection, and tail-docking. Piglets administered MEL + BUP and BUP demonstrated significantly fewer pain behaviours than piglets in the MEL and no treatment group. MEL + BUP + MAX and BUP piglets displayed significantly lower facial grimace scores than piglets in the no treatment group. There were no significant differences in emitted vocalisations between the analgesia-treated piglets and the no treatment group and both injection and tail-docking elicited piglet vocalisations of similar frequency, power, and energy. There were no significant differences in behaviour, facial grimacing or emitted vocalisations between male and female piglets. All treatment groups with buprenorphine were able to alleviate tail-docking-associated pain, suggesting that opioid administration is highly effective for managing piglet pain.

Surgical castration of piglets is a routine procedure on commercial pig farms, to prevent boar taint and reduce aggression. This procedure is known to cause pain, yet piglets are often not provided appropriate analgesia for relief. The objective of this study was to assess a multimodal approach to managing post-castration pain in piglets, using 0.4 mg kg-1 meloxicam (MEL), 0.04 mg kg-1 buprenorphine (BUP), and Maxilene® (MAX). Efficacy was evaluated using behavioural indicators, vocalisation, and facial grimace analysis. Male piglets were randomly assigned to one of ten possible treatments (n = 15 piglets per treatment group): MEL + BUP + MAX (castrated or uncastrated); MEL + BUP (castrated or uncastrated); BUP + MAX (castrated or uncastrated); MEL + MAX (castrated or uncastrated); saline (castrated control); or sham (uncastrated control). Castrated piglets in the MEL + BUP + MAX, MEL + BUP, and BUP + MAX treatment groups displayed significantly fewer pain behaviours than piglets administered saline. MEL + MAX was insufficient in reducing surgical castration pain behaviours. At 24 h post-procedure, saline and MEL + MAX-castrated piglets displayed significantly more pain behaviours than all other treatment groups and time-points. Facial grimace analysis indicated that MEL + MAX-castrated piglets had significantly higher grimace scores than MEL + BUP (castrated and uncastrated) and BUP + MAX-uncastrated. There were no significant differences in emitted vocalisations between the analgesia-treated and saline-castrated piglets. All treatment groups with buprenorphine were effective in alleviating castration-associated pain behaviours, suggesting that opioid administration is beneficial for managing piglet castration pain.

A growing body of research supports the value of a multimodal assessment approach, drawing on measures from different response modalities, for clarifying how core biobehavioral processes relate to various clinical problems and dimensions of psychopathology. Using data for 507 healthy adults, the current study was undertaken to integrate self-report and neurophysiological (brain potential) measures as a step toward a multimodal measurement model for the trait of affiliative capacity (AFF) – a biobehavioral construct relevant to adaptive and maladaptive social-interpersonal functioning. Individuals low in AFF exhibit a lack of interpersonal connectedness, deficient empathy, and an exploitative-aggressive social style that may be expressed transdiagnostically in antagonistic externalizing or distress psychopathology. Specific aims were to (1) integrate trait scale and brain potential indicators into a multimodal measure of AFF and (2) evaluate associations of this multimodal measure with criterion variables of different types. Results demonstrated (1) success in creating a multimodal measure of AFF from self-report and neural indicators, (2) effectiveness of this measure in predicting both clinical-diagnostic and neurophysiological criterion variables, and (3) transdiagnostic utility of the multimodal measure at both specific-disorder and broad symptom-dimension levels. Our findings further illustrate the value of psychoneurometric operationalizations of biobehavioral trait dimensions as referents for clarifying transdiagnostic relationships between biological systems variables and empirically defined dimensions of psychopathology.

It is well known that traditional Markov chain Monte Carlo (MCMC) methods can fail to effectively explore the state space for multimodal problems. Parallel tempering is a well-established population approach for such target distributions involving a collection of particles indexed by temperature. However, this method can suffer dramatically from the curse of dimensionality. In this paper we introduce an improvement on parallel tempering called QuanTA. A comprehensive theoretical analysis quantifying the improved efficiency and scalability of the approach is given. Under weak regularity conditions, QuanTA gives accelerated mixing through the temperature space. Empirical evidence of the effectiveness of this new algorithm is illustrated on canonical examples.

Engagement represents how much a user is interested in and willing to continue the current dialogue. Engagement recognition will provide an important clue for dialogue systems to generate adaptive behaviors for the user. This paper addresses engagement recognition based on multimodal listener behaviors of backchannels, laughing, head nodding, and eye gaze. In the annotation of engagement, the ground-truth data often differs from one annotator to another due to the subjectivity of the perception of engagement. To deal with this, we assume that each annotator has a latent character that affects his/her perception of engagement. We propose a hierarchical Bayesian model that estimates both engagement and the character of each annotator as latent variables. Furthermore, we integrate the engagement recognition model with automatic detection of the listener behaviors to realize online engagement recognition. Experimental results show that the proposed model improves recognition accuracy compared with other methods which do not consider the character such as majority voting. We also achieve online engagement recognition without degrading accuracy.

While artificial neural networks have been in existence for over half a century, it was not until year 2010 that they had made a significant impact on speech recognition with a deep form of such networks. This invited paper, based on my keynote talk given at Interspeech conference in Singapore in September 2014, will first reflect on the historical path to this transformative success, after providing brief reviews of earlier studies on (shallow) neural networks and on (deep) generative models relevant to the introduction of deep neural networks (DNN) to speech recognition several years ago. The role of well-timed academic-industrial collaboration is highlighted, so are the advances of big data, big compute, and the seamless integration between the application-domain knowledge of speech and general principles of deep learning. Then, an overview is given on sweeping achievements of deep learning in speech recognition since its initial success. Such achievements, summarized into six major areas in this article, have resulted in across-the-board, industry-wide deployment of deep learning in speech recognition systems. Next, more challenging applications of deep learning, natural language and multimodal processing, are selectively reviewed and analyzed. Examples include machine translation, knowledgebase completion, information retrieval, and automatic image captioning, where fresh ideas from deep learning, continuous-space embedding in particular, are shown to be revolutionizing these application areas albeit with less rapid pace than for speech and image recognition. Finally, a number of key issues in deep learning are discussed, and future directions are analyzed for perceptual tasks such as speech, image, and video, as well as for cognitive tasks involving natural language.

Interactive language use inherently involves a process of coordination, which often leads to matching behaviour between interlocutors in different semiotic channels. We study this process of interactive alignment from a multimodal perspective: using data from head-mounted eye-trackers in a corpus of face-to-face conversations, we measure which effect gaze fixations by speakers (on their own gestures, condition 1) and fixations by interlocutors (on the gestures by those speakers, condition 2) have on subsequent gesture production by those interlocutors. The results show there is a significant effect of interlocutor gaze (condition 2), but not of speaker gaze (condition 1) on the amount of gestural alignment, with an interaction between the conditions.

The role of artistry in transformative maintenance of law and custom is a theme widely researched and discussed in Aboriginal arts related literature. However, it is the aim of this paper to contribute to a wider discourse about learning and economic participation in remote Australia, and in particular the role of multimodality as a significant asset. The paper draws from relevant literature and two case studies; one from Keringke Arts, and one from Eastern Arrernte teacher and artist, Kathleen Kemarre Wallace. In customary form, multimodality combines and recombines various modalities — including dance, song, sand drawing, body painting and design, storytelling, stories, rhythm, petroglyph and ochre-painted rock art — enabling the intergenerational teaching and learning of rich cultural heritage in ways which connect that experience to the law and custom of the homelands. Multimodality, as it is used in this paper, draws on the concept of ‘form-relationality’; the way various modalities are combined and recombined, as elements which together describe a body of knowledge and yet separately provide myriad detail. Although beyond the scope of this paper, multimodality is also a mediating influence between contemporary and customary elements and contexts. This paper considers the complexity of multimodality as an asset in a contemporary arts market.

There are several investigational drugs in development for the treatment of depression. Some of the novel antidepressants in development target monoaminergic neurotransmission in accordance with the “monoamine hypothesis of depression.” However, the current conceptualization of antidepressant actions is that it is the downstream effects on protein synthesis and neuroplasticity that account for therapeutic efficacy, rather than the immediate effects on synaptic monoamine levels. Thus, a number of novel agents in development directly target components of this “neuroplasticity hypothesis of depression,” including hypothetically overactive glutamatergic neurotransmission and dysfunctional hypothalamic–pituitary–adrenal (HPA) axis functioning.

Creativity of designers can be enhanced by the application of appropriate triggers. The presence of triggers helps designers to search solution spaces. The searching of a solution space increases the possibility of finding creative solutions. Both representation and content of the triggers or stimulus to which the designers are exposed are believed to play a vital role in the representation and content of the outcome of the designers during problem solving. We studied the effect of representation of triggers on ideas generated by six design engineers while trying to solve a given problem. A variety of representations (video/animation and audio, text, explanation, and others) that are potentially useful to designers for five prespecified triggers were administered to each designer, who generated ideas in response to each trigger–representation combination individually. The effect of representations of these triggers on the content and representation of the solutions generated by the design engineers was studied. The results showed significant influence of the representation of the triggers on the representations, number, and quality of the resulting ideas that were generated.