We highlight the need for – and describe initial strategies to find – new digital-optical basis functions and performance merit functions to serve as a foundation for designing, analyzing, characterizing, testing, and comparing a range of computational imaging systems. Such functions will provide a firm theoretical foundation for computational sensing and imaging and enhanced design software, thereby broadly speeding the development of computational imaging systems.

Multiview video plus depth is emerging as the most flexible format for three-dimensional video representation, as witnessed by the current standardization efforts by ISO and ITU. In particular, in depth representation, arguably the most important information lies in object contours. As a consequence, an interesting approach consists in performing a lossless coding of object contours, possibly followed by a lossy coding of per-object depth values. In this context, we propose a new technique for lossless coding of object contours, based on the elastic deformation of curves. Using the square-root velocity representation for the elements of the space of curves, we can model a continuous evolution of elastic deformations between two reference contour curves. An elastically deformed version of the reference contours can be sent to the decoder with a reduced coding cost and used as side information to improve the lossless coding of the actual contour. Experimental results on several multiview video sequences show remarkable gains with respect to the reference techniques and to the state of the art.

Distributed video coding (DVC) is a coding paradigm that entails low complexity encoding by exploiting the source statistics at the decoder. To improve the DVC coding efficiency, this paper presents a novel adaptive technique for mode decision to control and take advantage of skip mode and intra mode in DVC initially proposed by Luong et al. in 2013. The adaptive mode decision (AMD) is not only based on quality of key frames but also the rate of Wyner–Ziv (WZ) frames. To improve noise distribution estimation for a more accurate mode decision, a residual motion compensation is proposed to estimate a current noise residue based on a previously decoded frame. The experimental results, integrating AMD in two efficient DVC codecs, show that the proposed AMD DVC significantly improves the rate distortion performance without increasing the encoding complexity. For a GOP size of 2 on the set of six test sequences, the average (Bjøntegaard) bitrate saving of the proposed codec is 35.5% on WZ frames compared with the DISCOVER codec. This saving is mainly achieved by AMD.

Deep learning has demonstrated its superiority in computer vision. Landsat images have specific characteristics compared with natural images. The spectral and texture features of the same class vary along with the imaging conditions. In this paper, we extend the use of deep learning to remote sensing image classification to large geographical regions, and explore a way to make deep learning classifiers transferable for different regions. We take Jingjinji region and Henan province in China as the study areas, and choose FCN, ResNet, and PSPNet as classifiers. The models are trained by different proportions of training samples from Jingjinji region. Then we use the trained models to predict results of the study areas. Experimental results show that the overall accuracy decreases when trained by small samples, but the recognition ability on mislabeled areas increases. All methods can obtain great performance when used to Jingjinji region while they all need to be fine-tuned with new training samples from Henan province, due to the reason that images of Henan province have different spectral features from the original trained area.

High dynamic range (HDR) technology is rapidly changing today's video landscape by offering spectacular visual experiences. The development in display technology to support higher luminance levels for commercial and consumer electronic devices such as TVs, smartphones, projectors etc., has created an exponential demand for delivering HDR content to viewers. The essential component of the HDR technology is “expanded contrast,” which allows richer black levels and enhanced brightness, providing dramatic contrast that reveals finer details. The use of “wide color gamut” allows wider color spectrum and richer colors providing aesthetically pleasing true-to-life feel. Such visual enhancements clearly establish HDR as one of the most significant upcoming video technologies.

In this paper, we review major technical advances in this exciting field of study. Quantization of HDR signals is reviewed in the context of transfer functions that convert optical signals to electrical signals and vice versa. They mainly consist of Perceptual Quantization and Hybrid-Log-Gamma approaches. Compression of HDR content is another broad area of study involving several coding approaches, often categorized in terms of backward-compatibility and single/dual layer methods. Some key industry applications of HDR processing systems are also discussed, followed by some future directions of HDR technology.

We recently proposed a bidirectional hierarchical anchoring (BIHA) of motion fields for highly scalable video coding. The BIHA scheme employs piecewise-smooth motion fields, and uses breakpoints to signal motion discontinuities. In this paper, we show how the fundamental building block of the BIHA scheme can be used to perform bidirectional, occlusion-aware temporal frame interpolation (BOA-TFI). From a “parent” motion field between two reference frames, we use information about motion discontinuities to compose motion fields from both reference frames to the target frame; these then get inverted so that they can be used to predict the target frame. During the motion inversion process, we compute a reliable occlusion mask, which is used to guide the bidirectional motion-compensated prediction of the target frame. The scheme can be used in any state-of-the-art codec, but is most beneficial if used in conjunction with a highly scalable video coder which employs piecewise-smooth motion fields with motion discontinuities. We evaluate the proposed BOA-TFI scheme on a large variety of natural and challenging computer-generated sequences, and our results compare favorably to state-of-the-art TFI methods.

In computational auditory scene analysis, the accurate estimation of binary mask or ratio mask plays a key role in noise masking. An inaccurate estimation often leads to some artifacts and temporal discontinuity in the synthesized speech. To overcome this problem, we propose a new ratio mask estimation method in terms of Wiener filtering in each Gammatone channel. In the reconstruction of Wiener filter, we utilize the relationship of the speech and noise power spectra in each Gammatone channel to build the objective function for the convex optimization of speech power. To improve the accuracy of estimation, the estimated ratio mask is further modified based on its adjacent time–frequency units, and then smoothed by interpolating with the estimated binary masks. The objective tests including the signal-to-noise ratio improvement, spectral distortion and intelligibility, and subjective listening test demonstrate the superiority of the proposed method compared with the reference methods.

This discussion of responsive media provides a perspective on the future of media experiences that are increasingly responsive to users' preference, alertness and their physical, digital, and social environment. By examining a range of future scenarios combining virtual-, remote-, and augmented-reality, autonomous vehicles, digital assistants and robots, we see that the responsiveness of media is what provides the key value. To reach the ultimate goal of augmented innovation in which thinking machines supplement humans, there are a number of technological and user-experience challenges that the research community needs to resolve. These challenges fall into a few key categories: throughput, latency, perception, intelligence, and interaction. While some challenges may be tackled purely technologically, others require insights from sociology and psychology to break new ground. The paper concludes that intelligent, responsive media will not fully supplant human intelligence, but will increasingly serve as augmentation to human creativity.

In this paper, we study the semantic segmentation of 3D LiDAR point cloud data in urban environments for autonomous driving, and a method utilizing the surface information of the ground plane was proposed. In practice, the resolution of a LiDAR sensor installed in a self-driving vehicle is relatively low and thus the acquired point cloud is indeed quite sparse. While recent work on dense point cloud segmentation has achieved promising results, the performance is relatively low when directly applied to sparse point clouds. This paper is focusing on semantic segmentation of the sparse point clouds obtained from 32-channel LiDAR sensor with deep neural networks. The main contribution is the integration of the ground information which is used to group ground points far away from each other. Qualitative and quantitative experiments on two large-scale point cloud datasets show that the proposed method outperforms the current state-of-the-art.

The world of intelligent and interactive systems is undergoing an era of unprecedented innovation and advanced development. With the rapid progress in natural sensing and perceptual computing technologies, devices and machines are increasingly being endowed with the abilities to sense and understand the world, navigate in the environment, and interact with humans in natural ways. Interfaces based on touch sensing and speech recognition are now ubiquitous, and the race is on to the next frontiers of machine intelligence and interactions based on three-dimensional (3D) sensing. In this paper, we review the recent progress in the development of real-time 3D-sensing technologies and their deployment in a new class of interactive and autonomous systems. As an example of a commercially available platform, we describe the Intel® RealSense technologies, as well as its deployments in a new class of interactive and autonomous systems and applications.

In this paper we combine video compression and modern image processing methods. We construct novel iterative filter methods for prediction signals based on Partial Differential Equation (PDE)-based methods. The central idea of the signal adaptive filters is explained and demonstrated geometrically. The meaning of particular parameters is discussed in detail. Furthermore, thorough parameter tests are introduced which improve the overall bitrate savings. It is shown that these filters enhance the rate-distortion performance of the state-of-the-art hybrid video codecs. In particular, based on mathematical denoising techniques, two types of diffusion filters are constructed: a uniform diffusion filter using a fixed filter mask and a signal adaptive diffusion filter that incorporates the structures of the underlying prediction signal. The latter has the advantage of not attenuating existing edges while the uniform filter is less complex. The filters are embedded into a software based on HEVC with additional QTBT (Quadtree plus Binary Tree) and MTT (Multi-Type-Tree) block structure. Overall, the diffusion filter method achieves average bitrate savings of 2.27% for Random Access having an average encoder runtime increase of 19% and 17% decoder runtime increase. For UHD (Ultra High Definition) test sequences, bitrate savings of up to 7.36% for Random Access are accomplished.

Artificial Intelligence (AI) was the inspiration that shaped computing as we know it today. In this article, I explore why and how AI would continue to inspire computing and reinvent it when Moore's Law is running out of steam. At the dawn of computing, Alan Turing proposed that instead of comprising many different specific machines, the computing machinery for AI should be a Universal Digital Computer, modeled after human computers, which carry out calculations with pencil on paper. Based on the belief that a digital computer would be significantly faster, more diligent and patient than a human, he anticipated that AI would be advanced as software. In modern terminology, a universal computer would be designed to understand a language known as an Instruction Set Architecture (ISA), and software would be translated into the ISA. Since then, universal computers have become exponentially faster and more energy efficient through Moore's Law, while software has grown more sophisticated. Even though software has not yet made a machine think, it has been changing how we live fundamentally. The computing revolution started when the software was decoupled from the computing machinery. Since the slowdown of Moore's Law in 2005, the universal computer is no longer improving exponentially in terms of speed and energy efficiency. It has to carry ISA legacy, and cannot be aggressively modified to save energy. Turing's proposition of AI as software is challenged, and the temptation of making many domain-specific AI machines emerges. Thanks to Deep Learning, software can stay decoupled from the computing machinery in the language of linear algebra, which it has in common with supercomputing. A new universal computer for AI understands such language natively to then become a Native Supercomputer. AI has been and will still be the inspiration for computing. The quest to make machines think continues amid the slowdown of Moore's Law. AI might not only maximize the remaining benefits of Moore's Law, but also revive Moore's Law beyond current technology.

The bandwidth limitation of wideband (WB) audio systems degrades the subjective quality and naturalness of audio signals. In this paper, a new method for blind bandwidth extension of WB audio signals is proposed based on non-linear prediction and hidden Markov model (HMM). The high-frequency (HF) components in the band of 7–14 kHz are artificially restored only from the low-frequency information of the WB audio. State-space reconstruction is used to convert the fine spectrum of WB audio to a multi-dimensional space, and a non-linear prediction based on nearest-neighbor mapping is employed in the state space to restore the fine spectrum of the HF components. The spectral envelope of the resulting HF components is estimated based on an HMM according to the features extracted from the WB audio. In addition, the proposed method and the reference methods are applied to the ITU-T G.722.1 WB audio codec for comparison with the ITU-T G.722.1C super WB audio codec. Objective quality evaluation results indicate that the proposed method is preferred over the reference bandwidth extension methods. Subjective listening results show that the proposed method has a comparable audio quality with G.722.1C and improves the extension performance compared with the reference methods.

The demands for Smart City are growing in the context of the rapid urbanization in the world. In the public safety domain, law enforcement agencies are facing new challenges to protect people and properties in the cities, and they are shifting their focus from the traditional crime investigation towards preventive actions that prevent accidents and incidents before it occurs, and inter-agency collaboration that brings the situation under control swiftly when it occurs. In this paper, we discuss new challenges that cities are facing, and describe NEC's solution based on intelligent multimedia surveillance architecture for Safe Cities to tackle these challenges.

State-of-the-art lossless image compression schemes, such as JPEG-LS and CALIC, have been proposed in the context-adaptive predictive coding framework. These schemes involve a prediction step followed by context-adaptive entropy coding of the residuals. However, the models for context determination proposed in the literature, have been designed using ad-hoc techniques. In this paper, we take an alternative approach where we fix a simpler context model and then rely on a systematic technique to efficiently exploit spatial correlation to achieve efficient compression. The essential idea is to decompose the image into binary bitmaps such that the spatial correlation that exists among non-binary symbols is captured as the correlation among few bit positions. The proposed scheme then encodes the bitmaps in a particular order based on the simple context model. However, instead of encoding a bitmap as a whole, we partition it into rectangular blocks, induced by a binary tree, and then independently encode the blocks. The motivation for partitioning is to explicitly identify the blocks within which the statistical correlation remains the same. On a set of standard test images, the proposed scheme, using the same predictor as JPEG-LS, achieved an overall bit-rate saving of 1.56% against JPEG-LS.

This work proposes a query-by-singing (QBS) content-based music retrieval (CBMR) system that uses Approximate Karbunen–Loeve transform for noise reduction. The proposed QBS-CBMR system uses a music clip as a search key. First, a 51-dimensional matrix containing 39-Mel-frequency cepstral coefficients (MFCCs) features and 12-Chroma features are extracted from an input music clip. Next, adapted symbolic aggregate approximation (adapted SAX) is used to transform each dimension of features into a symbolic sequence. Each symbolic sequence corresponding to each dimension of MFCCs is then converted into a structure called advanced fast pattern index (AFPI) tree. The similarity between the query music clip and the songs in the database is evaluated by calculating a partial score for each AFPI tree. The final score is obtained by calculating the weighted sum of all partial scores, where the weighting of each partial score is determined by its entropy. Experimental results show that the proposed music retrieval system performs robustly and accurately with the entropy weighting mechanism.

While deceptive behaviors are a natural part of human life, it is well known that human is generally bad at detecting deception. In this study, we present an automatic deception detection framework by comprehensively integrating prior domain knowledge in deceptive behavior understanding. Specifically, we compute acoustics, textual information, implicatures with non-verbal behaviors, and conversational temporal dynamics for improving automatic deception detection in dialogs. The proposed model reaches start-of-the-art performance on the Daily Deceptive Dialogues corpus of Mandarin (DDDM) database, 80.61% unweighted accuracy recall in deception recognition. In the further analyses, we reveal that (i) the deceivers’ deception behaviors can be observed from the interrogators’ behaviors in the conversational temporal dynamics features and (ii) some of the acoustic features (e.g. loudness and MFCC) and textual features are significant and effective indicators to detect deception behaviors.

In recent months, COVID-19 has become a global pandemic and had a huge impact on the world. People under different conditions have very different attitudes toward the epidemic. Due to the real-time and large-scale nature of social media, we can continuously obtain a massive amount of public opinion information related to the epidemic from social media. In particular, researchers may ask questions such as “how is the public reacting to COVID-19 in China during different stages of the pandemic?”, “what factors affect the public opinion orientation in China?”, and so on. To answer such questions, we analyze the pandemic-related public opinion information on Weibo, China's largest social media platform. Specifically, we have first collected a large amount of COVID-19-related public opinion microblogs. We then use a sentiment classifier to recognize and analyze different groups of users’ opinions. In the collected sentiment-orientated microblogs, we try to track the public opinion through different stages of the COVID-19 pandemic. Furthermore, we analyze more key factors that might have an impact on the public opinion of COVID-19 (e.g. users in different provinces or users with different education levels). Empirical results show that the public opinions vary along with the key factors of COVID-19. Furthermore, we analyze the public attitudes on different public-concerning topics, such as staying at home and quarantine. In summary, we uncover interesting patterns of users and events as an insight into the world through the lens of a major crisis.

This paper proposes a boundary operation technique of two-dimensional (2D) non-separable oversampled lapped transforms (NSOLT). The proposed technique is based on a lattice structure consisting of the 2D separable block discrete cosine transform and non-separable redundant support-extension processes. The atoms are allowed to be anisotropic with the oversampled, symmetric, real-valued, compact-supported, and overlapped property. First, the blockwise implementation is developed so that the atoms can be locally controlled. The local control of atoms is shown to maintain perfect reconstruction. This property leads an atom termination (AT) technique as a boundary operation. The technique overcomes the drawback of NSOLT that the popular symmetric extension method is invalid. Through some experimental results with iterative hard thresholding, the significance of AT is verified.

An infinite mixture model is applied to model-based speaker clustering with sampling-based optimization to make it possible to estimate the number of speakers. For this purpose, a framework of non-parametric Bayesian modeling is implemented with the Markov chain Monte Carlo and incorporated in the utterance-oriented speaker model. The proposed model is called the utterance-oriented Dirichlet process mixture model (UO-DPMM). The present paper demonstrates that UO-DPMM is successfully applied on large-scale data and outperforms the conventional hierarchical agglomerative clustering, especially for large amounts of utterances.