A novel technique based on dynamic stochastic resonance (DSR) in discrete cosine transform (DCT) domain has been proposed in this paper for the enhancement of dark as well as low-contrast images. In conventional DSR-based techniques, the performance of a system can be improved by addition of external noise. However, in the proposed DSR-based work, the intrinsic noise of an image has been utilized to create a noise-induced transition of a dark image to a state of good contrast. The proposed technique significantly enhances the image contrast and color information without losing any image or color data by optimization of bistable system parameters. The performance of the proposed methodology has been measured in terms of relative contrast enhancement factor, perceptual quality measure, and color enhancement factor. When compared with the existing enhancement techniques, such as adaptive histogram equalization, gamma correction, single-scale retinex, multi-scale retinex, modified high-pass filtering, multicontrast enhancement with dynamic range compression, color enhancement by scaling, edge-preserving multi-scale decomposition, automatic control of imaging tool, and various spatial/frequency-domain SR-based techniques, the proposed technique gives remarkable performance in terms of contrast and color enhancement while ascertaining good perceptual quality.

For the transmission of aerial surveillance videos taken from unmanned aerial vehicles (UAVs), region of interest (ROI)-based coding systems are of growing interest in order to cope with the limited channel capacities available. We present a fully automatic detection and coding system which is capable of transmitting high-resolution aerial surveillance videos at very low bit rates. Our coding system is based on the transmission of ROI areas only. We assume two different kinds of ROIs: in order to limit the transmission bit rate while simultaneously retaining a high-quality view of the ground, we only transmit new emerging areas (ROI-NA) for each frame instead of the entire frame. At the decoder side, the surface of the earth is reconstructed from transmitted ROI-NA by means of global motion compensation (GMC). In order to retain the movement of moving objects not conforming with the motion of the ground (like moving cars and their previously occluded ground), we additionally consider regions containing such objects as interesting (ROI-MO). Finally, both ROIs are used as input to an externally controlled video encoder. While we use GMC for the reconstruction of the ground from ROI-NA, we use meshed-based motion compensation in order to generate the pelwise difference in the luminance channel (difference image) between the mesh-based motion compensated and the current input image to detect the ROI-MO. High spots of energy within this difference image are used as seeds to select corresponding superpixels from an independent (temporally consistent) superpixel segmentation of the input image in order to obtain accurate shape information of ROI-MO. For a false positive detection rate (regions falsely classified as containing local motion) of less than 2% we detect more than 97% true positives (correctly detected ROI-MOs) in challenging scenarios. Furthermore, we propose to use a modified high-efficiency video coding (HEVC) video encoder. Retaining full HDTV video resolution at 30 fps and subjectively high quality we achieve bit rates of about 0.6–0.9 Mbit/s, which is a bit rate saving of about 90% compared to an unmodified HEVC encoder.

This paper proposes a novel approach for lossless coding of light field (LF) images based on a macro-pixel (MP) synthesis technique which synthesizes the entire LF image in one step. The reference views used in the synthesis process are selected based on four different view configurations and define the reference LF image. This image is stored as an array of reference MPs which collect one pixel from each reference view, being losslessly encoded as a base layer. A first contribution focuses on a novel network design for view synthesis which synthesizes the entire LF image as an array of synthesized MPs. A second contribution proposes a network model for coding which computes the MP prediction used for lossless encoding of the remaining views as an enhancement layer. Synthesis results show an average distortion of 29.82 dB based on four reference views and up to 36.19 dB based on 25 reference views. Compression results show an average improvement of 29.9% over the traditional lossless image codecs and 9.1% over the state-of-the-art.

Real-time magnetic resonance imaging (rtMRI) of the moving vocal tract during running speech production is an important emerging tool for speech production research providing dynamic information of a speaker's upper airway from the entire midsagittal plane or any other scan plane of interest. There have been several advances in the development of speech rtMRI and corresponding analysis tools, and their application to domains such as phonetics and phonological theory, articulatory modeling, and speaker characterization. An important recent development has been the open release of a database that includes speech rtMRI data from five male and five female speakers of American English each producing 460 phonetically balanced sentences. The purpose of the present paper is to give an overview and outlook of the advances in rtMRI as a tool for speech research and technology development.

This paper presents a moving object detection algorithm for H.264/AVC video streams that is applied in the compressed domain. The method is able to extract and analyze several syntax elements from any H.264/AVC-compliant bit stream. The number of analyzed syntax elements depends on the mode in which the method operates. The algorithm is able to perform either a spatiotemporal analysis in a single step or a two-step analysis that starts with a spatial analysis of each frame, followed by a temporal analysis of several subsequent frames. Thereby, in each mode either only (sub-)macroblock types and partition modes or, additionally, quantization parameters are analyzed. The evaluation of these syntax elements enables the algorithm to determine a “weight” for each 4×4 block of pixels that indicates the level of motion within this block. A final segmentation after creating these weights segments each frame to foreground and background and hence indicates the positions and sizes of all moving objects. Our experiments show that the algorithm is able to efficiently detect moving objects in the compressed domain and that it is configurable to process a large number of parallel bit streams in real time.

Biometric recognition technologies have become more important in the modern society due to their convenience with the recent informatization and the dissemination of network services. Among such technologies, face recognition is one of the most convenient and practical because it enables authentication from a distance without requiring any authentication operations manually. As far as we know, face recognition is susceptible to the changes in the appearance of faces due to aging, the surrounding lighting, and posture. There were a number of technical challenges that need to be resolved. Recently, remarkable progress has been made thanks to the advent of deep learning methods. In this position paper, we provide an overview of face recognition technology and introduce its related applications, including face presentation attack detection, gaze estimation, person re-identification and image data mining. We also discuss the research challenges that still need to be addressed and resolved.

We propose a new paradigm for no-reference image quality assessment (IQA) exploiting neurological and psychophysical properties of the human visual system (HVS)1. Physiological and psychological evidences exist that HVS has different behavioral patterns under low and high noise/artifact levels. In this paper, we propose a dual-model approach for blind IQA under near-threshold and suprathreshold noise conditions. The underlying assumption for the proposed dual-model approach is that for images with low-level near-threshold noise, HVS tries to gauge the strength of the noise, so image quality can be well approximated via measuring strength of the noise. On the other hand, for images with structures overwhelmed by high-level suprathreshold noise, perceptual quality assessment relies on a cognitive model and the HVS tries to recover meaningful contents from the noisy pixels using past experiences and prior knowledge encoded into an internal generative model of the brain. And image quality is therefore closely related to the agreement between the noisy observation and the internal generative model explainable part of the image. Specifically, under near-threshold noise condition, a noise level estimation algorithm using natural image statistics is used, while under suprathreshold condition, an active inference model based on the free-energy principle is adopted. The near- and suprathreshold models can be seamlessly integrated through a mathematical transformation between estimates from both models. The proposed dual-model algorithm has been tested on additive Gaussian noise contaminated images. Experimental results and comparative studies suggest that although being a no-reference approach, the proposed algorithm has prediction accuracy comparable with some of the best full-reference IQA methods. The dual-model perspective of IQA presented in this paper is expected to facilitate future research in the field of visual perceptual modeling.

Low-cost, compact, and accurate systems for in-home rehabilitation are needed in aging, aged, and hyper-aged groups. In this study, we developed an in-home rehabilitation system for patients with balance disorders by providing visual feedback of postural information in real-time. Our system measures the user's whole body motion and the center of pressure (COP) using a Kinect and Wii Balance Board (WBB). The accuracy of body motion for estimating the anterior folding and lateral bending angles was validated experimentally by comparing the estimates with the angles given by an optical motion capture system. Additional experiments showed that the COP has a small correlation coefficient with the angles, suggesting that WBB is necessary for measuring the COP.

A tone mapping operation (TMO) for HDR images with fixed-point arithmetic is proposed. A TMO generates a low dynamic range (LDR) image from a high dynamic range (HDR) image by compressing its dynamic range. Since HDR images are generally expressed in a floating-point data format, a TMO also deals with floating-point data even though resulting LDR images have integer data. As a result, conventional TMOs require many resources such as computational and memory cost. To reduce the resources, an integer TMO which treats a floating-point number as two 8-bit integer numbers was proposed. However, this method has the limitation of available input HDR image formats. The proposed method introduces an intermediate format to relieve the limitation of input formats, and expands the integer TMO for the intermediate format. The proposed integer TMO can be applied for multiple formats such as the RGBE and the OpenEXR. Moreover, the method can conduct all calculations in the TMO with fixed-point arithmetic. Using both integer data and fixed-point arithmetic, the method reduces not only the memory cost, but also the computational cost. The experimental and evaluation results show that the proposed method reduces the computational and memory cost, and gives almost same quality of LDR images, compared with the conventional method with floating-point arithmetic.

We propose the notion of latent acoustic topics to capture contextual information embedded within a collection of audio signals. The central idea is to learn a probability distribution over a set of latent topics of a given audio clip in an unsupervised manner, assuming that there exist latent acoustic topics and each audio clip can be described in terms of those latent acoustic topics. In this regard, we use the latent Dirichlet allocation (LDA) to implement the acoustic topic models over elemental acoustic units, referred as acoustic words, and perform text-like audio signal processing. Experiments on audio tag classification with the BBC sound effects library demonstrate the usefulness of the proposed latent audio context modeling schemes. In particular, the proposed method is shown to be superior to other latent structure analysis methods, such as latent semantic analysis and probabilistic latent semantic analysis. We also demonstrate that topic models can be used as complementary features to content-based features and offer about 9% relative improvement in audio classification when combined with the traditional Gaussian mixture model (GMM)–Support Vector Machine (SVM) technique.

This paper describes an automatic singing transcription (AST) method that estimates a human-readable musical score of a sung melody from an input music signal. Because of the considerable pitch and temporal variation of a singing voice, a naive cascading approach that estimates an F0 contour and quantizes it with estimated tatum times cannot avoid many pitch and rhythm errors. To solve this problem, we formulate a unified generative model of a music signal that consists of a semi-Markov language model representing the generative process of latent musical notes conditioned on musical keys and an acoustic model based on a convolutional recurrent neural network (CRNN) representing the generative process of an observed music signal from the notes. The resulting CRNN-HSMM hybrid model enables us to estimate the most-likely musical notes from a music signal with the Viterbi algorithm, while leveraging both the grammatical knowledge about musical notes and the expressive power of the CRNN. The experimental results showed that the proposed method outperformed the conventional state-of-the-art method and the integration of the musical language model with the acoustic model has a positive effect on the AST performance.

In conventional studies, cryptographic techniques are used to ensure the security of transaction between a seller and buyer in a fingerprinting system. However, the tracing protocol from a pirated copy has not been studied from the security point of view though the collusion resistance is considered by employing a collusion secure fingerprinting code. In this paper, we consider the secrecy of parameters for a fingerprinting code and burdens at a trusted center, and propose a secure tracing protocol jointly executed by a seller and a delegated server. Our main idea is to delegate authority to a server so that the center is required to operate only at the initialization phase in the system. When a pirated copy is found, a seller calculates a correlation score for each user's codeword in an encrypted domain, and identifies illegal users by sending the ciphertexts of scores as queries to the server. The information leakage from the server can be managed at the restriction of response from the server to check the maliciousness of the queries.

Concerns with global warming prompted many governments to mandate increased proportion of electricity generation from renewable sources. This, together with the desire to have more efficient and secure power generation and distribution, has driven research in the next-generation power grid, namely, the smart grid. Through integrating advanced information and communication technologies with power electronic and electric power technologies, smart grid will be highly reliable, efficient, and environmental-friendly. A key component of smart grid is the communication system. This paper explores the design goals and functions of the smart grid communication system, followed by an in-depth investigation on the communication requirements. Discussions on some of the recent developments related to smart grid communication systems are also introduced.

We present a statistical-modeling method for piano reduction, i.e. converting an ensemble score into piano scores, that can control performance difficulty. While previous studies have focused on describing the condition for playable piano scores, it depends on player's skill and can change continuously with the tempo. We thus computationally quantify performance difficulty as well as musical fidelity to the original score, and formulate the problem as optimization of musical fidelity under constraints on difficulty values. First, performance difficulty measures are developed by means of probabilistic generative models for piano scores and the relation to the rate of performance errors is studied. Second, to describe musical fidelity, we construct a probabilistic model integrating a prior piano-score model and a model representing how ensemble scores are likely to be edited. An iterative optimization algorithm for piano reduction is developed based on statistical inference of the model. We confirm the effect of the iterative procedure; we find that subjective difficulty and musical fidelity monotonically increase with controlled difficulty values; and we show that incorporating sequential dependence of pitches and fingering motion in the piano-score model improves the quality of reduction scores in high-difficulty cases.

Within the past decade, analyzing and modeling human behavior by processing large amounts of collected data has become an active research field in the area of human–machine interaction. The research community is striving to find principled ways to explain and represent important behavioral characteristics of humans, with the goal of developing more efficient and more effective cooperative interactions between humans, machines, and environment. This paper provides a summary of the progress we have achieved to date in our study, which has focused specifically on interactions between driver, vehicle, and driving environment. First, we describe the method of data collection used to develop our on-the-road driving data corpus. We then provide an overview of the data-driven, signal processing approaches we used to analyze and model driver behavior for a wide range of practical vehicle applications. Next, we perform experimental validation by observing the actual driving behavior of groups of real drivers. In particular, the vehicle applications of our research include driver identification, behavior prediction related to car following and lane changing, detection of emotional frustration, and improving driving safety through driver coaching. We hope this paper will provide some insight to researchers with an interest in this field, and help identify areas and applications where further research is needed.

Deep neural networks (DNNs) have the same structure as the neocognitron proposed in 1979 but have much better performance, which is because DNNs include many heuristic techniques such as pre-training, dropout, skip connections, batch normalization (BN), and stochastic depth. However, the reason why these techniques improve the performance is not fully understood. Recently, two tools for theoretical analyses have been proposed. One is to evaluate the generalization gap, defined as the difference between the expected loss and empirical loss, by calculating the algorithmic stability, and the other is to evaluate the convergence rate by calculating the eigenvalues of the Fisher information matrix of DNNs. This overview paper briefly introduces the tools and shows their usefulness by showing why the skip connections and BN improve the performance.

In this paper, we propose a novel method for protecting convolutional neural network models with a secret key set so that unauthorized users without the correct key set cannot access trained models. The method enables us to protect not only from copyright infringement but also the functionality of a model from unauthorized access without any noticeable overhead. We introduce three block-wise transformations with a secret key set to generate learnable transformed images: pixel shuffling, negative/positive transformation, and format-preserving Feistel-based encryption. Protected models are trained by using transformed images. The results of experiments with the CIFAR and ImageNet datasets show that the performance of a protected model was close to that of non-protected models when the key set was correct, while the accuracy severely dropped when an incorrect key set was given. The protected model was also demonstrated to be robust against various attacks. Compared with the state-of-the-art model protection with passports, the proposed method does not have any additional layers in the network, and therefore, there is no overhead during training and inference processes.

In this paper, we propose a novel hue-correction scheme for color-image-enhancement algorithms including deep-learning-based ones. Although hue-correction schemes for color-image enhancement have already been proposed, there are no schemes that can both perfectly remove perceptual hue-distortion on the basis of CIEDE2000 and be applicable to any image-enhancement algorithms. In contrast, the proposed scheme can perfectly remove hue distortion caused by any image-enhancement algorithm such as deep-learning-based ones on the basis of CIEDE2000. Furthermore, the use of a gamut-mapping method in the proposed scheme enables us to compress a color gamut into an output RGB color gamut, without hue changes. Experimental results show that the proposed scheme can completely correct hue distortion caused by image-enhancement algorithms while maintaining the performance of the algorithms and ensuring the color gamut of output images.

Automatic speech recognition (ASR) systems often make unrecoverable errors due to subsystem pruning (acoustic, language and pronunciation models); for example, pruning words due to acoustics using short-term context, prior to rescoring with long-term context based on linguistics. In this work, we model ASR as a phrase-based noisy transformation channel and propose an error correction system that can learn from the aggregate errors of all the independent modules constituting the ASR and attempt to invert those. The proposed system can exploit long-term context using a neural network language model and can better choose between existing ASR output possibilities as well as re-introduce previously pruned or unseen (Out-Of-Vocabulary) phrases. It provides corrections under poorly performing ASR conditions without degrading any accurate transcriptions; such corrections are greater on top of out-of-domain and mismatched data ASR. Our system consistently provides improvements over the baseline ASR, even when baseline is further optimized through Recurrent Neural Network (RNN) language model rescoring. This demonstrates that any ASR improvements can be exploited independently and that our proposed system can potentially still provide benefits on highly optimized ASR. Finally, we present an extensive analysis of the type of errors corrected by our system.

The standardization process for Versatile Video Coding (VVC), the next generation video coding standard, was launched in 2018, after several recent advances in video coding technologies had been investigated under the Joint Video Experts Team (JVET) of ITU-T VCEG and ISO/IEC MPEG experts. The recent standard development status (up to VVC working draft 2) shows that the VTM software, the test model for this VVC standard, can achieve over 23% average coding gain under random access configuration when compared to the HM software, the test model of HEVC standard. This paper gives a review of recently developed video coding technologies that have been either adopted into the VVC working draft as part of the standard or under further evaluation for potential inclusions.