Speech recognition is a sequence prediction problem. Besides employing various deep learning approaches for frame-level classification, sequence-level discriminative training has been proved to be indispensable to achieve the state-of-the-art performance in large vocabulary continuous speech recognition (LVCSR). However, keyword spotting (KWS), as one of the most common speech recognition tasks, almost only benefits from frame-level deep learning due to the difficulty of getting competing sequence hypotheses. The few studies on sequence discriminative training for KWS are limited for fixed vocabulary or LVCSR based methods and have not been compared to the state-of-the-art deep learning based KWS approaches. In this paper, a sequence discriminative training framework is proposed for both fixed vocabulary and unrestricted acoustic KWS. Sequence discriminative training for both sequence-level generative and discriminative models are systematically investigated. By introducing word-independent phone lattices or non-keyword blank symbols to construct competing hypotheses, feasible and efficient sequence discriminative training approaches are proposed for acoustic KWS. Experiments showed that the proposed approaches obtained consistent and significant improvement in both fixed vocabulary and unrestricted KWS tasks, compared to previous frame-level deep learning based acoustic KWS methods.

Recent advances in automatic speaker verification ASV lead to an increased interest in securing these systems for real-world applications. Malicious spoofing attempts against ASV systems can lead to serious security breaches. A spoofing attack within the context of ASV is a condition in which a potentially harmful person successfully masks as another, to the ASV system already known person by falsifying or manipulating data. While most previous work focuses on enhanced, spoof-aware features, end-to-end models can be a potential alternative. In this paper, we investigate the training of a raw wave front-ends for deep convolutional, long short-term memory LSTM and vanilla neural networks, which are analyzed for their suitability toward spoofing detection, regarding the influence of frame size, number of output neurons, and sequence length. A joint convolutional LSTM neural network CLDNN is proposed, which outperforms previous attempts on the BTAS2016 dataset 0.82% $\rightarrow$ 0.19% HTER, placing itself as the current state-of-the-art model for the dataset. We show that end-to-end approaches are appropriate for the important replay detection task and show that the proposed model is capable of distinguishing device-invariant spoofing attempts. Regarding the ASVspoof2015 dataset, the end-to-end solution achieves an equal error rate EER of 0.00% for the S1-S9 conditions. We show that the end-to-end approach based on a raw waveform input can outperform common cepstral features, without the use of context-dependent frame extensions. In addition, a cross-database domain mismatch scenario is also evaluated, which shows that the proposed CLDNN model trained on the BTAS2016 dataset achieves an EER of 25.7% on the ASVspoof2015 dataset.

Although great progress has been made in automatic speech recognition, significant performance degradation still exists in noisy environments. Our previous work has demonstrated the superior noise robustness of very deep convolutional neural networks VDCNN. Based on our work on VDCNNs, this paper proposes a more advanced model referred to as the very deep convolutional residual network VDCRN. This new model incorporates batch normalization and residual learning, showing more robustness than previous VDCNNs.Then, to alleviate the mismatch between the training and testing conditions, model adaptation and adaptive training are developed and compared for the new VDCRN. This paper focuses on factor aware training FAT and cluster adaptive training CAT. For FAT, a unified framework is explored. For CAT, two schemes are first explored to construct the bases in the canonical model; furthermore, a factorized version of CAT is designed to address multiple nonspeech variabilities in one model. Finally, a complete multipass system is proposed to achieve the best system performance in the noisy scenarios. The proposed new approaches are evaluated on three different tasks: Aurora4 simulated data with additive noise and channel distortion, CHiME4 both simulated and real data with additive noise and reverberation, and the AMI meeting transcription task real data with significant reverberation.The evaluation not only includes different noisy conditions, but also covers both simulated and real noisy data. The experiments show that the new VDCRN is more robust, and the adaptation on this model can further significantly reduce the word error rate WER. The proposed best architecture obtains consistent and very large improvements on all tasks compared to the baseline VDCNN or long short-term memory. Particularly, on Aurora4 a new milestone 5.67% WER is achieved by only improving acoustic modeling.

With the recent advances of the sequence-to-sequence framework, generation approaches for the short text conversation (STC) become attractive. The traditional sequence-to-sequence approaches for the STC often suffer from poor diversity and general reply without substantiality. It is also hard to control the topic or semantics of the selected reply from multiple generated candidates. In this paper, a novel external-memory-driven sequence-to-sequence learning approach is proposed to address these problems. A tensor of the external memory is constructed to represent interpretable topics or semantics. During generation, a controllable memory trigger is extracted given the input sequence, and a reply is then generated using the memory trigger as well as the sequence-to-sequence model. Experiments show that the proposed approach can generate much richer diversity than the traditional sequence-to-sequence training with attention. Meanwhile, it achieves better quality score in human evaluation. It is also observed that by manually manipulating the memory trigger, it is possible to interpretably guide the topics or semantics of the reply.

Connectionist temporal classification CTC has recently shown improved performance and efficiency in automatic speech recognition. One popular decoding implementation is to use a CTC model to predict the phone posteriors at each frame and then perform Viterbi beam search on a modified WFST network. This is still within the traditional frame synchronous decoding framework. In this paper, the peaky posterior property of CTC is carefully investigated and it is found that ignoring blank frames will not introduce additional search errors. Based on this phenomenon, a novel phone synchronous decoding framework is proposed by removing tremendous search redundancy due to blank frames, which results in significant search speed up. The framework naturally leads to an extremely compact phone-level acoustic space representation: CTC lattice. With CTC lattice, efficient and effective modular speech recognition approaches, second pass rescoring for large vocabulary continuous speech recognition LVCSR, and phone-based keyword spotting KWS, are also proposed in this paper. Experiments showed that phone synchronous decoding can achieve 3-4 times search speed up without performance degradation compared to frame synchronous decoding. Modular LVCSR with CTC lattice can achieve further WER improvement. KWS with CTC lattice not only achieved significant equal error rate improvement, but also greatly reduced the KWS model size and increased the search speed.

Recently biometric authentication has made progress in areas, such as speaker verification. However, some evidence shows that the technology is susceptible to malicious spoofing attacks, and thus dedicated countermeasures are needed to detect a variety of specific attack types. Inspired by the great success of deep learning in automatic speech recognition, we propose a detailed deep learning based feature engineering framework for spoofing detection in this paper. To incorporate deep learning into spoofing detection, this work proposes novel approaches for extracting and using features from deep learning models. In contrast to the traditional short-term spectral features, such as MFCC or PLP, outputs from the hidden layer of various deep models are employed as deep features for spoofing detection. Two frameworks are developed to extract deep features, including DNN-based frame-level feature extraction and RNN-based sequence-level feature extraction, and several structures are explored within each framework. Once the deep features are extracted, they can be used as a spoofing identity representation for each utterance, and the appropriate back-end classifier is then applied to make the final detection decision. These approaches were evaluated on the ASVspoof2015 Challenge data corpus. Experiments show that deep feature based systems achieve good performance, even without using any designed features such as phase and cochlea features common in spoofing detection, and obtain significant performance improvements compared to the traditional baselines. The EER of the best deep feature system achieves nearly 0.0% for all attack types from S1 to S9, and gets 1.1% on all averaged conditions (plus S10), which is very promising performance in ASVspoof2015 Challenge task.

Although great progress has been made in automatic speech recognition, significant performance degradation still exists in noisy environments. Recently, very deep convolutional neural networks (CNNs) have been successfully applied to computer vision and speech recognition tasks. Based on our previous work on very deep CNNs, in this paper this architecture is further developed to improve recognition accuracy for noise robust speech recognition. In the proposed very deep CNN architecture, we study the best configuration for the sizes of filters, pooling, and input feature maps: the sizes of filters and poolings are reduced and dimensions of input features are extended to allow for adding more convolutional layers. Then the appropriate pooling, padding, and input feature map selection strategies are investigated and applied to the very deep CNN to make it more robust for speech recognition. In addition, an in-depth analysis of the architecture reveals key characteristics, such as compact model scale, fast convergence speed, and noise robustness. The proposed new model is evaluated on two tasks: Aurora4 task with multiple additive noise types and channel mismatch, and the AMI meeting transcription task with significant reverberation. Experiments on both tasks show that the proposed very deep CNNs can significantly reduce word error rate (WER) for noise robust speech recognition. The best architecture obtains a 10.0% relative reduction over the traditional CNN on AMI, competitive with the long short-term memory recurrent neural networks (LSTM-RNN) acoustic model. On Aurora4, even without feature enhancement, model adaptation, and sequence training, it achieves a WER of 8.81%, a 17.0% relative improvement over the LSTM-RNN. To our knowledge, this is the best published result on Aurora4.

Dialogue state tracking (DST) is a process to estimate the distribution of the dialogue states as a dialogue progresses. Recent studies on constrained Markov Bayesian polynomial (CMBP) framework take the first step towards bridging the gap between rule-based and statistical approaches for DST. In this paper, the gap is further bridged by a novel framework -- recurrent polynomial network (RPN). RPN's unique structure enables the framework to have all the advantages of CMBP including efficiency, portability and interpretability. Additionally, RPN achieves more properties of statistical approaches than CMBP. RPN was evaluated on the data corpora of the second and the third Dialog State Tracking Challenge (DSTC-2/3). Experiments showed that RPN can significantly outperform both traditional rule-based approaches and statistical approaches with similar feature set. Compared with the state-of-the-art statistical DST approaches with a lot richer features, RPN is also competitive.

Statistical dialogue management is the core of cognitive spoken dialogue systems (SDS) and has attracted great research interest. In recent years, SDS with the ability of evolution is of particular interest and becomes the cuttingedge of SDS research. Dialogue state tracking (DST) is a process to estimate the distribution of the dialogue states at each dialogue turn, given the previous interaction history. It plays an important role in statistical dialogue management. To provide a common testbed for advancing the research of DST, international DST challenges (DSTC) have been organised and well-attended by major SDS groups in the world. This paper reviews recent progresses on rule-based and statistical approaches during the challenges. In particular, this paper is focused on evolvable DST approaches for dialogue domain extension. The two primary aspects for evolution, semantic parsing and tracker, are discussed. Semantic enhancement and a DST framework which bridges rule-based and statistical models are introduced in detail. By effectively incorporating prior knowledge of dialogue state transition and the ability of being data-driven, the new framework supports reliable domain extension with little data and can continuously improve with more data available. Thismakes it excellent candidate for DST evolution. Experiments show that the evolvable DST approaches can achieve the state-of-the-art performance and outperform all previously submitted trackers in the third DSTC.

Although context-dependent DNN-HMM systems have achieved significant improvements over GMM-HMM systems, significant performance degradation has been observed if the acoustic condition of the test data mismatches that of the training data. Hence, adaptation and adaptive training of DNN are of great research interest. Previous DNN adaptation works mainly focus on adapting parameters of a single DNN by applying linear transformations to feature or hidden-layer output; introducing vector representation of non-speech variability into the input. In these methods, large number of parameters are required to be estimated during adaptation. In this paper, the cluster adaptive training (CAT) framework is employed for DNN adaptive training. Here, multiple weight matrices are constructed to form the basis of a canonical parametric space. During adaptation, for a new acoustic condition, an interpolation vector is estimated to combine the weight basis into a single adapted weight matrix. Since only the interpolation vector need to be estimated during adaptation, the number of updated parameters is much smaller than existing DNN adaptation methods. The CAT-DNN approach was evaluated on an English switchboard task in unsupervised adaptation mode. It achieved significant WER reductions over the unadapted DNN-HMM, relative 7.6% to 10.6%, with only 10 parameters.