About Us

AI for assistive speech communication technologies

The proportional increase in the elderly population and the inappropriate use of portable audio devices have led to a rapid increase in incidents of hearing loss.

Untreated hearing loss can cause feelings of loneliness and isolation in the elderly and may lead to learning difficulties in students. Over the past few years, our group has investigated the application of machine learning and signal processing algorithms in FM assistive hearing systems [1, 2], hearing aids [3, 4], and cochlear implants (CIs) [5-7] to improve speech communication in hearing-impaired patients and the subsequent enhancement in their quality of life. In addition to assistive listening devices, we have also investigated the development of machine learning-based assistive speaking devices to enhance intelligibility in individuals with speech and language disorders [8].

Oral cancer ranks in the top five of all cancers in Taiwan. To treat the oral cancer, surgical processes are often required to have parts of the patients’ articulators removed. Because of the removal of parts of the articulator, a patient’s speech may be distorted and difficult to understand. To overcome this problem, we propose two voice conversion (VC) approaches: the first one is the joint dictionary training non-negative matrix factorization (JD-NMF), and the second one is the end-to-end generative adversarial network (GAN)-based unsupervised VC model [9]. Experimental results show that both approaches can be applied to convert the distorted speech signals to the ones with improved intelligibility.

Deep learning-based speech signal processing

In this investigation, we focused on deriving novel deep learning-based algorithms for denoising [10], dereverberation [11], and channel compensation [12] on speech signals. The goal is to enhance the speech signals in order to achieving improved human-human and human-machine communication efficacy.

We investigated approaches to enhance speech intelligibility [13] and quality [14] to facilitate higher speech recognition rates and improved communication. We also proposed end-to-end waveform enhancement to directly improve the intelligibility and quality of speech. In addition, we have developed a novel integrated deep and ensemble learning algorithm (IDEA) [15] and an environment-adaptive algorithm (based on the domain adversarial training criterion) [16] for speech signal processing to address possible mismatched issues in real-world applications.

Multimodal speech signal processing

Communication can be verbal or nonverbal. Verbal communication includes speaking and listening.

The speaker relays verbal information while the listener focuses on auditory signals and the visual cues for speech recognition. The visual signals may include articulatory movements, facial expressions, and co-speech gestures of the speaker, which constitutes the nonverbal part of the communication process. In various applied speech technologies, it has been shown that audio-visual integration can assist in human information exchange and the development of human-computer interfaces. We have conducted studies on the incorporation of audio and video information to facilitate improved speech signal processing performance.

Currently, we have developed novel algorithms by fusing the audio and visual information for emotion recognition [17], oral presentation scoring [18], and speech enhancement [19]

Increasing Compactness of Deep Learning based Speech Enhancement Models

Most recent studies on deep learning-based speech enhancement (SE) have focused on improving the denoising performance. However, successful SE application requires the achievement of a balance between denoising performance and computational cost in real scenarios.We have investigated two approaches to effectively compress deep learning models so that the SE can be performed at edge sides.

These approaches are model pruning and parameter quantization. In model pruning, a computation-performance optimization (CPO) algorithm was developed [20] for the removal of redundant channels in a neural network, as shown in Fig. 4. For parameter quantization, we proposed an exponent-only floating point quantized neural network (EOFP-QNN) to compress the model and enhance inference efficiency [21]. Both the model pruning and parameter quantization techniques can significantly reduce model size and increase inference efficiency with an acceptable drop in performance.

Speech Enhancement with Direct Evaluation Metric Optimization

During the training process for an SE model, an objective function is used to optimize the model parameters. In the existing literature, there is an inconsistency between the model optimization criterion and the evaluation criterion for enhanced speech.

For example, in the measurement of intelligibility, most of the evaluation metrics are based on short-time objective intelligibility (STOI) measure, while a frame based mean square error (MSE) between the enhanced speech and clean reference is widely used in the process of optimizing the model. Due to this inconsistency, there is no guarantee that the trained model will facilitate optimal performance in different applications [13]. We therefore investigated several algorithms with the aim of directly optimizing model parameters based on evaluation metrics including STOI [13], perceptual evaluation of speech quality (PESQ) [14], and automatic speech recognition (ASR) [22].

Reinforcement learning and GAN-based methods have also been exploited to facilitate optimization given that some evaluation metrics are complex and not differentiable. Experimental results show that by using the same specific evaluation metric as the objective function, the SE model can be trained to yield superior performance compared to the MSE to achieve a desired outcome.

Multimodal Pathological Voice Classification

We have conducted research on pathological voice classification based on medical records [23] and voice signals [24]. The results showed that voice disorders can be accurately identified using voice signals and medical records when advanced signal processing and machine learning methods are utilized. Based on the voice data, we organized a pathological Voice Detection Challenge in IEEE Big Data 2018 [25], which attracted 109 participating teams from 27 different countries.

More recently, we investigated the combination of acoustic signals and medical records and derived a multimodal deep learning model. The proposed model consists of two stages: the first stage processes acoustic features and medical data individually and the second stage integrates the outputs from the first stage to perform classification. The proposed multimodal deep learning frameworks were evaluated using 589 samples collected from Far Eastern Memorial Hospital, consisting of three categories of vocal disease, i.e. glottic neoplasm, phonotraumatic lesions, and vocal paralysis. We obtained promising experimental results compared to systems that use only acoustic signals or medical records.