Scaling Speech Technology to 1, 000+ Languages - Vineel Pratap; Andros Tjandra; Bowen Shi; Paden Tomasello; Arun Babu; Sayani Kundu; Ali Elkahky; Zhaoheng Ni; Apoorv Vyas; Maryam Fazel-Zarandi; Alexei Baevski; Yossi Adi; et al. (J. Mach. Learn. Res., 2024)

Stack-and-Delay: A New Codebook Pattern for Music Generation - Gaël Le Lan; Varun Nagaraja; Ernie Chang; David Kant; Zhaoheng Ni; Yangyang Shi; Forrest N. Iandola; Vikas Chandra (IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14-19, 2024, 2024)

An Empirical Study on the Impact of Positional Encoding in Transformer-Based Monaural Speech Enhancement - Qiquan Zhang; Meng Ge; Hongxu Zhu; Eliathamby Ambikairajah; Qi Song; Zhaoheng Ni; Haizhou Li (IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14-19, 2024, 2024)

On the Open Prompt Challenge in Conditional Audio Generation - Ernie Chang; Sidd Srinivasan; Mahi Luthra; Pin-Jie Lin; Varun Nagaraja; Forrest N. Iandola; Zechun Liu; Zhaoheng Ni; Changsheng Zhao; Yangyang Shi; Vikas Chandra (IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14-19, 2024, 2024)

Less Peaky and More Accurate CTC Forced Alignment by Label Priors - Ruizhe Huang; Xiaohui Zhang; Zhaoheng Ni; Li Sun; Moto Hira; Jeff Hwang; Vimal Manohar; Vineel Pratap; Matthew Wiesner; Shinji Watanabe; Daniel Povey; Sanjeev Khudanpur (IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14-19, 2024, 2024)

Folding Attention: Memory and Power Optimization for On-Device Transformer-Based Streaming Speech Recognition - Yang Li; Liangzhen Lai; Yuan Shangguan; Forrest N. Iandola; Zhaoheng Ni; Ernie Chang; Yangyang Shi; Vikas Chandra (IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14-19, 2024, 2024)

Foleygen: Visually-Guided Audio Generation - Xinhao Mei; Varun Nagaraja; Gaël Le Lan; Zhaoheng Ni; Ernie Chang; Yangyang Shi; Vikas Chandra (34th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2024, London, UK, September 22-25, 2024, 2024)

Serialized Speech Information Guidance with Overlapped Encoding Separation for Multi-Speaker Automatic Speech Recognition - Hao Shi; Yuan Gao; Zhaoheng Ni; Tatsuya Kawahara (IEEE Spoken Language Technology Workshop, SLT 2024, Macao, December 2-5, 2024, 2024)

Data Efficient Reflow for Few Step Audio Generation - Lemeng Wu; Zhaoheng Ni; Bowen Shi; Gaël Le Lan; Anurag Kumar; Varun Nagaraja; Xinhao Mei; Yunyang Xiong; Bilge Soran; Raghuraman Krishnamoorthi; Wei-Ning Hsu; Yangyang Shi; et al. (IEEE Spoken Language Technology Workshop, SLT 2024, Macao, December 2-5, 2024, 2024)

Massively Multilingual Forced Aligner Leveraging Self-Supervised Discrete Units - Hirofumi Inaguma; Ilia Kulikov; Zhaoheng Ni; Sravya Popuri; Paden Tomasello (IEEE Spoken Language Technology Workshop, SLT 2024, Macao, December 2-5, 2024, 2024)

An Empirical Study on the Impact of Positional Encoding in Transformer-based Monaural Speech Enhancement - Qiquan Zhang; Meng Ge; Hongxu Zhu; Eliathamby Ambikairajah; Qi Song; Zhaoheng Ni; Haizhou Li (CoRR, 2024)

Less Peaky and More Accurate CTC Forced Alignment by Label Priors - Ruizhe Huang; Xiaohui Zhang; Zhaoheng Ni; Li Sun; Moto Hira; Jeff Hwang; Vimal Manohar; Vineel Pratap; Matthew Wiesner; Shinji Watanabe; Daniel Povey; Sanjeev Khudanpur (CoRR, 2024)

URGENT Challenge: Universality, Robustness, and Generalizability For Speech Enhancement - Wangyou Zhang; Robin Scheibler; Kohei Saijo; Samuele Cornell; Chenda Li; Zhaoheng Ni; Anurag Kumar; Jan Pirklbauer; Marvin Sach; Shinji Watanabe; Tim Fingscheidt; Yanmin Qian (CoRR, 2024)

High Fidelity Text-Guided Music Generation and Editing via Single-Stage Flow Matching - Gaël Le Lan; Bowen Shi; Zhaoheng Ni; Sidd Srinivasan; Anurag Kumar; Brian Ellis; David Kant; Varun Nagaraja; Ernie Chang; Wei-Ning Hsu; Yangyang Shi; Vikas Chandra (CoRR, 2024)

Serialized Speech Information Guidance with Overlapped Encoding Separation for Multi-Speaker Automatic Speech Recognition - Hao Shi; Yuan Gao; Zhaoheng Ni; Tatsuya Kawahara (CoRR, 2024)

SyncFlow: Toward Temporally Aligned Joint Audio-Video Generation from Text - Haohe Liu; Gaël Le Lan; Xinhao Mei; Zhaoheng Ni; Anurag Kumar; Varun Nagaraja; Wenwu Wang; Mark D. Plumbley; Yangyang Shi; Vikas Chandra (CoRR, 2024)

Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decoding - Jiahui Zhao; Hao Shi; Chenrui Cui; Tianrui Wang; Hexin Liu; Zhaoheng Ni; Lingxuan Ye; Longbiao Wang (CoRR, 2024)

Software Design and User Interface of ESPnet-SE++: Speech Enhancement for Robust Speech Processing - Yen-Ju Lu; Xuankai Chang; Chenda Li; Wangyou Zhang; Samuele Cornell; Zhaoheng Ni; Yoshiki Masuyama; Brian Yan; Robin Scheibler; Zhong-Qiu Wang; Yu Tsao; Yanmin Qian; et al. (J. Open Source Softw., 2023)

A Time-Frequency Attention Module for Neural Speech Enhancement - Qiquan Zhang; Xinyuan Qian; Zhaoheng Ni; Aaron Nicolson; Eliathamby Ambikairajah; Haizhou Li (IEEE ACM Trans. Audio Speech Lang. Process., 2023)

ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkit - Brian Yan; Jiatong Shi; Yun Tang; Hirofumi Inaguma; Yifan Peng; Siddharth Dalmia; Peter Polak; Patrick Fernandes; Dan Berrebbi; Tomoki Hayashi; Xiaohui Zhang; Zhaoheng Ni; et al. (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, ACL 2023, Toronto, Canada, July 10-12, 2023, 2023)

TorchAudio 2.1: Advancing Speech Recognition, Self-Supervised Learning, and Audio Processing Components for Pytorch - Jeff Hwang; Moto Hira; Caroline Chen; Xiaohui Zhang; Zhaoheng Ni; Guangzhi Sun; Pingchuan Ma; Ruizhe Huang; Vineel Pratap; Yuekai Zhang; Anurag Kumar; Chin-Yun Yu; et al. (IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2023, Taipei, Taiwan, December 16-20, 2023, 2023)

Torchaudio-Squim: Reference-Less Speech Quality and Intelligibility Measures in Torchaudio - Anurag Kumar; Ke Tan; Zhaoheng Ni; Pranay Manocha; Xiaohui Zhang; Ethan Henderson; Buye Xu (IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023, 2023)

Ripple Sparse Self-Attention for Monaural Speech Enhancement - Qiquan Zhang; Hongxu Zhu; Qi Song; Xinyuan Qian; Zhaoheng Ni; Haizhou Li (IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023, 2023)

Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Compute - William Chen; Xuankai Chang; Yifan Peng; Zhaoheng Ni; Soumi Maiti; Shinji Watanabe (24th Annual Conference of the International Speech Communication Association, Interspeech 2023, Dublin, Ireland, August 20-24, 2023, 2023)

Software Design and User Interface of ESPnet-SE++: Speech Enhancement for Robust Speech Processing (espnet-v.202310) (Version 1) - Yen-Ju Lu; Xuankai Chang; Chenda Li; Wangyou Zhang; Samuele Cornell; Zhaoheng Ni; Yoshiki Masuyama; Brian Yan; Robin Scheibler; Zhong-Qiu Wang; Yu Tsao; Yanmin Qian; et al. (Zenodo, 2023)
Accessed on YYYY-MM-DD.

Ripple sparse self-attention for monaural speech enhancement - Qiquan Zhang; Hongxu Zhu; Qi Song; Xinyuan Qian; Zhaoheng Ni; Haizhou Li (CoRR, 2023)

Scaling Speech Technology to 1, 000+ Languages - Vineel Pratap; Andros Tjandra; Bowen Shi; Paden Tomasello; Arun Babu; Sayani Kundu; Ali Elkahky; Zhaoheng Ni; Apoorv Vyas; Maryam Fazel-Zarandi; Alexei Baevski; Yossi Adi; et al. (CoRR, 2023)

Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Compute - William Chen; Xuankai Chang; Yifan Peng; Zhaoheng Ni; Soumi Maiti; Shinji Watanabe (CoRR, 2023)

Enhance audio generation controllability through representation similarity regularization - Yangyang Shi; Gaël Le Lan; Varun Nagaraja; Zhaoheng Ni; Xinhao Mei; Ernie Chang; Forrest N. Iandola; Yang Liu; Vikas Chandra (CoRR, 2023)

Stack-and-Delay: a new codebook pattern for music generation - Gaël Le Lan; Varun Nagaraja; Ernie Chang; David Kant; Zhaoheng Ni; Yangyang Shi; Forrest N. Iandola; Vikas Chandra (CoRR, 2023)

FoleyGen: Visually-Guided Audio Generation - Xinhao Mei; Varun Nagaraja; Gaël Le Lan; Zhaoheng Ni; Ernie Chang; Yangyang Shi; Vikas Chandra (CoRR, 2023)

TorchAudio 2.1: Advancing speech recognition, self-supervised learning, and audio processing components for PyTorch - Jeff Hwang; Moto Hira; Caroline Chen; Xiaohui Zhang; Zhaoheng Ni; Guangzhi Sun; Pingchuan Ma; Ruizhe Huang; Vineel Pratap; Yuekai Zhang; Anurag Kumar; Chin-Yun Yu; et al. (CoRR, 2023)

On The Open Prompt Challenge In Conditional Audio Generation - Ernie Chang; Sidd Srinivasan; Mahi Luthra; Pin-Jie Lin; Varun Nagaraja; Forrest N. Iandola; Zechun Liu; Zhaoheng Ni; Changsheng Zhao; Yangyang Shi; Vikas Chandra (CoRR, 2023)

Torchaudio: Building Blocks for Audio and Speech Processing - Yao-Yuan Yang; Moto Hira; Zhaoheng Ni; Artyom Astafurov; Caroline Chen; Christian Puhrsch; David Pollack; Dmitriy Genzel; Donny Greenberg; Edward Z. Yang; Jason Lian; Jeff Hwang; et al. (IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022, 2022)

Time-Frequency Attention for Monaural Speech Enhancement - Qiquan Zhang; Qi Song; Zhaoheng Ni; Aaron Nicolson; Haizhou Li (IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022, 2022)

Towards Low-Distortion Multi-Channel Speech Enhancement: The ESPNET-Se Submission to the L3DAS22 Challenge - Yen-Ju Lu; Samuele Cornell; Xuankai Chang; Wangyou Zhang; Chenda Li; Zhaoheng Ni; Zhong-Qiu Wang; Shinji Watanabe (IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022, 2022)

ESPnet-SE++: Speech Enhancement for Robust Speech Recognition, Translation, and Understanding - Yen-Ju Lu; Xuankai Chang; Chenda Li; Wangyou Zhang; Samuele Cornell; Zhaoheng Ni; Yoshiki Masuyama; Brian Yan; Robin Scheibler; Zhong-Qiu Wang; Yu Tsao; Yanmin Qian; et al. (23rd Annual Conference of the International Speech Communication Association, Interspeech 2022, Incheon, Korea, September 18-22, 2022, 2022)

Towards Low-distortion Multi-channel Speech Enhancement: The ESPNet-SE Submission to The L3DAS22 Challenge - Yen-Ju Lu; Samuele Cornell; Xuankai Chang; Wangyou Zhang; Chenda Li; Zhaoheng Ni; Zhong-Qiu Wang; Shinji Watanabe (CoRR, 2022)

ESPnet-SE++: Speech Enhancement for Robust Speech Recognition, Translation, and Understanding - Yen-Ju Lu; Xuankai Chang; Chenda Li; Wangyou Zhang; Samuele Cornell; Zhaoheng Ni; Yoshiki Masuyama; Brian Yan; Robin Scheibler; Zhong-Qiu Wang; Yu Tsao; Yanmin Qian; et al. (CoRR, 2022)

WPD++: An Improved Neural Beamformer for Simultaneous Speech Separation and Dereverberation - Zhaoheng Ni; Yong Xu; Meng Yu; Bo Wu; Shi-Xiong Zhang; Dong Yu; Michael I. Mandel (IEEE Spoken Language Technology Workshop, SLT 2021, Shenzhen, China, January 19-22, 2021, 2021)

TorchAudio: Building Blocks for Audio and Speech Processing - Yao-Yuan Yang; Moto Hira; Zhaoheng Ni; Anjali Chourdia; Artyom Astafurov; Caroline Chen; Ching-Feng Yeh; Christian Puhrsch; David Pollack; Dmitriy Genzel; Donny Greenberg; Edward Z. Yang; et al. (CoRR, 2021)

Time-Frequency Attention for Monaural Speech Enhancement - Qiquan Zhang; Qi Song; Zhaoheng Ni; Aaron Nicolson; Haizhou Li (CoRR, 2021)

Mask-Dependent Phase Estimation for Monaural Speaker Separation - Zhaoheng Ni; Michael I. Mandel (2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020, 2020)

WPD++: An Improved Neural Beamformer for Simultaneous Speech Separation and Dereverberation - Zhaoheng Ni; Yong Xu; Meng Yu; Bo Wu; Shi-Xiong Zhang; Dong Yu; Michael I. Mandel (CoRR, 2020)

Enhancement of Spatial Clustering-Based Time-Frequency Masks using LSTM Neural Networks - Félix Grèzes; Zhaoheng Ni; Viet Anh Trinh; Michael I. Mandel (CoRR, 2020)

Improved MVDR Beamforming Using LSTM Speech Models to Clean Spatial Clustering Masks - Zhaoheng Ni; Félix Grèzes; Viet Anh Trinh; Michael I. Mandel (CoRR, 2020)

Combining Spatial Clustering with LSTM Speech Models for Multichannel Speech Enhancement - Félix Grèzes; Zhaoheng Ni; Viet Anh Trinh; Michael I. Mandel (CoRR, 2020)

Onssen: an open-source speech separation and enhancement library - Zhaoheng Ni; Michael I. Mandel (CoRR, 2019)

Unusable Spoken Response Detection with BLSTM Neural Networks - Zhaoheng Ni; Rutuja Ubale; Yao Qian; Michael I. Mandel; Su-Youn Yoon; Abhinav Misra; David Suendermann-Oeft (11th International Symposium on Chinese Spoken Language Processing, ISCSLP 2018, Taipei City, Taiwan, November 26-29, 2018, 2018)

Sound Signal Processing with Seq2Tree Network - Weicheng Ma; Kai Cao; Zhaoheng Ni; Peter Chin; Xiang Li (Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018, 2018)

Confused or not Confused?: Disentangling Brain Activity from EEG Data Using Bidirectional LSTM Recurrent Neural Networks - Zhaoheng Ni; Ahmet Cem Yuksel; Xiuyan Ni; Michael I. Mandel; Lei Xie (Proceedings of the 8th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, BCB 2017, Boston, MA, USA, August 20-23, 2017, 2017)

A Sep2Tree Model for Recognizing Synthetic Bach Chorales - Xiuyan Ni; Weicheng Ma; Zhaoheng Ni; Robert M. Haralick (Proceedings of the 2017 International Computer Music Conference, ICMC 2017, Shanghai, China, October 16-20, 2017, 2017)