Ziyang Ma (马子阳)

Ph.D. student, Shanghai Jiao Tong University. 800 Dongchuan RD. Minhang District, Shanghai, China. E-mail: zym.22@sjtu.edu.cn

Biography

Hi👋 nice to meet you!

Currently I am a Ph.D. student of Shanghai Jiao Tong University (SJTU) and SJTU Artificial Intelligence Institute, and a member in Cross Media (X-) Language Intelligence Lab (X-LANCE) of the Department of Computer Science and Engineering, co-supervised by Prof. Xie Chen, Yanmin Qian and working closely with Prof. Kai Yu. As the first Ph.D. supervised by Prof. Chen, I will try my best in the next five exciting years! 💪

I was a research assistant at InteLligent media research center (iLearn), working closely with Prof. Xuemeng Song and Liqiang Nie during my undergraduate years.

My research usually follows the KISS philosophy. My recent work focuses on speech, language, audio and music processing with Self-Supervised Learning (SSL) and Large Language Model (LLM). If you are also interested, please feel free to contact me.

Education

• Ph.D., Computer Science and Engineering, Shanghai Jiao Tong University, 2022.09-Now

• B.E., Computer Science and Technology, Shandong University, 2018.09-2022.06

Interests

• Self-Supervised Learning

• Speech and Audio Processing

• Natural Language Processing

• Multimedia and Multimodal

NEWS

• [2024.5] BAT was accepted by ICML 2024.

• [2024.4] MER24 Challenge@IJCAI and MRAC24 Workshop@ACM Multimedia are coming! [Baseline Paper][Baseline Code][Challenge Homepage]

• [2024.4] EAT was accepted by IJCAI 2024.

• [2024.3] 🎉 We won the 1st place in Categorical Emotion Recognition at Odyssey 2024 Emotion Recognition Challenge.

• [2024.1] Check out our Repo for EAT, a new audio representation model with both effectiveness and efficiency.

• [2023.12] Check out our Repo for emotion2vec, the first universal speech emotion representation model.

• [2023.12] 4 papers were accpeted by IEEE ICASSP2024.

• [2023.9] Check out our Repo for Fast-HuBERT. We accelerate HuBERT pre-training in 5.2X speedup without performance drop.

• [2023.9] 2 papers were accpeted by IEEE ASRU2023.

• [2023.8] MT4SSL was nominated in ISCA Interspeech Best Student Paper Shortlist. Congrats!

Research

Selected Publications

Thanks to all the collaborators for their great work!

Check out Google Scholar for more information.

Speech, Language, Audio, Music Processing with SSL

• Wenxi Chen, Yuzhe Liang, Ziyang Ma, Zhisheng Zheng, Xie Chen.
  EAT: Self-Supervised Pre-Training with Efficient Audio Transformer.
  in International Joint Conference on Artificial Intelligence (IJCAI), 2024.
  

• Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, Xie Chen.
  emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation.
  in arXiv, 2023.
  

• Guanrou Yang, Ziyang Ma, Zhisheng Zheng, Yakun Song, Zhikang Niu, Xie Chen.
  Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learning.
  in IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2023.
  

• Ziyang Ma, Zhisheng Zheng, Guanrou Yang, Yu Wang, Chao Zhang, Xie Chen.
  Pushing the Limits of Unsupervised Unit Discovery for SSL Speech Representation.
  in INTERSPEECH, 2023.
  

• Ziyang Ma, Zhisheng Zheng, Changli Tang, Yujin Wang, Xie Chen.
  MT4SSL: Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targets.
  Oral & Best Student Paper Shortlist in INTERSPEECH, 2023.
  

Speech, Language, Audio, Music Processing with LLM

• MuPT: A Generative Symbolic Music Pretrained Transformer.
  in arXiv, 2024.
  

• ChatMusician: Understanding and Generating Music Intrinsically with LLM.
  in arXiv, 2024.
  

• Ziyang Ma, Guanrou Yang, Yifan Yang, Zhifu Gao, Jiaming Wang, Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, Xie Chen.
  An Embarrassingly Simple Approach for LLM with Strong ASR Capacity.
  in arXiv, 2024.
  

• Zhisheng Zheng, Puyuan Peng, Ziyang Ma, Xie Chen, Eunsol Choi, David Harwath.
  BAT: Learning to Reason about Spatial Sounds with Large Language Models.
  in International Conference on Machine Learning (ICML), 2024.
  

• LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT.
  in arXiv, 2023.
  

• Ziyang Ma, Wen Wu, Zhisheng Zheng, Yiwei Guo, Qian Chen, Shiliang Zhang, Xie Chen.
  Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognition.
  Oral in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024.
  

Experiences

Research Intern, Speech Lab, Alibaba DAMO Academy, 2023.06-2024.02

• Investigate emotional speech dialogue system with large language model.

• Supervised by Shiliang Zhang and working closely with Zhifu Gao and Qian Chen.

Research Intern, NLC Group, Microsoft Research Asia(MSRA), 2022.02-2022.08

• Investigate joint pre-training of speech and text to help improve the accuracy of ASR and other downstream tasks.

• Led by Furu Wei, supervised by Shujie Liu, and working closely with Yu Wu and Long Zhou.

Research Intern, Video Group, MEGVII Research, 2021.04-2021.06

• Investigate re-identification of vehicle with Transformer architecture.

• Supervised by Chi Zhang.

Research Assistant, InteLligent media research center (iLearn), Shandong University, 2020.09-2021.09

• Investigate temporal moment localization in untrimmed videos.

• Supervised by Xuemeng Song and Liqiang Nie.

Academic Service

Conference Reviewer / TPC Member

• ACL Rolling Review (ACL ARR) 2024

• IEEE International Conference on Acoustics, Speech and Signal Processing (IEEE ICASSP) 2023, 2024

• AAAI Conference on Artificial Intelligence 2022

• ACM International Conference on Multimedia (ACM MM) 2022

Journal Reviewer

• IEEE Signal Processing Letters (IEEE SPL)

• IEEE Transactions on Multimedia (IEEE TMM)

• IEEE Transactions on Circuits and Systems for Video Technology (IEEE TCAVT)

Accomplishments

Awards

• SPS Travel Grant, IEEE, 2024.02

• Best Presentation Award in Student Forum, the 18th National Conference on Man-Machine Speech Communication (NCMMSC), 2023.12

• Interspeech Best Student Paper Shortlist, ISCA, 2023.08

• Excellent Graduate, Department of Education, Shandong Province, China, 2022.06

• "Intelligent Pedestal" Scholarship, Huawei, 2021.12

• SIGMM Student Travel Grant, ACM, 2021.11

• National Scholarship, Ministry of Education, China, 2021.10

Competitions

• 1st in Odyssey 2024 Emotion Recognition Challenge Task1: Categorical Emotion Recognition, Odyssey 2024 The Speaker and Language Recognition Workshop, 2024.03.

• 3rd in DCASE2023 Challenge Task4b: Sound Event Detection with Soft Labels, IEEE AASP Challenge on Detection and Classification of Acoustic Scenes and Events, 2023.06.

• 3rd in Avatar Track of AIWIN, the 5th World Artificial Intelligence Conference(WAIC2022), Shanghai, China, 2022.09.[Report][Invited Talk]

• Finalist(Top 284 in 26112 teams) in Mathematical Contest in Modeling (MCM), Consortium for Mathematics and Its Application, America, 2021.02

• First Prize(Top 293 in 45689 teams) in Contemporary Undergraduate Mathematical Contest in Modeling (CUMCM), China Society for Industrial and Applied Mathematics, China, 2020.09

Activities

• [SpeechHome Invited Talk]: INTERSPEECH 2023 Pre-presentation , 2023.07

• [Datawhale Invited Talk]: How to conduct audio-driven talking head? An introduction and solution sharing, 2022.11

• Member of Datawhale, 2022.09-Now

• Teaching Assistant, Computer Science and Technology, Shandong University, 2021.03-2021.06

• Member of Elite Class, Computer Science and Technology, Shandong University, 2020.09-2022.06