Yisu Zong

Yisu Zong

Publications

First Author

Learning Control of Neural Sound Effects Synthesis from Physically Inspired Models

Published in 2025 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2025)

Sound effects model design commonly uses digital signal processing techniques with full control ability, but it is difficult to achieve realism within a limited number of parameters. Recently, neural sound effects synthesis methods have emerged as a promising approach for generating highquality and realistic sounds, but the process of synthesizing the desired sound poses difficulties in terms of control. This paper presents a real-time neural synthesis model guided by a physically inspired model, enabling the generation of high-quality sounds while inheriting the control interface of the physically inspired model. We showcase the superior performance of our model in terms of sound quality and control.

A Machine learning method to evaluate and improve sound effects synthesis model design

Published in Audio Engineering Society (AES) 2024 International Audio for Games Conference

Procedural audio models have great potential in sound effects production and design, they can be incredibly high quality and have high interactivity with the users. However, they also often have many free parameters that may not be specified just from an understanding of the phenomenon, making it very difficult for users to create the desired sound. Moreover, their potential and generalization ability are rarely explored fully due to their complexity. To address these problems, this work introduces a hybrid machine learning method to evaluate the overall sound matching performance of a real sound dataset. First, we train a parameter estimation network using synthesis sound samples. Through the differentiable implementation of the sound synthesis model, we use both parameter and spectral loss in this self-supervised stage. Then, we perform adversarial training by spectral loss plus adversarial loss using real sound samples. We evaluate our approach for an example of an explosion sound synthesis model. We experiment with different model designs and conduct a subjective listening test. We demonstrate that this is an effective method to evaluate the overall performance of a sound synthesis model, and its capability to speed up the sound model design process.

Co-author

Discerning real from synthetic: analysis and perceptual evaluation of sound effects

Published in Audio Engineering Society (AES) 2024 International Audio for Games Conference

In audio post-production, the adoption of sound synthesis offers a viable alternative for searching and recording samples in creating soundscapes. However, a central concern arises regarding the ability of synthetic sounds to match the perceived authenticity of library samples. This paper introduces an analytical approach, examining authentic and synthetic samples in five categories(burning embers, pouring water, explosions, popping bubbles and church bells) by delving into audio descriptors that distinguish both types. We focus in the utilization of machine learning classification models and a perceptual evaluation experiment. The perceptual evaluation was between five distinct synthesis techniques – granular, additive, subtractive, physically informed, and modal synthesis – revealed that subtractive synthesis is perceived as more realistic in explosion sounds, while additive synthesis works better with pouring water sounds. This study provides valuable insights into the audio descriptors that may require modification in specific synthetic models, paving the way for a deeper understanding of sound synthesis methods and facilitating their integration into the sound design process.

Optimization Techniques for a Physical Model of Human Vocalisation

Published in Proceedings of the 26th International Conference on Digital Audio Effects (DAFx23)

We present a non-supervised approach to optimize and evaluate the synthesis of non-speech audio effects from a speech production model. We use the Pink Trombone synthesizer as a case study of a simplified production model of the vocal tract to target nonspeech human audio signals – yawnings. We selected and optimized the control parameters of the synthesizer to minimize the difference between real and generated audio. We validated the most common optimization techniques reported in the literature and a specifically designed neural network. We evaluated several popular quality metrics as error functions. These include both objective quality metrics and subjective-equivalent metrics. We compared the results in terms of total error and computational demand. Results show that genetic and swarm optimizers outperform least squares algorithms at the cost of executing slower and that specific combinations of optimizers and audio representations offer significantly different results. The proposed methodology could be used in benchmarking other physical models and audio types.