StoryFX: Augmenting Long-form Audio Stories with Generated Sound Effects Through Large Language Models and Text-to-Audio Generators

Author

Thant Zaw, Thinn Thinn Wai


Abstract

 Contemporary long-form audio stories in audiobook and podcast formats are limited to speech-only methods for storytelling. Integrating sound effects (SFX) through discrete event sounds and continuous ambient soundscapes can enhance listener engagement, emotional impact, and perceived presence, but manual sound design is labor-intensive and costly. This paper presents StoryFX, an end-to-end framework for automatically translating narrative text into synchronized sound-effect compositions using large language models (LLMs) for semantic orchestration, text-to-audio (TTA) models for sound synthesis, and digital audio workstations (DAWs) for audio composition. StoryFX introduces a temporal-semantic ontology for decomposing audiobook sound design into interpretable subtasks, enabling task-specific prompting and stepwise evaluation, together with a collection of multi-strategy prompting templates for inference optimization. Qwen 2.5 32B Instruct and Woosh were employed as foundation models to adapt a human-annotated story excerpt. We evaluate the framework using Qwen2.5-32B-Instruct for orchestration and Woosh for sound generation on a human-annotated audiobook excerpt. 



Keywords

Text-to-Audio Generation, Large Language Models, Prompt Engineering, Audio Systems, Audio Storytelling, Immersive Audio, Audiobooks, Podcasting, Sound Studies



Full Text:

Download Paper PDF


References


  • Döring, B. Müller, S. Rohr, J. Ruhrmann, and M. Schäfer, “Listen and Read: The Battle for Attention: A New Report About Key Audience Behaviour in the Age of eBooks, Audiobooks and Podcasts,” Publishing Research Quarterly, vol. 38, no. 1, pp. 40–52, Jan. 2022, doi: 10.1007/s12109-021-09853-2.
  • McClung and K. Johnson, “Examining the Motives of Podcast Users,” Journal of Radio & Audio Media, vol. 17, no. 1, pp. 82–95, May 2010, doi: 10.1080/19376521003719391.
  • Best, C. Clark, R. Perry, and L. Riad, “Audiobooks: A survey of UK adults in 2021,” National Literacy Trust, London, U.K, 2022.
  • Tosunay Gencelli, “The Competition of Eyes and Ears: Audiobook Consumption and Reading Practices Among Storytel Türkiye Users,” Publishing Research Quarterly, vol. 41, no. 2, pp. 216–238, Jun. 2025, doi: 10.1007/s12109-025-10031-x.
  • Ji, B. Liu, J. Xu, and J. Gong, “Why Do We Listen to Audiobooks? The Role of Narrator Performance, BGM, Telepresence, and Emotional Connectedness,” SAGE Open, vol. 14, no. 2, Apr. 2024, doi: 10.1177/21582440241257357.
  • Chignell, "Radio drama", In The Routledge Companion to British Media History; Routledge, pp. 356-365, 2014,
  • Verma, Theater of the mind : imagination, aesthetics, and American radio drama. Chicago ; London: The University Of Chicago Press, 2012.
  • Rodero, “See It on a Radio Story,” Communication Research, vol. 39, no. 4, pp. 458–479, Dec. 2010, doi: 10.1177/0093650210386947.
  • Cuadrado, I. Lopez-Cobo, T. Mateos-Blanco, and A. Tajadura-Jiménez, “Arousing the Sound: A Field Study on the Emotional Impact on Children of Arousing Sound Design and 3D Audio Spatialization in an Audio Story,” Frontiers in Psychology, vol. 11, no. 737, May 2020, doi: 10.3389/fpsyg.2020.00737.
  • M. Bradley and P. J. Lang, “Affective reactions to acoustic stimuli,” Psychophysiology, vol. 37, no. 2, pp. 204–215, Mar. 2000, 10.1111/1469-8986.3720204
  • Bhide, E. Goins, and J. Geigel, “Experimental Analysis of Spatial Sound for Storytelling in Virtual Reality,” Frameless, vol. 1, no. 1, pp. 1–3, Dec. 2019, doi: 10.14448/frameless.01.007.
  • X. Zhao et al., “A Survey of Large Language Models,” arXiv (Cornell University), Mar. 2023, doi: 10.48550/arxiv.2303.18223.
  • Brown et al., “Language Models Are Few-Shot Learners,” ArXiv (Cornell University), vol. 4, May 2020, doi: 10.48550/arxiv.2005.14165.
  • Jason Zhanshun Wei et al., “Finetuned Language Models Are Zero-Shot Learners,” Sep. 2021, doi: 10.48550/arxiv.2109.01652.
  • Z. Wei et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” arXiv, Jan. 2022, doi: 10.48550/arxiv.2201.11903.
  • E. J. Hu et al., “LoRA: Low-rank adaptation of large language models,” arXiv (Cornell University), Jan. 2021, doi: 10.48550/arxiv.2106.09685.
  • Sahoo, A. K. Singh, S. Saha, V. Jain, S. Mondal, and A. Chadha, “A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications,” arXiv (Cornell University), Feb. 2024, doi: 10.48550/arxiv.2402.07927.
  • OpenAI, “GPT-4 Technical Report,” arXiv (Cornell University), Mar. 2023, doi: 10.48550/arxiv.2303.08774.
  • Comanici et al., “Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities,” arXiv (Cornell University), Jul. 2025, doi: 10.48550/arxiv.2507.06261.
  • Dubey et al., “The Llama 3 Herd of Models,” arXiv (Cornell University), Nov. 2024, doi: 10.48550/arxiv.2407.21783.
  • Kreuk et al., “AudioGen: Textually Guided Audio Generation,” arXiv (Cornell University), Sep. 2022, doi: 10.48550/arxiv.2209.15352.
  • Liu et al., “AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 2871–2883, 2024, doi: 10.1109/taslp.2024.3399607.
  • Hadjeres et al., “Woosh: A Sound Effects Foundation Model,” arXiv (Cornell University), Apr. 2026, doi: 10.48550/arxiv.2604.01929.
  • Ge, C. Xuan, R. Song, C. Zou, W. Liu, and J. Zhou, “From Text to Sound,” Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, Jul. 2019, doi: 10.1145/3331184.3331274.
  • Mac and M. Sra, “Sonic Storyteller: Augmenting Oral Storytelling with Spatial Sound Effects,” Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST Adjunct ’23), San Francisco, CA, USA, Oct. 2023, pp. 1–3, doi: 10.1145/3586182.3616642.
  • Liu et al., “WavJourney: Compositional Audio Creation with Large Language Models,” IEEE Transactions on Audio, Speech and Language Processing, pp. 1–15, 2025, doi: 10.1109/taslpro.2025.3574867.
  • Gu et al., “A survey on LLM-as-a-Judge,” The Innovation, p. 101253, Jan. 2026, doi: 10.1016/j.xinn.2025.101253.
  • Rousso, E. Cohen, J. Keshet, and E. Chodroff, “Tradition or Innovation: A Comparison of Modern ASR Methods for Forced Alignment,” Interspeech 2022, pp. 1525–1529, Sep. 2024, doi: 10.21437/interspeech.2024-429.
  • Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, doi: 10.18653/v1/d19-1410.


 

Share your valuable work from Social Media Buttons