# Audio-Agent: Leveraging LLMs For Audio Generation, Editing and Composition

## Abstract

We introduce Audio-Agent, a multimodal framework for audio generation, editing and composition based on text or video inputs. Conventional approaches for text-to-audio (TTA) tasks often make single-pass inferences from text descriptions. While straightforward, this design struggles to produce high-quality audio when given complex text conditions. In our method, we utilize a pre-trained TTA diffusion network as the audio generation agent to work in tandem with GPT-4, which decomposes the text condition into atomic, specific instructions, and calls the agent for audio generation. Consequently, Audio-Agent generates high-quality audio that is closely aligned with the provided text or video while also supporting variable-length generation. For video-to-audio (VTA) tasks, most existing methods require training a timestamp detector to synchronize video events with generated audio, a process that can be tedious and time-consuming. We propose a simpler approach by fine-tuning a pre-trained Large Language Model (LLM), e.g., Gemma2-2B-it, to obtain both semantic and temporal conditions to bridge video and audio modality. Thus our framework provides a comprehensive solution for both TTA and VTA tasks without substantial computational overhead in training.

Figure 2: Overview of the TTA part. We use GPT-4 to convert a complex audio generation process into multiple generation steps and combine inference results.

Figure 3: Overview of the generation backbone. We build on top of the pre-trained Auffusion model for both TTA and VTA generation.

## Table of Contents

- Text-to-Audio generation
- Video-to-Audio generation
- Conversation example
- Long audio example

## Text-to-Audio generation

Single Caption:

Young children are whistling and laughing | Plastic clanking as a horse trots and a woman talks in the background | A child laughs, a man speaks, and people laugh
AudioGen: Your browser does not support the audio element. AudioLDM2: Your browser does not support the audio element. Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element. | AudioGen: Your browser does not support the audio element. AudioLDM2: Your browser does not support the audio element. Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element. | AudioGen: Your browser does not support the audio element. AudioLDM2: Your browser does not support the audio element. Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element.

People speaking with loud bangs followed by a slow motion rumble | A man speaks followed by loud snoring | A man whistling followed by a man yelling as plastic rustles and clanks in the background
AudioGen: Your browser does not support the audio element. AudioLDM2: Your browser does not support the audio element. Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element. | AudioGen: Your browser does not support the audio element. AudioLDM2: Your browser does not support the audio element. Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element. | AudioGen: Your browser does not support the audio element. AudioLDM2: Your browser does not support the audio element. Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element.

A clock chimes and ticktocks
AudioGen: Your browser does not support the audio element. AudioLDM2: Your browser does not support the audio element. Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element.

Two Captions:

Repetitive faint snoring followed by two men speaking | A croaking frog with brief bird chirps followed by a man talking as birds chirp in the background followed by a loud popping | Pigeons cooing and bird wings flapping as footsteps shuffle on paper followed by motor sounds with male speaking
AudioGen: Your browser does not support the audio element. AudioLDM2: Your browser does not support the audio element. Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element. | AudioGen: Your browser does not support the audio element. AudioLDM2: Your browser does not support the audio element. Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element. | AudioGen: Your browser does not support the audio element. AudioLDM2: Your browser does not support the audio element. Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element.

A vehicle accelerating in the distance then driving by followed by multiple gunfire sounds, and men speak | Whistling and then a female singing followed by woman speaking in a quiet environment | Distant thumping with some lights wind followed by water splashing occurs while a person quacks to imitate a duck and an adult female laughs
AudioGen: Your browser does not support the audio element. AudioLDM2: Your browser does not support the audio element. Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element. | AudioGen: Your browser does not support the audio element. AudioLDM2: Your browser does not support the audio element. Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element. | AudioGen: Your browser does not support the audio element. AudioLDM2: Your browser does not support the audio element. Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element.

A female voice and a duck quacking followed by wind noise on microphone with waves splashing in the background
AudioGen: Your browser does not support the audio element. AudioLDM2: Your browser does not support the audio element. Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element.

Complex Captions:

A man enters his car and drives away | A couple decorates a room, hangs pictures, and admires their work | A woman packs a suitcase, locks her house, and walks to the bus station
AudioGen: Your browser does not support the audio element. AudioLDM2: Your browser does not support the audio element. Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element. | AudioGen: Your browser does not support the audio element. AudioLDM2: Your browser does not support the audio element. Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element. | AudioGen: Your browser does not support the audio element. AudioLDM2: Your browser does not support the audio element. Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element.

## Video-to-Audio generation

Your browser does not support the video element. FoleyCrafter Your browser does not support the video element. Ours | Your browser does not support the video element. FoleyCrafter Your browser does not support the video element. Ours | Your browser does not support the video element. FoleyCrafter Your browser does not support the video element. Ours

Your browser does not support the video element. FoleyCrafter Your browser does not support the video element. Ours | Your browser does not support the video element. FoleyCrafter Your browser does not support the video element. Ours | Your browser does not support the video element. FoleyCrafter Your browser does not support the video element. Ours

## Conversation example

We provide audio output for Figure 4

A man enters his car and drives away. | Add "a man talks" | Edit "driving away" by "playing loud music"
Your browser does not support the audio element. | Your browser does not support the audio element. | Your browser does not support the audio element.

Combination with reasoning the missing part
Your browser does not support the audio element.

## Long audio example

We provide audio output for Figure 5

A river stream of water flowing followed by typing on a computer keyboard | A vehicle engine revving then accelerating at a high rate as a metal surface is whipped followed by tires skidding followed by a door shutting and a female speaking | A woman delivering a speech followed by a male speech and static
Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element. | Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element. | Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element.

Continuous white noise followed by a vehicle driving as a man and woman are talking and laughing
Auffusion: Your browser does not support the audio element. Our Method: Your browser does not support the audio element.