# Taming Data and Transformers for Audio Generation

Anonymous Authors

## Abstract

Generating ambient sounds and effects is a challenging task due to data scarcity and often insufficient caption quality, making it difficult to em- ploy large-scale generative models for the task. In this work, we tackle this problem by introducing two new models. First, we propose AutoCap , a high- quality and efficient automatic audio captioning model. By using a compact audio representation and leveraging audio metadata, AutoCap substantially enhances caption quality, reaching CIDEr score of 83.2, marking a 3.2% improvement from the best available captioning model at four times faster inference speed. Second, we propose GenAu, a scalable transformer-based audio generation architecture that we scale up to 1.25B parameters. Using AutoCap to generate caption clips from existing audio datasets, we demon- strate the benefits of data scaling with synthetic captions as well as model size scaling. When compared to state-of-the-art audio generators trained at similar size and data scale, GenAu obtains significant improvements of 4.7% in FAD score, 22.7% in IS, and 13.5% in CLAP score, indicating significantly improved quality of generated audio compared to previous works. Moreover, we propose an efficient and scalable pipeline for collecting audio datasets, enabling us to compile 57M ambient audio clips, forming AutoReCap-XL, the largest available audio-text dataset, which is 100 times larger than existing ones. Our code, model checkpoints, and dataset will be made publicly available upon acceptance.

Left: An overview of the proposed architecture for our AutoCap model. Frozen CLAP and HTSAT audio encoders produce the audio representation. To reduce the large number of tokens produced by the HTSAT encoder, we use a Q-Former, reducing the amount of input tokens by a factor of 4. A pretrained BART encoder-decoder aggregates the tokens, producing the output caption.
Right: Overview of our GenAu model based on an FIT-based latent audio generator. A frozen 1D-VAE produces the latent audio representation. Input patches are divided into groups and processed by `local' attention layers. `read' and `write' operations implemented as cross attention layers transfer information between patches and latents. Finally, `global' attention layers process latent tokens with attention spanning over all groups, enabling global communication.

## Comparison of AutoCap with other audio captioning methods:

Here we compare our captioning method with ENCLAP and CoNeTTE. We show the original audio from the AudioCaps test set in the first column and as well as ground truth caption in the second. In the thrird column we show caption predicted by our method. Fourth and Fifth columns corresponds to the baselines. Our method consistently generates more descriptive and accurate captions. For instance, it is the only method that captures the water splashing in the first example and identifies both birds chirping and insects buzzing in the second. Our method generates audio clips with overall better quality, realism, and prompt alignment. It is the only method that captures all events in the first example and produces the most realistic audio for the second. Additionally, it is the only method to generate the horse 'growling' in the third example.

Input | Groundtruth Caption | Ours | ENCLAP | CoNeTTE
--- | --- | --- | --- | ---
Your browser does not support the audio element.
| A man talking as ocean waves trickle and splash while wind blows into a microphone | A man speaks as wind blows and water splashes | A man is speaking and wind is blowing | A man is speaking and wind is blowing
Your browser does not support the audio element.
| An adult male speaks, birds chirp in the background, and many insects are buzzing | Birds chirp in the distance, followed by a man speaking nearby, after which insects buzz nearby | Birds are chirping and a man speaks | A man speaking with birds chirping in the background.
Your browser does not support the audio element.
| A telephone dialing tone followed by a plastic switch flipping on and off | A telephone dialing followed by a series of plastic clicking then plastic clanking before plastic thumps on a surface | A telephone dialing followed by a series of electronic beeps | A telephone ringing followed by a beep.
Your browser does not support the audio element.
| A running train and then a train whistle | A train moves getting closer and a horn is triggered | A train running on railroad tracks followed by a train horn blowing as wind blows into a microphone | A train horn blows and a steam whistle is blowing
Your browser does not support the audio element.
| A female speaking with some rustling followed by another female speaking | Dishes are being moved and a woman laughs and speaks | A woman speaking followed by clanking | A woman is speaking and a child is laughing.
Your browser does not support the audio element.
| A child is speaking followed by a door moving | A child speaks followed by a loud crash and a scream | A young girl speaks followed by a loud bang | A woman speaking followed by a door opening and closing.
Your browser does not support the audio element.
| Water splashing as a baby is laughing and birds chirp in the background | A baby laughs and splashes, and an adult female speaks | A baby laughs and splashes in water | A baby is laughing and people are talking.
Your browser does not support the audio element.
| Leaves rustling in the wind with dogs barking and birds chirping | Birds chirp in the distance, and then a dog barks nearby | Birds chirp and a dog barks | A dog is barking and a person is walking.
Your browser does not support the audio element.
| Tapping followed by water spraying and more tapping | Some light rustling followed by a clank then water pouring | A faucet is turned on and runs | A toilet is flushed and water is running.

## Comparison of GenAu with text-to-audio generation methods:

We compare our method with state-of-the-art approaches on non-cherry-picked examples. The first row lists the method names, while the first column contains the input text used to generate the audio.

Input | Ours | Make-an-audio | AudioLDM | AudioLDM2 | Stable Audio | Tango
--- | --- | --- | --- | --- | --- | ---
A muffled man talking as a goat baas before and after two goats baaing in the distance while wind blows into a microphone
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
A small child and woman speak with splashing water
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
Horses growl and clop hooves.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
A woman speaks with chirping frogs and distant music playing
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
A vehicle driving by while splashing water as a stream of water trickles and flows followed by a thunder roaring in the distance while wind blows into a microphone
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
Large church bells ring as rain falls on a hard surface and wind blows lightly into a microphone
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
A man speaks with a high frequency hum with some banging and clanking
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.

## Ablating different variants of GenAu:

We compare different variants of our method. The top row shows the captions used for generation, followed by the second row, which displays a U-Net baseline. The next three rows present our small model trained on AudioCaps, on the full dataset without recaptioning, and on the fully recaptioned dataset. The final two rows show our large model trained on AudioCaps and the fully recaptioned dataset.

| A person briefly talks followed quickly by toilet flushing and another voice from another person | A gun cocking then firing as metal clanks on a hard surface followed by a man talking during an electronic laser effect as gunshots and explosions go off in the distance | Motorcycle starting and taking off | A female laughs, snoring occurs, and an adult male speaks in the background
--- | --- | --- | --- | ---
W/ U-Net
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
Small (AudioCaps)
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
Small (AutoReCap w/o Recaptioning)
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
Small (AutoReCap)
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
Large (AudioCaps)
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
Large (AutoReCap)
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.
| Your browser does not support the audio element.

## Random samples from AutoReCap-XL:

High resolution videos were excluded due to the size limitations.

Your browser does not support the video tag.
A motor vehicle engine is running and accelerates, and a dog barks
Your browser does not support the video tag.
A woman laughs and claps
Your browser does not support the video tag.
A car door is opened and closed
Your browser does not support the video tag.
A machine is making a series of rapid clicks and clicks
Your browser does not support the video tag.
A man yells followed by a gunshot
Your browser does not support the video tag.
A gun is fired several times
Your browser does not support the video tag.
Crickets chirp in the distance, followed by a gunshot
Your browser does not support the video tag.
Footsteps on a hard surface
Your browser does not support the video tag.
A race car passes by
Your browser does not support the video tag.
A police car driving by with sirens on
Your browser does not support the video tag.
Humming and vibrating of a power tool
Your browser does not support the video tag.
Birds chirp in the distance, followed by a snake hissing nearby

## Non-curated list of samples generated by GenAu:

Here we provide an non-curated list of samples generated with GenAU-L using AudioCaps test captions.

Input | Sample
--- | ---
A man speaks followed by a toilet flush
| Your browser does not support the audio element.
A man making a horn sound and then speaking
| Your browser does not support the audio element.
A woman talks and a baby whispers
| Your browser does not support the audio element.
A group of people laughing followed by farting
| Your browser does not support the audio element.
A woman talking followed by a group of people laughing as plastic crinkles
| Your browser does not support the audio element.
Sustained industrial engine noise
| Your browser does not support the audio element.
A vehicle engine revving as a crowd of people talk
| Your browser does not support the audio element.
A speedboat is racing across water with loud wind noise
| Your browser does not support the audio element.
A kid crying as a man and a woman talk followed by a car door opening then closing
| Your browser does not support the audio element.
A cat meowing and young female speaking
| Your browser does not support the audio element.
Police car siren starts with two horn blasts then becomes a high pitched wail
| Your browser does not support the audio element.
Rustling pigeons coo
| Your browser does not support the audio element.
A man and woman laughing followed by a man shouting then a woman laughing as a child laughs
| Your browser does not support the audio element.
A man talking while wood clanks on a metal pan followed by gravel crunching as food and oil sizzle
| Your browser does not support the audio element.
A motor is running, and metal clanging is present
| Your browser does not support the audio element.
A muffled man talking as a goat baas before and after two goats baaing in the distance while wind blows into a microphone
| Your browser does not support the audio element.
An adult male is speaking in a quiet environment
| Your browser does not support the audio element.
A small motor is buzzing and water is running, splashing and gurgling
| Your browser does not support the audio element.
A man talking followed by a brush scrapping then liquid spraying in the background
| Your browser does not support the audio element.
An idle vehicle engine running followed by a gear cranking then revving
| Your browser does not support the audio element.
A cat meows and hisses
| Your browser does not support the audio element.
Ocean waves crashing as a man talks in the distance and wind heavily blows into a microphone
| Your browser does not support the audio element.
A man speaking then a baby crying, duck quacking in background and finally a woman speaking
| Your browser does not support the audio element.
Some groaning followed by a woman speaking
| Your browser does not support the audio element.
A crowd murmurs as a siren blares and then stops at a distance
| Your browser does not support the audio element.
A large bell rings out multiple times
| Your browser does not support the audio element.
A cat meowing once with a thud
| Your browser does not support the audio element.
A drilling sound with humming in the background
| Your browser does not support the audio element.
A cat meowing twice
| Your browser does not support the audio element.
A rolling train blows its horn multiple times
| Your browser does not support the audio element.
A man speaks and a vehicle passes
| Your browser does not support the audio element.
A woman is speaking over a microphone
| Your browser does not support the audio element.
A vehicle engine revving then accelerating at a high rate as a metal surface is whipped followed by tires skidding
| Your browser does not support the audio element.
Instrumental music playing as a woman speaks followed by rain pouring then rain falling on a surface
| Your browser does not support the audio element.
A mid-size motor vehicle engine accelerates and is accompanied by hissing and spinning tires, then it decelerates and an adult male begins to speak
| Your browser does not support the audio element.
Humming of an accelerating engine with wind passing and rustling
| Your browser does not support the audio element.
Humming and vibrating of a power tool with some high frequency squealing
| Your browser does not support the audio element.
A car is passing by with leaves rustling
| Your browser does not support the audio element.
Train engine as it travels
| Your browser does not support the audio element.
A dog barks with distant birds chirping then people speak
| Your browser does not support the audio element.
Some child speaking in the distant and a toilet flushing
| Your browser does not support the audio element.
A man talking while bongos play followed by frogs croaking
| Your browser does not support the audio element.
A group of people talking in the background as compressed air sprays while a tin can rattles followed by a man talking
| Your browser does not support the audio element.
An adult female speaks in a quiet environment
| Your browser does not support the audio element.
A subway train signal plays followed by a bell chiming followed by a horn honking as a crowd of people talk in the background
| Your browser does not support the audio element.
Wind blowing and water splashing
| Your browser does not support the audio element.
Loud humming followed by hissing
| Your browser does not support the audio element.
Thuds on floor
| Your browser does not support the audio element.
Waves and wind rake a shore
| Your browser does not support the audio element.
Clicking and sputtering of a running engine with people speaking and wind blowing
| Your browser does not support the audio element.
Splashing water with children speaking and people screaming with a distant blow of a whistle
| Your browser does not support the audio element.
A man is speaking followed by a tap and motorcycle turning on
| Your browser does not support the audio element.
Metal shuffling followed by plastic clicking as wind blows into a microphone
| Your browser does not support the audio element.
A woman speaking
| Your browser does not support the audio element.
A stream of water trickling as plastic clanks against a metal surface followed by water pouring down a drain alongside a camera muffling
| Your browser does not support the audio element.
An adult male gives a speech
| Your browser does not support the audio element.
An aircraft engine running then slowing down after a plastic click
| Your browser does not support the audio element.
A sound of vibrating motor
| Your browser does not support the audio element.
Male speaking, laughter and shouting and clapping
| Your browser does not support the audio element.
A woman speaks briefly, and a muffled engine rumbles
| Your browser does not support the audio element.
Plastic clacking followed by as person breathing then liquid pouring into containers
| Your browser does not support the audio element.
A woman and a man talking as another man talks softly and papers shuffle in the background
| Your browser does not support the audio element.
Someone burps and then laughs
| Your browser does not support the audio element.
Chirping birds near and far
| Your browser does not support the audio element.
A dog barking as a man is talking while birds chirp and wind blows into a microphone
| Your browser does not support the audio element.
A sudden horn blare as a train passes
| Your browser does not support the audio element.
A man is speaking while typing
| Your browser does not support the audio element.
Motorcycle starting then driving away
| Your browser does not support the audio element.
A group of men speaking as cannons fire while rain falls and water splashes followed by thunder roaring
| Your browser does not support the audio element.
A man speaking followed by another man speaking with some rustling
| Your browser does not support the audio element.
Waves roll slowly and water swirls as the wind blows
| Your browser does not support the audio element.
A train sounds horn while traveling on train track
| Your browser does not support the audio element.
Water splashing and trickling as wind blows into a microphone while a man speaks over a radio
| Your browser does not support the audio element.
Some rustling then silence then traffic passing in the distance with a cat meowing
| Your browser does not support the audio element.
A vehicle motor running idle followed by a car horn honking then a group of men groaning
| Your browser does not support the audio element.
A car engine is revving while driving
| Your browser does not support the audio element.
Cats meowing and then wind
| Your browser does not support the audio element.
There is a mature male talking to some animals
| Your browser does not support the audio element.
A man speaking on a microphone as a crowd of people laugh followed by glass clinking
| Your browser does not support the audio element.
An engine and speech on a loudspeaker
| Your browser does not support the audio element.
An idle vehicle engine running normally before stuttering
| Your browser does not support the audio element.
An engine revving and then tires squealing
| Your browser does not support the audio element.
Chainsaw being run
| Your browser does not support the audio element.
A man speaks with low speech in the background
| Your browser does not support the audio element.
Race cars are racing followed by people talking
| Your browser does not support the audio element.
The sound of horn from a car approaching from a distance
| Your browser does not support the audio element.
A long burp ends in a sigh
| Your browser does not support the audio element.
A man speaking as vehicles drive by and leaves rustling
| Your browser does not support the audio element.
A chainsaw cutting as wood is cracking
| Your browser does not support the audio element.
A cat meows and a woman speaks
| Your browser does not support the audio element.
A gun cocking then firing as metal clanks on a hard surface followed by a man talking during an electronic laser effect as gunshots and explosions go off in the distance
| Your browser does not support the audio element.
A male voice and a machine buzzing
| Your browser does not support the audio element.
Speech followed by quietness and a man speaks and laughs
| Your browser does not support the audio element.
A man speaks with some clinking and clanking
| Your browser does not support the audio element.
A man speaking as rain lightly falls followed by thunder
| Your browser does not support the audio element.
A bell is ringing
| Your browser does not support the audio element.
A woman is speaking from a microphone
| Your browser does not support the audio element.
Wind is blowing and heavy rain is falling and splashing
| Your browser does not support the audio element.
Muffled sounds followed by metal being hit
| Your browser does not support the audio element.
A vehicle driving as a man and woman are talking and laughing
| Your browser does not support the audio element.