Diff-TTSG

Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesis

We present Diff-TTSG, the first diffusion model that jointly learns to synthesise speech and gestures together. Our method is probabilistic and non-autoregressive, and can be trained on small datasets from scratch. In addition, to showcase the efficacy of these systems and pave the way for their evaluation, we describe a set of careful uni- and multi-modal subjective tests for evaluating integrated speech and gesture synthesis systems.

Stimuli from the evaluation test

Speech-only evaluation

You walk around Dublin city centre and even if you try and strike up a conversation with somebody it’s impossible because everyone has their headphones in. And again, I would listen to podcasts sometimes with my headphones in walking around the streets.

NAT	Diff-TTSG	T2-ISG	Grad-TTS

And then a few weeks later after that my parents were away my granny was minding us and again I don’t know why I told my brother to do this but I was like here.

NAT	Diff-TTSG	T2-ISG	Grad-TTS

But I remember once my parents were just downstairs in the kitchen and this is when mobile phones just began coming out. So, like my oldest brother and my oldest sister had a mobile phone each I’m pretty sure.

NAT	Diff-TTSG	T2-ISG	Grad-TTS

Eventually got to a point where I was like okay I need to stop doing this sort of stuff Like it just doesn’t make any sense as to why because I was getting hurt like there was times where like, I was like tearing muscles and I never broke a bone which I’m pretty proud of.

NAT	Diff-TTSG	T2-ISG	Grad-TTS

Gesture-only evaluation (no audio)

Currently loaded: Diff-TTSG 1

If you like touched it, it was excruciatingly sore. And I went up to the teachers I was like look I'm after like really damaging my finger I might have to go to the doctors.

Text prompt #	NAT	Diff-TTSG	T2-ISG	[Grad-TTS]+M
1
2
3
4

Speech-and-gesture evaluation

Matched	Mismatched

*Note: Matched versus mismatched stimuli were not labelled in the study and presented in random order.

Currently loaded: Diff-TTSG 1

Yeah and then obviously there, there's certain choirs that come down to the church. There's a woman called, I can't remember her name. But she has an incredible voice. Like an amazing voice.

Text prompt #	NAT	Diff-TTSG	T2-ISG
1
2
3
4

Additional examples from proposed system (Diff-TTSG)

Beat gestures

Currently loaded: Example 1

And the train stopped, The door opened. I got out first, then Jack Kane got out, Ronan got out, Louise got out.

Positive-negative emotional pairs

Positive emotion	Negative emotion

Currently loaded: Example 1

Positive: I went to a comedy show last night, and it was absolutely hilarious. The jokes were fresh and clever, and I laughed so hard my sides hurt.
Negative: I tried meditating to relieve stress, but it just made me feel more anxious. I couldn't stop thinking about all the things I needed to do, and it felt like a waste of time.

Importance of the diffusion model

To illustrate the importance of using diffusion in modelling both speech and motion, these stimuli compare synthesis from condition D-TTSG to synthesis directly from the μ values predicted by the D-TTSG decoder and Conformer.

μ (before diffusion)	Final output (after diffusion)

Currently loaded: Example 1

Positive: I went to a comedy show last night, and it was absolutely hilarious. The jokes were fresh and clever, and I laughed so hard my sides hurt.
Negative: I tried meditating to relieve stress, but it just made me feel more anxious. I couldn't stop thinking about all the things I needed to do, and it felt like a waste of time.