Onoma-to-wave: Environmental Sound Synthesis from Onomatopoeic Words

Okamoto, Yuki; Imoto, Keisuke; Takamichi, Shinnosuke; Yamanishi, Ryosuke; Fukumori, Takahiro; Yamashita, Yoichi

doi:10.1561/116.00000049

Cited by 10 publications

(7 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The sound event feature was added to the output of the variance adaptor via a linear layer. Other configurations, e.g., model configuration and training schedule, followed the open-sourced implementation of vTTS [15] 6 . Original onoma-to-wave paper [6] uses a Tacotron-inspired synthesis model [24], but in this paper, we used FastSpeech2-inspired model for both conventional onoma-to-wave and the proposed visual onoma-to-wave for fair comparison.…”

Section: Methodsmentioning

confidence: 99%

“…There are two types of methods for textbased control: describing all attributes of the sound in a sentence and separately describing the overall impression and detailed structure. Yang's method [4] is of the former, and onoma-to-wave [6], is of the latter. Since onomatopoeias efficiently imitate temporal changes of the sound, they are considered straightforward to control the detailed structure of the sound [11], [12].…”

Section: Sound Synthesis From Discrete Symbolsmentioning

confidence: 99%

“…Following the onoma-to-wave paper [6], we conducted fivescale mean opinion score (MOS) evaluations on three criteria: acceptance of synthetic sounds relative to onomatopoeias, expressiveness of the sounds relative to onomatopoeia, and naturalness of the sounds. Native Japanese speakers recruited through crowdsourcing participated in this evaluation.…”

Section: Evaluationsmentioning

confidence: 99%

“…This method synthesizes a sound that appropriates the given label but cannot control the detailed structure of the sound, e.g., duration and temporal changes in tone. Another method is synthesis from a text [4]- [6]. For synthesizing sounds, onoma-to-wave [6] (Figure 1 (a)) uses an onomatopoeia, a word imitating the sound (i.e., the text representation of sounds).…”

mentioning

confidence: 99%

“…Another method is synthesis from a text [4]- [6]. For synthesizing sounds, onoma-to-wave [6] (Figure 1 (a)) uses an onomatopoeia, a word imitating the sound (i.e., the text representation of sounds). This method can control the detailed structure of the synthesized sound by the input onomatopoeia text.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Visual onoma-to-wave: environmental sound synthesis from visual onomatopoeias and sound-source images

Ohnaka¹,

Takamichi²,

Imoto³

et al. 2022

Preprint

View full text Add to dashboard Cite

We propose a method for synthesizing environmental sounds from visually represented onomatopoeias and sound sources. An onomatopoeia is a word that imitates a sound structure, i.e., the text representation of sound. From this perspective, onoma-to-wave has been proposed to synthesize environmental sounds from the desired onomatopoeia texts. Onomatopoeias have another representation: visual-text representations of sounds in comics, advertisements, and virtual reality. A visual onomatopoeia (visual text of onomatopoeia) contains rich information that is not present in the text, such as a long-short duration of the image, so the use of this representation is expected to synthesize diverse sounds. Therefore, we propose visual onoma-to-wave for environmental sound synthesis from visual onomatopoeia. The method can transfer visual concepts of the visual text and sound-source image to the synthesized sound. We also propose a data augmentation method focusing on the repetition of onomatopoeias to enhance the performance of our method. An experimental evaluation shows that the methods can synthesize diverse environmental sounds from visual text and sound-source images.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Sound Synthesis From Discrete Symbolsmentioning

confidence: 99%

Section: Evaluationsmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations