Self-content-based audio inpainting

Bahat, Yuval; Schechner, Yoav Y.; Elad, Michael

doi:10.1016/j.sigpro.2014.11.023

Cited by 31 publications

(36 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sparse approximation in the time-frequency domain has been explored in [1,43], but silence will be introduced when gap exceeds 50ms. Self-similarity has been employed to inpaint gaps up to 0.25 seconds using time-evolving features [7]. Recently, using similarity graphs, [39] proposes to inpaint long music segments, but it cannot handle segments shorter than 3 seconds.…”

Section: Related Workmentioning

confidence: 99%

Vision-Infused Deep Audio Inpainting

Zhou

Liu

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Multi-modality perception is essential to develop interactive intelligence. In this work, we consider a new task of visual information-infused audio inpainting, i.e. synthesizing missing audio segments that correspond to their accompanying videos. We identify two key aspects for a successful inpainter: (1) It is desirable to operate on spectrograms instead of raw audios. Recent advances in deep semantic image inpainting could be leveraged to go beyond the limitations of traditional audio inpainting.(2) To synthesize visually indicated audio, a visualaudio joint feature space needs to be learned with synchronization of audio and video. To facilitate a largescale study, we collect a new multi-modality instrumentplaying dataset called MUSIC-Extra-Solo (MUSICES) by enriching MUSIC dataset [54]. Extensive experiments demonstrate that our framework is capable of inpainting realistic and varying audio segments with or without visual contexts. More importantly, our synthesized audio segments are coherent with their video counterparts, showing the effectiveness of our proposed Vision-Infused Audio Inpainter (VIAI). Code, models, dataset and video results are available at https://hangz-nju-cuhk. github.io/projects/AudioInpainting.

show abstract

Section: Related Workmentioning

confidence: 99%

Vision-Infused Deep Audio Inpainting

Zhou

Liu

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

show abstract

“…Its main purpose is to make the restored image meet the requirements of human vision, so that people who are not familiar with the original image cannot notice the restoration trace [17,33]. With the rapid development of computer and multimedia technology, the inpainting technology has been widely used in many fields [20], such as scratches restoration of old photos and precious literatures, protection of cultural relics [8,18], robot vision, film and television special effects production, and so on [3,37].…”

Section: Introductionmentioning

confidence: 99%

An image inpainting method for object removal based on difference degree constraint

Zhang

Chang

2020

Multimed Tools Appl

View full text Add to dashboard Cite

In the inpainting method for object removal, SSD (Sum of Squared Differences) is commonly used to measure the degree of similarity between the exemplar patch and the target patch, which has a very important impact on the restoration results. Although the matching rule is relatively simple, it is likely to lead to the occurrence of mismatch error. Even worse, the error may be accumulated along with the process continues. Finally some unexpected objects may be introduced into the target region, making the result unable to meet the requirements of visual consistency. In view of these problems, we propose an inpainting method for object removal based on difference degree constraint. Firstly, we define the MSD (Mean of Squared Differences) and use it to measure the degree of differences between corresponding pixels at known positions in the target patch and the exemplar patch. Secondly, we define the SMD (Square of Mean Differences) and use it to measure the degree of differences between the pixels at known positions in the target patch and the pixels at unknown positions in the exemplar patch. Thirdly, based on MSD and SMD, we define a new matching rule and use it to find the most similar exemplar patch in the source region. Finally, we use the exemplar patch to restore the target patch. Experimental results show that the proposed method can effectively prevent the occurrence of mismatch error and improve the restoration effect.

show abstract

“…Another interesting technique is to exploit both, past and future frames present in the jitter buffer such as using hidden Markov models (HMM) for estimating lost frame parameters is presented in [20]; or using audio inpainting to reconstruct missing parts of audio signal basing on the good received parts, in condition that the duration of missing parts is lower than 50ms [21]. Sparsity based techniques [22] or, based on self-content [23]. In this paper we propose an algorithm that exploits the bit representation of G.722.2 coded frames to insert a part of the previous frames to recover them in case of their lost.…”

Section: Introductionmentioning

confidence: 99%

An Enhanced Interleaving Frame Loss Concealment Method for Voice Over IP Network Services

Gueham

Merazka

2018

2018 26th European Signal Processing Conference (EUSIPCO)

View full text Add to dashboard Cite

This paper focuses on AMR WB G.722.2 speech codec, and discusses the unused bandwidth resources of the senders by using a Word16(16 bit) to encode the sent frames. A packet loss concealment (PLC) method for G.722.2 speech codec is proposed in order to overcome this problem and increases the efficiency of this codec by improving the quality of decoded speech under burst frame loss conditions over frame-switched networks. Objective and subjective experimental results confirm that our proposed algorithm could achieve better speech quality. Our proposed method achieves a PESQ value higher than 2 at 20% frame erasure rate and ensure the compatibility between our modified decoder and the non-modified G.722.2 coder.

show abstract

Self-content-based audio inpainting

Cited by 31 publications

References 21 publications

Vision-Infused Deep Audio Inpainting

Vision-Infused Deep Audio Inpainting

An image inpainting method for object removal based on difference degree constraint

An Enhanced Interleaving Frame Loss Concealment Method for Voice Over IP Network Services

Contact Info

Product

Resources

About