As two of the five traditional human senses (sight, hearing, taste, smell, and touch), vision and sound are basic sources through which humans understand the world. Often correlated during natural events, these two modalities combine to jointly affect human perception. In this paper, we pose the task of generating sound given visual input. Such capabilities could help enable applications in virtual reality (generating sound for virtual scenes automatically) or provide additional accessibility to images or videos for people with visual impairments. As a first step in this direction, we apply learning-based methods to generate raw waveform samples given input video frames. We evaluate our models on a dataset of videos containing a variety of sounds (such as ambient sounds and sounds from people/animals). Our experiments show that the generated sounds are fairly realistic and have good temporal synchronization with the visual inputs.

Abstract. Based on life-long observations of physical, chemical, and biologic phenomena in the natural world, humans can often easily picture in their minds what an object will look like in the future. But, what about computers? In this paper, we learn computational models of object transformations from time-lapse videos. In particular, we explore the use of generative models to create depictions of objects at future times. These models explore several different prediction tasks: generating a future state given a single depiction of an object, generating a future state given two depictions of an object at different times, and generating future states recursively in a recurrent framework. We provide both qualitative and quantitative evaluations of the generated results, and also conduct a human evaluation to compare variations of our models.

Given a video of an activity, can we predict what will happen next? In this paper we explore two simple tasks related to temporal prediction in egocentric videos of everyday activities. We provide both human experiments to understand how well people can perform on these tasks and computational models for prediction. Experiments indicate that humans and computers can do well on temporal prediction and that personalization to a particular individual or environment provides significantly increased performance. Developing methods for temporal prediction could have far reaching benefits for robots or intelligent agents to anticipate what a person will do, before they do it.

The guided mode resonance (GMR) effect is widely used in biosensing due to its advantages of narrow linewidth and high efficiency. However, the optimization of a figure of merit (FOM) has not been considered for most GMR sensors. Aimed at obtaining a higher FOM of GMR sensors, we proposed an effective design method for the optimization of FOM. Combining the analytical model and numerical simulations, the FOM of “grating–waveguide” GMR sensors for the wavelength and angular shift detection schemes were investigated systematically. In contrast with previously reported values, higher FOM values were obtained using this method. For the “waveguide–grating” GMR sensors, a linear relationship between the grating period and groove depth was obtained, which leads to excellent FOM values for both the angular and wavelength resonance. Such higher performance GMR sensors will pave the way to lower detection limits in biosensing.

No abstract