Video summarization provides a condensed or summarized version of a video stream by analyzing the video content. Automatic summarization of consumer videos is an important tool that facilitates efficient browsing, searching, and album creation in the large amount of consumer video collections. This paper studies automatic video summarization in the consumer domain where most previous methods can not be easily applied due to the challenging issues for content analysis, i.e., the consumer videos are captured with uncontrolled conditions such as uneven lighting, clutter, and large camera motion, and with poor-quality sound track as a mix of multiple sound sources under severe noises. To pursue reliable summarization, a case study with real consumer users is conducted, from which a set of consumer-oriented guidelines are obtained. The guidelines reflect the high-level semantic rules, in both visual and audio aspects, which are recognized by consumers as important to produce good video summaries. Following these guidelines, an automatic video summarization algorithm is developed where both visual and audio information are used to generate improved summaries. The experimental evaluations from consumer raters show the effectiveness of our approach.