Abstract-The growing number of videos on the internet requires effective management strategies. Video summarization is a method for managing video data which provides succinct versions of the videos for efficient browsing and retrieval. The inter-frame disparity based key frame extraction is a popular scheme for summarizing videos. However, the performance of such schemes is limited by the fact that most of these techniques select the first frame in a shot as key frame which may not be representative of the shot. In this paper, we propose a saliency inspired inter-frame difference based summarization scheme which selects the most important frame of the shot based on color contrast saliency. The preliminary experimental results prove the efficacy of the proposed method.Index Terms-Video summarization, key frame extraction, image saliency, visual attention model, frame difference measure.