Digital applications typically describe their privacy policy in lengthy and vague documents (called PrPs), but these are rarely read by users, who remain unaware of privacy risks associated with the use of these digital applications. Thus, users need to become more aware of digital applications' policies and, thus, more confident about their choices. To raise privacy awareness, we implemented the CAP-A portal, a crowdsourcing platform which aggregates knowledge as extracted from PrP documents and motivates users in performing privacy-related tasks. The Rewarding Framework is one of the most critical components of the platform. It enhances user motivation and engagement by combining features from existing successful rewarding theories. In this work, we describe this Rewarding Framework, and show how it supports users to increase their privacy knowledge level by engaging them to perform privacy-related tasks, such as annotating PrP documents in a crowdsourcing environment. The proposed Rewarding Framework was validated by pilots ran in the frame of the European project CAP-A and by a user evaluation focused on its impact in terms of engagement and raising privacy awareness. The results show that the Rewarding Framework improves engagement and motivation, and increases users' privacy awareness.