With technology at the fingertips of most undergraduate students, it has been difficult for instructors to fully engage students in the classroom, which has resulted in the creation of several innovative assessment platforms, such as mTuner. mTuner integrates several cognitive learning strategies within an assessment, with the goal of actually enhancing learning, as opposed to just measuring it. In the current study, students’ level of engagement and performance with mTuner were compared to their final multiple-choice paper and pencil exam mark to determine the efficacy of mTuner in achieving improved learning outcomes. Results indicated that students had high performance scores on the mTuner assessment despite their limited engagement in the cognitive learning features, putting to question mTuner’s facilitation of long-term learning. Implications and future recommendations of mTuner implementation in educational environments are discussed.