This position paper describes our approach to recognizing tasks of mobile users in a smart environment. Such high-level interpretation of behavior enables context-aware applications to adapt to the users' needs and intentions. In the AI community, plan recognition techniques have proven their applicability in recognizing the tasks of software agents in a controlled environment. However, these approaches fall short in recognizing tasks of people in a real-world environment. Therefore, we propose several extensions to plan recognition techniques by using constraints and hybrid reasoning algorithms. In addition, we propose to improve the plan recognition process with multi-step processing of context information. We also discuss how our approach leverages some of the difficulties of plan recognition in smart environments.