Policy mix analysis has been applied in research on energy, climate, urban and transport policy, and more recently biodiversity conservation and ecosystem services. However, policy mix analysis has thus far been employed at a high conceptual level, focusing on describing interactions between instrument types. Policy mix analysis rarely describes instrument ‘structure’ or functional characteristics in a way that would answer the question ‘what constitutes an instrument’? We describe how the rules‐in‐use taxonomy of the Institutional Analysis and Development (IAD) framework, developed for research on common pool resource management, can be used to characterize conservation policy instrument interactions. We demonstrate the approach on the well‐known payments for ecosystem services (PES) program in Costa Rica and cross‐compliance policies, arguing that PES is a policy mix rather than a single economic instrument. Our analysis shows how design features of PES described in the economics literature map to ‘rules‐in‐use’ in the IAD framework. The framework provides a terminology for defining what constitutes institutional context, comparing economic, regulatory and information instruments, and studying their interactions. The rules‐in‐use taxonomy of IAD is a ‘structural’ diagnostic approach, which needs to be combined with other tools that analyse the role and ‘agency’ of actors, as part of integrative environmental governance research. Copyright © 2017 John Wiley & Sons, Ltd and ERP Environment