BackgroundMany promising technological innovations in health and social care are characterized by nonadoption or abandonment by individuals or by failed attempts to scale up locally, spread distantly, or sustain the innovation long term at the organization or system level.ObjectiveOur objective was to produce an evidence-based, theory-informed, and pragmatic framework to help predict and evaluate the success of a technology-supported health or social care program.MethodsThe study had 2 parallel components: (1) secondary research (hermeneutic systematic review) to identify key domains, and (2) empirical case studies of technology implementation to explore, test, and refine these domains. We studied 6 technology-supported programs—video outpatient consultations, global positioning system tracking for cognitive impairment, pendant alarm services, remote biomarker monitoring for heart failure, care organizing software, and integrated case management via data sharing—using longitudinal ethnography and action research for up to 3 years across more than 20 organizations. Data were collected at micro level (individual technology users), meso level (organizational processes and systems), and macro level (national policy and wider context). Analysis and synthesis was aided by sociotechnically informed theories of individual, organizational, and system change. The draft framework was shared with colleagues who were introducing or evaluating other technology-supported health or care programs and refined in response to feedback.ResultsThe literature review identified 28 previous technology implementation frameworks, of which 14 had taken a dynamic systems approach (including 2 integrative reviews of previous work). Our empirical dataset consisted of over 400 hours of ethnographic observation, 165 semistructured interviews, and 200 documents. The final nonadoption, abandonment, scale-up, spread, and sustainability (NASSS) framework included questions in 7 domains: the condition or illness, the technology, the value proposition, the adopter system (comprising professional staff, patient, and lay caregivers), the organization(s), the wider (institutional and societal) context, and the interaction and mutual adaptation between all these domains over time. Our empirical case studies raised a variety of challenges across all 7 domains, each classified as simple (straightforward, predictable, few components), complicated (multiple interacting components or issues), or complex (dynamic, unpredictable, not easily disaggregated into constituent components). Programs characterized by complicatedness proved difficult but not impossible to implement. Those characterized by complexity in multiple NASSS domains rarely, if ever, became mainstreamed. The framework showed promise when applied (both prospectively and retrospectively) to other programs.ConclusionsSubject to further empirical testing, NASSS could be applied across a range of technological innovations in health and social care. It has several potential uses: (1) to inform the desi...

Background Numerous frameworks for supporting, evaluating and reporting patient and public involvement in research exist. The literature is diverse and theoretically heterogeneous. Objectives To identify and synthesize published frameworks, consider whether and how these have been used, and apply design principles to improve usability. Search strategy Keyword search of six databases; hand search of eight journals; ancestry and snowball search; requests to experts. Inclusion criteria Published, systematic approaches (frameworks) designed to support, evaluate or report on patient or public involvement in health‐related research. Data extraction and synthesis Data were extracted on provenance; collaborators and sponsors; theoretical basis; lay input; intended user(s) and use(s); topics covered; examples of use; critiques; and updates. We used the Canadian Centre for Excellence on Partnerships with Patients and Public (CEPPP) evaluation tool and hermeneutic methodology to grade and synthesize the frameworks. In five co‐design workshops, we tested evidence‐based resources based on the review findings. Results Our final data set consisted of 65 frameworks, most of which scored highly on the CEPPP tool. They had different provenances, intended purposes, strengths and limitations. We grouped them into five categories: power‐focused; priority‐setting; study‐focused; report‐focused; and partnership‐focused. Frameworks were used mainly by the groups who developed them. The empirical component of our study generated a structured format and evidence‐based facilitator notes for a “build your own framework” co‐design workshop. Conclusion The plethora of frameworks combined with evidence of limited transferability suggests that a single, off‐the‐shelf framework may be less useful than a menu of evidence‐based resources which stakeholders can use to co‐design their own frameworks.

BackgroundThe 2014 UK Research Excellence Framework (REF2014) generated a unique database of impact case studies, each describing a body of research and impact beyond academia. We sought to explore the nature and mechanism of impact in a sample of these.MethodsThe study design was manual content analysis of a large sample of impact case studies (producing mainly quantitative data), plus in-depth interpretive analysis of a smaller sub-sample (for qualitative detail), thereby generating both breadth and depth. For all 162 impact case studies submitted to sub-panel A2 in REF2014, we extracted data on study design(s), stated impacts and audiences, mechanisms of impact, and efforts to achieve impact. We analysed four case studies (selected as exemplars of the range of approaches to impact) in depth, including contacting the authors for their narratives of impact efforts.ResultsMost impact case studies described quantitative research (most commonly, trials) and depicted a direct, linear link between research and impact. Research was said to have influenced a guideline in 122 case studies, changed policy in 88, changed practice in 84, improved morbidity in 44 and reduced mortality in 25. Qualitative and participatory research designs were rare, and only one case study described a co-production model of impact. Eighty-two case studies described strong and ongoing linkages with policymakers, but only 38 described targeted knowledge translation activities. In 40 case studies, no active efforts to achieve impact were described. Models of good implementation practice were characterised by an ethical commitment by researchers, strong institutional support and a proactive, interdisciplinary approach to impact activities.ConclusionREF2014 both inspired and documented significant efforts by UK researchers to achieve impact. But in contrast with the published evidence on research impact (which depicts much as occurring indirectly through non-linear mechanisms), this sub-panel seems to have captured mainly direct and relatively short-term impacts one step removed from patient outcomes. Limited impacts on morbidity and mortality, and researchers’ relatively low emphasis on the processes and interactions through which indirect impacts may occur, are concerns. These findings have implications for multi-stakeholder research collaborations such as UK National Institute for Health Research Collaborations for Leadership in Applied Health Research and Care, which are built on non-linear models of impact.

Background Showing how engagement adds value for all stakeholders can be an effective motivator for broader implementation of patient engagement. However, it is unclear what methods can best be used to evaluate patient engagement. This paper is focused on ways to evaluate patient engagement at three decision‐making points in the medicines research and development process: research priority setting, clinical trial design and early dialogues with regulators and health technology assessment bodies. Objective Our aim was to review the literature on monitoring and evaluation of patient engagement, with a focus on indicators and methods. Search strategy and inclusion criteria We undertook a scoping literature review using a systematic search, including academic and grey literature with a focus on evaluation approaches or outcomes associated with patient engagement. No date limits were applied other than a cut‐off of publications after July 2018. Data extraction and synthesis Data were extracted from 91 publications, coded and thematically analysed. Main results A total of 18 benefits and 5 costs of patient engagement were identified, mapped with 28 possible indicators for their evaluation. Several quantitative and qualitative methods were found for the evaluation of benefits and costs of patient engagement. Discussion and conclusions Currently available indicators and methods are of some use in measuring impact but are not sufficient to understand the pathway to impact, nor whether interaction between researchers and patients leads to change. We suggest that the impacts of patient engagement can best be determined not by applying single indicators, but a coherent set of measures.

IntroductionResearch to date into assisted living technologies broadly consists of 3 generations: technical design, experimental trials and qualitative studies of the patient experience. We describe a fourth-generation paradigm: studies of assisted living technologies in their organisational, social, political and policy context. Fourth-generation studies are necessarily organic and emergent; they view technology as part of a dynamic, networked and potentially unstable system. They use co-design methods to generate and stabilise local solutions, taking account of context.Methods and analysisSCALS (Studies in Co-creating Assisted Living Solutions) consists (currently) of 5 organisational case studies, each an English health or social care organisation striving to introduce technology-supported services to support independent living in people with health and/or social care needs. Treating these cases as complex systems, we seek to explore interdependencies, emergence and conflict. We employ a co-design approach informed by the principles of action research to help participating organisations establish, refine and evaluate their service. To that end, we are conducting in-depth ethnographic studies of people's experience of assisted living technologies (micro level), embedded in evolving organisational case studies that use interviews, ethnography and document analysis (meso level), and exploring the wider national and international context for assisted living technologies and policy (macro level). Data will be analysed using a sociotechnical framework developed from structuration theory.Ethics and disseminationResearch ethics approval for the first 4 case studies has been granted. An important outcome will be lessons learned from individual co-design case studies. We will document the studies’ credibility and rigour, and assess the transferability of findings to other settings while also recognising unique aspects of the contexts in which they were generated. Academic outputs will include a cross-case analysis and progress in theory and method of fourth-generation assisted living technology research. We will produce practical guidance for organisations, policymakers, designers and service users.