Apples and Oranges: An International Comparison of the Public's Experience of Justiciable Problems and the Methodological Issues Affecting Comparative Study

Abstract

Since the mid‐1990s, at least 28 large‐scale national surveys of the public's experience of justiciable problems have been conducted in at least 15 separate jurisdictions, reflecting widespread legal aid reform activity. While the majority of these surveys take their structure from Genn's Paths to Justice survey (1999), they vary significantly in length, scope, mode of administration, types of problems included, survey reference period, data structure, data analysis, and question formulation. This article draws on surveys from across the world, contrasting their methodologies, comparing their headline findings, and setting out the potential for bias as a consequence of methodological variation. The article also presents findings from five online experiments testing the impact of various question formulations on problem prevalence, use of advice, and formal processes. Specifically, the experiments test whether varying the reference period, describing problems as “legal,” offering detailed as opposed to simple problem descriptions, and describing problems as “difficult to solve” had an impact on reported prevalence of justiciable problems, and whether presenting lists as opposed to a series of individual questions had an impact on reported use of advice and processes. The experiments demonstrated that modest differences in question formulation yield significantly different results. Specifically, alteration of survey reference period did not result in a proportional change in reported problem prevalence, introducing problems as either “legal” or “difficult to solve” significantly reduced reported prevalence, and introducing use of advice/processes as multiple questions rather than as lists significantly increased reported use. The risks involved in comparative analysis (and particularly in looking beyond methodology when attempting to explain jurisdictional variation) are discussed. In relation to future studies, the importance of understanding the impact of methodological change, learning the lessons of the past, making technical details transparent, and making data available are highlighted.

Moral communication and legal uncertainty in small and medium enterprises

“…These findings bear a striking resemblance to the plethora of studies on individuals' use of the law (See only Pleasence et al , 2016). Still, in-depth investigations into subaltern legal phenomena, legal uncertainty included, in small business are few and far between.…”

“…To address methodological issues and to capture the diversity of problems and responses, we used a reliable and paradigmatic method of carrying out empirical research in the sociology of law and empirical legal studies, which is the paths to justice (P2J) questionnaire (Genn, 1999; Pleasence et al , 2013, 2016). It consists of asking respondents questions about the experienced “life” problems, and if they were legal, in asking a series of nested detailing questions, allowing to determine the features of such problems and the types of remedial actions taken.…”

Legal Needs Surveys and Access to Justice

“…14 Neglecting areas of memory will reduce problem reporting, affecting accuracy and potentially limiting the prospects of statistical analysis. In the specific context of legal needs surveys, experimental evidence indicates that more detailed questions result in increased reporting of problems, although the experiments were not conclusive, and the impact of varying levels of detail differed depending on the problem type (Pleasence et al, 2016).…”

“…31 Although the impact of even small methodological differences on results precludes definitive incidence rates being determined across or within individual jurisdictions (Pleasence et al 2016). Also see Chapters 2 and 3.…”

“…Experimental evidence shows that asking about only "difficult to solve" problems can decrease problem reporting by around 30% (Pleasence et al, 2016). This hardly constitutes preventing "the floodgates opening" and is hugely problematic in conceptual terms; as it conflates problem occurrence, problem resolving strategy and legal capability.…”