Making the Invisible Visible: Guidelines for the Coding Process in Meta-Analyses

Abstract

This article contributes to the practice of coding in meta-analyses by offering direction and advice for experienced and novice meta-analysts on the “how” of coding. The coding process, the invisible architecture of any meta-analysis, has received comparably little attention in methodological resources, leaving the research community with insufficient guidance on “how” it should be rigorously planned (i.e., cohere with the research objective), conducted (i.e., make reliable and valid coding decisions), and reported (i.e., in a sufficiently transparent manner for readers to comprehend the authors’ decision-making). A lack of rigor in these areas can lead to erroneous results, which is problematic for entire research communities who build their future knowledge upon meta-analyses. Along four steps, the guidelines presented here elucidate “how” the coding process can be performed in a coherent, efficient, and credible manner that enables connectivity with future research, thereby enhancing the reliability and validity of meta-analytic findings. Our recommendations also support editors and reviewers in advising authors on how to improve the rigor of their coding and ultimately establish higher quality standards in meta-analytic research.

Designing Categories for a Mixed-Method Research on Competence Development and Professional Identity through Collegial Advice in Nursing Education in Germany

“…In this way, the coding result becomes highly independent of subjective impressions of the coders. Due to the elaborate and small-step documented process, the category system and the underlying descriptions appear reliable and valid, oriented to Villiger, Schweiger and Baldauf [ 29 ]. The measure of reliability, in this case intercoder reliability, is represented by the determination of the Kappa [ 30 ].…”

“…She is experienced in qualitative research and comes from a complementary discipline. Therefore she matches the criteria of a second coder [ 29 ]. She was given various materials (e.g., research questions, interview guide, transcript G1).…”

The fintech transformation of banking: Governance dynamics and socio-economic outcomes in spatial contexts

“…According to the framework (Figure 1) that adopts recognized theories with their established concepts, we applied selective (closed) and deductive (theory driven) coding (Villiger et al, 2021;Tranfield, Denyer, and Smart, 2003). To code the types of the major constructs, we used the terms included in Figure 1 and explained earlier in the Literature Review and Theory Development (Hoon, 2013).…”

“…For instance, the types indicated for governance inform whether fintech mechanisms transform it to market or network or hierarchy. These coding terms were used as keywords for the search within the final sample of articles (Villiger et al, 2021). In the absence of particular code terms, we inferred and classified the types of the major constructs based on their descriptions, such as hierarchical governance inferred from power relations being top-down and dominated by particular entities.…”

Review Research as Scientific Inquiry

“…(2) Execution Execution is about demonstrating operational competence in collecting the studies for and then conducting the review research, especially in terms of appropriate collection of studies and proficient use of appropriate analytic techniques (e.g., meta-analysis). For example, Villiger et al (2022) contend that coding in a review "is susceptible to high degrees of subjective decision-making, […] the same rigor called for during data collection for primary research is likewise appropriate for the coding process in meta-analyses" (p. 730). The qualities of being thorough, transparent, replicable, and logical contribute directly to this outcome (Tranfield et al, 2003).…”