A Quantitative Content Analysis of Survey Research Methods Over a 40-Year Time Period in the Journal of Agricultural Education

Abstract

To provide a benchmark of survey methods used in agricultural education research, a content analysis of Journal of Agricultural Education (JAE) research articles was conducted to determine populations studied, survey contact and response mode use, the use of incentives, and response and participation rates reported. Findings revealed survey research is still the most commonly used research methodology for articles published in the JAE. School-based agricultural education (SBAE) teachers are the population most frequently surveyed with contact and response modes shifting from mailed paper surveys to online surveys. Incentives were rarely used in the sampled articles. Response and participation rates have declined 20 -30% from 1979 to 2019, increasing the possibility for nonresponse error. Recommendations for practice include providing more specific information in manuscripts related to tailoring survey methods for a particular population and reporting participation and response rates more accurately. Further research was recommended for determining which survey modes are most successful for populations commonly studied in agricultural education research, conducting a cost analysis of using incentives, and exploring new technologies to reach populations via research.

Determining the Effects of Response Mode and Incentives on Survey Response Rates of School-based Agricultural Education Teachers: An Experimental Study

“…Attitude toward surveys, questionnaire appearance and content, number of contacts made, providing incentives, and data collection mode used are all major categories influencing response rates and within each of these categories there are multiple variables at work (Ary et al, 2014;Fraenkel et al, 2019;James & Bolstein, 1990;Leeper, 2019;Mertler & Charles, 2011;Rogelberg et al, 2001;Ye, 2007). To narrow down the variables of interest, we chose to study the use of incentives and survey response modes because of their ability to have larger impacts on survey response rates and because of how they are currently used in agricultural education research (Dillman et al, 2014;Doss et al, 2021).…”

“…Further research should be conducted on the effects of contact mode used to improve response rates rather than response mode. Mixed contact modes were used across all experimental groups; however, this is not really used often in agricultural education research (Doss et al, 2021). Communication was the largest expense in this study, even above incentives.…”

“…Agricultural education is by nature an applied, social science discipline where scholars use survey research methodologies most frequently when studying populations (Doss et al, 2021;Dyer et al, 2003). However, survey response rates are declining in agricultural education research, possibly affecting the strength of the most frequently used methodology in the discipline.…”

“…Increasing the number of contacts with participants, providing incentives, and survey mode selected can have the greatest effect on survey response rates (Ary et al, 2014;Dillman et al, 2014;Fraenkel et al, 2019). In agricultural education survey research, it is common to contact participants multiple times; however, few studies use an incentive nor use a survey mode other than web-based (Doss et al, 2021). This raises the question: do these current practices achieve the best possible response rate?…”