Reliability between online raters with varying familiarities of a region: Microscale Audit of Pedestrian Streetscapes (MAPS)

Abstract

Background To test inter-rater reliability of the online Microscale Audit of Pedestrian Streetscapes (MAPS) tool between raters with varying familiarities of Phoenix, Arizona. Methods The online MAPS tool, based on the MAPS in-field audit tool and scoring system, was used for audits. Sixty route pairs, 141 segment pairs, and 92 crossing pairs in Phoenix were included. Each route, segment or crossing was audited by two independent raters: one rater in Phoenix and the other in San Diego, California, respectively. Item, subscale scores, and total scores reliability analyses were computed using Kappa or intra-class correlation coefficient (ICC). Results The route overall score had substantial reliability (ICC: 0.832). Of the route subscale and overall scores, sixteen out of twenty had moderate to substantial reliability (ICC: 0.616–0.906), and the four subscales had fair reliability (ICC: 0.409–0.563). Sixteen out of twenty scores in segment and crossing sections demonstrated fair to substantial reliability (ICC: 0.448–0.897), and the remaining four had slight reliability (ICC: 0.348–0.364). Conclusions Most of the online MAPS items, subscales, and overall scores demonstrated fair to substantial reliability between raters with varied familiarities of the Phoenix area. Results support use of online MAPS to measure microscale elements of the built environment by raters unfamiliar with a region.

Online versus in-person comparison of Microscale Audit of Pedestrian Streetscapes (MAPS) assessments: reliability of alternate methods

“…The 120-item original [ 3 ] and 60-item Abbreviated MAPS [ 12 ] field audit tools were validated in four age groups from three US regions. Same method inter-rater reliability was established for the original MAPS using in-person raters [ 28 ] and for the online MAPS Abbreviated Online using online raters [ 27 ]. The MAPS Abbreviated Online was designed for use with Google Earth, a free geographic software program based on the Google Maps service which displays both satellite and street level images of Earth in high resolution.…”

“…The number of crossings, segments and cul-de-sacs varied by route, though all routes had at least one segment. Most individual items and subscale scores in the MAPS original [ 28 ], Abbreviated in-person [ 12 ] and Abbreviated Online [ 27 ] tools have demonstrated moderate to excellent inter-rater reliability.…”

“…Only two studies have addressed the issue of area-familiarity between raters using virtual microscale audit tools. Zhu and colleagues [ 27 ] demonstrated mostly substantial to excellent inter-rater reliability between online raters with different familiarities of routes in Phoenix using the MAPS Abbreviated Online audit. Wilson and colleagues [ 18 ] directly assessed differences in audit site familiarity between in-person and online raters using the Active Neighborhood Checklist and found high overall agreement between in-person raters familiar with the assessment areas (i.e., St. Louis and Indianapolis) and online raters unfamiliar with the assessment areas.…”

Is walkability equally distributed among downtowners? Evaluating the pedestrian streetscapes of eight European capitals using a micro-scale audit approach

“…So far, several micro-scale walkability audit tools have been developed, including the following: (1) the Irvine-Minnesota Inventory [15,19]; (2) the SPOTLIGHT-Virtual Audit Tool [20]; (3) the Walkability Audit Tool of Centers for Disease Control and Prevention (CDC) [21]; (4) the Pedestrian Environmental Data Scan (PEDS) [22]; and (5) the Microscale Audit of Pedestrian Streetscapes (MAPS) [18]. The MAPS audit tool is one of the most widely studied instruments in active transportation and physical activity research [18] and has been systematically tested for use either in online or field observations [23]. MAPS is a 120-item tool developed in the United States for physical activity research.…”

Exploring effective micro-level items for evaluating inclusive walking facilities on urban streets (applied in Johor Bahru, Malaysia)

“…In recent years, Google tools have been applied in pedestrian assessment studies (Bader et al, 2015;Griew et al, 2013) to reduce the audit cost and time. Some efforts have also been made to assess the feasibility of these tools (Aghaabbasi, Moeinaddini, Shah, Asadi-Shekari, & Kermani, 2018;Bethlehem et al, 2014;Kelly, Wilson, Baker, Miller, & Schootman, 2013;Phillips et al, 2017;Vanwolleghem, Van Dyck, Ducheyne, De Bourdeaudhuij, & Cardon, 2014;Wu et al, 2014;Zhua et al, 2017). Most of these studies mention that applying virtual assessment for detailed attributes and facilities remains time-consuming and sometimes impossible.…”