Automated Detection of HONcode Website Conformity Compared to Manual Detection: An Evaluation

Abstract

BackgroundTo earn HONcode certification, a website must conform to the 8 principles of the HONcode of Conduct In the current manual process of certification, a HONcode expert assesses the candidate website using precise guidelines for each principle. In the scope of the European project KHRESMOI, the Health on the Net (HON) Foundation has developed an automated system to assist in detecting a website’s HONcode conformity. Automated assistance in conducting HONcode reviews can expedite the current time-consuming tasks of HONcode certification and ongoing surveillance. Additionally, an automated tool used as a plugin to a general search engine might help to detect health websites that respect HONcode principles but have not yet been certified.ObjectiveThe goal of this study was to determine whether the automated system is capable of performing as good as human experts for the task of identifying HONcode principles on health websites.MethodsUsing manual evaluation by HONcode senior experts as a baseline, this study compared the capability of the automated HONcode detection system to that of the HONcode senior experts. A set of 27 health-related websites were manually assessed for compliance to each of the 8 HONcode principles by senior HONcode experts. The same set of websites were processed by the automated system for HONcode compliance detection based on supervised machine learning. The results obtained by these two methods were then compared.ResultsFor the privacy criterion, the automated system obtained the same results as the human expert for 17 of 27 sites (14 true positives and 3 true negatives) without noise (0 false positives). The remaining 10 false negative instances for the privacy criterion represented tolerable behavior because it is important that all automatically detected principle conformities are accurate (ie, specificity [100%] is preferred over sensitivity [58%] for the privacy criterion). In addition, the automated system had precision of at least 75%, with a recall of more than 50% for contact details (100% precision, 69% recall), authority (85% precision, 52% recall), and reference (75% precision, 56% recall). The results also revealed issues for some criteria such as date. Changing the “document” definition (ie, using the sentence instead of whole document as a unit of classification) within the automated system resolved some but not all of them.ConclusionsStudy results indicate concordance between automated and expert manual compliance detection for authority, privacy, reference, and contact details. Results also indicate that using the same general parameters for automated detection of each criterion produces suboptimal results. Future work to configure optimal system parameters for each HONcode principle would improve results. The potential utility of integrating automated detection of HONcode conformity into future search engines is also discussed.

Inflammatory bowel disease: An evaluation of health information on the internet

“…Therefore, the automated assistance in conducting HONcode reviews can expedite the current time-consuming tasks of HONcode certification and ongoing surveillance. A recent study showed that there is concordance between automated and expert manual compliance detection for the criteria[18]. In this research we have used the electronic system available at: .…”

Effect of the Named Entity Recognition and Sliding Window on the HONcode Automated Detection of HONcode Criteria for Mass Health Online Content

“…We used the standard systematic evaluation scheme with 80% of the collection employed to train the system and 20% to assess the algorithm. The Nave Bayes machine learning algorithm with a single word tokenization and a space reduction of 70% was used for the classifier as a result of previous publication work ( (Boyer and Dolamic, 2014;Boyer and Dolamic, 2015) and in this research). The evaluation of the classifier on the remaining 20% of the collection yielded good results for the Complementarity and Date Attribution criteria with a precision of 83% and 94% respectively, and a recall of 95% for both.…”

“…After extensively testing this system with the sample collection, the next logical step was to verify its performance on real life examples, thus comparing it to the current manual HONcode certification process. In the report (Boyer and Dolamic, 2015), we compared the automated detection of the HONcode criteria with a manual process and we obtained mitigated results especially for the Date Attribution and Complementarity criteria. We also identified further research paths to improve our first evaluation in real life settings.…”

“…The Complementarity criterion-related problem illustrated here was also present for other criteria such as Date Attribution. To deal with this issue (Boyer and Dolamic, 2015), we tested the effectiveness of using the sentence as the unit of classification. This approach was experimented for the Privacy and Date Attribution criteria on a selection of websites and proved very conclusive, especially in relation to recall for both criteria.…”

Consumer Health Search on the Web: Study of Web Page Understandability and Its Integration in Ranking Algorithms (Preprint)

“…Ad-hoc solutions to support the general public in searching and accessing health information on the Web have been implemented, typically supported by government initiatives or medical practitioner associations, e.g., HealthOnNet.org (HON [11]) and HealthDirect.gov.au, among others. These solutions aim to provide better health information to the general public.…”