Algorithmic methods to explore the automation of the appraisal of structured and unstructured digital data

Abstract

<jats:sec> <jats:title content-type="abstract-subheading">Purpose</jats:title> <jats:p>This paper aims to describe an interdisciplinary and innovative research conducted in Switzerland, at the Geneva School of Business Administration HES-SO and supported by the State Archives of Neuchâtel (Office des archives de l'État de Neuchâtel, OAEN). The problem to be addressed is one of the most classical ones: how to extract and discriminate relevant data in a huge amount of diversified and complex data record formats and contents. The goal of this study is to provide a framework and a proof of concept for a software that helps taking defensible decisions on the retention and disposal of records and data proposed to the OAEN. For this purpose, the authors designed two axes: the archival axis, to propose archival metrics for the appraisal of structured and unstructured data, and the data mining axis to propose algorithmic methods as complementary or/and additional metrics for the appraisal process.</jats:p> </jats:sec> <jats:sec> <jats:title content-type="abstract-subheading">Design/methodology/approach</jats:title> <jats:p>Based on two axes, this exploratory study designs and tests the feasibility of archival metrics that are paired to data mining metrics, to advance, as much as possible, the digital appraisal process in a systematic or even automatic way. Under Axis 1, the authors have initiated three steps: first, the design of a conceptual framework to records data appraisal with a detailed three-dimensional approach (trustworthiness, exploitability, representativeness). In addition, the authors defined the main principles and postulates to guide the operationalization of the conceptual dimensions. Second, the operationalization proposed metrics expressed in terms of variables supported by a quantitative method for their measurement and scoring. Third, the authors shared this conceptual framework proposing the dimensions and operationalized variables (metrics) with experienced professionals to validate them. The expert’s feedback finally gave the authors an idea on: the relevance and the feasibility of these metrics. Those two aspects may demonstrate the acceptability of such method in a real-life archival practice. In parallel, Axis 2 proposes functionalities to cover not only macro analysis for data but also the algorithmic methods to enable the computation of digital archival and data mining metrics. Based on that, three use cases were proposed to imagine plausible and illustrative scenarios for the application of such a solution.</jats:p> </jats:sec> <jats:sec> <jats:title content-type="abstract-subheading">Findings</jats:title> <jats:p>The main results demonstrate the feasibility of measuring the value of data and records with a reproducible method. More specifically, for Axis 1, the authors applied the metrics in a flexible and modular way. The authors defined also the main principles needed to enable computational scoring method. The results obtained through the expert’s consultation on the relevance of 42 metrics indicate an acceptance rate above 80%. In addition, the results show that 60% of all metrics can be automated. Regarding Axis 2, 33 functionalities were developed and proposed under six main types: macro analysis, microanalysis, statistics, retrieval, administration and, finally, the decision modeling and machine learning. The relevance of metrics and functionalities is based on the theoretical validity and computational character of their method. These results are largely satisfactory and promising.</jats:p> </jats:sec> <jats:sec> <jats:title content-type="abstract-subheading">Originality/value</jats:title> <jats:p>This study offers a valuable aid to improve the validity and performance of archival appraisal processes and decision-making. Transferability and applicability of these archival and data mining metrics could be considered for other types of data. An adaptation of this method and its metrics could be tested on research data, medical data or banking data.</jats:p> </jats:sec>

Prise en main automatisée du vrac numérique audiovisuel d’un service communication

“…Des outils intéressants sont développés par des archives nationales, des gouvernements ou encore des compagnies privées. Ils facilitent les tâches des professionnels dans l'analyse des contenus volumineux ou non structurés (Bavaud et al, 2022 ;Oguey & Schneiter, 2018) afin de permettre une meilleure efficacité de traitement, la maîtrise de la volumétrie, la localisation effective des actifs informationnels, ce qui vient directement contribuer à l'amélioration de la prise de décision et de la recherche scientifique (Makhlouf Shabou, 2023b l'intégrité des données d'entrée, la mise en place des critères de sélection justes et la conformité des données en bout de chaîne (Makhlouf Shabou et al, 2020).…”

“…Les institutions sont alors confrontées à une problématique d'envergure : la maîtrise de leurs données qui renferment un potentiel inexploité (Texier, 2022a). En effet, ces données pourraient aider à la prise de décision stratégique, favoriser l'innovation ou encore alimenter les archives historiques (Makhlouf Shabou et al, 2020 ;Makhlouf Shabou, 2023a).…”

“…De nombreuses institutions se retrouvent avec des masses de données difficiles Revue électronique suisse de science de l'information | n°24 | 2024 3 https://doi.org/10.55790/journals/ressi.2024.e1522 à naviguer et trop souvent inexploitables. L'automatisation peut apporter une réponse efficace et tangible au travail quotidien des archivistes, et optimiser la gouvernance de l'information de l'institution (Makhlouf Shabou et al, 2020).…”

Pragmatism and Methodology

“…In the end, when it comes to typologies, we need to move past simple dichotomies like qualitative and quantitative data and, in many ways, the field of methodology is doing so by operating increasingly with new concepts such as structured and unstructured data (Bolla & Anandan, 2018;Fong et al, 2015;Shabou et al, 2020). While structured data are generally quantitative and unstructured data are generally qualitative, these new notions shift the emphasis away from a binary toward the idea that there can be degrees of structure.…”

Coûts écologiques de nos pratiques archivistiques

“…Le développement de nouveaux outils s'avère de plus en plus indispensable afin d'aider les archivistes à traiter une diversité et un volume plus importants de documents et données. En attendant de pouvoir mettre en place une automatisation de l'évaluation des données structurées et non structurées, 48 les archivistes peuvent déjà accéder à des logiciels pouvant fournir une aide concrète. Par exemple, Archifiltre, développé par la Fabrique numérique des ministères sociaux de France, propose différentes fonctionnalités, comme la visualisation d'arborescences, l'enrichissement de métadonnées ou encore le repérage de doublons grâce à un calcul d'empreintes.…”