Collective knowledge: organizing research projects as a database of reusable components and portable workflows with common interfaces

Abstract

This article provides the motivation and overview of the Collective Knowledge Framework (CK or cKnowledge). The CK concept is to decompose research projects into reusable components that encapsulate research artifacts and provide unified application programming interfaces (APIs), command-line interfaces (CLIs), meta descriptions and common automation actions for related artifacts. The CK framework is used to organize and manage research projects as a database of such components. Inspired by the USB ‘plug and play’ approach for hardware, CK also helps to assemble portable workflows that can automatically plug in compatible components from different users and vendors (models, datasets, frameworks, compilers, tools). Such workflows can build and run algorithms on different platforms and environments in a unified way using the customizable CK program pipeline with software detection plugins and the automatic installation of missing packages. This article presents a number of industrial projects in which the modular CK approach was successfully validated in order to automate benchmarking, auto-tuning and co-design of efficient software and hardware for machine learning and artificial intelligence in terms of speed, accuracy, energy, size and various costs. The CK framework also helped to automate the artifact evaluation process at several computer science conferences as well as to make it easier to reproduce, compare and reuse research techniques from published papers, deploy them in production, and automatically adapt them to continuously changing datasets, models and systems. The long-term goal is to accelerate innovation by connecting researchers and practitioners to share and reuse all their knowledge, best practices, artifacts, workflows and experimental results in a common, portable and reproducible format at cKnowledge.io . This article is part of the theme issue ‘Reliability and reproducibility in computational science: implementing verification, validation and uncertainty quantification in silico ’.

Research artifacts and citations in computer systems papers

“…For example, a new study on software artifacts in CS research observed that while artifact sharing rate is increasing, the bidirectional links between artifacts and papers do not always exist or last very long, as we have also found ( Hata et al, 2021 ). Some of the reasons that researchers struggle to reproduce experimental results and reuse research code from scientific papers are the continuously changing software and hardware, lack of common APIs, stochastic behavior of computer systems, and a lack of a common experimental methodology ( Fursin, 2021 ), as well as copyright restrictions ( Stodden, 2008 ).…”

Building Domain-Specific Machine Learning Workflows: A Conceptual Framework for the State-of-the-Practice

“…That is, they provide a website for domain experts to search for a workflow which suits their needs 48 . We also point to the impressive Collective Knowledge framework [40] for a repository focusing on AI, ML, and system research 49 .…”

Applying DevOps to Machine Learning Processes: A Systematic Mapping

“…Os autores desse estudo apontam que a previsão eficaz de scripts de IaC propensos a falhas permite que metodologias de DevOps sejam aplicadas com enfoque nessa área, auxiliando na garantia de qualidade. Em A6, [Fursin 2021] propõe um framework para a organizac ¸ão de projetos de pesquisa como um banco de dados de componentes reutilizáveis e fluxo de dados automatizado. O artigo A7, de [García et al 2020], apresenta uma arquitetura distribuída para prover àqueles que trabalham com aprendizado de máquina ferramentas e servic ¸os de nuvem para a melhoria do processo de implantac ¸ão dos modelos.…”