Council Data Project: Software for Municipal Data Collection, Analysis, and Publication

Abstract

Cities, counties, and states throughout the USA are bound by law to archive recordings of public meetings. Most local governments comply with these laws by posting documents, audio, or video recordings online. As there is no set standard for municipal data archives however, parsing and processing such data is typically time consuming and highly dependent on each municipality. Council Data Project (CDP) is a set of open-source tools that improve the accessibility of local government data by systematically collecting, transforming, and republishing this data to the web. The data re-published by CDP is packaged and presented within a searchable web application that vastly simplifies the process of finding specific information within the archived data. We envision this project being used by a variety of groups including civic technologists hoping to promote government transparency, researchers focused on public policy, natural language processing, machine learning, or information retrieval and discovery, and many others. Statement of NeedComparative research into municipal governance in the USA is often prohibitively difficult due to a broad federal system where states, counties, and cities divide legislative powers differently. This has contributed to the lack of large-scale quantitative studies of municipal government, and impeded necessary research into effective procedural elements of administrative and legislative processes (Trounstine, 2009). Council Data Project enables large-scale quantitative studies by generating standardized municipal governance corpora -including legislative voting records, timestamped transcripts, and full legislative matter attachments (related reports, presentations, amendments, etc.).

Speakerbox: Few-Shot Learning for Speaker Identification with Transformers

“…We are utilizing Speakerbox-trained models to annotate municipal council meeting transcripts provided by the Council Data Project (Brown et al, 2021). In our initial research, we first annotated ~10 hours of audio using the Gecko platform in ~12 hours of time, we then used our diarization and labeling method to annotate an additional ~21 hours of audio in ~6 hours of time.…”

Integrative urban AI to expand coverage, access, and equity of urban data

“…To enable such large-scale studies, we have begun work on “Council Data Project,” [ 6 ] a suite of tools for deploying and managing infrastructure for rapidly generating, archiving, and analyzing transcript datasets of municipal council meeting content. Council Data Project (CDP) is easily deployable and generalizes to many different meeting venues, but is specifically built with municipal council meetings in mind.…”

Councils in Action: Automating the Curation of Municipal Governance Data for Research

“…Over the last two years, the Council Data Project (CDP) has developed an open‐source software platform that significantly improves access and engagement with local government data (Brown et al, 2021). In the following paper we provide a brief background of CDP, and how this open‐source platform enables large‐scale, longitudinal, comparative research into municipal governance.…”

“…CDP can be deployed in any municipality with minimal configuration by a developer or city IT department. Using a simple Python‐based “cookie‐cutter template” (Cookiecutter, 2013) a developer can configure a new CDP deployment with just a two‐line installation process and fully deploy the instance once provided a function to gather events (Brown et al, 2021). Once fully deployed, an instance of CDP will collect and process municipal meeting minutes, agendas, voting records, and crucially, the recorded video or audio of each meeting that is archived by a municipality.…”

“…For every legislative meeting that CDP processes, the system generates a transcript from the provided video using a machine learning model that converts recorded audio to text (aka speech‐to‐text processing). On a continuous schedule CDP infrastructure then uses this corpus of transcripts to generate and update a keyword based index to enable search of meetings by keyword (Brown et al, 2021).…”