NovelTM Datasets for English-Language Fiction, 1700-2009

Abstract

This report accompanies a collection of 210,266 volumes, predicted to be fiction, that researchers are encouraged to borrow for their own work. We divide the collection into seven subsets with different emphases (for instance, one where books written by men and women are represented equally, and one composed of only the most prominent and widely-held books). Comparing the pictures produced by these different subsets allows us to assess the resilience or fragility of recent quantitative arguments about literary history. Readers can also simply browse the report as a description of English-language fiction in HathiTrust Digital Library. This report describes a collection of 210,266 volumes of fiction drawn from HathiTrust Digital Library. 1 Our aim is to provide researchers greater access to English-language fiction in the Hathi collection and to advance understanding of the nature of the collection. Intellectual property laws keep us from providing the texts themselves, but researchers can use the volume IDs in our metadata tables to locate volumes in HathiTrust, or download extracted feature files that are openly available on the web. 2 In this report, we define seven samples of Hathi's English-language fiction that can be freely used by scholars for a range of purposes. We have offered seven distinct subsets because we do not think any single dataset will be universally valuable for all research questions. Some scholars are interested in literary production; others are interested in reception, and care mainly about widelyread works. Some scholars need a large collection of books. Others would prefer a smaller, manually-groomed list. So we selected volumes using a range of different criteria, and invested time in manually labeling some of the smaller lists. Through the process of manual annotation, we were able to illuminate some of the broad demographic contours of fiction over the last two centuries as represented in academic libraries. Thus, we see this report as ideally benefitting both practitioners of

HATHI 1M: Introducing a Million Page Historical Prose Dataset in English from the Hathi Trust

“…1. Fiction (411 documents): We randomly sample two or three volumes per year from the class of English-language fiction drawn from the "most frequently reproduced" fiction in Underwood et al (2020). Our training data is thus designed to represent a yearly cross-section of the most reproduced English-language fiction between 1800 and 2000.…”

“…While admirable technical solutions have been implemented to make copyright-restricted material accessible, these solutions require high levels of technical expertise that exceed most researchers. 1 It is with these challenges and opportunities in mind that we build on prior work (Underwood et al, 2020) to generate a large sample of richly annotated prose data in English drawn from the Hathi Trust Digital Library. The Hathi Trust Digital Library represents the largest collection of digitized historical documents in English, with over 17 million volumes.…”

“…For the period under investigation, writing in prose is by far the dominant mode during these years. While prior work has developed models for the detection of prose fiction during this period (Underwood et al, 2020), we build models for the creation of symmetrical collections of fictional and non-fictional modes of writing. Our work is motivated by theoretical frameworks grounded in theories of social differentiation (Luhmann, 1995), where the meaning and function of different modes of communication evolve in distinction to one another.…”

“…Single model. Our data is generated from a single predictive model for each mode of writing (fiction/non-fiction) across the entire sampled time-frame based on manually reviewed training data derived from prior work (Underwood et al, 2020). While there is still much work to be done regarding the implications of applying predictive models on collections spanning long historical time frames, as we show the use of a single model overcomes important anomalies introduced by the conjoining of multiple models from prior work (see Figure 4).…”

New York Times Hardcover Fiction Bestsellers, 1931—2020

“…HathiTrust volume identifiers were matched based on string comparisons against the "volumemeta" dataset described by Ted Underwood et al (2020). I used the shorttitle and author metadata fields.…”