Term Weighting

Where does research design fall short? Mental health related‐stigma as example

“…We chose to set the unit of analysis at the single word level or ‘unigram’, as single‐word variations (e.g., depression/schizophrenia) have previously been shown to influence the implicit beliefs and affects engaged by the public (Foster, 2006; Walsh & Foster, 2020). However, as not all words are equally informative for elucidating a document's meaning, term‐weighting is a crucial next step in the text indexing process (El‐Khair, 2009). This often relies on a measure of word frequency (El‐Khair, 2009; Robertson, 2004).…”

“…However, as not all words are equally informative for elucidating a document's meaning, term‐weighting is a crucial next step in the text indexing process (El‐Khair, 2009). This often relies on a measure of word frequency (El‐Khair, 2009; Robertson, 2004). Decisions about weighting are typically informed by two principles: representativity and discrimination (Chartier & Meunier, 2011; El‐Khair, 2009).…”

“…This often relies on a measure of word frequency (El‐Khair, 2009; Robertson, 2004). Decisions about weighting are typically informed by two principles: representativity and discrimination (Chartier & Meunier, 2011; El‐Khair, 2009). The representativity principle proposes that a word used frequently is more likely to describe its context (Chartier & Meunier, 2011; El‐Khair, 2009).…”

“…Decisions about weighting are typically informed by two principles: representativity and discrimination (Chartier & Meunier, 2011; El‐Khair, 2009). The representativity principle proposes that a word used frequently is more likely to describe its context (Chartier & Meunier, 2011; El‐Khair, 2009). The discrimination principle offers that a frequently used word in one place but not another is more likely to be meaningful (Chartier & Meunier, 2011; El‐Khair, 2009).…”

“…The representativity principle proposes that a word used frequently is more likely to describe its context (Chartier & Meunier, 2011; El‐Khair, 2009). The discrimination principle offers that a frequently used word in one place but not another is more likely to be meaningful (Chartier & Meunier, 2011; El‐Khair, 2009). These principles have now been diffused throughout language models in the form of term frequency‐inverse document frequency (TF‐IDF) scores (Chartier & Meunier, 2011; Robertson, 2004).…”