A System for Ranking Organizations Using Social Scale Analysis

Abstract

Abstract-In this paper we utilize feature extraction and model fitting techniques to process the rhetoric found in the web sites of 23 Indonesian religious organizations -comprising a total of 37,000 articles dating from 2005 to 2011 -to profile their ideology and activity patterns along a hypothesized radical/counter-radical scale. We rank these organizations by assigning them to probable positions on the scale. We show that the developed Rasch model fits the data using Andersen's LR-test. We create a gold standard of the ranking of these organizations through an expertise elicitation tool. We compute expert-to-expert agreements, and we present experimental results comparing the performance of three different baseline methods to show that the Rasch model not only outperforms our baseline methods, but it is also the only system that performs at expert-level accuracy.

LookingGlass

“…In the context of our project, both social movements (subjects) and their socio-economic, political, or religious beliefs, goals and practices (items) are mapped simultaneously on a set of scales via expert inputs and text mining algorithms. Recently we developed algorithms [3,4,5,6] that utilize large amounts of multilingual text collected from a wide variety of organizations' media outlets (e.g. web sites, blogs, news, RSS feeds, tweets, leaders' speeches etc.)…”

“…Binary labeling as counter-radical vs. radical does not capture the overlaps, movement and interactivity among these actors. We observe [6] that both counter-radical and radical movements in Muslim societies exhibit distinct combinations of discrete states comprising various social, political, and religious beliefs, goals, attitudes and practices, and their discriminating perspectives can be semi-automatically identified and mapped onto latent linear continuums or scales.…”

MultiScale Modeling of Islamic Organizations in UK

“…We employed the Rasch model presented in Section III-D to rank both the organizations (subjects) and corresponding perspectives (items) on each scale as an alternative ranking algorithm alongside the QUIC algorithm. Rasch Modeling algorithm 4 also produces a metric [10] to validate the fitness of the model. A p-value, returned by the test, indicates the goodness of fit and a pvalue 5 higher than 0.05 indicates no presence of lack of fit.…”

“…In our prior work [4], we utilized a simple term frequency -inverse document frequency (TF-IDF) [6] based technique to generate a candidate list of keywords that can be utilized as items during scaling analysis. Top 100 n-grams from each organization's web site were aggregated into a list of candidate items, and we asked social scientists on our team to scan the list manually, and select relevant keywords indicating different perspectives.…”

“…Earlier we developed an algorithm [4] that utilize large amounts of multilingual text collected from a wide variety of organizations media outlets (e.g. web sites, blogs, news, RSS feeds, leaders speeches etc.)…”