Instance selection of linear complexity for big data

Arnaiz‐González, Álvar; Díez-Pastor, José-Francisco; Rodrı́guez, Juan J.; García‐Osorio, César

doi:10.1016/j.knosys.2016.05.056

Cited by 65 publications

(17 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Ideally, data reduction would lead to a dataset of smaller size and dimension that can be handled more efficiently. Various types of data reduction methods have been developed, which focus on feature extraction, dimensionality reduction, instance selection, noise removal, and outlier detection to reduce, refine and clean spatial big data [20][21][22][23][24]. With the aim of selecting the most important flow instances pertaining to the purpose of analysis, the head/tail break falls among the instance-based approaches.…”

Section: Head/tail Break As a Methods Of Data Reductionmentioning

confidence: 99%

Boosting Computational Effectiveness in Big Spatial Flow Data Analysis with Intelligent Data Reduction

Tao

Zi-tong

et al. 2020

IJGI

View full text Add to dashboard Cite

One of the enduring issues of spatial origin-destination (OD) flow data analysis is the computational inefficiency or even the impossibility to handle large datasets. Despite the recent advancements in high performance computing (HPC) and the ready availability of powerful computing infrastructure, we argue that the best solutions are based on a thorough understanding of the fundamental properties of the data. This paper focuses on overcoming the computational challenge through data reduction that intelligently takes advantage of the heavy-tailed distributional property of most flow datasets. We specifically propose the classification technique of head/tail breaks to this end. We test this approach with representative algorithms from three common method families, namely flowAMOEBA from flow clustering, Louvain from network community detection, and PageRank from network centrality algorithms. A variety of flow datasets are adopted for the experiments, including inter-city travel flows, cellphone call flows, and synthetic flows. We propose a standard evaluation framework to evaluate the applicability of not only the selected three algorithms, but any given method in a systematic way. The results prove that head/tail breaks can significantly improve the computational capability and efficiency of flow data analyses while preserving result quality, on condition that the analysis emphasizes the “head” part of the dataset or the flows with high absolute values. We recommend considering this easy-to-implement data reduction technique before analyzing a large flow dataset.

show abstract

Section: Head/tail Break As a Methods Of Data Reductionmentioning

confidence: 99%

Boosting Computational Effectiveness in Big Spatial Flow Data Analysis with Intelligent Data Reduction

Tao

Zi-tong

et al. 2020

IJGI

View full text Add to dashboard Cite

show abstract

“…While removing valid observations is considered a sin by many analysts working with discrete choice models, in fields like machine-learning down-scaling of data sets is more common practice (e.g. Arnaiz-González et al 2016;Loyola et al 2016). As we will argue in this paper, using a carefully sampled subset of choice observations can give nearly identical estimation results as compared to using the complete dataset.…”

Section: Introductionmentioning

confidence: 93%

Information theoretic-based sampling of observations

Cranenburgh

Bliemer

2019

Journal of Choice Modelling

View full text Add to dashboard Cite

Due to the surge in the amount of data that are being collected, analysts are increasingly faced with very large data sets. Estimation of sophisticated discrete choice models (such as Mixed Logit models) based on these typically large data sets can be computationally burdensome, or even infeasible. Hitherto, analysts tried to overcome these computational burdens by reverting to less computationally demanding choice models or by taking advantage of the increase in computational resources. In this paper we take a different approach: we develop a new method called Sampling of Observations (SoO) which scales down the size of the choice data set, prior to the estimation. More specifically, based on information-theoretic principles this method extracts a subset of observations from the data which is much smaller in volume than the original data set, yet produces statistically nearly identical results. We show that this method can be used to estimate sophisticated discrete choice models based on data sets that were originally too large to conduct sophisticated choice analysis.

show abstract

“…We are planning to extend the use of kNN-IS to instance selection techniques for big data [45], where it reports good results. Another direction for future work is to extend the application of the presented kNN-IS approach to a big data semi-supervised learning [46] context.…”

Section: Conclusion and Further Workmentioning

confidence: 99%

kNN-IS: An Iterative Spark-based design of the k-Nearest Neighbors classifier for big data

Maillo

Ramírez

Triguero

et al. 2017

Knowledge-Based Systems

256

126

View full text Add to dashboard Cite

A note on versions:The version presented here may differ from the published version or from the version of record. If you wish to cite this item you are advised to consult the publisher's version. Please see the repository url above for details on accessing the published version and note that access may require a subscription. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.ACCEPTED MANUSCRIPT A C C E P T E D M A N U S C R I P T AbstractThe k-Nearest Neighbors classifier is a simple yet effective widely renowned method in data mining. The actual application of this model in the big data domain is not feasible due to time and memory restrictions. Several distributed alternatives based on MapReduce have been proposed to enable this method to handle large-scale data. However, their performance can be further improved with new designs that fit with newly arising technologies.In this work we provide a new solution to perform an exact k-nearest neighbor classification based on Spark. We take advantage of its in-memory operations to classify big amounts of unseen cases against a big training dataset. The map phase computes the k-nearest neighbors in different training data splits. Afterwards, multiple reducers process the definitive neighbors from the list obtained in the map phase. The key point of this proposal lies on the management of the test set, keeping it in memory when possible. Otherwise, it is split into a minimum number of pieces, applying a MapReduce per chunk, using the caching skills of Spark to reuse the previously partitioned * Corresponding author. ACCEPTED MANUSCRIPT A C C E P T E D M A N U S C R I P T training set. In our experiments we study the differences between Hadoop and Spark implementations with datasets up to 11 million instances, showing the scaling-up capabilities of the proposed approach. As a result of this work an open-source Spark package is available.

show abstract

Instance selection of linear complexity for big data

Cited by 65 publications

References 44 publications

Boosting Computational Effectiveness in Big Spatial Flow Data Analysis with Intelligent Data Reduction

Boosting Computational Effectiveness in Big Spatial Flow Data Analysis with Intelligent Data Reduction

Information theoretic-based sampling of observations

kNN-IS: An Iterative Spark-based design of the k-Nearest Neighbors classifier for big data

Contact Info

Product

Resources

About