Research Directions for Engineering Big Data Analytics Software

Otero, Carlos E.; Peter, Adrian M.

doi:10.1109/mis.2014.76

Cited by 60 publications

(49 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Based on this context model, the authors then sketch research challenges in requirements engineering, architectures, and testing and maintenance. The results are similar to those in [49], although the models are quite different (conceptual versus more mathematical/formal).…”

Section: Related Researchsupporting

confidence: 78%

See 1 more Smart Citation

Big Data and software engineering: prospects for mutual enrichment

Arndt

2017

Iran J Comput Sci

View full text Add to dashboard Cite

Software engineering has evolved over the last 50 years, initially as a response to the so-called software crisis (the problems that organizations had producing quality software systems on time and on budget) of the 1960s and 1970s. Software engineering (SE) has been defined as "the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software; that is, the application of engineering to software". Software engineering has developed a number of approaches to areas such as software requirements, software design, software testing, and software maintenance. Software development processes such as the waterfall model, incremental development, and the spiral model have been successfully applied to produce high-quality software on time and under budget. More recently, agile software development has gained popularity as an alternative to the more traditional development methods for development of complex systems. Within the last decade or so, advances in technologies such as mobile computing, social networks, cloud computing, and the Internet of things have given rise to massive datasets which have been given the name Big Data (BD). Big Data has been defined as data with 3Vs-high volume, velocity, and variety. Big Data contains so much data that low probability events are captured in the data. These events can be discovered using analytics methods and turned into actionable intelligence which can be used by businesses to gain a competitive advantage. Unfortunately, the very scale of BD often renders inadequate SQL-based relational database systems which have formed the backbone of data intensive systems for the last 30 years, requiring new NoSQL technologies to be effective. In this paper, we will explore how well-established SE technology can be adapted to support successful development of BD projects, as well as how BD techniques can be used to increase the utility of SE processes and techniques. Thus, BD and SE may mutually support and enrich each other.

show abstract

Section: Related Researchsupporting

confidence: 78%

“…Otero and Peter [49] attempt to set a research agenda for engineering Big Data analytics software. They look at several problems in the engineering of BD software: the requirements problem, the design problem, the construction problem, and the testing problem.…”

Section: Related Researchmentioning

confidence: 99%

Big Data and software engineering: prospects for mutual enrichment

Arndt

2017

Iran J Comput Sci

View full text Add to dashboard Cite

show abstract

“…If a MR is violated, for any pair of source and follow-up test cases, the tester reports a failure in the program. MT has been successfully applied to test many different types of software, such as numerical programs (Zhou et al 2004), embedded software (Kuo et al 2011), analysis of feature models (Segura et al 2010), machine learning Xie et al 2011), testing service oriented applications (Chan et al 2007), and big data analytics (Otero and Peter 2015). A simple and classical example of MT is to test the correctness of an implementation of a program that computes the sin(x) trigonometric function, using some wellknown mathematical properties of the function as MRs (Table 3).…”

Section: Metamorphic Testingmentioning

confidence: 99%

How to test bioinformatics software?

et al. 2015

View full text Add to dashboard Cite

Bioinformatics is the application of computational, mathematical and statistical techniques to solve problems in biology and medicine. Bioinformatics programs developed for computational simulation and large-scale data analysis are widely used in almost all areas of biophysics. The appropriate choice of algorithms and correct implementation of these algorithms are critical for obtaining reliable computational results. Nonetheless, it is often very difficult to systematically test these programs as it is often hard to verify the correctness of the output, and to effectively generate failure-revealing test cases. Software testing is an important process of verification and validation of scientific software, but very few studies have directly dealt with the issues of bioinformatics software testing. In this work, we review important concepts and state-ofthe-art methods in the field of software testing. We also discuss recent reports on adapting and implementing software testing methodologies in the bioinformatics field, with specific examples drawn from systems biology and genomic medicine.

show abstract

“…In order to deal with the challenges brought by big data, there is very active ongoing development of new tools, algorithms, computational frameworks, analytic platforms and deployment strategies. Current big data analyssi optimization focuses on scaling up/out analytic platforms [10] [11] and using approximation algorithms [12] [13].…”

Section: Introductionmentioning

confidence: 99%

Content-Aware Partial Compression for Textual Big Data Analysis in Hadoop

Dong

Herbert

2018

IEEE Trans. Big Data

View full text Add to dashboard Cite

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Abstract-A substantial amount of information in companies and on the Internet is present in the form of text. The value of this semi-structured and unstructured data has been widely acknowledged, with consequent scientific and commercial exploitation. The ever-increasing data production, however, pushes data analytic platforms to their limit. Compression as an effective means to reduce data size has been employed by many emerging data analytic platforms, whom the main purpose of data compression is to save storage space and reduce data transmission cost over the network. Since general purpose compression methods endeavour to achieve higher compression ratios by leveraging data transformation techniques and contextual data, this context-dependency forces the access to the compressed data to be sequential. Processing such compressed data in parallel, such as desirable in a distributed environment, is extremely challenging. This work proposes techniques for more efficient textual big data analysis with an emphasis on content-aware compression schemes suitable for the Hadoop analytic platform. The compression schemes have been evaluated for a number of standard MapReduce analysis tasks using a collection of public and private real-world datasets. In comparison with existing solutions, they have shown substantial improvement in performance and significant reduction in system resource requirements.

show abstract

Research Directions for Engineering Big Data Analytics Software

Cited by 60 publications

References 19 publications

Big Data and software engineering: prospects for mutual enrichment

Big Data and software engineering: prospects for mutual enrichment

How to test bioinformatics software?

Content-Aware Partial Compression for Textual Big Data Analysis in Hadoop

Contact Info

Product

Resources

About