Big data challenges: Impact, potential responses and research needs

Abstract

Although reports on big data success stories have been accumulating in the media, most organizations dealing with high-volume, high-velocity and high-variety information assets still face challenges. Only a thorough understanding of these challenges puts organizations into a position in which they can make an informed decision for or against big data, and, if the decision is positive, overcome the challenges smoothly. The combination of a series of interviews with leading experts from enterprises, associations and research institutions, and focused literature reviews allowed not only identifying and describing eight key challenges but also characterizing their impact, outlining potential responses to them and proposing directions for future research. The challenges faced were found to be not only technological in nature. Organizational and people-related matters as well as the legislative framework are also relevant. For large enterprises and startups specialized in big data, it is typically easier to overcome the challenges than it is for other enterprises and public administration bodies

Automatic Testing of Design Faults in MapReduce Applications

“…The configurations can be created manually to cover each test coverage item of the t-Wise selected, but the MRTest-t-Wise technique generates these configurations automatically. The following pseudo-code describes how the configurations are generated: In order to generate a configuration that covers each one of the test coverage items (1, 2, 3), the MRTest-t-Wise covers the first parameter with random values, then the second, and so on (4,5,6,7,8). For example, if the first parameter should be P1: >1, i.e., more Mappers, then a random value is selected greater or equal to 2 to guarantee >1 Mappers, and so on with the remainder of the parameters.…”

“…In these cases MRTest-t-Wise uses a backtracking approach. First it fulfils randomly the first parameter (4, 5, 6), then it executes the part of the program affected by this parameter (7,8) and tries to cover the second, and so on. When the generator discovers at runtime that one parameter cannot be covered, then it backtracks, changing the value of previous parameters (9, 10).…”

“…This work was supported in part by the Spanish Ministry of Science and Technology under the PERTEST project (TIN2013-46928-C3-1-R), the Spanish Ministry of Economy and Competitiveness under both TestEAMoS (TIN2016-76956-C3-1-R) and POLOLAS (TIN2016-76956-C3-2-R) projects, the Principality of Asturias (Spain) under both the GRUPIN14-007 project and Severo Ochoa pre-doctoral grants (BP16215), the Italian MIUR under GAUSS project (PRIN 2015, 2015KWREMX), and ERDF funds. [5] and different technological issues [3], [6], [7]. According to Gartner, it is expected that 60% of Big Data projects will fail to go beyond piloting and will be abandoned during 2017 [8].…”

Testing MapReduce programs: A systematic mapping study

“…Despite the importance of these applications, some studies predicted that 60% of Big Data projects fail to go beyond piloting and would be abandoned in 2017 . There are several challenges and concerns: poor data quality,() lack of technological skills,() and various different technological issues such as complexity, maturity, operability, and technical problems . Those problems complicate development, and MapReduce applications could potentially be implemented with faults.…”

Towards Ex Vivo Testing of MapReduce Applications

“…Although these programs are considered critical for several Fortune 1000 enterprises [2], there are several challenges and concerns: poor data quality [3], [4], lack of technological skills [5], [6], and different technological issues such as, among others, complexity [7], maturity [8], operability [9] and technical problems [3]. Some of the previously stated problems complicate the construction of the projects and could lead to failures.…”