Big data-based acquisition and storage system (ASS) plays an important role in the design of industrial data platform. Many big data frameworks have been integrated compression and serialization method. These methods cannot meet the needs of industrial production information management for requiring time-consuming and mass storage. Based on the existing big data frameworks, we propose an enhanced industrial big data platform in order to reduce the data processing time while requiring fewer data storage space. Specifically, this paper focuses on evaluating the impact of multiple compression and serialization methods on big data platform performance and tries to choose optimal compression and serialization methods for the industrial data platform. Compared to the methods integrated in Hadoop and Spark, the experimental results showed the data compression time of the platform has been reduced by 73.9% with a less than 96% the size of data compressed, furthermore, the data serialization time has been reduced by 80.8%. With the increasing amount of data, it takes less time to compare with benchmark methods.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.