A Proposed Technique for Conversion of Unstructured Agro-Data to Semi-Structured or Structured Data

Sambrekar, Kuldeep; Rajpurohit, Vijay S.; Joshi, Jui

doi:10.1109/iccubea.2018.8697432

Cited by 9 publications

(3 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Semi-structured data are irregular data that may be insufficient and have a structure that changes as new data are entered; therefore, their structures are unpredictable [44]. This means they are neither table-oriented in a relational database model nor ordered in object-oriented databases.…”

Section: Semi-structured Datamentioning

confidence: 99%

Data Type and Data Sources for Agricultural Big Data and Machine Learning

et al. 2022

View full text Add to dashboard Cite

Sustainable agriculture is currently being challenged under climate change scenarios since extreme environmental processes disrupt and diminish global food production. For example, drought-induced increases in plant diseases and rainfall caused a decrease in food production. Machine Learning and Agricultural Big Data are high-performance computing technologies that allow analyzing a large amount of data to understand agricultural production. Machine Learning and Agricultural Big Data are high-performance computing technologies that allow the processing and analysis of large amounts of heterogeneous data for which intelligent IT and high-resolution remote sensing techniques are required. However, the selection of ML algorithms depends on the types of data to be used. Therefore, agricultural scientists need to understand the data and the sources from which they are derived. These data can be structured, such as temperature and humidity data, which are usually numerical (e.g., float); semi-structured, such as those from spreadsheets and information repositories, since these data types are not previously defined and are stored in No-SQL databases; and unstructured, such as those from files such as PDF, TIFF, and satellite images, since they have not been processed and therefore are not stored in any database but in repositories (e.g., Hadoop). This study provides insight into the data types used in Agricultural Big Data along with their main challenges and trends. It analyzes 43 papers selected through the protocol proposed by Kitchenham and Charters and validated with the PRISMA criteria. It was found that the primary data sources are Databases, Sensors, Cameras, GPS, and Remote Sensing, which capture data stored in Platforms such as Hadoop, Cloud Computing, and Google Earth Engine. In the future, Data Lakes will allow for data integration across different platforms, as they provide representation models of other data types and the relationships between them, improving the quality of the data to be integrated.

show abstract

Section: Semi-structured Datamentioning

confidence: 99%

Data Type and Data Sources for Agricultural Big Data and Machine Learning

et al. 2022

View full text Add to dashboard Cite

show abstract

“…In addition, agriculture data are stored in clusters, and it is difficult to handle heterogeneous data. Therefore, a uniform format was reported by (Sambrekar, Rajpurohit, and Joshi [44], using Couchbase and NoSQL, and it was found that the time duration for fetching records is fast. Apart from this, different frameworks have been developed by different organizations to make decisions, but no framework has been proposed for value creation.…”

Section: How Does the Data Harmonization Resolve The Issues Of Heterogeneity?mentioning

confidence: 99%

“…Domain Contributions [43] Oil and Gas High performance measure [44] Agriculture High performance, high availability, and high scalability, using the latest techniques [45] General-Purpose Data generation, storing, fetching, analysis, visualization, and decision-making [46] Banking Helps in auditing the multisource data [47] Healthcare Facilitate for navigation of HL7 FHIR core resources [48] General-Purpose Delivering automatic services to interoperable system [49] Healthcare Helps in developing an automatic system for disordered patient [50] Education To motivate researchers and academicians about the latest techniques [10] Healthcare Useful for decisions of scientific, clinical, and administrative work [51] Education Facilitate in online learning, storage, processing, and academic activities [52] General-Purpose Recommendation system, opinion mining, and parallelism can be targeted [53] Oil and Gas Helpful for decision-makers during exploration, drilling, and production [54] General-Purpose It will facilitate for fetching data and performance measure [65] Healthcare Helpful for disease prevention, tracking, and policy making [11] Healthcare Helps in boosting statistical power of sustainable and robust data [55] Infrastructure Geographic based smart city for aggregation, visualization, and analysis [56] Healthcare Helps in predicting the clinical codes of patient stays…”

Section: Study Referencementioning

confidence: 99%

Data Harmonization for Heterogeneous Datasets: A Systematic Literature Review

et al. 2021

View full text Add to dashboard Cite

As data size increases drastically, its variety also increases. Investigating such heterogeneous data is one of the most challenging tasks in information management and data analytics. The heterogeneity and decentralization of data sources affect data visualization and prediction, thereby influencing analytical results accordingly. Data harmonization (DH) corresponds to a field that unifies the representation of such a disparate nature of data. Over the years, multiple solutions have been developed to minimize the heterogeneity aspects and disparity in formats of big-data types. In this study, a systematic review of the literature was conducted to assess the state-of-the-art DH techniques. This study aimed to understand the issues faced due to heterogeneity, the need for DH and the techniques that deal with substantial heterogeneous textual datasets. The process produced 1355 articles, but among them, only 70 articles were found to be relevant through inclusion and exclusion criteria methods. The result shows that the heterogeneity of structured, semi-structured, and unstructured (SSU) data can be managed by using DH and its core techniques, such as text preprocessing, Natural Language Preprocessing (NLP), machine learning (ML), and deep learning (DL). These techniques are applied to many real-world applications centered on the information-retrieval domain. Several assessment criteria were implemented to measure the efficiency of these techniques, such as precision, recall, F-1, accuracy, and time. A detailed explanation of each research question, common techniques, and performance measures is also discussed. Lastly, we present readers with a detailed discussion of the existing work, contributions, and managerial and academic implications, along with the conclusion, limitations, and future research directions.

show abstract