Mining user–user communities for a weighted bipartite network using spark GraphFrames and Flink Gelly

Rao, T. Ramalingeswara; Ghosh, Soumya K.; Goswami, Adrijit

doi:10.1007/s11227-020-03488-4

Cited by 8 publications

(4 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ramalingeswara Rao et al (2021) examined distributed significant data processing. A weighted split network contains the top k user-to-user communities.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

An Efficient Approach to Extract and Store Big Semantic Web Data Using Hadoop and Apache Spark GraphX

Salih Mohammed,

Ju Maa

2024

ADCAIJ

View full text Add to dashboard Cite

The volume of data is growing at an astonishingly high speed. Traditional techniques for storing and processing data, such as relational and centralized databases, have become inefficient and time-consuming. Linked data and the Semantic Web make internet data machine-readable. Because of the increasing volume of linked data and Semantic Web data, storing and working with them using traditional approaches is not enough, and this causes limited hardware resources. To solve this problem, storing datasets using distributed and clustered methods is essential. Hadoop can store datasets because it can use many hard disks for distributed data clustering; Apache Spark can be used for parallel data processing more efficiently than Hadoop MapReduce because Spark uses memory instead of the hard disk. Semantic Web data has been stored and processed in this paper using Apache Spark GraphX and the Hadoop Distributed File System (HDFS). Spark's in-memory processing and distributed computing enable efficient data analysis of massive datasets stored in HDFS. Spark GraphX allows graph-based semantic web data processing. The fundamental objective of this work is to provide a way for efficiently combining Semantic Web and big data technologies to utilize their combined strengths in data analysis and processing. First, the proposed approach uses the SPARQL query language to extract Semantic Web data from DBpedia datasets. DBpedia is a hugely available Semantic Web dataset built on Wikipedia. Secondly, the extracted Semantic Web data was converted to the GraphX data format; vertices and edges files were generated. The conversion process is implemented using Apache Spark GraphX. Third, both vertices and edge tables are stored in HDFS and are available for visualization and analysis operations. Furthermore, the proposed techniques improve the data storage efficiency by reducing the amount of storage space by half when converting from Semantic Web Data to a GraphX file, meaning the RDF size is around 133.8 and GraphX is 75.3. Adopting parallel data processing provided by Apache Spark in the proposed technique reduces the required data processing and analysis time. This article concludes that Apache Spark GraphX can enhance Semantic Web and Big Data technologies. We minimize data size and processing time by converting Semantic Web data to GraphX format, enabling efficient data management and seamless integration.

show abstract

“…Ramalingeswara Rao et al (2021) examined distributed significant data processing. A weighted split network contains the top k user-to-user communities.…”

Section: Related Workmentioning

confidence: 99%

“…GraphFrames can also be used to create vertex and edge tables as follows (Ramalingeswara Rao et al, 2021) Vertex using DataFrame, vertices = Spark.createDataFrame ([(3,"rxin","student") ,(7,"jgonzal","postdoc") ,etc], ["id","name","role"])…”

Section: Spark Graphx and Graphframementioning

confidence: 99%

An Efficient Approach to Extract and Store Big Semantic Web Data Using Hadoop and Apache Spark GraphX

Salih Mohammed,

Ju Maa

2024

ADCAIJ

View full text Add to dashboard Cite

show abstract

“…He et al [27] advocated adding feedback modification to the recommendation system in light of the project's appealing effect on users, and they were able to boost suggestion novelty in numerous datasets. Zhou et al [28] used the bipartite network to execute a onedimensional projection to produce the user-user connection graph, and then used the Jaccard similarity function [29] to generate a recommendation list based on the resources that users transferred to each other.…”

Section: Related Workmentioning

confidence: 99%

A Novel Recommendation Algorithm Integrates Resource Allocation and Resource Transfer in Weighted Bipartite Network

Sun,

Shi,

Liu

et al. 2024

Big Data Min. Anal.

View full text Add to dashboard Cite

Grid-based recommendation algorithms view users and items as abstract nodes, and the information utilised by the algorithm is hidden in the selection relationships between users and items. Although these relationships can be easily handled, much useful information is overlooked, resulting in a less accurate recommendation algorithm. The aim of this paper is to propose improvements on the standard substance diffusion algorithm, taking into account the influence of the user's rating on the recommended item, adding a moderating factor, and optimising the initial resource allocation vector and resource transfer matrix in the recommendation algorithm. An average ranking score evaluation index is introduced to quantify user satisfaction with the recommendation results. Experiments are conducted on the MovieLens training dataset, and the experimental results show that the proposed algorithm outperforms classical collaborative filtering systems and network structure based recommendation systems in terms of recommendation accuracy and hit rate.

show abstract

“…Because the sigmoidal function is an S-shaped curve, the function image grows around the point of central symmetry and changes significantly and changes very little at both ends of the curve. In the distribution function according to formula (3), if the value of the shape parameter Y is smaller, the growth range of the function image S-shaped curve around the center point will be larger [21].…”

Section: Introduction To the Logistic Algorithmmentioning

confidence: 99%

Statistics and Analysis of Targeted Poverty Alleviation Information Integrated with Big Data Mining Algorithm

Mei-zhen

Gao

2022

Security and Communication Networks

View full text Add to dashboard Cite

To gain a more comprehensive and systematic understanding of the impact of government assistance to poor households on poverty reduction targets, a targeted poverty alleviation information statistics and analysis integrated with big data mining algorithm is proposed. Combined with the big data knowledge of the new era, according to the machine learning (ML) pipeline module in spark, a big data computing framework, combined with known data mining algorithms, massive sample data are used to replace random stratified sampling data for modeling and analysis, and random forest model, logistic model, and newly proposed waterfall model are constructed for poor households. Finally, through the comparative evaluation of several poor household identification models, the results show that when 100 real data test the accuracy of the three poor household models, the random forest model and logistic model are slightly reduced, which are 82% and 72%, respectively, but the waterfall model is basically unchanged, which is 83%, and the three models have little change. The new waterfall design proposed in this article has the advantage of a high percentage of sample reuse and can effectively prevent overfitting, and there is no need for massive data. It is a stable and reliable new model. The combination of targeted poverty reduction algorithms and big information technology and mining data can get the most common causes more accurate and convincing results. The right rib trunk and rib are often separated from the common cause because of the population.

show abstract

Mining user–user communities for a weighted bipartite network using spark GraphFrames and Flink Gelly

Cited by 8 publications

References 52 publications

An Efficient Approach to Extract and Store Big Semantic Web Data Using Hadoop and Apache Spark GraphX

An Efficient Approach to Extract and Store Big Semantic Web Data Using Hadoop and Apache Spark GraphX

A Novel Recommendation Algorithm Integrates Resource Allocation and Resource Transfer in Weighted Bipartite Network

Statistics and Analysis of Targeted Poverty Alleviation Information Integrated with Big Data Mining Algorithm

Contact Info

Product

Resources

About