CollabLearn: An Uncertainty-Aware Crowd-AI Collaboration System for Cultural Heritage Damage Assessment

Zhang, Yang; Zong, Ruohan; Kou, Ziyi; Shang, Lanyu; Wang, Dong

doi:10.1109/tcss.2021.3109143

Cited by 10 publications

(3 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our work is also related to crowd-AI hybrid systems that leverage human intelligence to solve the complex AI-driven computational problems (Goldberg, Wang, and Grant 2017;Zhang et al 2021;Jarrett et al 2014;Zhang et al 2020b;Sener and Savarese 2018;Zhang et al 2019). For example, Jarrett et.…”

Section: Crowd-ai Hybrid Systemsmentioning

confidence: 99%

StreamCollab: A Streaming Crowd-AI Collaborative System to Smart Urban Infrastructure Monitoring in Social Sensing

Zhang

Shang

Zong

et al. 2021

HCOMP

View full text Add to dashboard Cite

Social sensing has emerged as a pervasive and scalable sensing paradigm to collect observations of the physical world from human sensors. A key advantage of social sensing is its infrastructure-free nature. In this paper, we focus on a streaming urban infrastructure monitoring (Streaming UIM) problem in social sensing. The goal is to automatically detect the urban infrastructure damages from the streaming imagery data posted on social media by exploring the collective power of both AI and human intelligence from crowdsourcing systems. Our work is motivated by the limitation of current AI and crowdsourcing solutions that either fail in many critical time-sensitive UIM application scenarios or are not easily generalizable to monitor the damage of different types of urban infrastructures. We identify two critical challenges in solving our problem: i) it is difficult to dynamically integrate AI and crowd intelligence to effectively identify and fix the failure cases of AI solutions; ii) it is non-trivial to obtain accurate human intelligence from unreliable crowd workers in streaming UIM applications. In this paper, we propose StreamCollab, a streaming crowd-AI collaborative system that explores the collaborative intelligence from AI and crowd to solve the streaming UIM problem. The evaluation results on a real-world urban infrastructure imagery dataset collected from social media demonstrate that StreamCollab consistently outperforms both state-of-the-art AI and crowd-AI baselines in UIM accuracy while maintaining the lowest computational cost.

show abstract

Section: Crowd-ai Hybrid Systemsmentioning

confidence: 99%

StreamCollab: A Streaming Crowd-AI Collaborative System to Smart Urban Infrastructure Monitoring in Social Sensing

Zhang

Shang

Zong

et al. 2021

HCOMP

View full text Add to dashboard Cite

show abstract

“…In our SocialCrowd framework, we leverage human intelligence from crowdsourcing systems (e.g., Amazon MTurk) to quantitatively estimate the NSI of misclassified photos. Our motivation for incorporating crowdsourcing efforts is to leverage the crowd workers' extensive background knowledge and experiences to accurately estimate NSI based on their considerations of the relevant social contexts (Zuccon et al 2011;Savenkov, Weitzner, and Agichtein 2016;Zhang et al 2021b). The estimated NSI values from different class categories are then leveraged to construct an NSI-aware graph network to accurately classify photos with minimized misclassification NSI.…”

Section: Introductionmentioning

confidence: 99%

A Crowd-AI Collaborative Duo Relational Graph Learning Framework towards Social Impact Aware Photo Classification

Zhang

Kou

Shang

et al. 2023

AAAI

View full text Add to dashboard Cite

In artificial intelligence (AI), negative social impact (NSI) represents the negative effect on the society as a result of mistakes conducted by AI agents. While the photo classification problem has been widely studied in the AI community, the NSI made by photo misclassification is largely ignored due to the lack of quantitative measurements of the NSI and effective approaches to reduce it. In this paper, we focus on an NSI-aware photo classification problem where the goal is to develop a novel crowd-AI collaborative learning framework that leverages online crowd workers to quantitatively estimate and effectively reduce the NSI of misclassified photos. Our problem is motivated by the limitations of current NSI-aware photo classification approaches that either 1) cannot accurately estimate NSI because they simply model NSI as the semantic difference between true and misclassified categories or 2) require costly human annotations to estimate NSI of pairwise class categories. To address such limitations, we develop SocialCrowd, a crowdsourcing-based NSI-aware photo classification framework that explicitly reduces the NSI of photo misclassification by designing a duo relational NSI-aware graph with the NSI estimated by online crowd workers. The evaluation results on two large-scale image datasets show that SocialCrowd not only reduces the NSI of photo misclassification but also improves the classification accuracy on both datasets.

show abstract

“…In recent years, many scholars have explored the methods of cultural heritage damage assessment [18] and applied them to heritage protection. Zhang (2021) improved the artificial intelligence algorithm to build the CHDA application [19], which accurately locates the damaged areas of cultural heritage by exploring image data posted on social media during disaster events. Tejedor et al (2022) analyzed the degree of damage to cultural heritage through non-destructive testing (NDT) technology [20][21][22][23][24].…”

Section: Introductionmentioning

confidence: 99%

Research on Measuring Methods and Influencing Factors of Spatial Damage Degree of Historic Sites: A Case Study of Three Ancient Cities in Shanxi, China

Zhao,

Han

2023

Buildings

View full text Add to dashboard Cite

Historic sites are important components of every city’s cultural history because they preserve rich historical knowledge and distinctive values passed down from previous generations to the present. Due to the progress of urbanization and modernization, many historic sites face pressure from damage and transformation. In this paper, a method for assessing cultural heritage damage was developed to measure the extent of spatial damage in historic sites. Using sample data obtained in Xiyang, Qi, and Xiaoyi, all historic cities in Shanxi Province, Mainland China, and combined weights were estimated using the Delphi technique and the CRITIC weight method. Following this, the Spatial Damage Degree Model (SDDM) based on K-means cluster analysis and K-nearest neighbor (KNN) classification was developed. The findings show that the model efficiently solves the problem of assessing spatial damage levels in historic sites. Through multiple linear regression analysis, it was shown that the damage to historic sites was predominantly caused by three factors: natural erosion, construction damage, and planning and policy. SDDM was used to calculate the spatial damage levels of historic sites, allowing conservators to fully comprehend the features and concerns related to historic sites. As a result, more scientific and rational preservation approaches can be developed, improving the efficiency of historic site restoration and conservation, and encouraging the sustainable development of urban and rural heritage.

show abstract

CollabLearn: An Uncertainty-Aware Crowd-AI Collaboration System for Cultural Heritage Damage Assessment

Cited by 10 publications

References 52 publications

StreamCollab: A Streaming Crowd-AI Collaborative System to Smart Urban Infrastructure Monitoring in Social Sensing

StreamCollab: A Streaming Crowd-AI Collaborative System to Smart Urban Infrastructure Monitoring in Social Sensing

A Crowd-AI Collaborative Duo Relational Graph Learning Framework towards Social Impact Aware Photo Classification

Research on Measuring Methods and Influencing Factors of Spatial Damage Degree of Historic Sites: A Case Study of Three Ancient Cities in Shanxi, China

Contact Info

Product

Resources

About