Towards Human-centric Digital Twins: Leveraging Computer Vision and Graph Models to Predict Outdoor Comfort

Liu, Pengyuan; Zhao, Tianhong; Luo, Junjie; Lei, Binyu; Frei, Mario; Miller, Clayton; Biljecki, Filip

doi:10.1016/j.scs.2023.104480

Cited by 25 publications

(4 citation statements)

References 80 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Urban planners and policymakers can gain valuable insights and empirical evidence to guide decisions in sustainable urban planning and design by utilizing CV techniques to investigate and assess the spatial correlation and underlying factors contributing to diverse urban challenges and prospects [ 3 ]. The emergence of spatial-explicit geospatial AI (GeoAI), which integrates spatial attributes and reasoning into AI computations, is increasingly prevalent in urban studies [ 134 , 135 ], offering diverse applications and opportunities for environmentally sustainable urban planning and development.…”

Section: Results: Analysis and Synthesismentioning

confidence: 99%

See 1 more Smart Citation

The synergistic interplay of artificial intelligence and digital twin in environmentally planning sustainable smart cities: A comprehensive systematic review

Bibri,

Huang,

Jagatheesaperumal

et al. 2024

Environmental Science and Ecotechnology

View full text Add to dashboard Cite

Section: Results: Analysis and Synthesismentioning

confidence: 99%

“… Liu et al. [ 135 ] Genetic and evolutionary AI algorithms Freight parking management Real-time parking connectivity, logistics efficiency optimization, urban resource allocation Integration of cognitive DTs with logistics, improved logistics efficiency, and resource allocation optimization. Manocha et al.…”

Section: Results: Analysis and Synthesismentioning

confidence: 99%

The synergistic interplay of artificial intelligence and digital twin in environmentally planning sustainable smart cities: A comprehensive systematic review

Bibri,

Huang,

Jagatheesaperumal

et al. 2024

Environmental Science and Ecotechnology

View full text Add to dashboard Cite

“…We deliberately chose not to include spatial coordinates during pretraining because we wanted to learn a general representation of gene expression variation across both modalities, fully supervised by gene expression alone. Nevertheless, we anticipate that future iterations of Nicheformer will account for spatial relationships of cells by encoding spatial neighbor graphs, for example, and potentially leveraging graph transformer architectures 96–98 for the pretraining stage on spatial transcriptomics data. Graph transformers excel at modeling relationships between nodes in graphs, making them ideal for capturing nearest-neighbor effects on a cell’s transcriptome.…”

Section: Discussionmentioning

confidence: 99%

Nicheformer: a foundation model for single-cell and spatial omics

Schaar,

Tejada-Lapuerta,

Palla

et al. 2024

Preprint

View full text Add to dashboard Cite

Tissue makeup and the corresponding orchestration of vital biological activities, ranging from development and differentiation to immune response and regeneration, rely fundamentally on the cellular microenvironment and the interactions between cells. Spatial single-cell genomics allows probing such interactions in an unbiased and, increasingly, scalable fashion. To learn a unified cell representation that accounts for local dependencies in the cellular microenvironment and the underlying cell interactions, we propose to generalize recent foundation modeling approaches for disassociated single-cell transcriptomics to the spatial omics setting. Our model, Nicheformer, is a transformer-based foundation model that combines human and mouse dissociated single-cell and targeted spatial transcriptomics data to learn a cellular representation useful for a large variety of downstream tasks. Nicheformer is pretrained on over 57 million dissociated and 53 million spatially resolved cells across 73 tissues from both human and mouse. Subsequently, the model is fine-tuned on spatial tasks for spatial omics data to decode spatially resolved cellular information. We demonstrate the usefulness of Nicheformer in both zero-shot-like as well as fine-tuning scenarios on a novel set of spatially-relevant downstream tasks such as spatial density prediction or niche and region label prediction. In particular, we show that Nicheformer enables the prediction of the spatial context of dissociated cells, allowing the transfer of rich spatial information to scRNA-seq datasets. We define a series of novel spatial prediction problems and observe consistent top performance of Nicheformer, demonstrating the advantage of the improved model capacity of the underlying transformer. Altogether, our large-scale resource of more than 110 million cells in a partial spatial context, together with the set of novel spatial learning tasks and the Nicheformer model itself, will pave the way for the next generation of machine-learning models for spatial single-cell analysis.

show abstract

“…While many 3D urban visual analytics systems have been proposed, few empirical studies were done in these scenarios to evaluate the visualizations, interactions and the overall human experience and exposure within urban spaces (see Section 10.7). This oversight suggests a slight potential disconnect with the latest human‐centric developments in digital twins and geospatial science [RS19,LZL*23]. As discussed throughout this survey, the 3D urban environment adds a great deal of complexity to the navigation, interactions, and occlusion handling.…”

Section: Discussion Research Directions and Opportunitiesmentioning

confidence: 99%

The State of the Art in Visual Analytics for 3D Urban Data

Miranda,

Ortner,

Moreira

et al. 2024

Computer Graphics Forum

View full text Add to dashboard Cite

Urbanization has amplified the importance of three‐dimensional structures in urban environments for a wide range of phenomena that are of significant interest to diverse stakeholders. With the growing availability of 3D urban data, numerous studies have focused on developing visual analysis techniques tailored to the unique characteristics of urban environments. However, incorporating the third dimension into visual analytics introduces additional challenges in designing effective visual tools to tackle urban data's diverse complexities. In this paper, we present a survey on visual analytics of 3D urban data. Our work characterizes published works along three main dimensions (why, what, and how), considering use cases, analysis tasks, data, visualizations, and interactions. We provide a fine‐grained categorization of published works from visualization journals and conferences, as well as from a myriad of urban domains, including urban planning, architecture, and engineering. By incorporating perspectives from both urban and visualization experts, we identify literature gaps, motivate visualization researchers to understand challenges and opportunities, and indicate future research directions.

show abstract

Towards Human-centric Digital Twins: Leveraging Computer Vision and Graph Models to Predict Outdoor Comfort

Cited by 25 publications

References 80 publications

The synergistic interplay of artificial intelligence and digital twin in environmentally planning sustainable smart cities: A comprehensive systematic review

The synergistic interplay of artificial intelligence and digital twin in environmentally planning sustainable smart cities: A comprehensive systematic review

Nicheformer: a foundation model for single-cell and spatial omics

The State of the Art in Visual Analytics for 3D Urban Data

Contact Info

Product

Resources

About