A Novel Scenarios Engineering Methodology for Foundation Models in Metaverse

Abstract

Foundation models are used to train a broad system of general data to build adaptations to new bottlenecks. Typically, they contain hundreds of billions of hyperparameters that have been trained with hundreds of gigabytes of data. However, this type of black-box vulnerability places foundation models at risk of data poisoning attacks that are designed to pass on misinformation or purposely introduce machine bias. Moreover, ordinary researchers have not been able to completely participate due to the rise in deployment standards. This study introduces the theoretical framework of scenarios engineering (SE) for building accessible and reliable foundation models in metaverse, namely, "SE-enabled foundation models in metaverse." Particularly, the research framework comprises a six-layer architecture (infrastructure layer, operation layer, knowledge layer, intelligence layer, management layer, and interaction layer), which can provide controllability, trustworthiness, and interactivity for the foundation models in metaverse. This creates closed-loop, virtual-real, and human-machine environments that provides the best indices and goals for the foundation models, which allows us to fully validate and calibrate the corresponding models. Then, examples of use cases from the automotive industry are listed to provide transparency on the possible use and benefits of our approach. Finally, the open research topics of related frameworks are discussed.

Metarobotics for Industry and Society: Vision, Technologies, and Opportunities

“…Local AI/ML models (LMs) are trained on-premise using sensitive data of citizens and open IoT data to update public foundation models (FMs) without disclosing raw data via Federated Learning (FL). Scenarios Engineering is a methodology to develop FMs in the Metaverse [30]. Resulting Federated Foundation Models (FFM) [31] with relay mechanisms to support energy-limited devices [32] take advantage of edge AI for an intelligent (e.g., context-and resource-aware) and continuous access to applications from society to industry with human-centered values.…”

“…Learned multi-modal correlations between images and tactile features allow it to adapt robots and inform citizens about various environment properties like roughness [47]. Cross-modal reasoning can be achieved using interconnected multi-modal knowledge graphs (MMKG) [30], [48] that evolve from structured crowd sourcing, IoT, and synthetic data. This leads to a swarm of cogDTs (as MMKG-nodes) with maturing intelligence in terms of the depth and diversity of knowledge integration, as well as edge and cloudlet processing rates (see figure 5).…”

Scenarios Engineering: Enabling Trustworthy and Effective AI for Autonomous Vehicles

“…Scenarios engineering (SE) [2], [3] is proposed to achieve trustworthy and effective AI for AVs. It was inspired by parallel intelligent [4], [5] (PI), which provides an effective way of making small data into big data and then refining big data into deep intelligence for specific tasks.…”

Toward Fair and Thrilling Autonomous Racing: Governance Rules and Performance Metrics for the Autonomous One

“…In its beta season, Roborace introduced the Metaverse-based approach, incorporating virtual and dynamic elements on the racetrack, thereby enhancing the suspense even in single-player time-trial races. Although critics have pointed out potential visibility issues for spectators due to the introduction of virtual objects [31], the affordability and safety improvements that these virtual objects bring to the race cannot be ignored [32,33].…”