OpenBenchmark: Repeatable and Reproducible Internet of Things Experimentation on Testbeds

Vučinić, Mališa; Skrbic, Bozidar; Kočan, Enis; Pejanović-Djurišić, Milica; Watteyne, Thomas

doi:10.1109/infcomw.2019.8845160

Cited by 6 publications

(6 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, we provide a comparison of five testbeds under consideration. Specifically, we consider the following key points: (1) architecture, (2) resources available for wireless experiments, (3) operating system which can be deployed, (4) IoT capabilities, (5) limitations, (6) automatic configuration, (7) SDN/OpenFlow capability, (8) data that can be collected, and (9) machine learning capability. The comparison is shown in Table 1 and is explained in the following subsections.…”

Section: Theoretical Comparison Of the Testbedsmentioning

confidence: 99%

See 1 more Smart Citation

Future Wireless Networking Experiments Escaping Simulations

et al. 2022

View full text Add to dashboard Cite

In computer networking, simulations are widely used to test and analyse new protocols and ideas. Currently, there are a number of open real testbeds available to test the new protocols. In the EU, for example, there are Fed4Fire testbeds, while in the US, there are POWDER and COSMOS testbeds. Several other countries, including Japan, Brazil, India, and China, have also developed next-generation testbeds. Compared to simulations, these testbeds offer a more realistic way to test protocols and prototypes. In this paper, we examine some available wireless testbeds from the EU and the US, which are part of an open-call EU project under the NGIAtlantic H2020 initiative to conduct Software-Defined Networking (SDN) experiments on intelligent Internet of Things (IoT) networks. Furthermore, the paper presents benchmarking results and failure recovery results from each of the considered testbeds using a variety of wireless network topologies. The paper compares the testbeds based on throughput, latency, jitter, resources available, and failure recovery time, by sending different types of traffic. The results demonstrate the feasibility of performing wireless experiments on different testbeds in the US and the EU. Further, issues faced during experimentation on EU and US testbeds are also reported.

show abstract

Section: Theoretical Comparison Of the Testbedsmentioning

confidence: 99%

“…The goal is to find the bottlenecks of the considered testbeds. Numerous papers in the literature focus on a particular scenario on a particular testbed, including [7][8][9][10][11][12]. The objective of this paper is to report the results of comparing a similar network scenario deployed on different testbeds.…”

Section: Introductionmentioning

confidence: 99%

Future Wireless Networking Experiments Escaping Simulations

et al. 2022

View full text Add to dashboard Cite

show abstract

“…This challenge has been recognized in the community, spurring initiatives such as the IoT Benchmarking consortium (https://iotbench.ethz. ch/ (accessed on 12 December 2021)), OpenBenchmark [150], and others discussed in Section 5.3-yet several tasks remain unresolved.…”

Section: Future Research Areas and Current Challengesmentioning

confidence: 99%

“…Furthermore, most evaluations use different settings for traffic patterns, duration, RPL configuration and convergence, software versions, and so on—making comparison inherently difficult. This challenge has been recognized in the community, spurring initiatives such as the IoT Benchmarking consortium ( (accessed on 12 December 2021)), OpenBenchmark [ 150 ], and others discussed in Section 5.3 —yet several tasks remain unresolved.…”

Section: Future Research Areas and Current Challengesmentioning

confidence: 99%

A Survey of 802.15.4 TSCH Schedulers for a Standardized Industrial Internet of Things

Urke

Kure

Øvsthus

2021

Sensors

View full text Add to dashboard Cite

Concepts such as Industry 4.0 and Cyber-Physical Systems may bring forward a new industrial revolution. These concepts require extensive connectivity far beyond what is provided by traditional industrial networks. The Industrial Internet of Things (IIoT) bridges this gap by employing wireless connectivity and IP networking. In order for wireless networks to meet the strict requirements of the industrial domain, the Time Slotted Channel Hopping (TSCH) MAC is often employed. The properties of a TSCH network are defined by the schedule, which dictates transmission opportunities for all nodes. We survey the literature for these schedulers, describe and organize them according to their operation: Centralized, Collaborative, Autonomous, Hybrid, and Static. For each category and the field as a whole, we provide a holistic view and describe historical trends, highlight key developments, and identify trends, such as the attention towards autonomous mechanisms. Each of the 76 schedulers is analyzed into their common components to allow for comparison between schedulers and a deeper understanding of functionality and key properties. This reveals trends such as increasing complexity and the utilization of centralized principles in several collaborative schedulers. Further, each scheduler is evaluated qualitatively to identify its objectives. Altogether this allows us to point out challenges in existing work and identify areas for future research, including fault tolerance, scalability, non-convergecast traffic patterns, and hybrid scheduling strategies.

show abstract

“…1). The platform takes care of testbed 1 The article is an extension of the paper [7] published in the INFOCOM 2019 CNERT workshop. This version complements with the produced KPIs through an extensive experimentation campaign performed using the Open-Benchmark platform.…”

Section: Introductionmentioning

confidence: 99%

Key Performance Indicators of the Reference 6TiSCH Implementation in Internet-of-Things Scenarios

et al. 2020

Self Cite

View full text Add to dashboard Cite

Tens of thousands of wireless industrial monitoring deployments exist today, logging more than 18 billion operating hours. These solutions have been around for over a decade and are based on standards such as WirelessHART and ISA100.11a to provide performance guarantees to the applications. The new trend in industry deployments is the convergence of operational and information technologies happening through the Industrial Internet of Things (IIoT) paradigm. The challenge is to bridge the performance of these well-proven industrial standards with the interoperability of IP-based systems. The Internet Engineering Task Force (IETF), the organization behind most of the technical solutions of the Internet, has produced a set of specifications with this requirement in mind. The output of this effort is the 6TiSCH protocol stack based on open standards, such as those that have played a key role in the Internet's ubiquitous adoption. The standardization of 6TiSCH is done. The state-of-the-art research work focus is on important, but niche, optimizations and performance evaluations of the 6TiSCH stack. This paper takes a different approachit evaluates the performance of the standards-compliant 6TiSCH solution from the end user point of view. It does so on two experimental testbeds, in typical IoT test scenarios based on a well-defined experimentation methodology. We provide a set of Key Performance Indicators (KPIs) useful for the end user to decide whether the 6TiSCH technology is a good fit performance-wise for a particular use case. We demonstrate reliability of a vanilla open-source implementation of 6TiSCH above 99.99%, upstream latency on the order of a second and radio duty cycle well below 1%.

show abstract

OpenBenchmark: Repeatable and Reproducible Internet of Things Experimentation on Testbeds

Cited by 6 publications

References 13 publications

Future Wireless Networking Experiments Escaping Simulations

Future Wireless Networking Experiments Escaping Simulations

A Survey of 802.15.4 TSCH Schedulers for a Standardized Industrial Internet of Things

Key Performance Indicators of the Reference 6TiSCH Implementation in Internet-of-Things Scenarios

Contact Info

Product

Resources

About