Setting the Yardstick: A Quantitative Metric for Effectively Measuring Tactile Internet

Verburg, Joseph; Kroep, H.J.C.; Gokhale, Vineet; Prasad, R. Venkatesha; Rao, Vijay S.

doi:10.1109/infocom41043.2020.9155540

Cited by 10 publications

(3 citation statements)

References 28 publications

(33 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are a few requirements for an experiment that benefit the statistical relevance of the test results. 1 The participants should perform the same task multiple times under different settings. 2 To maximize the perception, the participants should concentrate.…”

Section: Subjective Analysismentioning

confidence: 99%

ETVO: Effectively Measuring Tactile Internet with Experimental Validation

Kroep¹,

Gokhale²,

Verburg³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The next frontier in communications is teleoperation -manipulation and control of remote environments with feedback. Compared to conventional networked applications, teleoperation poses widely different requirements, ultra-low latency (ULL) is primary. Realizing ULL communication demands significant redesign of conventional networking techniques, and the network infrastructure envisioned for achieving this is termed as Tactile Internet (TI). The design of the network infrastructure and meaningful performance metrics are crucial for seamless TI communication. However, existing performance metrics fall severely short of comprehensively characterizing TI performance. We take the first step towards bridging this gap. We take Dynamic Time Warping(DTW) as the basis of our work and identify necessary changes for characterizing TI performance. Through substantial refinements to DTW, we design Effective Time-and Value-Offset (ETVO) -a new method for measuring the fine-grained performance of TI systems. Through an in-depth objective analysis, we demonstrate the improvements of ETVO over DTW. Through human-in-the-loop subjective experiments, we demonstrate how and why existing QoS and QoE methods fall short of estimating the TI session performance accurately. Using subjective experiments, we demonstrate the behavior of the proposed metrics, their ability to match theoretically derived performance, and finally their ability to reflect user satisfaction in a practical setting. The results are highly encouraging.

show abstract

Section: Subjective Analysismentioning

confidence: 99%

ETVO: Effectively Measuring Tactile Internet with Experimental Validation

Kroep¹,

Gokhale²,

Verburg³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The works in [6], [9], [14] use RMSE as the objective metric. Recently, another work [15] proposed using both temporal and position errors for characterizing the quality of TI interaction. We showed blind spots for RMSE (Sec.…”

Section: B Performance Metrics In Timentioning

confidence: 99%

“…2) Subjective Metrics: Literature provides many works that have conducted a subjective evaluation of TI experiments. Collecting subjective grades by including human subjects in the is the most commonly used metric [5], [16], [9], [15]. However, this approach is often cumbersome and time-consuming.…”

Section: B Performance Metrics In Timentioning

confidence: 99%

Blind Spots of Objective Measures: Exploiting Imperceivable Errors for Immersive Tactile Internet

Kroep

Gokhale

Prasad

2022

2022 ACM/IEEE 13th International Conference on Cyber-Physical Systems (ICCPS)

Self Cite

View full text Add to dashboard Cite

Tactile Internet (TI) enables the transfer of human skills over the Internet, enabling teleoperation with force feedback. Advancements are being made rapidly at several fronts to realize a functional TI soon. Generally, TI is expected to faithfully reproduce operator's actions at the other end, where a robotic arm emulates it while providing force feedback to the operator. Performance of TI is usually characterized using objective metrics such as network delay, packet losses, and RMSE. Pari passu, subjective evaluations are used as additional validation, and performance evaluation itself is not primarily based on user experience. Hence objective evaluation, which generally minimizes error (signal mismatch), is oblivious to subjective experience.In this paper, we argue that user-centric designs of TI solutions are necessary. We first consider a few common TI errors and examine their perceivability. The idea is to reduce the impact of perceivable errors and exploit the imperceivable errors to our advantage, while the objective metrics may indicate that the errors are high. To harness the imperceivable errors, we design Adaptive Offset Framework (AOF) to improve the TI signal reconstruction under realistic network settings. We use AOF to highlight the contradictory inferences drawn by objective and subjective evaluations while realizing that subjective evaluations are closer to ground truth. This strongly suggests the existence of 'blind spots of objective measures'. Further, we show that AOF significantly improves the user grade, up to 3 points (on a scale of 10) compared to the standard reconstruction method.

show abstract