On the Credibility of Manet Simulations

Abstract

Computer P u b l i s h e d b y t h e I E E E C o m p u t e r S o c i e t ycols because designers can use it to evaluate theoretical systems. However, if the simulation doesn't reflect an important aspect of reality, it can't give insight into the operating characteristics of the system the developers are studying. Generalization and lack of rigor can lead to inaccurate data, which can result in wrong conclusions or inappropriate implementation decisions.Consider, for example, the inappropriate exercise of feature isolation. Investigators can use simulation to isolate target features by varying only those features, while holding all other parameters constant. Many manet simulations compare protocols by varying only the protocol in each simulation run.Unfortunately, the constant factors can (passively) inject inaccuracies into the simulation. This type of unintended side effect can occur when, for example, investigators run a simulation with protocol A and subsequently run the identical simulation with protocol B. Intuitively, with all other settings the same, the protocol that performs the best head-to-head should have the better performance. However, this isn't necessarily true.In reality, static features can dominate performance. For example, protocol A might outperform B at 10-megabits-per-second (mbps) bandwidth, but the converse might be true at 100 mbps. Worse yet, even if the simulation results reflect these two situations, they'd say little about performance in a 50-mbps or 500-mbps network. Omitting such intermediate-valued tests can lead to false conclusions. Determining which features to vary, how much to vary them, and in what combinations with which other features to simulate them is difficult. Often, the volume of interaction possibilities precludes thorough simulation, leading to less than rigorous validaSimulation is useful for evaluating protocol performance and operation. However, the lack of rigor with which it's applied threatens the credibility of the published research within the manet research community. Todd R. Andel and Alec YasinsacFlorida State University M obile ad hoc networks (manets) allow rapid deployment because they don't depend on a fixed infrastructure. Manet nodes can participate as the source, the destination, or an intermediate router. This flexibility is attractive for military applications, disaster-response situations, and academic environments where fixed networking infrastructures might not be available.Simulation has proven to be a valuable tool in many areas where analytical methods aren't applicable and experimentation isn't feasible. Researchers generally use simulation to analyze system performance prior to physical design or to compare multiple alternatives over a wide range of conditions. Unfortunately, errors in simulation models or improper data analysis often produce incorrect or misleading results.The mainstream approach in the manet research community follows the development, simulation, and publish process. Manet publications typically include performance s...

Enhanced Secure Trusted AODV (ESTA) Protocol to Mitigate Blackhole Attack in Mobile Ad Hoc Networks

“…The authors of [17,18] has raised a very critical aspect related to the research being done in the field of MANETs specially related to the use of simulation tools. In a survey [17], it was concluded that most of the research articles related this field used simulation for experimentation and random waypoint models was used by 64% of simulations.…”

MANET protocol simulations considered harmful: the case for benchmarking

“…The work in [5] reports a survey similar to [8] in which they studied 280 papers on simulations of peer-to-peer systems and found that 71 papers did not even state which simulator was used.…”

“…The work described in [8] also finds in 2006 that very little has changed in the MANET community in terms of simulation-based evaluation methodology. Almost 90 percent of papers do not even specify the simulator version, and over half do not specify the number of simulation runs.…”

“…Comparing the popularity of individual simulators used in these published papers to a survey of the 2000-2005 MobiHoc papers that used simulation-based evaluations [8], we found that Network Simulator 2 (ns-2) tied with custom simulators (13 of the 47 simulated papers for each, or 27.7 percent each). This is quite different from the previous study, which ranked ns-2 at 43.8 percent and self-developed simulators at 27.3 percent.…”

Comparison of two routing metrics in OLSR on a grid based mesh network

“…Traditionally ad hoc and mesh networking research has mostly been carried out using simulation tools but many recent studies [1] have revealed the inherent limitations these have in modelling the physical layer and aspects of the MAC layer. Researchers should acknowledge that the results from a simulation tool only give a rough estimate of performance.…”