R&D in wireless networking typically depends on experimentation to make realistic evaluations, since simulation is inherently a simplification of the real-world. However, experimentation is limited in aspects where simulation excels, such as repeatability and reproducibility.
Real wireless experiments are hardly repeatable. Given the same input they can produce very different output results, since wireless communications are influenced by external random phenomena such as noise, interference, and multipath. Real experiments are also difficult to reproduce: either the original community testbed is unavailable – offline or running other experiments – or the custom testbed used is inaccessible.
Fed4FIRE+ wireless testbeds such as w-iLab.t and NITOS, although deployed in controlled environments, do not fully address the problem. The CONCRETE tool used in such testbeds assures the repeatability and reproducibility of experiments, but ignores executions whose results are also representative of the system operation and often reveal unpredicted behaviour that must be understood.
What if we could make any wireless experiment repeatable and reproducible under the same exact conditions? What if we could share the same Fed4FIRE+ testbed execution conditions among an "infinite" number of users? What if we could run wireless experiments faster than in real time?
INESC TEC has been developing the Offline Experimentation (OE) approach that combines the best of simulation and experimentation to achieve the above-mentioned goals. By relying on Network Simulator 3 (ns-3) and its good simulation capabilities from the MAC to the application layer, we have been exploring how ns-3 can be used to replicate real-world wireless experiments using real traces containing 1) position of nodes and 2) the quality of each radio link.
The SIMBED project aimed at running a set of wireless experiments on top of the controlled environments of w-ilab.t and NITOS Fed4FIRE+ testbeds to further validate the OE approach. For that purpose, we configured different fixed and mobile experimental scenarios, representative of Wi-Fi range of operation, and measured the attained network performance using metrics such as throughput and Round-Trip Time (RTT). Then, we repeated each experiment using, both, Pure Simulation (PS) and OE approaches based on ns-3, also measuring the network performance for the same set of executions of experiments for all the different scenarios.
By comparing the performance metrics of each real experiment with its PS and OE counterparts, we were able to measure the relative error of each simulation approach relatively to the real experiments, as well as the accuracy gains introduced by the OE approach when compared to the PS traditional alternative. The main results show that it is possible to repeat and reproduce real experiments in ns-3, using the OE approach, achieving closer to real performance than using the PS approach. For all the experiments performed in SIMBED, using the OE approach resulted in an average accuracy gain of 59% when comparing to the PS approach.
These results were important for validating a PhD thesis contribution related to the OE approach, as well as for producing two conference papers and one journal paper. The SIMBED results increased our confidence on the accuracy of the OE approach and are envisioned to foster the adoption of the OE approach by the networking community, in complement to the use of real experimentation.