Wireless networking R&D depends on experimentation to make realistic evaluations of networking solutions, as simulation is inherently a simplification of the real-world. However, despite more realistic, experimentation is limited in aspects where simulation excels, such as repeatability and reproducibility.
Real wireless experiments may be difficult to repeat. For the same input they can produce very different output results, since wireless communications are influenced by external phenomena such as noise, interference, and multipath. Even if repeatable, experiments may still be difficult to reproduce. Namely, other researchers may be unable to reproduce an experiment and confirm previous experimental results right away, because either the testbed is unavailable – offline or running other experiments when using a community testbed –, or inaccessible at all when a custom testbed was originally used.
Fed4FIRE+ testbeds such as w-iLab.t, although operating in controlled environment, do not fully address the problem. This is even more evident in testbeds running in non-controlled and very dynamic environments, such as CityLab, where they may suffer from radio interference and competition from existing networks sharing the same radio spectrum.
What if we could make any wireless experiment repeatable and reproducible? What if we could share the same Fed4FIRE+ testbed execution conditions among an "infinite" number of users? What if we could run wireless experiments faster than in real time?
INESC TEC has developed the Offline Experimentation (OE) approach that combines the best of simulation and experimentation to achieve the above-mentioned goals. By relying on Network Simulator 3 (ns-3) and its good simulation capabilities from the MAC to the Application layer, we have been exploring how ns-3 can be used to replicate real-world wireless experiments using real traces containing 1) position of nodes and 2) the quality of each radio link.
The previous SIMBED project validated the OE approach for controlled environments and helped identifying some of its limitations. The OE approach was then improved to support experiments using Multiple-In-Multiple-Out (MIMO) and shared radio spectrum with concurrent networks. To further validate the improved OE approach, the SIMBED+ project aimed at running a set of experiments on top of the controlled and non-controlled environments of w-iLab.t and CityLab Fed4FIRE+ testbeds. For that purpose, we configured different fixed experimental scenarios, representative of Wi-Fi range of operation, subjected to controlled and non-controlled interference from concurrent experiments. For each experiment the achieved network throughput was measured. Then, we repeated each experiment using, both, Pure Simulation (PS) and OE approaches (now with MIMO and channel occupancy information) based on ns-3, also measuring the network throughput for the same set of experiments.
To compare the network throughput of the real experiments with their PS and OE counterparts, Cumulative Distribution Function (CDF) curves were used, plotting all 1-second average throughput samples. For all the experiments performed in SIMBED+, using the improved OE approach resulted in network throughput considerably closer to real than using the PS approach. This is even more evident in non-controlled environments using MIMO and shared radio spectrum.
These results were important for further validating the OE approach, producing two conference papers and one journal paper. The SIMBED+ results increased our confidence on the OE approach ability of reproducing past experiments, and are envisioned to foster the adoption of the OE approach by the networking community, in complement to the use of real experimentation.