1Automated hostapd/wpa_supplicant testing with mac80211_hwsim
2------------------------------------------------------------
3
4This directory contains testing infrastructure and test cases to run
5automated tests of full hostapd and wpa_supplicant functionality. This
6testing is done with the help of mac80211_hwsim which is Linux kernel
7driver that simulates IEEE 802.11 radios without requiring any
8additional hardware. This setup most of the hostapd and wpa_supplicant
9functionality (and large parts of the Linux cfg80211 and mac80211
10functionality for that matter) to be tested.
11
12mac80211_hwsim is loaded with five simulated radios to allow different
13device combinations to be tested. wlantest is used analyze raw packets
14captured through the hwsim0 monitor interface that capture all frames
15sent on all channels. wlantest is used to store the frames for
16analysis. Three wpa_supplicant processes are used to control three
17virtual radios and one hostapd process is used to dynamically control
18the other two virtual radios. wpa_supplicant/hostapd test functionality
19is used to verify that data connection (both unicast and broadcast)
20works between two netdevs.
21
22The python scripts and tools in this directory control test case
23execution. They interact wpa_supplicant and hostapd through control
24interfaces to perform the operations. In addition, wlantest_cli is used
25to verify that operations have been performed correctly and that the
26network connection works in the expected way.
27
28These test cases are run automatically against the hostap.git commits
29for regression testing and to help in keeping the hostap.git master
30branch in stable state. Results from these tests are available here:
31http://buildbot.w1.fi/hwsim/
32
33
34Building binaries for testing
35-----------------------------
36
37You will need to build (or use already built) components to be
38tested. These are available in the hostap.git repository and can be
39built for example as follows:
40
41cd ../../wpa_supplicant
42cp ../tests/hwsim/example-wpa_supplicant.config .config
43make clean
44make
45cd ../hostapd
46cp ../tests/hwsim/example-hostapd.config .config
47make clean
48make hostapd hostapd_cli hlr_auc_gw
49cd ../wlantest
50make clean
51make
52
53Alternatively, the build.sh script here can be used to run these steps
54with conditional creation of .config files only if they do not exist.
55
56The test scripts can find the binaries in the locations where they were
57built. It is also possible to install wlantest_cli somewhere on the path
58to use pre-built tools.
59
60Please note that some of the configuration parameters used to enable
61more testing coverage may require development packages that may not be
62installed by default in many distributions. For example, following
63Debian/Ubuntu packages are likely to be needed:
64- binutils-dev
65- libsqlite3-dev
66- libpcap-dev
67
68example-setup.txt provides more complete step-by-step example on how a
69test setup can be built.
70
71
72wpaspy
73------
74
75The python scripts use wpaspy.py to interact with the wpa_supplicant
76control interface, but the run-tests.py script adds the (relative)
77path into the environment so it doesn't need to be installed.
78
79
80mac80211_hwsim
81--------------
82
83mac80211_hwsim kernel module is available from the upstream Linux
84kernel. Some Linux distributions enable it by default. If that's not the
85case, you can either enable it in the kernel configuration
86(CONFIG_MAC80211_HWSIM=m) and rebuild your kernel or use Backports with
87CPTCFG_MAC80211_HWSIM=m to replace the wireless LAN components in the
88base kernel.
89
90
91sudo
92----
93
94Some parts of the testing process requires root privileges. The test
95scripts are currently using sudo to achieve this. To be able to run the
96tests, you'll probably want to enable sudo with a timeout to not expire
97password entry very quickly. For example, use this in the sudoers file:
98
99Defaults        env_reset,timestamp_timeout=180
100
101Or on a dedicated test system, you could even disable password prompting
102with this in sudoers:
103
104%sudo   ALL=NOPASSWD: ALL
105
106
107Other network interfaces
108------------------------
109
110Some of the test scripts are still using hardcoded interface names, so
111the easiest way of making things work is to avoid using other network
112devices that may use conflicting interface names. For example, unload
113any wireless LAN driver before running the tests and make sure that
114wlan0..4 gets assigned as the interface names for the mac80211_hwsim
115radios. It may also be possible to rename the interface expectations in
116run-tests.py to allow other names to be used.
117
118Please also note that some commonly enabled tools, like NetworkManager,
119may end up trying to control new network interfaces automatically. This
120can result in conflicts with the test scripts and you may need to
121disable such network services or at least mark the mac80211_hwsim wlan#
122interfaces as umanaged. As an example, this can be done in
123/etc/NetworkManager/NetworkManager.conf with following addition:
124
125[keyfile]
126unmanaged-devices=mac:02:00:00:00:00:00;mac:02:00:00:00:01:00;mac:02:00:00:00:02:00;mac:02:00:00:00:03:00;mac:02:00:00:00:04:00
127
128
129Running tests
130-------------
131
132Simplest way to run a full set of the test cases is by running
133run-all.sh in tests/hwsim directory. This will use start.sh to load the
134mac80211_hwsim module and start wpa_supplicant, hostapd, and various
135test tools. run-tests.sh is then used to run through all the defined
136test cases and stop.sh to stop the programs and unload the kernel
137module.
138
139run-all.sh can be used to run the same test cases under different
140conditions:
141
142# run normal test cases
143./run-all.sh
144
145# run normal test cases under valgrind
146./run-all.sh valgrind
147
148# run normal test cases with Linux tracing
149./run-all.sh trace
150
151# run normal test cases with multi channel support (see details below)
152./run-all.sh channels=<num of channels>
153
154run-all.sh directs debug logs into the logs subdirectory (or $LOGDIR if
155present in the environment). Log file names include the current UNIX
156timestamp and a postfix to identify the specific log:
157- *.log0 = wpa_supplicant debug log for the first radio
158- *.log1 = wpa_supplicant debug log for the second radio
159- *.log2 = wpa_supplicant debug log for the third radio
160- *.hostapd = hostapd debug log
161- hwsim0 = wlantest debug log
162- hwsim0.pcapng = capture with all frames exchanged during the tests
163- *.log = debug prints from the test scripts
164- trace.dat = Linux tracing record (if enabled)
165- hlr_auc_gw - hlr_auc_gw (EAP-SIM/AKA/AKA' authentication) log
166- auth_serv - hostapd as RADIUS authentication server log
167
168
169For manual testing, ./start.sh can be used to initialize interfaces and
170programs and run-tests.py to execute one or more test
171cases. run-tests.py output verbosity can be controlled with -d (more
172verbose debug output) and -q (less verbose output) on the command
173line. "-f <module name>" (pointing to file test_<module name>.py) can be
174used to specify that all test cases from a single file are to be
175run. Test name as the last command line argument can be specified that a
176single test case is to be run (e.g., "./run-tests.py ap_pmf_required").
177
178Notice that some tests require the driver to support concurrent
179operation on multi channels in order to run. These tests will be skipped
180in case the driver does not support multi channels. To enable support
181for multi channel, the number of supported channel is passed as an
182argument to run-all.sh or start.sh
183
184
185Adding/modifying test cases
186---------------------------
187
188All the test cases are defined in the test_*.py files. These are python
189scripts that can use the local helper classes to interact with the test
190components. While various python constructs can be used in the scripts,
191only a minimal level of python knowledge should really be needed to
192modify and add new test cases. The easiest starting point for this is
193likely to take a look at some of the example scripts. When working on a
194new test, run-tests.py with -d and the test case name on the command
195line is a convenient way of verifying functionality.
196
197run-tests.py will automatically import all test cases from the test_*.py
198files in this directory. All functions starting with the "test_" prefix
199in these files are assumed to be test cases. Each test case is named by
200the function name following the "test_" prefix.
201
202
203Results database
204----------------
205
206run-tests.py can be requested to write results from the execution of
207each test case into an sqlite database. The "-S <path to database>" and
208"-b <build id>" command line arguments can be used to do that. The
209database must have been prepared before this, e.g., with following:
210
211cat | sqlite3 /tmp/example.db <<EOF
212CREATE TABLE results (test,result,run,time,duration,build,commitid);
213CREATE INDEX results_idx ON results (test);
214CREATE INDEX results_idx2 ON results (run);
215CREATE TABLE tests (test,description);
216CREATE UNIQUE INDEX tests_idx ON tests (test);
217CREATE TABLE logs (test,run,type,contents);
218CREATE INDEX logs_idx ON logs (test);
219CREATE INDEX logs_idx2 ON logs (run);
220EOF
221