Writing tests is extra work, compared to the way many small (and some not\[hy]so\[hy]small) software shops operate. For this reason, the testing requirement may be turned off.

The win is that the tests hang around forever, catching minor and major slips before they become embarrassing "features" in a released product. Prevention is cheaper than cure in this case, the tests save work down the track.

All of the "extra work" of writing tests is a long\[hy]term win, where old problems never again reappear. All of the "extra work" of reviewing changes means that another pair of eyes sees the code and finds potential problems before they manifest themselves in shipped product. All of the "extra work" of integration ensures that the baseline always works, and is always self\[hy]consistent. All of the "extra work" of having a baseline and separate development directories allows multiple parallel development, with no inter\[hy]developer interference; and the baseline always works, it is never in an "in\[hy]between" state. In each case, not doing this "extra work" is a false economy.

The existence of these tests, though, is what determines which projects are most suited to Aegis and which are not. It should be noted that suitability is a continuous scale, not black\[hy]and\[hy]white. With effort and resources, almost anything fits.

Projects most suited to supervision by Aegis are straight programs. What the non\[hy]systems\[hy]programmers out there call "tools" and sometimes "applications". These are programs which take a pile of input, chew on it, and emit a pile of output. The tests can then compare actual outputs with expected outputs.

As an example, you could be writing a sed (1) look\[hy]alike, a public domain clone of the X sed utility. You could write tests which exercise every feature (insertion, deletion, etc.) and generate the expected output with the real X sed. You write the code, and run the tests; you can immediately see if the output matches expectations.

This is a simple example. More complex examples exist, such as Aegis itself. The Aegis program is used to supervise its own development. Tests consist of sequences of commands and expected results are tested for.

Other types of software have been developed using Aegis: compilers and interpreters, client\[hy]server model software, magnetic tape utilities, graphics software such as a ray\[hy]tracer. The range is vast, but it is not all types of software.

For many years there have been full\[hy]screen applications on text terminals. In more recent times there is increasing use of graphical interfaces.

In developing these types of programs it is still possible to use Aegis, but several options need to be explored.

There are screen emulators for both full\[hy]screen text and X11 available. Using these emulators, it is possible to test the user interface, and test via the user interface. As yet, the author knows of no freely available emulators suitable for testing via Aegis. If you find one, please let me know.

You may choose to use Aegis simply for its ability to provide controlled access to a large source. You still get the history and change mechanisms, the baseline model, the enforced review. You simply don't test all changes, because figuring out what is on the screen, and testing it against expectations, is too hard.

If the program has a command line interface, in addition to the full\[hy]screen or GUI interface, the functionality accessible from the command line may be tested using Aegis.

It is possible that "limited testing" actually means "no testing", if you have no functionality accessible from the command line.

Another alternative is to provide hooks into your program allowing you to substitute a file for user input, and to be able to trigger the dump of a "screen image". The simulated user input can then be fed to the program, and the screen dump (in some terminal\[hy]independent form) can be compared against expectations.

This is easier for full\[hy]screen applications, than for X11 applications. You need to judge the cost\[hy]benefit trade\[hy]off. Cost of development, cost of storage space for X11 images, cost of not testing.

The Aegis program provides a manual test facility. It was originally intended for programs which required some physical action from a user, such as "unplug Ethernet cable now" or "mount tape XG356B now". It can also be used to have a user confirm that some on\[hy]screen activity has happened.

The problem with manual tests is that they simply don't happen. It is far more pleasant to say "run the automatic tests" and go for a cup of coffee, than to wait while the computer thinks of mindless things to ask you to do. This is human nature: if it can be automated, it is more likely to happen.

Many folks think of testing as taking the final product and testing it. It is also possible to build specialized unit tests, which exercise specific portions of the code. These tests can then be administrated by Aegis, even if the full\[hy]blown GUI cannot be.

Another class of software is things like operating system kernels and firmware; things which are "stand alone". This isolated nature makes it the most difficult to test: to test it you want to provide physical input and watch the physical output. By its very nature, it is hard to put into a shell script, and thus hard to write an Aegis test for.

The above chapter was written in 1991. At this writing (1999) there are projects like \(muLinux and operating systems like VxWorks. These are all embedded, and all have excellent network and download support. It is entirely possible (with design support!) to write automatically testable embedded systems.

It is not impossible, just that few of us have the resources to do it. You need to have a test system and a testing system: the test system has all of its input and outputs connected to the outputs and inputs of the testing system. That is, the testing system controls and drives the test system, and watches what happens.

For example, in the olden days before everyone had PC and graphics terminals, there were only serial interfaces available. Many operating system vendors tested their products by using computers connected to each serial line to simulate "user input". The system can be rebooted this way, and using dual\[hy]ported disks allows different versions of a kernel to be tried, or other test conditions created.

For software houses which write kernels, or device drivers for kernels, or some other kernel work, this is bad news: the Aegis program is probably not for you. It is possible, but there may be more cost\[hy]effective development strategies. Of course, you could always use the rest of Aegis, and ignore the testing part.

However, Aegis has been used quite successfully to develop Linux kernel modules. With suitable sudo(1) configuration to permit access to insmod(1) &co, developers can write test scripts which load device drivers, try them out, and unload them again, all without universal root access.

Also, the advent of modern tools, such as VMware, which allow one operating system to "host" another, may also permit straightforward testing of kernels and operating systems.

Firmware is a similar deal: you need some way to download the code to be tested into the test system, and write\[hy]protect it to simulate ROM, and have the necessary hardware to drive the inputs and watch the outputs.

As you can see, this is generally not available to run\[hy]of\[hy]the\[hy]mill software houses, but then they rarely write firmware, either. Those that do write firmware usually have the download capabilities, and some kind of remote operation facility.

However, this omits the possibility of not only cross compiling your code for the target system, but also compiling your code to run natively on the build system. The firmware (in the host incarnation) then falls into one of the categories above, and may be readily tested. This does not relieve you of also testing the firmware, but it increases the probability that the firmware isn't completely useless before you download it.

By using an object oriented language, such as C++, the polymorphism necessary to cope with multiple environments can be elegantly hidden behind a pure abstract base class. Alternatively, by using a consistent API, you can accomplish the necessary sleight\[hy]of\[hy]hand at link time.

The unit test method mentioned earlier is also very useful for firmware, even if the device "as a whole" cannot be tested. vim: set ts=8 sw=4 et :