All rights reserved.
%sccs.include.redist.roff%
@(#)paper.ms 5.3 (Berkeley) 05/26/92
use roff -ms
Berkeley UNIX Yesterday, Today and Tomorrow .AU Keith Bostic .AU Marshall Kirk McKusick .AU Michael J. Karels .AI Computer Systems Research Group Computer Science Division Department of Electrical Engineering and Computer Science University of California, Berkeley Berkeley, California 94720 .AB This paper presents a brief overview of the historic Berkeley releases and the role that Berkeley has played in the UNIX\(dg .FS \(dgUNIX is a registered trademark of UNIX System Laboratories. .FE world and discusses the role that University and research releases should play in the future. .AE A Technical History of the Berkeley Project
This is a brief technical history of Berkeley UNIX. For more details, [McKusick85] is strongly recommended along with the other papers listed in the Reference section. Specifically, we apologize that space does not permit mentioning most of the major contributors and influences on the Berkeley system!
The University of California, Berkeley, first ran Bell Laboratories Version 4 UNIX on a PDP-11/45, in January of 1974. Later that spring, a PDP-11/40 was configured with the newly available Version 5. Version 6 was running on a PDP-11/70 by the fall of 1975, with the arrival of two new graduate students, Bill Joy and Chuck Haley. They initially concentrating on making improvements to a Pascal system written by Ken Thompson while on sabbatical at Berkeley. With this completed, they turned their attention to the ed editor, eventually producing a new version they called ex. By the end of the summer of 1976, Joy and Haley began to take an interest in exploring the internals of the UNIX kernel.
In 1977, a steady stream of requests for the enhanced Pascal system had begun. Early that year, Joy put together the first ``Berkeley Software Distribution'', including the Pascal system and the source to ex. Over the next year, about thirty copies of this distribution were sent out. At around the same time, Joy began work on what was to become vi. By mid-1978, the software distribution clearly needed to be updated. The Pascal system had been made markedly more robust through feedback from its expanding user community, and had been split into two passes so that it could be run on PDP-11/34s. The result of the update was the ``Second Berkeley Software Distribution'' a name that was quickly abbreviated to 2BSD. Along with the Pascal system, vi and termcap for several terminals were included. Once again Bill Joy single-handedly put together distributions, answered the phone, and incorporated user feedback into the system. Over the next year nearly seventy-five tapes were shipped. Although Joy soon moved on to other projects, the 2BSD distribution continued to expand. Today the latest version of this distribution, 2.10BSD is still alive and being used by people all around the world on PDP-11's.
Early in 1978, Berkeley obtained a newly announced Digital Equipment Corp. (DEC) VAX\(dd .FS \(ddVAX is a registered trademark of Digital Equipment Corporation. .FE 11-780's. Shortly after the arrival of the VAX, Bell Laboratories provided Berkeley with a copy of their 32/V port of UNIX to the VAX. Although 32/V supported a Version 7 UNIX environment on the VAX, it did not take advantage of the virtual memory capability of the VAX hardware. Like its predecessors on the PDP-11, it was a swapping system. Ozalp Babaoglu, a Berkeley graduate student, set about finding a way of implementing a working set paging system on the VAX. As Babaoglu neared the completion of his first attempt at an implementation, he approached Bill Joy for some help in understanding the intricacies of the UNIX kernel. Intrigued by Babaoglu's approach, Joy joined in helping to integrate the code into 32/V and then with the ensuing debugging.
Joy realized that the 32-bit VAX would soon make the 16-bit PDP-11 obsolete, and he began to port the 2BSD software to the VAX. By the end of 1979, a complete distribution had been created. This distribution included the virtual memory kernel, the standard 32/V utilities, and the 2BSD additions, which by now included the Pascal system, ex/vi, the Cshell and several other utilities. In December, 1979, Joy shipped the first of nearly a hundred copies of 3BSD, the first VAX distribution from Berkeley.
In the fall of 1979, the Defense Advanced Research Projects Agency, DARPA, accepted a Berkeley proposal to develop an enhanced version of 3BSD for the DARPA community. With the already good reputation of 3BSD supporting the proposal, the Berkeley project was funded. Joy took charge of the project, which was to become the Computer Systems Research Group (CSRG). Joy soon incorporated job control, added auto reboot, a 1K block file system, and support for the latest VAX machine, the VAX-11/750. By October, 1980, a polished distribution that also included the Franz Lisp system and an enhanced mail handling system was released as 4BSD. During its nine-month lifetime, nearly 150 copies were shipped.
With the increasingly wide distribution and visibility of Berkeley UNIX, several critics began to emerge. The major objection cited was the performance of various benchmarks as compared to the DEC VMS system. Over the course of several months, Joy systematically tuned the kernel, soon matching VMS's performance. Rather than continue shipping 4BSD, the tuned system, with the addition of Robert Elz's auto configuration code, was released as 4.1BSD in June, 1981. Over its two year lifetime about 400 distributions were shipped.
With the release of 4.1BSD, much of the debate over performance died down. DARPA again funded Berkeley, this time with the intention of adding new features to the BSD system. These new features eventually included Berkeley reliable signals, the fast filesystem, disk quotas, and the socket interface with TCP/IP networking support. Around this time Joy left the CSRG for Sun Microsystems, and Sam Leffler took over responsibility for completing the project. In August, 1983, this system was released as 4.2BSD [Joy83]. The popularity of 4.2BSD was impressive; within eighteen months more copies of 4.2BSD had been shipped than of all the previous Berkeley software distributions combined.
As with 4BSD, the major criticism of 4.2BSD was performance. The problem, not surprisingly, was that the new facilities had not been tuned and that many of the kernel data structures were not well suited to their new uses. In addition, many of the interfaces, particularly in the networking area had been left unfinished. As Sam Leffler had left the CSRG for Lucasfilm, the tuning and enhancement of 4.2BSD was largely done under the direction of Michael Karels and Kirk McKusick [Leffler84] [McKusick85]. This system was released in April of 1986 as 4.3BSD, and had greatly enhanced performance and reliability over 4.2BSD, along with several new features.
As it had done in 4.2BSD, the CSRG then embarked on a new development phase to update other major components of the system, and design and integrate new functionality. The 4.4BSD release, scheduled for fall of 1992, will contain the results of several major new projects. Among these projects are an OSI network protocol suite integrated with existing ISO applications, an IEEE POSIX 1003.1 standard interface, a highly tuned TCP/IP networking interface, support for Sun Microsystem's Network File System, the integration of a log-structured file system, an integration of the MACH virtual memory system, volume labels and user-level database support.
There will have been four interim releases made by the CSRG between 4.3BSD and the upcoming 4.4BSD release. The first two of these releases, 4.3BSD-Tahoe and 4.3BSD-Reno were intended to distribute a subset of the new functionality found in 4.4BSD available to vendors. The 4.3BSD-Tahoe release, made in the summer of 1988, was the first Berkeley release to support two architectures. This goal was made possible by the reimplementation of much of the machine specific kernel source and a fundamental restructuring of the source code pool so that binaries for more than one architecture could be constructed from a single source pool. The two supported architectures were the VAX and the Computer Consoles Inc. Power 6/32 (the Tahoe). Since this release, architecture support for the Intel 386/486, the Sun Microsystems SPARCstation\(dg .FS \(dgAll SPARC trademarks are trademarks or registered trademarks of SPARC International, Inc. SPARCstation is licensed exclusively to Sun Microsystems, Inc. .FE I and II, the DECstation 3100 and 5000 and the Hewlett-Packard 300 have been added as well. The 4.3BSD-Reno release, made in the summer of 1990, was intended to make the Network File System code available to vendors using Berkeley-derived systems, such as The Open Software Foundation (OSF). This code had been written by Rick Macklem at the University of Guelph and integrated by the CSRG, under a new version of the kernel file system switch.
Two other interim releases, the first and second release of the ``Berkeley Network Software Distribution'', usually abbreviated as NET/1 and NET/2, were intended to make the source code of the 4BSD system available to and redistributable by anyone. Over the years of development by the CSRG and others, an increasingly larger percentage of the system was not derived from the original AT&T 32/V distribution. In the spring of 1988 Berkeley made its first distribution not requiring an AT&T source license, NET/1. This distribution primarily contained the networking portions of the system, from the utilities all the way through to the kernel device drivers, although other items such as login and other files were included for various reasons. This release was extremely popular with many vendors with their own versions of UNIX but who wished to run the Berkeley TCP/IP code and with vendors wishing to create smart networking cards, not to mention the users that wanted access to the source code for class work or other research purposes.
Around this time, the CSRG also began to search out freely redistributable versions of the UNIX utilities and to rewrite, or encourage BSD users to rewrite, those that were not available elsewhere. This was an immensely time-consuming task, involving contributions by hundreds of programmers from all around the world. In the summer of 1991, Berkeley released NET/2, which, like NET/1, did not require an AT&T source license. The NET/2 release included about 80% of the source code found in the 4.3BSD-Reno release. This release has proved to be immensely popular, with hundreds of thousands of copies taken from the public network archives and an unknown number redistributed by other organizations.
The Role of the Berkeley Project in the UNIX WorldThe role that Berkeley has played in the UNIX world has been a constantly changing one. In the 1970's, Berkeley was among the first participants in the UNIX research community, acting as host to several researchers on sabbatical from Bell Laboratories. This cooperation typified the harmony that was characteristic of the early UNIX community, as led by Bell Laboratories. Work that was contributed to the Laboratories by different members of the community, Berkeley among them, helped produce a rapidly expanding set of tools and facilities. With the commercialization of UNIX, the Bell Laboratories researchers were no longer able to act as a clearinghouse for the ongoing UNIX research. As the research community continued to modify the UNIX system, it found that it needed an organization that could produce leading edge research releases. Because of its early involvement in UNIX and its history of releasing UNIX-based tools, the CSRG quickly filled this role.
For the first half of the 1980's, Berkeley served as the focus of the leading edge of UNIX research. The Berkeley system was widely used, ported and considered the arbiter of what should comprise a UNIX system. By the mid-1980's, largely because the networking component of the Berkeley system was unique and unavailable from vendors for a period of time, Berkeley was forced into the role of a vendor [McKusick89]. This role expanded to the point that there were two major variants of UNIX, System V and BSD, and resulted in a breach in the UNIX world that is only gradually being healed. Acting as a vendor required an immense amount of time, money and effort by the CSRG. Thousands of hours were devoted to release engineering, thousands more to participation in the emerging UNIX standards and thousands more in distribution and user support. Over the years it became increasingly clear to the people associated with the Berkeley UNIX project that its limited funding and manpower were insufficient to complete its historical task of designing, implementing and supporting a complete, reliable, leading edge system. As each portion of the system became more complex and additional features were added, more and more effort had to be expended to keep the system at a high level of quality, and less and less effort was available to move the system technically forward. Fortunately, during the last half of the 1980's, as the UNIX interface became the consensus choice for an industry standard, and the number of vendors marketing, selling and supporting UNIX systems grew, Berkeley has been able to start to return to its historical orientation of doing leading-edge research instead of customer support. Berkeley UNIX Tomorrow
For UNIX to become the system of choice for a large segment of the industry, potential customers must have confidence that the product is supported, that future versions will continue to be developed and enhanced, and that future versions will be upwardly compatible with all past applications. In addition, vendors desiring to maximize their return on investment require that the source code for their systems be proprietary and are unwilling to make it available to users under any but the most onerous restrictions. Many of these changes, while acceptable for most users, are diametrically opposed to what has made UNIX the research platform of choice: low cost, wide availability of source code, and leading edge technology.
System development can be likened to the process of evolution. While gene mutation is critical to the advancement of the species, only one in 100 mutations produces a useful feature; the rest result in needless or detrimental changes. The mere existence of an environment for mutation is not enough -- some organization must bear responsibility for brutally pruning the weak, outdated and useless ideas. UNIX was fortunate in this sense. Unlike other projects beset by competing groups jealously guarding their work from one another, UNIX thrived in an open and cooperative community willing to channel its ideas through a central clearinghouse (first Bell Laboratories and later the CSRG), in spite of the clearinghouse's reputation for selective technical scrutiny.
Here one must distinguish between the selection process provided by research and commercial organizations. Research organizations can base pruning decisions strictly on the coherence of the system and the technical merit of the idea. They need not concern themselves with how changes might affect past variants of the system. Commercial organizations, though, must ensure that changes will not affect programs built to an obsolete interface. For example, paging might be a great idea, but it will cause problems for software that depends on the execution predictability of a swap-based system, making it impossible for paging to replace swapping. As a result, both schemes must be maintained, dramatically increasing the complexity of the system. As the system becomes more complex, its evolutionary paths will become increasingly restricted.
Here the role of a dynamic research version of UNIX becomes clear. While it is only directly used by a small group of people, it provides an important role as the feedstock for the commercial versions of UNIX. Over the long term, it is reasonable to expect that the most useful functionality of the research systems will be grafted into the commercial versions. Examples of ideas that began with BSD and moved into commercial systems include the fast filesystem, TCP/IP networking, and nearly half of the commands and utilities.
The CSRG spends a significant amount of time collecting prototypes of projects throughout the research world and molding them together into a coherent and usable system. Many of the ideas do not work out and are dropped in later releases. The ability to experiment without concern for past applications is critical. The resulting system is a third the size and a fraction of the complexity of its roughly equivalently functioned commercial brethren. This lean and mean approach allows the system to evolve rapidly (the nightmare of every commercial user, but the dream of every researcher). A recent example of this type of experimentation is the prototyping of various proposed POSIX utilities and interfaces by the CSRG. When drafts of the standard were implemented, basic flaws in the specification became apparent. These flaws and suggested solutions were presented to the standards committees, resulting in changes to the standard ensuring that the ratified standard could be efficiently and correctly implemented. The research system users also benefit from having a reference implementation of the standard almost from the day that it is finalized.
Another major influence on the UNIX systems of the future will be the NET/2 release. At least three separate groups (two in the U.S. and one in Europe) have added the necessary source code to the NET/2 release to make it a fully functional UNIX system. As the NET/2 release was not proprietary to any person or organization other than the University of California and may be freely redistributed, the cost of a UNIX system with source code will be less in the future than in the current UNIX market by two orders of magnitude. The UNIX single-server release by the Carnegie Mellon University MACH group will also use the NET/2 release as a starting point, making their release freely redistributable without a UNIX source license. The advent of cheaply available sources will make it far easier than ever before for research groups and users to develop and exchange software.
The role of designing and implementing a leading-edge research version of UNIX is one that Berkeley is uniquely equipped to fill. Future Berkeley releases will be oriented, as they were in the early days of Berkeley UNIX, toward the development and integration of a few well-chosen pieces of new research into a leading-edge system.
References:Joy, W., E. Cooper, R. Fabry, S. Leffler, M. McKusick, D. Mosher, ``4.2BSD System Manual,'' 4.2BSD UNIX Programmer's Manual, Vol 2c, Document #68 August 1983.