1Proposal: NetBSD System Installation Packages 2============================================= 3 4CONTENTS 5-------- 60. Introduction 71. System Packages 8 1.1 Package Format 9 1.2 Package Granularity 10 1.2.1 Root/User/Share separation 112. Package Sets 12 2.1 Set format 133. Creation of Packages and Sets 144. Modifications to the NetBSD installation process 15A. Working Plan 16 17------------------------------------------------------------------------ 18 190. Introduction 20 21 The current NetBSD installation process involves the downloading 22 of binary `sets', which the user can choose among at install time. 23 A set is a tarred, gzipped set of files, to be untarred relative 24 to '/'. No facility exists to choose convenient subsets of the files 25 in a set to be installed, or to remove a set which has been installed. 26 27 The current granularity of sets is very large, being divided into: 28 29 base -- general system binaries 30 comp -- compilers and related tools 31 etc -- system configuration files 32 games -- games and other amusements 33 man -- system manual pages 34 misc -- items not falling into other categories 35 secr -- items not exportable under US law 36 text -- text processing tools 37 xbase -- general X11R6 binaries 38 xcomp -- X11R6 development items 39 xfont -- X11R6 fonts 40 xserver -- X11R6 servers for various video hardware 41 42 Users who wish to install part of a set need to either install 43 the full set and then determine which files they need to remove, 44 or abandon the normal install process, and figure out which files 45 to unpack by hand. Similarly, if a set is later determined to 46 be unnecessary, the only way to remove it is to figure out which 47 files on the system belonged to that set, and remove them by hand. 48 49 When it comes time to upgrade a system which has been installed this 50 way, the usual procedure is to unpack a new version of each installed 51 set over the previous version. When a file is moved, renamed, or 52 removed in a newer version of a set, the old version often remains on 53 the system for some time. In at least one recent instance (the move 54 of /sbin/mountd to /usr/sbin/mountd) this has resulted in much 55 confusion, and large amounts of traffic on the relevant mailing lists. 56 57 The remainder of this document describes a proposed method of handling 58 these and other problems with the current install set system by 59 moving to the use of fine-grained `system packages', based on the 60 currently existing package system for third-party software, and 61 allowing users to choose among either `package sets' at the same 62 granularity as our current install sets, or individual `packages' 63 at a much finer level of granularity. In either case, the new system 64 would also greatly simplify upgrading or removal of such packages 65 and sets at a later time, and would allow tracking of dependencies 66 between the various sets and packages distributed as part of NetBSD. 67 68 First, the format of system packages in the proposed system is 69 discussed, followed by the format of package sets, which will serve 70 as a replacement for the current install sets. The creation of 71 packages in an automated fashion from a NetBSD source tree is 72 discussed as is the effect of this system on the NetBSD installation 73 process. An appendix discusses my work plan to implement this new 74 system. 75 76 It is hoped that this document will serve as a basis for discussion 77 of what is involved in changing NetBSD to use system packages for 78 system installation and upgrades, and that after several iterations 79 of discussion and revision, it will serve as a plan for the actual 80 implementation of this system. 81 82------------------------------------------------------------------------ 83 841. System Packages 85 86 System packages will be the basic building blocks of a NetBSD system. 87 At install time, the user will choose which system packages to install, 88 subject to dependencies between packages. After system install, 89 users will be able to install additional packages or remove installed 90 packages. When it comes time to upgrade the system, packages can 91 be removed and reinstalled in a reliable fashion. All of this 92 functionality is already available for third-party software via the 93 use of the software package system in /usr/pkgsrc. This proposal 94 extends that functionality to the NetBSD system itself. 95 961.1 Package Format 97 98 System packages will be identical in format to the binary packages 99 used by the current third-party package system. This will allow the 100 same tools to be used for working with system packages as are 101 currently used for working with third-party packages. This will also 102 also allow the system to benefit from the fact that the workings of 103 the current package system are well understood. 104 1051.2 Package Granularity 106 107 System packages will be at the granularity of groups of related tools 108 and their support files. Thus, `Kerberos', `UUCP', `Text formatting' 109 and `amd' might each be packages which depended on nothing but a few 110 base packages, while `C Development' and `Fortran development' might 111 be separate packages which each depended upon `Binutils' and `Base 112 EGCS utilities' packages. Packages sets, described below, would add 113 the ability to choose entire broad categories of software to install, 114 like todays install sets, while maintaining the ability to remove 115 individual packages later. 116 1171.2.1 Root/User/Share separation 118 119 In order to support a variety of system configurations, it is crucial 120 that the new package system support the possibility of some part of 121 a system residing on a server and possibly being shared between 122 multiple machines on a network. A machine which has some filesystems 123 local and some shared must, at the very least, be able to add and 124 remove packages from local filesystems, and should be able to 125 determine what packages have been added or removed from the volumes 126 mounted over the network. 127 128 The most common shared configurations are to have a system share 129 /usr/share from the network, and have all other filesystems local, 130 or to share the entirety of /usr from the network, and maintain 131 local root and /var hierarchies, possibly as a single filesystem. 132 Other commonly shared hierarchies include /usr/X11R6 and /usr/pkg. 133 134 Two steps are necessary to support this type of sharing: the system 135 must be able to check separate repositories for packages installed 136 on different filesystems, and packages must be designed so as to 137 allow a client to install only those parts of the system which reside 138 on local filesystems. 139 140 The first of these is addressed by a set of patches described by 141 Alistair Crooks in a post to the netbsd-current mailing list on 142 Friday, September 18, 1998. These patches, which have not yet been 143 committed cause third-party software packages installed in /usr/pkg 144 to be registered in /usr/pkg/etc/pkg, and packages installed in 145 /usr/X11R6 to be registered in /usr/X11R6/etc/pkg. This could be 146 extended easily to allow sharing of system package installations by 147 having the new system X11R6 packages also use /usr/X11R6/etc/pkg 148 for package registration, to have system packages installed in /usr 149 use /usr/etc/pkg for package registration, and to have system 150 packages installed in / and /var use /etc/pkg for package 151 registration. This would allow all of the types of filesystem 152 sharing described above, without introducing too much complication 153 into the package system. 154 155 The second step, that of insuring that a client can choose to install 156 only the parts of the system which reside on local volumes can be 157 most easily addressed by careful consideration of package contents. 158 A look through the contents of the current install sets suggests 159 that relatively few packages will in fact need to install in more 160 than one of /, /usr, /usr/share and /usr/X11R6. Were such packages 161 split into separate components, based on filesystem boundaries, 162 users would easily be able to install only the parts which are local 163 in their particular configuration. 164 165------------------------------------------------------------------------ 166 1672. Package Sets 168 169 In moving to fine-grained system packages, it is important that 170 beginning users still be able to select broad categories of software 171 to install at once. The introduction of `package sets', analogous 172 in granularity, but not mechanism, to the current binary install sets 173 addresses this concern, while maintaining the ability of more advanced 174 users to choose among individual packages at install time, and 175 maintaining the ability to remove, upgrade, or add individual 176 packages at a later time. 177 178 These package sets will maintain the same layout as the current 179 install sets, so that a user who chooses the same sets as he would 180 have chosen now will see the same results. In the new system, 181 however, these sets will be made up of binary packages, and installing 182 a set will simply result in the installation of the constituent 183 packages. 184 1852.1 Set format 186 187 A set will be a tar archive containing the packages which make up the 188 set plus a contents file. At the least, the index file will contain 189 the name of each included package, plus a one line description of each 190 package's contents. Installation utilities will offer the option of 191 installing the whole set, or choosing among individual packages, 192 based on the descriptions in the contents file. It is expected that the 193 contents file itself will be automatically generated from the one-line 194 descriptions provided in each package's pkg/COMMENT file. 195 196 When a set is installed, the contents file will be recorded in a 197 manner similar to the registration of package information in the 198 current third-party package system. This will allow users to remove 199 an entire set at a later date, without needing to know what individual 200 packages came from that set. 201 202------------------------------------------------------------------------ 203 2043. Creation of Packages and Sets 205 206 Under the current distribution-building system, the Makefile in 207 /usr/src/etc creates binary install sets from an installed system, 208 based on the set lists in /usr/src/distrib/sets/lists. In the new 209 system, a new directory hierarchy, /usr/src/distrib/pkg, will 210 contain Makefiles and data files relevant to the creation of 211 system packages and package sets. 212 213 The directory /usr/src/distrib/pkg/sets will contain a directory 214 for each package set, and each of these directories will contain 215 a directory for each package in that set. The Makefile in 216 /usr/src/distrib/pkg/sets will recurse into these set directories 217 to build each set. The individual set Makefiles will recurse into 218 each package directory to build the individual packages, and will 219 then create a set file from the constituent packages and from the 220 contents file, which will be automatically generated from the 221 package directories. 222 223 The package directories will resemble the package directories for 224 third-party software packages in /usr/pkgsrc, except that they will 225 probably rely on the files making up the package already being 226 present in ${DESTDIR}, rather than building them directly. This 227 assumption is already present in the current distribution package 228 Makefile code, and is probably reasonable to keep. 229 230------------------------------------------------------------------------ 231 2324. Modifications to the NetBSD installation process 233 234 Once the NetBSD system is available as system packages and package 235 sets, it will be possible to modify the various installation tools 236 to use these sets to install the system. It is expected that 237 installation tools will default to allow users to choose among 238 package sets at install time, but allow an `advanced mode' in which 239 packages could be selected and deselected on an individual basis. 240 241 This will require that the various package tools (at least pkg_add) 242 be present on install media to be used with system packages. 243 Modifications to sysinst and other install tools are beyond the 244 current scope of this proposal, but will be necessary to take 245 advantage of the new capabilities provided by this system. 246 247------------------------------------------------------------------------ 248 249A. Working Plan 250 251 My current plan for implementing system packages and package sets 252 for NetBSD consists of four steps. All of these steps should be 253 taken in the CVS source tree (segregated into src/distrib/pkg, of 254 course), and hopefully will involve other contributors in addition 255 to myself: 256 257 1.) Hammer this proposal into a more detailed specification 258 259 I am submitting this proposal now in the hopes that it 260 will spark discussion which will lead to a refinement 261 of the planned system package system. Once some sort 262 of consensus is reached on the relevant mailing lists, 263 I will begin work in earnest on implementing this. 264 265 2.) Create the /usr/src/distrib/pkg hierarchy, and a template 266 package 267 268 The first step in actually implementing this system will 269 be to create either an actual or mocked-up system package 270 which can be used as a template for creation of the 271 remaining system packages. 272 273 3.) Create system packages 274 275 I expect that this step will involve most of the actual 276 work in implementing the new system. Packages will have 277 to be created for each functional group of binaries 278 currently shipped with NetBSD. A lot of discussion and 279 design will have to go into the decisions as to how 280 many packages should make up each set and what files 281 belong in which packages. 282 283 4.) Create Package Sets 284 285 Once all system packages exist, it will be necessary to 286 put together some code to automatically generate set 287 contents files and to create sets from each directory 288 of packages in /usr/src/distrib/pkg/sets. 289 290 Once these steps are complete, NetBSD will have system packages, 291 and it will be possible to begin looking at modifying the NetBSD 292 install process to use them. It is important to note that none 293 of these changes will require modifying the current installation 294 set building code in any way, so the use of the current system 295 can continue unhindered while the new system is being implemented. 296 297------------------------------------------------------------------------ 298$Id: PROPOSAL,v 1.2 2004/01/17 05:30:01 lukem Exp $ 299