1Proposal: NetBSD System Installation Packages 2============================================= 3 4CONTENTS 5-------- 60. Introduction 71. System Packages 8 1.1 Package Format 9 1.2 Package Granularity 10 1.2.1 Root/User/Share separation 112. Package Sets 12 2.1 Set format 133. Creation of Packages and Sets 144. Modifications to the NetBSD installation process 15A. Working Plan 16 17------------------------------------------------------------------------ 18 190. Introduction 20 21 The current NetBSD installation process involves the downloading 22 of binary `sets', which the user can choose among at install time. 23 A set is a tarred, gzipped set of files, to be untarred relative 24 to '/'. No facility exists to choose convenient subsets of the files 25 in a set to be installed, or to remove a set which has been installed. 26 27 The current granularity of sets is very large, being divided into: 28 29 base -- general system binaries 30 comp -- compilers and related tools 31 etc -- system configuration files 32 games -- games and other amusements 33 man -- system manual pages 34 misc -- items not falling into other categories 35 secr -- items not exportable under US law 36 text -- text processing tools 37 xbase -- general X11R6 binaries 38 xcomp -- X11R6 development items 39 xcontrib - random binaries from the X11R6 `contrib' tree 40 xfont -- X11R6 fonts 41 xserver -- X11R6 servers for various video hardware 42 43 Users who wish to install part of a set need to either install 44 the full set and then determine which files they need to remove, 45 or abandon the normal install process, and figure out which files 46 to unpack by hand. Similarly, if a set is later determined to 47 be unnecessary, the only way to remove it is to figure out which 48 files on the system belonged to that set, and remove them by hand. 49 50 When it comes time to upgrade a system which has been installed this 51 way, the usual procedure is to unpack a new version of each installed 52 set over the previous version. When a file is moved, renamed, or 53 removed in a newer version of a set, the old version often remains on 54 the system for some time. In at least one recent instance (the move 55 of /sbin/mountd to /usr/sbin/mountd) this has resulted in much 56 confusion, and large amounts of traffic on the relevant mailing lists. 57 58 The remainder of this document describes a proposed method of handling 59 these and other problems with the current install set system by 60 moving to the use of fine-grained `system packages', based on the 61 currently existing package system for third-party software, and 62 allowing users to choose among either `package sets' at the same 63 granularity as our current install sets, or individual `packages' 64 at a much finer level of granularity. In either case, the new system 65 would also greatly simplify upgrading or removal of such packages 66 and sets at a later time, and would allow tracking of dependencies 67 between the various sets and packages distributed as part of NetBSD. 68 69 First, the format of system packages in the proposed system is 70 discussed, followed by the format of package sets, which will serve 71 as a replacement for the current install sets. The creation of 72 packages in an automated fashion from a NetBSD source tree is 73 discussed as is the effect of this system on the NetBSD installation 74 process. An appendix discusses my work plan to implement this new 75 system. 76 77 It is hoped that this document will serve as a basis for discussion 78 of what is involved in changing NetBSD to use system packages for 79 system installation and upgrades, and that after several iterations 80 of discussion and revision, it will serve as a plan for the actual 81 implementation of this system. 82 83------------------------------------------------------------------------ 84 851. System Packages 86 87 System packages will be the basic building blocks of a NetBSD system. 88 At install time, the user will choose which system packages to install, 89 subject to dependencies between packages. After system install, 90 users will be able to install additional packages or remove installed 91 packages. When it comes time to upgrade the system, packages can 92 be removed and reinstalled in a reliable fashion. All of this 93 functionality is already available for third-party software via the 94 use of the software package system in /usr/pkgsrc. This proposal 95 extends that functionality to the NetBSD system itself. 96 971.1 Package Format 98 99 System packages will be identical in format to the binary packages 100 used by the current third-party package system. This will allow the 101 same tools to be used for working with system packages as are 102 currently used for working with third-party packages. This will also 103 also allow the system to benefit from the fact that the workings of 104 the current package system are well understood. 105 1061.2 Package Granularity 107 108 System packages will be at the granularity of groups of related tools 109 and their support files. Thus, `Kerberos', `UUCP', `Text formatting' 110 and `amd' might each be packages which depended on nothing but a few 111 base packages, while `C Development' and `Fortran development' might 112 be separate packages which each depended upon `Binutils' and `Base 113 EGCS utilities' packages. Packages sets, described below, would add 114 the ability to choose entire broad categories of software to install, 115 like todays install sets, while maintaining the ability to remove 116 individual packages later. 117 1181.2.1 Root/User/Share separation 119 120 In order to support a variety of system configurations, it is crucial 121 that the new package system support the possibility of some part of 122 a system residing on a server and possibly being shared between 123 multiple machines on a network. A machine which has some filesystems 124 local and some shared must, at the very least, be able to add and 125 remove packages from local filesystems, and should be able to 126 determine what packages have been added or removed from the volumes 127 mounted over the network. 128 129 The most common shared configurations are to have a system share 130 /usr/share from the network, and have all other filesystems local, 131 or to share the entirety of /usr from the network, and maintain 132 local root and /var hierarchies, possibly as a single filesystem. 133 Other commonly shared hierarchies include /usr/X11R6 and /usr/pkg. 134 135 Two steps are necessary to support this type of sharing: the system 136 must be able to check separate repositories for packages installed 137 on different filesystems, and packages must be designed so as to 138 allow a client to install only those parts of the system which reside 139 on local filesystems. 140 141 The first of these is addressed by a set of patches described by 142 Alistair Crooks in a post to the netbsd-current mailing list on 143 Friday, September 18, 1998. These patches, which have not yet been 144 committed cause third-party software packages installed in /usr/pkg 145 to be registered in /usr/pkg/etc/pkg, and packages installed in 146 /usr/X11R6 to be registered in /usr/X11R6/etc/pkg. This could be 147 extended easily to allow sharing of system package installations by 148 having the new system X11R6 packages also use /usr/X11R6/etc/pkg 149 for package registration, to have system packages installed in /usr 150 use /usr/etc/pkg for package registration, and to have system 151 packages installed in / and /var use /etc/pkg for package 152 registration. This would allow all of the types of filesystem 153 sharing described above, without introducing too much complication 154 into the package system. 155 156 The second step, that of insuring that a client can choose to install 157 only the parts of the system which reside on local volumes can be 158 most easily addressed by careful consideration of package contents. 159 A look through the contents of the current install sets suggests 160 that relatively few packages will in fact need to install in more 161 than one of /, /usr, /usr/share and /usr/X11R6. Were such packages 162 split into separate components, based on filesystem boundaries, 163 users would easily be able to install only the parts which are local 164 in their particular configuration. 165 166------------------------------------------------------------------------ 167 1682. Package Sets 169 170 In moving to fine-grained system packages, it is important that 171 beginning users still be able to select broad categories of software 172 to install at once. The introduction of `package sets', analogous 173 in granularity, but not mechanism, to the current binary install sets 174 addresses this concern, while maintaining the ability of more advanced 175 users to choose among individual packages at install time, and 176 maintaining the ability to remove, upgrade, or add individual 177 packages at a later time. 178 179 These package sets will maintain the same layout as the current 180 install sets, so that a user who chooses the same sets as he would 181 have chosen now will see the same results. In the new system, 182 however, these sets will be made up of binary packages, and installing 183 a set will simply result in the installation of the constituent 184 packages. 185 1862.1 Set format 187 188 A set will be a tar archive containing the packages which make up the 189 set plus a contents file. At the least, the index file will contain 190 the name of each included package, plus a one line description of each 191 package's contents. Installation utilities will offer the option of 192 installing the whole set, or choosing among individual packages, 193 based on the descriptions in the contents file. It is expected that the 194 contents file itself will be automatically generated from the one-line 195 descriptions provided in each package's pkg/COMMENT file. 196 197 When a set is installed, the contents file will be recorded in a 198 manner similar to the registration of package information in the 199 current third-party package system. This will allow users to remove 200 an entire set at a later date, without needing to know what individual 201 packages came from that set. 202 203------------------------------------------------------------------------ 204 2053. Creation of Packages and Sets 206 207 Under the current distribution-building system, the Makefile in 208 /usr/src/etc creates binary install sets from an installed system, 209 based on the set lists in /usr/src/distrib/sets/lists. In the new 210 system, a new directory hierarchy, /usr/src/distrib/pkg, will 211 contain Makefiles and data files relevant to the creation of 212 system packages and package sets. 213 214 The directory /usr/src/distrib/pkg/sets will contain a directory 215 for each package set, and each of these directories will contain 216 a directory for each package in that set. The Makefile in 217 /usr/src/distrib/pkg/sets will recurse into these set directories 218 to build each set. The individual set Makefiles will recurse into 219 each package directory to build the individual packages, and will 220 then create a set file from the constituent packages and from the 221 contents file, which will be automatically generated from the 222 package directories. 223 224 The package directories will resemble the package directories for 225 third-party software packages in /usr/pkgsrc, except that they will 226 probably rely on the files making up the package already being 227 present in ${DESTDIR}, rather than building them directly. This 228 assumption is already present in the current distribution package 229 Makefile code, and is probably reasonable to keep. 230 231------------------------------------------------------------------------ 232 2334. Modifications to the NetBSD installation process 234 235 Once the NetBSD system is available as system packages and package 236 sets, it will be possible to modify the various installation tools 237 to use these sets to install the system. It is expected that 238 installation tools will default to allow users to choose among 239 package sets at install time, but allow an `advanced mode' in which 240 packages could be selected and deselected on an individual basis. 241 242 This will require that the various package tools (at least pkg_add) 243 be present on install media to be used with system packages. 244 Modifications to sysinst and other install tools are beyond the 245 current scope of this proposal, but will be necessary to take 246 advantage of the new capabilities provided by this system. 247 248------------------------------------------------------------------------ 249 250A. Working Plan 251 252 My current plan for implementing system packages and package sets 253 for NetBSD consists of four steps. All of these steps should be 254 taken in the CVS source tree (segregated into src/distrib/pkg, of 255 course), and hopefully will involve other contributors in addition 256 to myself: 257 258 1.) Hammer this proposal into a more detailed specification 259 260 I am submitting this proposal now in the hopes that it 261 will spark discussion which will lead to a refinement 262 of the planned system package system. Once some sort 263 of consensus is reached on the relevant mailing lists, 264 I will begin work in earnest on implementing this. 265 266 2.) Create the /usr/src/distrib/pkg hierarchy, and a template 267 package 268 269 The first step in actually implementing this system will 270 be to create either an actual or mocked-up system package 271 which can be used as a template for creation of the 272 remaining system packages. 273 274 3.) Create system packages 275 276 I expect that this step will involve most of the actual 277 work in implementing the new system. Packages will have 278 to be created for each functional group of binaries 279 currently shipped with NetBSD. A lot of discussion and 280 design will have to go into the decisions as to how 281 many packages should make up each set and what files 282 belong in which packages. 283 284 4.) Create Package Sets 285 286 Once all system packages exist, it will be necessary to 287 put together some code to automatically generate set 288 contents files and to create sets from each directory 289 of packages in /usr/src/distrib/pkg/sets. 290 291 Once these steps are complete, NetBSD will have system packages, 292 and it will be possible to begin looking at modifying the NetBSD 293 install process to use them. It is important to note that none 294 of these changes will require modifying the current installation 295 set building code in any way, so the use of the current system 296 can continue unhindered while the new system is being implemented. 297 298------------------------------------------------------------------------ 299$Id: PROPOSAL,v 1.1.1.1 2002/01/07 22:46:17 jwise Exp $ 300