1.\" $OpenBSD: bulk.8,v 1.6 2023/04/26 19:12:49 jmc Exp $ 2.\" 3.\" Copyright (c) 2016 Marc Espie <espie@openbsd.org> 4.\" 5.\" 6.\" Permission to use, copy, modify, and distribute this software for any 7.\" purpose with or without fee is hereby granted, provided that the above 8.\" copyright notice and this permission notice appear in all copies. 9.\" 10.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 11.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 12.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 13.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 14.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 15.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 16.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 17.Dd $Mdocdate: April 26 2023 $ 18.Dt BULK 8 19.Os 20.Sh NAME 21.Nm bulk 22.Nd building OpenBSD packages in bulk 23.Sh DESCRIPTION 24There are quite a few steps necessary to build packages on a cluster. 25They are: 26.Pp 27.Bl -enum -compact -offset indent 28.It 29Choose master machine setup and create partitions. 30.It 31Setup chrooted builds on the master. 32.It 33Add slaves and do a full bulk. 34.It 35Clean up and do subsequent bulks. 36.It 37Perform additional maintenance. 38.El 39.Ss 1. Choose master machine setup and create partitions 40Setup a master machine with enough room for a chroot, say 41.Pa /build . 42Assuming you are using a cluster of machines, 43this chroot should contain NFS exportable partitions for distfiles, 44plists, and packages (one single partition can be used for simplicity). 45A full setup currently requires on the order of 70GB for distfiles 46and 50GB for packages. 47Expect these numbers to grow. 48100GB for each should last a few years. 49.Pp 50It is possible to build packages without a chroot, but the space 51requirement difference is negligible (a full 52.Ox 53install is less than 1GB), and having everything chrooted means you may 54install useful tools to help with the process outside of the chroot 55.Po 56for instance 57.Xr rsync 1 58.Pc . 59.Pp 60Reserve one "scratch" partition under the chroot for WRKOBJDIR 61(for instance, mfs, async, or SSD). 62This partition should be roughly 10GB if you want to be able to 63build all ports. 64This can often double as 65.Pa /tmp 66under the chroot. 67.Pp 68Alternately, you can setup your whole chroot as a scratch partition, 69and reserve one more permanent space under it for distfiles, 70packages, and plists. 71.Pp 72Choose a strategy for the ports tree itself. 73There must be a copy under 74.Pa /build . 75You can either copy it from outside the chroot, have it in an NFS 76partition, or manually make sure all machines on the cluster have the 77same ports tree (cvs checkout, rsync ...). 78.Pp 79Note that logs are only produced on the master, and thus do not 80need to be nfs exportable, nor even inside the chroot. 81.Pp 82.Ox 83now comes with default users for package builds, namely _pbuild and _pfetch. 84.Pp 85The default 86.Xr login.conf 5 87is appropriate for most setups, but _pbuild's datasize-cur will need 88to be bumped for a few ports, like pypy. 89Likewise, maxproc-cur is too small for machines with more than 4-6 cpus. 90.Pp 91Note that _pbuild does not need network access, and is now blocked by default 92in 93.Xr pf 4 . 94.Pp 95Recent 96.Ox 97systems do not need any kind of 98.Xr doas 1 99setup for bulk ports builds, as 100.Xr dpb 1 101is run as root and drops permissions appropriately. 102.Pp 103However, you may still want to setup 104.Xr doas 1 105for root, if you want to manually fix ports, as 106.Ar PORTS_PRIVSEP 107relies on it. 108.Ss 2. Setup chrooted builds on the master 109Populate the initial chroot with 110.Xr proot 1 . 111Point DISTDIR, PACKAGE_REPOSITORY, PLIST_REPOSITORY, WRKOBJDIR 112to appropriate locations. 113.Pp 114Pay attention to nodev and wxallowed warnings. 115A chroot setup that can't have devices won't work at all. 116A setup without wxallowed in /usr/local and WRKOBJDIR won't 117build a lot of things. 118.Pp 119Check that this setup can build ports by running 120.Li dpb -B /build 121as root. 122Fix any issues related to file ownership at this point. 123See 124.Xr dpb 1 125for details. 126.Pp 127If your WRKOBJDIR is a temporary partition, make sure it 128belongs to _pbuild:_pbuild after a reboot. 129.Ss 3. Add slaves and do a full bulk 130Create identical slave machines with the same release material. 131Have them import the NFS partitions from the master, they 132don't need root access to the partitions. 133Set up 134.Xr ssh 1 135so that the master can connect to the slaves, using ssh protocol 2, 136as root, preferably without a password or passphrase (however, 137.Xr dpb 1 138uses a master connection, so a password would be required just once per host). 139.Pp 140Note that code on slave machines will only run as _pbuild 141(during builds) or root (during dependency installation). 142Slave machines only require highly restricted network access. 143They just need to act as nfs clients to the master and to be reachable 144through ssh from the master. 145.Pp 146Use a similar 147.Xr proot 1 148config to populate each slave. 149.Pp 150You should now be able to build ports on the slaves. 151A simple config will just have 152.Bd -literal -offset indent 153DEFAULT chroot=/build 154localhost 155host1 156\&... 157.Ed 158.Pp 159Check that the full config can still build ports. 160.Pp 161You're now ready for a full bulk. 162Beware that even fast configs (3 amd64 with 8 cores each) may take over 24 hours 163to finish. 164It's generally appropriate to run 165.Xr dpb 1 166under 167.Xr tmux 1 . 168.Ss 4. Clean up and do subsequent bulks 169Before running the next bulk, you should set up rotating logs and move the 170PACKAGE_REPOSITORY away. 171Save the PLIST_REPOSITORY and DISTDIR though. 172PLIST_REPOSITORY will catch problems in packing-lists. 173.Pa ${PLIST_REPOSITORY}/${ARCH}/history 174is also used to store 175.Xr sha256 1 176history, necessary to reorder files inside packages to speed updates up. 177.Pp 178The DISTDIR contains history information as well as DISTDIR/build-stats 179to speed further runs up. 180.Pp 181How you wipe things out depends on your setup. 182If you run 183.Xr proot 1 184again each time, most things will get cleaned up automatically 185.Po 186.Pa /build/usr/local , /build/var/db/pkg ... 187.Pc . 188Note that known directories such as WRKOBJDIR do not get cleaned up 189automatically, so you may want to set up a STARTUP_SCRIPT in 190.Xr dpb 1 . 191.Ss 5. Perform additional maintenance 192.Xr clean-old-distfiles 1 193should be run occasionally since the DISTDIR will continue growing. 194.Pp 195.Xr pkg_check-problems 1 196should be run occasionally to find out conflicts and dependency issues. 197.Sh SEE ALSO 198.Xr clean-old-distfiles 1 , 199.Xr dpb 1 , 200.Xr pkg_check-problems 1 , 201.Xr proot 1 , 202.Xr register-plist 1 , 203.Xr tmux 1 , 204.Xr bsd.port.mk 5 , 205.Xr release 8 206