1.\" Copyright (c) 2019 The DragonFly Project. All rights reserved. 2.\" 3.\" This code is derived from software contributed to The DragonFly Project 4.\" by Matthew Dillon <dillon@backplane.com> 5.\" This code is based on a concept originally developed by John R. Marino. 6.\" 7.\" Redistribution and use in source and binary forms, with or without 8.\" modification, are permitted provided that the following conditions 9.\" are met: 10.\" 11.\" 1. Redistributions of source code must retain the above copyright 12.\" notice, this list of conditions and the following disclaimer. 13.\" 2. Redistributions in binary form must reproduce the above copyright 14.\" notice, this list of conditions and the following disclaimer in 15.\" the documentation and/or other materials provided with the 16.\" distribution. 17.\" 3. Neither the name of The DragonFly Project nor the names of its 18.\" contributors may be used to endorse or promote products derived 19.\" from this software without specific, prior written permission. 20.\" 21.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 22.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 23.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 24.\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 25.\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 26.\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING, 27.\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 28.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED 29.\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 30.\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT 31.\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 32.\" SUCH DAMAGE. 33.\" 34.Dd August 21, 2019 35.Dt DSYNC 8 36.Os 37.Sh NAME 38.Nm dsynth 39.Nd dsynth bulk dports builder utility 40.Sh SYNOPSIS 41.Nm 42.Op Fl dhvyDS 43.Op Fl s Ar n 44.Op Fl m Ar gb 45.Ar directive 46.Op origins 47.Nm 48.Ar help 49.Sh DESCRIPTION 50The 51.Nm 52utility allows a user to build and maintain part or all of dports 53locally. 54.Nm 55figures out the dependency topology of the dport(s) you 56is capable of building any number of ports concurrently based 57on the configuration parameters you supply. 58.Pp 59.Nm 60is based on an application called 61.Xr synth 1 62which was written by John Marino in ADA and served as the conceptual base 63for this program. 64.Nm 65is written in C and designed to be as portable as possible given a 66ports-style infrastructure. 67.Pp 68Our recommended build topology is with a configuration as follows: 69.Bd -literal 70[Global Configuration] 71profile_selected= LiveSystem 72 73[LiveSystem] 74Operating_system= DragonFly 75Directory_packages= /build/synth/live_packages 76Directory_repository= /build/synth/live_packages/All 77Directory_portsdir= /build/synth/dports 78Directory_options= /build/synth/options 79Directory_distfiles= /build/synth/distfiles 80Directory_buildbase= /build/synth/build 81Directory_logs= /build/synth/logs 82Directory_ccache= disabled 83Directory_system= / 84Number_of_builders= 8 85Max_jobs_per_builder= 8 86Display_with_ncurses= true 87.Ed 88.Pp 89This places all major directories under 90.Pa /build/synth . 91If you want to use the same dports and the same distfiles as your base 92system, you can null-mount /usr/distfiles onto /build/synth/distfiles 93and /usr/dports onto /build/synth/dports with 94.Pa /etc/fstab 95entries as follows: 96.Bd -literal 97# Device Mountpoint FStype Options DumpPass# 98/usr/distfiles /build/synth/distfiles null rw 4 4 99/usr/dports /build/synth/dports null rw 4 4 100.Ed 101.Pp 102Please set the number of builders and the maximum number of jobs per 103builder according to available system resources. Remember that the total 104load on the system can be as high as (builders x jobs), and at least 4x 105that value in processes. Systems are typically restricted by memory and 106cpu horsepower. Start conservative and ramp up according to what your 107system can handle. 108A good rule of thumb is to set workers to the number of cpu threads your 109machine has or to 1/2 the number of gigabytes of memory your system has, 110whichever is lower. Then set the jobs per worker to no more than the 111number of cpu threads your machine has. 112.Pp 113.Nm 114has numerous feature to manage machine load and swap usage to 115prevent a machine from being overloaded, allowing more workers 116to be configured than you might otherwise think is reasonable 117(which helps a lot when building the smaller ports). 118However, users running this program should be aware that very high loads 119and modest swap use are still likely to develop when building a large 120number of ports or when building very large ports like chromium. 121If the system is not dedicated to building packages you can reduce the 122impact to the rest of the system by running 123.Nm 124at nice +20 and also by reducing the number of workers and number of 125jobs per worker somewhat. 126.Pp 127We recommend that a minimum of 64GB of SSD-based swap be configured, 128or twice as much swap as main memory, whichever is the higher value. 129.Pp 130We recommend a minimum of 500GB of storage be configured in 131.Pa /build 132or wherever you have configured various directories. 133A full set of distfiles requires at least 120GB, a full dports including 134the git repo requires at least 1.5GB, and a full set of built packages 135requires at least 75GB. If using a filesystem such as HAMMER or HAMMER2 136which frees space overnight, double all of those numbers. 137.Pp 138The actual build infrastructure uses tmpfs... memory and swap, and does 139not use regular filesystem space. 140.Pp 141.Sh OPTIONS 142.Bl -tag -width indent 143.It Fl d[d...] 144Run in debug mode. If specified two or more times this will turn off 145ncurses and output the primary log (00_last_results.log) to the standard 146output, along with additional spew. 147.It Fl h 148Quickly output a synopsis of options and directives and exit. 149.It Fl m Ar gb 150Override the default package dependency memory target, in gigabytes. 151The default is 1/2 physical memory. The number of workers will be limited 152such that the aggregate size of package dependencies installed in each 153worker slot does not exceed this value. 154.Pp 155This handles a well-known effect where the sheer amount of data that has 156to be installed in tmpfs filesystems for large ports, when multiplied by 157the number of worker slots, can force excessive paging to occur and leave 158preciously little memory available to actually run compiles. Some paging 159is necessary to maintain maximum cpu utilization, but excessive paging 160can cause the whole machine to essentially become idle for extended 161periods of time. 162.It Fl v 163Quickly output the version and exit. 164.It Fl y 165Automatically answer 'y'es to any questions. 166.It Fl s Ar n 167.Nm 168usually slow-starts the worker slots, beginning with one slot and increasing 169by one every 5 seconds until the maximum configured number of workers is 170reached. 171This gives 172.Nm 173a slower ramp that it can load manage against. 174Specifying 0 disables the slow-start feature and the maximum number of 175worker slots (limited by the dependency graph) will be loaded immediately. 176.It Fl D 177Turn on DEVELOPER mode when building ports. 178.It Fl S[S] 179Turn off curses for script friendliness. The output will be log 00 and 180should be redirected to /dev/null or something similar. 181If you supply the options twice, color output escapes will also be 182turned off. 183You may also wish to use the 184.Fl y 185option for scripting dsynth. 186.El 187.Sh DIRECTIVES 188.Pp 189Generally 190.Nm 191is run with a directive and some directives allow a list of ports to be 192specified. This list should be space-delimited in DIR/SUBDIR format, 193for example: 194.Ar www/chromium . 195For directives with an optional ports list, your current installed set 196of ports will be used if you do not specify a list. 197.Pp 198.Bl -tag -width indent 199.It Cm status 200This will do a dry-run of 201.Cm upgrade-system 202but not actually build anything. 203.It Cm cleanup 204This will clean up any left-over mounts from prior builds. 205.Nm 206attempts to clean up all processes and mounts when you interrupt 207a build but doesn't always succeed. 208.It Cm init 209Creates and initializes the 210.Pa /etc/dsynth 211directory if it does not exst. 212This directive will complain and exit if either 213.Pa /etc/dsynth 214or 215.Pa /usr/local/etc/dsynth 216exists. It will not create 217.Pa /etc/dsynth 218in this situation. 219.It Cm configure 220NOT CURRENTLY IMPLEMENTED 221.It Cm upgrade-system 222NOT CURRENTLY IMPLEMENTED. Incrementally build and upgrade your locally 223installed packages, then upgrade your local system with them. 224.It Cm prepare-system 225Incrementally build and upgrade your locally installed packages, but 226do not upgrade your system with them. 227.It Cm rebuild-repository 228Build or rebuild the database files for the configured repository. 229.It Cm purge-distfiles 230Delete any obsolete source distribution files. 231.It Cm status-everything 232This will do a dry-run of a full bulk build of everything, 233but not actually build anything. 234.It Cm everything 235This will build the entire dports tree and then rebuild the repository 236when it finishes. 237.It Cm version 238This is for synth compatibility. The version of 239.Nm 240will be printed and the program will exit. 241.It Cm help 242Output a synopsis of options and directives and exit. 243.It Cm status Op Ar ports 244Do a dry-run with 'build' of the given list. 245.It Cm build Op Ar ports 246Incrementally build dports based on the given list. When 247done, ask whether the repository should be rebuilt or not. 248.It Cm just-build Op Ar ports 249Incrementally build dports based on the given list, then 250exits. No post-build steps will be taken. 251.It Cm install Op Ar ports 252NOT CURRENTLY IMPLEMENTED. 'build' based on the supplied 253list (or using currently installed packages), then rebuild 254the repository and uprgade the system without asking any further 255questions. 256.It Cm force Op Ar ports 257This is the same as 'build' but will delete existing packages 258first. Dependencies are not deleted unless they are out of date. 259.It Cm test Op Ar ports 260This is the same as 'build' but sets the environment variable 261.La DEVELOPER=yes 262and pre-deletes specified packages. Dependencies are not deleted 263unless they are out of date. 264.El 265 266.Sh FILES 267.Bl -tag -width ".It Pa <fs>/abc/defghi/<name>" -compact 268.It Pa /etc/dsynth/synth.ini 269The primary configuration file. If not found, 270.Nm 271will also look in 272.Pa /usr/local/etc/dsynth/synth.ini . 273.Pp 274.It Pa /etc/dsynth/LiveSystem-make.conf 275Typically contains the environment variables that will be set in 276the workers. 277.Nm 278firewalls the environment it is run under from the environment it 279provides to the workers. 280.Pp 281.It Pa /build/synth/build 282Recommended setting for 283.La Directory_buildbase , 284contains the build infrastructure... typically a template, mirrored 285system directories, and mount points for all the worker slots. 286The template will be [re]generated if 'pkg' needs to be built or 287if the 288.Pa .template.good 289file in this directory is deleted. 290.Pp 291.It Pa /build/synth/distfiles 292Recommended setting for 293.La Directory_distfiles , 294ports to a directory into which 295.Nm 296will download any source distribution files required for building. 297.Pp 298.It Pa /build/synth/dports 299Recommended setting for 300.La Directory_portsdir , 301points to a checked out dports repo. 302Note that 303.Nm 304does not automatically 'git pull' or otherwise synchronize the dports repo, 305you must do that yourself prior to starting a build. 306.Pp 307.It Pa /build/synth/live_packages 308Recommended setting for 309.La Directory_packages , 310points to a directory which will contain the completed application 311packages. 312.Pp 313.It Pa /build/synth/logs 314Recommended setting for 315.La Directory_logs , 316all log files will be placed in this directory. 317Special management logfiles begin with the numeral '0' for easily 318location. 319The logfiles for ports while and after building are stored in the 320form subdir____portname.log, with three underscores. 321.Pp 322.It Pa /build/synth/options 323Recommended setting for 324.La Directory_options , 325where options overrides for specific ports may be located. 326.Pp 327.It Pa / 328Recommended setting for 329.La Directory_system , 330which 331.Nm 332uses as a basis for creating the jails or chroots in each worker slot 333during building. 334No part of the system root is ever NULL-mounted read-write... it is always 335NULL-mounted read-only. 336Some elements from the system base will be mirrored in the build-base 337as an optimization. 338.Pp 339Note that the packages directory and the distfiles directory is mounted 340read-write in jails or chroots. All other r/w filesystems in the workers 341are 342.Xr tmpfs 5 343based filesystems and will be created and torn-down for each port. 344.El 345.Sh EXIT STATUS 346.Ex -std 347.Sh SEE ALSO 348.Xr synth 1 349.Xr dports 7 350.Sh HISTORY 351The 352.Nm 353utility first appeared in 354.Dx 5.7 . 355.Sh AUTHORS 356.An Matthew Dillon Aq Mt dillon@backplane.com 357