1.\" 2.\" Copyright (c) 2019 The DragonFly Project. All rights reserved. 3.\" 4.\" This code is derived from software contributed to The DragonFly Project 5.\" by Matthew Dillon <dillon@backplane.com> 6.\" This code is based on a concept originally developed by John R. Marino. 7.\" 8.\" Redistribution and use in source and binary forms, with or without 9.\" modification, are permitted provided that the following conditions 10.\" are met: 11.\" 12.\" 1. Redistributions of source code must retain the above copyright 13.\" notice, this list of conditions and the following disclaimer. 14.\" 2. Redistributions in binary form must reproduce the above copyright 15.\" notice, this list of conditions and the following disclaimer in 16.\" the documentation and/or other materials provided with the 17.\" distribution. 18.\" 3. Neither the name of The DragonFly Project nor the names of its 19.\" contributors may be used to endorse or promote products derived 20.\" from this software without specific, prior written permission. 21.\" 22.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 23.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 24.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 25.\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 26.\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 27.\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING, 28.\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 29.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED 30.\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 31.\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT 32.\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 33.\" SUCH DAMAGE. 34.\" 35.Dd October 24, 2019 36.Dt DSYNTH 1 37.Os 38.Sh NAME 39.Nm dsynth 40.Nd dsynth bulk dports builder utility 41.Sh SYNOPSIS 42.Nm 43.Op Fl dhvyDSN 44.Op Fl p Ar profile 45.Op Fl s Ar n 46.Op Fl m Ar gb 47.Ar directive 48.Op origins 49.Nm 50.Ar help 51.Sh DESCRIPTION 52The 53.Nm 54utility allows a user to build and maintain part or all of dports 55locally. 56.Nm 57figures out the dependency topology of the dport(s) for you and 58is capable of building any number of ports concurrently based 59on the configuration parameters you supply. 60.Pp 61.Nm 62is based on an application called 63.Xr synth 1 64which was written by John Marino in Ada and served as the conceptual base 65for this program. 66.Nm 67is written in C and designed to be as portable as possible given a 68ports-style infrastructure. 69.Pp 70Our recommended build topology is with a configuration as follows: 71.Bd -literal 72[Global Configuration] 73profile_selected= LiveSystem 74 75[LiveSystem] 76Operating_system= DragonFly 77Directory_packages= /build/synth/live_packages 78Directory_repository= /build/synth/live_packages/All 79Directory_portsdir= /build/synth/dports 80Directory_options= /build/synth/options 81Directory_distfiles= /build/synth/distfiles 82Directory_buildbase= /build/synth/build 83Directory_logs= /build/synth/logs 84Directory_ccache= disabled 85Directory_system= / 86Package_suffix= .txz 87Number_of_builders= 8 88Max_jobs_per_builder= 8 89Display_with_ncurses= true 90.Ed 91.Pp 92This places all major directories under 93.Pa /build/synth . 94If you want to use the same dports and the same distfiles as your base 95system, you can null-mount /usr/distfiles onto /build/synth/distfiles 96and /usr/dports onto /build/synth/dports with 97.Pa /etc/fstab 98entries as follows: 99.Bd -literal 100# Device Mountpoint FStype Options DumpPass# 101/usr/distfiles /build/synth/distfiles null rw 4 4 102/usr/dports /build/synth/dports null rw 4 4 103.Ed 104.Pp 105Please set the number of builders and the maximum number of jobs per 106builder according to available system resources. 107Remember that the total 108load on the system can be as high as (builders x jobs), and at least 4x 109that value in processes. 110Systems are typically restricted by memory and CPU horsepower. 111Start conservative and ramp up according to what your system can handle. 112A good rule of thumb is to set workers to the number of CPU threads your 113machine has or to 1/2 the number of gigabytes of memory your system has, 114whichever is lower. 115Then set the jobs per worker to no more than the 116number of CPU threads your machine has. 117.Pp 118.Nm 119has numerous features to manage machine load and swap usage to 120prevent a machine from being overloaded, allowing more workers 121to be configured than you might otherwise think is reasonable 122(which helps a lot when building the smaller ports). 123However, users running this program should be aware that very high loads 124and modest swap use are still likely to develop when building a large 125number of ports or when building very large ports like chromium. 126If the system is not dedicated to building packages you can reduce the 127impact to the rest of the system by running 128.Nm 129at nice +20 and also by reducing the number of workers and number of 130jobs per worker somewhat. 131.Pp 132We recommend that a minimum of 64GB of SSD-based swap be configured, 133or twice as much swap as main memory, whichever is the higher value. 134.Pp 135We recommend a minimum of 500GB of storage be configured in 136.Pa /build 137or wherever you have configured various directories. 138A full set of distfiles requires at least 120GB, a full dports including 139the git repo requires at least 1.5GB, and a full set of built packages 140requires at least 75GB. 141If using a filesystem such as HAMMER or HAMMER2 142which frees space overnight, double all of those numbers. 143.Pp 144The actual build infrastructure uses tmpfs... memory and swap, and does 145not use regular filesystem space. 146.Sh OPTIONS 147.Bl -tag -width indent 148.It Fl d[d...] 149Run in debug mode. 150If specified two or more times this will turn off 151ncurses and output the primary log (00_last_results.log) to the standard 152output, along with additional spew. 153.It Fl h 154Quickly output a synopsis of options and directives and exit. 155.It Fl m Ar gb 156Override the default package dependency memory target, in gigabytes. 157The default is 1/2 physical memory. 158The number of workers will be limited 159such that the aggregate size of package dependencies installed in each 160worker slot does not exceed this value. 161.Pp 162This handles a well-known effect where the sheer amount of data that has 163to be installed in tmpfs filesystems for large ports, when multiplied by 164the number of worker slots, can force excessive paging to occur and leave 165preciously little memory available to actually run compiles. 166Some paging 167is necessary to maintain maximum CPU utilization, but excessive paging 168can cause the whole machine to essentially become idle for extended 169periods of time. 170.It Fl v 171Quickly output the version and exit. 172.It Fl y 173Automatically answer 'y'es to any questions. 174.It Fl p Ar profile 175Override the global profile default in 176.Pa /etc/dsynth/dsynth.ini , 177allowing you to trivially run whatever profile you like without having to 178edit the configuration file when switching. 179In addition, you can now run any number of dsynth's concurrently on the same 180machine without having to use a jail, each with a different profile, 181as long as the packages, repository, buildbase, and logs directories 182are different. 183.Pp 184Note that the distfiles directory can be shared and will not conflict 185or get confused with concurrent fetches. 186.It Fl s Ar n 187.Nm 188usually slow-starts the worker slots, beginning with one slot and increasing 189by one every 5 seconds until the maximum configured number of workers is 190reached. 191This gives 192.Nm 193a slower ramp that it can load manage against. 194Specifying 0 disables the slow-start feature and the maximum number of 195worker slots (limited by the dependency graph) will be loaded immediately. 196.It Fl D 197Turn on DEVELOPER mode when building ports. 198.It Fl S[S] 199Turn off curses for script friendliness. 200The output will be log 00 and 201should be redirected to /dev/null or something similar. 202If you supply the options twice, color output escapes will also be 203turned off. 204You may also wish to use the 205.Fl y 206option for scripting dsynth. 207.It Fl N 208Normally 209.Nm 210nices its sub-processes to +10. 211This option disables the feature. 212.El 213.Sh DIRECTIVES 214Generally 215.Nm 216is run with a directive and some directives allow a list of ports to be 217specified. 218This list should be space-delimited in DIR/SUBDIR format, for example: 219.Ar www/chromium . 220For directives with an optional ports list, your current installed set 221of ports will be used if you do not specify a list. 222.Bl -tag -width indent 223.It Cm status 224This will do a dry-run of 225.Cm upgrade-system 226but not actually build anything. 227.It Cm cleanup 228This will clean up any left-over mounts from prior builds. 229.Nm 230attempts to clean up all processes and mounts when you interrupt 231a build but doesn't always succeed. 232.It Cm init 233Creates and initializes the 234.Pa /etc/dsynth 235directory if it does not exst. 236This directive will complain and exit if either 237.Pa /etc/dsynth 238or 239.Pa /usr/local/etc/dsynth 240exists. 241It will not create 242.Pa /etc/dsynth 243in this situation. 244.It Cm configure 245NOT CURRENTLY IMPLEMENTED 246.It Cm upgrade-system 247NOT CURRENTLY IMPLEMENTED. 248Incrementally build and upgrade your locally 249installed packages, then upgrade your local system with them. 250.It Cm prepare-system 251Incrementally build and upgrade your locally installed packages, but 252do not upgrade your system with them. 253.It Cm rebuild-repository 254Build or rebuild the database files for the configured repository. 255.It Cm purge-distfiles 256Delete any obsolete source distribution files. 257.It Cm status-everything 258This will do a dry-run of a full bulk build of everything, 259but not actually build anything. 260.It Cm everything 261This will build the entire dports tree and then rebuild the repository 262when it finishes. 263.It Cm version 264This is for synth compatibility. 265The version of 266.Nm 267will be printed and the program will exit. 268.It Cm help 269Output a synopsis of options and directives and exit. 270.It Cm status Op Ar ports 271Do a dry-run with 'build' of the given list. 272.It Cm build Op Ar ports 273Incrementally build dports based on the given list. 274When done, ask whether the repository should be rebuilt or not. 275.It Cm just-build Op Ar ports 276Incrementally build dports based on the given list, then 277exits. 278No post-build steps will be taken. 279.It Cm install Op Ar ports 280NOT CURRENTLY IMPLEMENTED. 'build' based on the supplied 281list (or using currently installed packages), then rebuild 282the repository and upgrade the system without asking any further 283questions. 284.It Cm force Op Ar ports 285This is the same as 'build' but will delete existing packages first. 286Dependencies are not deleted unless they are out of date. 287.It Cm test Op Ar ports 288This is the same as 'build' but sets the environment variable 289.Ev DEVELOPER 290to 291.Sq yes 292and pre-deletes specified packages. 293Dependencies are not deleted unless they are out of date. 294.It Cm debug Op Ar ports 295This is the same as 'build' but leaves the chroot mounts intact 296upon completion. 297.El 298.Sh FILES 299.Bl -tag -width ".It Pa <fs>/abc/defghi/<name>" -compact 300.It Pa /etc/dsynth/dsynth.ini 301The primary configuration file. 302If not found, 303.Nm 304will also look in 305.Pa /usr/local/etc/dsynth/dsynth.ini . 306.Pp 307.It Pa /etc/dsynth/LiveSystem-make.conf 308Typically contains the environment variables that will be set in 309the workers. 310.Nm 311firewalls the environment it is run under from the environment it 312provides to the workers. 313.Pp 314.It Pa /build/synth/build 315Recommended setting for 316.Va Directory_buildbase , 317contains the build infrastructure... typically a template, mirrored 318system directories, and mount points for all the worker slots. 319The template will be [re]generated if 'pkg' needs to be built or 320if the 321.Pa .template.good 322file in this directory is deleted. 323.Pp 324.It Pa /build/synth/distfiles 325Recommended setting for 326.Va Directory_distfiles , 327ports to a directory into which 328.Nm 329will download any source distribution files required for building. 330.Pp 331.It Pa /build/synth/dports 332Recommended setting for 333.Va Directory_portsdir , 334points to a checked out dports repo. 335Note that 336.Nm 337does not automatically 'git pull' or otherwise synchronize the dports repo, 338you must do that yourself prior to starting a build. 339.Pp 340.It Pa /build/synth/live_packages 341Recommended setting for 342.Va Directory_packages , 343points to a directory which will contain the completed application 344packages. 345.Pp 346.It Pa /build/synth/logs 347Recommended setting for 348.Va Directory_logs , 349all log files will be placed in this directory. 350Special management logfiles begin with the numeral '0' for easily 351location. 352The logfiles for ports while and after building are stored in the 353form subdir____portname.log, with three underscores. 354.Pp 355.It Pa /build/synth/options 356Recommended setting for 357.Va Directory_options , 358where options overrides for specific ports may be located. 359.Pp 360.It Pa / 361Recommended setting for 362.Va Directory_system , 363which 364.Nm 365uses as a basis for creating the jails or chroots in each worker slot 366during building. 367No part of the system root is ever NULL-mounted read-write... it is always 368NULL-mounted read-only. 369Some elements from the system base will be mirrored in the build-base 370as an optimization. 371.Pp 372Note that the packages directory and the distfiles directory is mounted 373read-write in jails or chroots. 374All other r/w filesystems in the workers are 375.Xr tmpfs 5 376based filesystems and will be created and torn-down for each port. 377.Pp 378.It Pa .txz 379.It Pa .tgz 380.It Pa .tar 381.It Pa .tbz 382The recommended setting for 383.Va Package_suffix 384is either 385.Pa .txz 386or 387.Pa .tgz . 388Use 389.Pa .txz 390for better compression at the cost of somewhat slower bulk builds due 391to the time overhead for compression and decompression, or 392use 393.Pa .tgz 394for modest compression and very fast compression and decompression. 395Due to the way the builder works, package dependencies are fresly 396installed into the chroot slot for each package being built, so 397decompression time matters. 398.El 399.Sh EXIT STATUS 400.Ex -std 401.Sh SEE ALSO 402.Xr synth 1 , 403.Xr dports 7 404.Sh HISTORY 405The 406.Nm 407utility first appeared in 408.Dx 5.7 . 409.Sh AUTHORS 410.An Matthew Dillon Aq Mt dillon@backplane.com 411