1.\" Copyright (c) 2019 The DragonFly Project. All rights reserved. 2.\" 3.\" This code is derived from software contributed to The DragonFly Project 4.\" by Matthew Dillon <dillon@backplane.com> 5.\" This code is based on a concept originally developed by John R. Marino. 6.\" 7.\" Redistribution and use in source and binary forms, with or without 8.\" modification, are permitted provided that the following conditions 9.\" are met: 10.\" 11.\" 1. Redistributions of source code must retain the above copyright 12.\" notice, this list of conditions and the following disclaimer. 13.\" 2. Redistributions in binary form must reproduce the above copyright 14.\" notice, this list of conditions and the following disclaimer in 15.\" the documentation and/or other materials provided with the 16.\" distribution. 17.\" 3. Neither the name of The DragonFly Project nor the names of its 18.\" contributors may be used to endorse or promote products derived 19.\" from this software without specific, prior written permission. 20.\" 21.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 22.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 23.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 24.\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 25.\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 26.\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING, 27.\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 28.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED 29.\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 30.\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT 31.\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 32.\" SUCH DAMAGE. 33.\" 34.Dd August 21, 2019 35.Dt DSYNC 8 36.Os 37.Sh NAME 38.Nm dsynth 39.Nd dsynth bulk dports builder utility 40.Sh SYNOPSIS 41.Nm 42.Op Fl dhvyDS 43.Op Fl s Ar n 44.Op Fl m Ar gb 45.Ar directive 46.Op origins 47.Nm 48.Ar help 49.Sh DESCRIPTION 50The 51.Nm 52utility allows a user to build and maintain part or all of dports 53locally. 54.Nm 55figures out the dependency topology of the dport(s) you 56is capable of building any number of ports concurrently based 57on the configuration parameters you supply. 58.Pp 59.Nm 60is based on an application called 61.Xr synth 1 62which was written by John Marino in ADA and served as the conceptual base 63for this program. 64.Nm 65is written in C and designed to be as portable as possible given a 66ports-style infrastructure. 67.Pp 68Our recommended build topology is with a configuration as follows: 69.Bd -literal 70[Global Configuration] 71profile_selected= LiveSystem 72 73[LiveSystem] 74Operating_system= DragonFly 75Directory_packages= /build/synth/live_packages 76Directory_repository= /build/synth/live_packages/All 77Directory_portsdir= /build/synth/dports 78Directory_options= /build/synth/options 79Directory_distfiles= /build/synth/distfiles 80Directory_buildbase= /build/synth/build 81Directory_logs= /build/synth/logs 82Directory_ccache= disabled 83Directory_system= / 84Number_of_builders= 8 85Max_jobs_per_builder= 8 86Display_with_ncurses= true 87.Ed 88.Pp 89This places all major directories under 90.Pa /build/synth . 91If you want to use the same dports and the same distfiles as your base 92system, you can null-mount /usr/distfiles onto /build/synth/distfiles 93and /usr/dports onto /build/synth/dports with 94.Pa /etc/fstab 95entries as follows: 96.Bd -literal 97# Device Mountpoint FStype Options DumpPass# 98/usr/distfiles /build/synth/distfiles null rw 4 4 99/usr/dports /build/synth/dports null rw 4 4 100.Ed 101.Pp 102Please set the number of builders and the maximum number of jobs per 103builder according to available system resources. Remember that the total 104load on the system can be as high as (builders x jobs), and at least 4x 105that value in processes. Systems are typically restricted by memory and 106cpu horsepower. Start conservative and ramp up according to what your 107system can handle. 108A good rule of thumb is to set workers to the number of cpu threads your 109machine has or to 1/2 the number of gigabytes of memory your system has, 110whichever is lower. Then set the jobs per worker to no more than the 111number of cpu threads your machine has. 112.Pp 113.Nm 114has numerous features to manage machine load and swap usage to 115prevent a machine from being overloaded, allowing more workers 116to be configured than you might otherwise think is reasonable 117(which helps a lot when building the smaller ports). 118However, users running this program should be aware that very high loads 119and modest swap use are still likely to develop when building a large 120number of ports or when building very large ports like chromium. 121If the system is not dedicated to building packages you can reduce the 122impact to the rest of the system by running 123.Nm 124at nice +20 and also by reducing the number of workers and number of 125jobs per worker somewhat. 126.Pp 127We recommend that a minimum of 64GB of SSD-based swap be configured, 128or twice as much swap as main memory, whichever is the higher value. 129.Pp 130We recommend a minimum of 500GB of storage be configured in 131.Pa /build 132or wherever you have configured various directories. 133A full set of distfiles requires at least 120GB, a full dports including 134the git repo requires at least 1.5GB, and a full set of built packages 135requires at least 75GB. If using a filesystem such as HAMMER or HAMMER2 136which frees space overnight, double all of those numbers. 137.Pp 138The actual build infrastructure uses tmpfs... memory and swap, and does 139not use regular filesystem space. 140.Sh OPTIONS 141.Bl -tag -width indent 142.It Fl d[d...] 143Run in debug mode. If specified two or more times this will turn off 144ncurses and output the primary log (00_last_results.log) to the standard 145output, along with additional spew. 146.It Fl h 147Quickly output a synopsis of options and directives and exit. 148.It Fl m Ar gb 149Override the default package dependency memory target, in gigabytes. 150The default is 1/2 physical memory. The number of workers will be limited 151such that the aggregate size of package dependencies installed in each 152worker slot does not exceed this value. 153.Pp 154This handles a well-known effect where the sheer amount of data that has 155to be installed in tmpfs filesystems for large ports, when multiplied by 156the number of worker slots, can force excessive paging to occur and leave 157preciously little memory available to actually run compiles. Some paging 158is necessary to maintain maximum cpu utilization, but excessive paging 159can cause the whole machine to essentially become idle for extended 160periods of time. 161.It Fl v 162Quickly output the version and exit. 163.It Fl y 164Automatically answer 'y'es to any questions. 165.It Fl s Ar n 166.Nm 167usually slow-starts the worker slots, beginning with one slot and increasing 168by one every 5 seconds until the maximum configured number of workers is 169reached. 170This gives 171.Nm 172a slower ramp that it can load manage against. 173Specifying 0 disables the slow-start feature and the maximum number of 174worker slots (limited by the dependency graph) will be loaded immediately. 175.It Fl D 176Turn on DEVELOPER mode when building ports. 177.It Fl S[S] 178Turn off curses for script friendliness. The output will be log 00 and 179should be redirected to /dev/null or something similar. 180If you supply the options twice, color output escapes will also be 181turned off. 182You may also wish to use the 183.Fl y 184option for scripting dsynth. 185.El 186.Sh DIRECTIVES 187Generally 188.Nm 189is run with a directive and some directives allow a list of ports to be 190specified. This list should be space-delimited in DIR/SUBDIR format, 191for example: 192.Ar www/chromium . 193For directives with an optional ports list, your current installed set 194of ports will be used if you do not specify a list. 195.Bl -tag -width indent 196.It Cm status 197This will do a dry-run of 198.Cm upgrade-system 199but not actually build anything. 200.It Cm cleanup 201This will clean up any left-over mounts from prior builds. 202.Nm 203attempts to clean up all processes and mounts when you interrupt 204a build but doesn't always succeed. 205.It Cm init 206Creates and initializes the 207.Pa /etc/dsynth 208directory if it does not exst. 209This directive will complain and exit if either 210.Pa /etc/dsynth 211or 212.Pa /usr/local/etc/dsynth 213exists. It will not create 214.Pa /etc/dsynth 215in this situation. 216.It Cm configure 217NOT CURRENTLY IMPLEMENTED 218.It Cm upgrade-system 219NOT CURRENTLY IMPLEMENTED. Incrementally build and upgrade your locally 220installed packages, then upgrade your local system with them. 221.It Cm prepare-system 222Incrementally build and upgrade your locally installed packages, but 223do not upgrade your system with them. 224.It Cm rebuild-repository 225Build or rebuild the database files for the configured repository. 226.It Cm purge-distfiles 227Delete any obsolete source distribution files. 228.It Cm status-everything 229This will do a dry-run of a full bulk build of everything, 230but not actually build anything. 231.It Cm everything 232This will build the entire dports tree and then rebuild the repository 233when it finishes. 234.It Cm version 235This is for synth compatibility. The version of 236.Nm 237will be printed and the program will exit. 238.It Cm help 239Output a synopsis of options and directives and exit. 240.It Cm status Op Ar ports 241Do a dry-run with 'build' of the given list. 242.It Cm build Op Ar ports 243Incrementally build dports based on the given list. When 244done, ask whether the repository should be rebuilt or not. 245.It Cm just-build Op Ar ports 246Incrementally build dports based on the given list, then 247exits. No post-build steps will be taken. 248.It Cm install Op Ar ports 249NOT CURRENTLY IMPLEMENTED. 'build' based on the supplied 250list (or using currently installed packages), then rebuild 251the repository and upgrade the system without asking any further 252questions. 253.It Cm force Op Ar ports 254This is the same as 'build' but will delete existing packages 255first. Dependencies are not deleted unless they are out of date. 256.It Cm test Op Ar ports 257This is the same as 'build' but sets the environment variable 258.La DEVELOPER=yes 259and pre-deletes specified packages. Dependencies are not deleted 260unless they are out of date. 261.El 262.Sh FILES 263.Bl -tag -width ".It Pa <fs>/abc/defghi/<name>" -compact 264.It Pa /etc/dsynth/synth.ini 265The primary configuration file. If not found, 266.Nm 267will also look in 268.Pa /usr/local/etc/dsynth/synth.ini . 269.Pp 270.It Pa /etc/dsynth/LiveSystem-make.conf 271Typically contains the environment variables that will be set in 272the workers. 273.Nm 274firewalls the environment it is run under from the environment it 275provides to the workers. 276.Pp 277.It Pa /build/synth/build 278Recommended setting for 279.La Directory_buildbase , 280contains the build infrastructure... typically a template, mirrored 281system directories, and mount points for all the worker slots. 282The template will be [re]generated if 'pkg' needs to be built or 283if the 284.Pa .template.good 285file in this directory is deleted. 286.Pp 287.It Pa /build/synth/distfiles 288Recommended setting for 289.La Directory_distfiles , 290ports to a directory into which 291.Nm 292will download any source distribution files required for building. 293.Pp 294.It Pa /build/synth/dports 295Recommended setting for 296.La Directory_portsdir , 297points to a checked out dports repo. 298Note that 299.Nm 300does not automatically 'git pull' or otherwise synchronize the dports repo, 301you must do that yourself prior to starting a build. 302.Pp 303.It Pa /build/synth/live_packages 304Recommended setting for 305.La Directory_packages , 306points to a directory which will contain the completed application 307packages. 308.Pp 309.It Pa /build/synth/logs 310Recommended setting for 311.La Directory_logs , 312all log files will be placed in this directory. 313Special management logfiles begin with the numeral '0' for easily 314location. 315The logfiles for ports while and after building are stored in the 316form subdir____portname.log, with three underscores. 317.Pp 318.It Pa /build/synth/options 319Recommended setting for 320.La Directory_options , 321where options overrides for specific ports may be located. 322.Pp 323.It Pa / 324Recommended setting for 325.La Directory_system , 326which 327.Nm 328uses as a basis for creating the jails or chroots in each worker slot 329during building. 330No part of the system root is ever NULL-mounted read-write... it is always 331NULL-mounted read-only. 332Some elements from the system base will be mirrored in the build-base 333as an optimization. 334.Pp 335Note that the packages directory and the distfiles directory is mounted 336read-write in jails or chroots. All other r/w filesystems in the workers 337are 338.Xr tmpfs 5 339based filesystems and will be created and torn-down for each port. 340.El 341.Sh EXIT STATUS 342.Ex -std 343.Sh SEE ALSO 344.Xr synth 1 , 345.Xr dports 7 346.Sh HISTORY 347The 348.Nm 349utility first appeared in 350.Dx 5.7 . 351.Sh AUTHORS 352.An Matthew Dillon Aq Mt dillon@backplane.com 353