1.\" 2.\" Copyright (c) 2021 The DragonFly Project. All rights reserved. 3.\" 4.\" This code is derived from software contributed to The DragonFly Project 5.\" by Matthew Dillon <dillon@backplane.com> 6.\" This code is based on a concept originally developed by John R. Marino. 7.\" 8.\" Redistribution and use in source and binary forms, with or without 9.\" modification, are permitted provided that the following conditions 10.\" are met: 11.\" 12.\" 1. Redistributions of source code must retain the above copyright 13.\" notice, this list of conditions and the following disclaimer. 14.\" 2. Redistributions in binary form must reproduce the above copyright 15.\" notice, this list of conditions and the following disclaimer in 16.\" the documentation and/or other materials provided with the 17.\" distribution. 18.\" 3. Neither the name of The DragonFly Project nor the names of its 19.\" contributors may be used to endorse or promote products derived 20.\" from this software without specific, prior written permission. 21.\" 22.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 23.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 24.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 25.\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 26.\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 27.\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING, 28.\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 29.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED 30.\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 31.\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT 32.\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 33.\" SUCH DAMAGE. 34.\" 35.Dd January 26, 2021 36.Dt DSYNTH 1 37.Os 38.Sh NAME 39.Nm dsynth 40.Nd dsynth bulk dports builder utility 41.Sh SYNOPSIS 42.Nm 43.Op Fl dhvxyDSN 44.Op Fl p Ar profile 45.Op Fl s Ar n 46.Op Fl m Ar gb 47.Op Fl M Ar scale 48.Ar directive 49.Op origins 50.Nm 51.Ar help 52.Sh DESCRIPTION 53The 54.Nm 55utility allows a user to build and maintain part or all of dports 56locally. 57.Nm 58figures out the dependency topology of the dport(s) for you and 59is capable of building any number of ports concurrently based 60on the configuration parameters you supply. 61.Pp 62also detects changes made to the ports tree and rebuilds packages 63and anything that depends on those packages as needed. 64.Pp 65.Nm 66is based on an application called 67.Xr synth 1 68which was written by John Marino in Ada and served as the conceptual base 69for this program. 70.Nm 71is written in C and designed to be as portable as possible given a 72ports-style infrastructure. 73.Pp 74Our recommended build topology is with a configuration as follows: 75.Bd -literal 76[Global Configuration] 77profile_selected= LiveSystem 78 79[LiveSystem] 80Operating_system= DragonFly 81Directory_packages= /build/synth/live_packages 82Directory_repository= /build/synth/live_packages/All 83Directory_portsdir= /build/synth/dports 84Directory_options= /build/synth/options 85Directory_distfiles= /build/synth/distfiles 86Directory_buildbase= /build/synth/build 87Directory_logs= /build/synth/logs 88Directory_ccache= disabled 89Directory_system= / 90Package_suffix= .txz 91Number_of_builders= 8 92Max_jobs_per_builder= 8 93Display_with_ncurses= true 94.Ed 95.Pp 96This places all major directories under 97.Pa /build/synth . 98If you want to use the same dports and the same distfiles as your base 99system, you can null-mount /usr/distfiles onto /build/synth/distfiles 100and /usr/dports onto /build/synth/dports with 101.Pa /etc/fstab 102entries as follows: 103.Bd -literal 104# Device Mountpoint FStype Options DumpPass# 105/usr/distfiles /build/synth/distfiles null rw 4 4 106/usr/dports /build/synth/dports null rw 4 4 107.Ed 108.Pp 109Please set the number of builders and the maximum number of jobs per 110builder according to available system resources. 111Remember that the total 112load on the system can be as high as (builders x jobs), and at least 4x 113that value in processes. 114Systems are typically restricted by memory and CPU horsepower. 115Start conservative and ramp up according to what your system can handle. 116A good rule of thumb is to set workers to the number of CPU threads your 117machine has or to 1/2 the number of gigabytes of memory your system has, 118whichever is lower. 119Then set the jobs per worker to no more than the 120number of CPU threads your machine has. 121.Pp 122.Nm 123has numerous features to manage machine load and swap usage to 124prevent a machine from being overloaded, allowing more workers 125to be configured than you might otherwise think is reasonable 126(which helps a lot when building the smaller ports). 127However, users running this program should be aware that very high loads 128and modest swap use are still likely to develop when building a large 129number of ports or when building very large ports like chromium. 130If the system is not dedicated to building packages you can reduce the 131impact to the rest of the system by running 132.Nm 133at nice +20 and also by reducing the number of workers and number of 134jobs per worker somewhat. 135.Pp 136We recommend that a minimum of 64GB of SSD-based swap be configured, 137or twice as much swap as main memory, whichever is the higher value. 138.Pp 139We recommend a minimum of 500GB of storage be configured in 140.Pa /build 141or wherever you have configured various directories. 142A full set of distfiles requires at least 120GB, a full dports including 143the git repo requires at least 1.5GB, and a full set of built packages 144requires at least 75GB. 145If using a filesystem such as HAMMER or HAMMER2 146which frees space overnight, double all of those numbers. 147.Pp 148The actual build infrastructure uses tmpfs... memory and swap, and does 149not use regular filesystem space. 150.Sh OPTIONS 151.Bl -tag -width indent 152.It Fl d[d...] 153Run in debug mode. 154If specified two or more times this will turn off 155ncurses and output the primary log (00_last_results.log) to the standard 156output, along with additional spew. 157.It Fl h 158Quickly output a synopsis of options and directives and exit. 159.It Fl m Ar gb 160Override the default package dependency memory target, in gigabytes. 161The default is 1/2 physical memory. 162The number of workers will be limited 163such that the aggregate size of package dependencies installed in each 164worker slot does not exceed this value. 165.Pp 166This handles a well-known effect where the sheer amount of data that has 167to be installed in tmpfs filesystems for large ports, when multiplied by 168the number of worker slots, can force excessive paging to occur and leave 169preciously little memory available to actually run compiles. 170Some paging 171is necessary to maintain maximum CPU utilization, but excessive paging 172can cause the whole machine to essentially become idle for extended 173periods of time. 174.It Fl M Ar scale 175Override dynamic workers calculation by a specific scale factor. 176Specifying 1.0 leaves it unchanged, 0.8 will reduce the number of jobs by 17720%, and 1.2 will increase the number of jobs by 20%. And so forth. 178.Pp 179This option may be specified in the range 0.01 to 99.0. Out of range values 180will simply be clipped. 181.It Fl p Ar profile 182Override the global profile default in 183.Pa /etc/dsynth/dsynth.ini , 184allowing you to trivially run whatever profile you like without having to 185edit the configuration file when switching. 186In addition, you can now run any number of dsynth's concurrently on the same 187machine without having to use a jail, each with a different profile, 188as long as the packages, repository, buildbase, and logs directories 189are different. 190.Pp 191Note that the distfiles directory can be shared and will not conflict 192or get confused with concurrent fetches. 193.It Fl s Ar n 194.Nm 195usually slow-starts the worker slots, beginning with one slot and increasing 196by one every 5 seconds until the maximum configured number of workers is 197reached. 198This gives 199.Nm 200a slower ramp that it can load manage against. 201Specifying 0 disables the slow-start feature and the maximum number of 202worker slots (limited by the dependency graph) will be loaded immediately. 203.It Fl v 204Quickly output the version and exit. 205.It Fl x 206.It Fl xx 207Normally dsynth builds a package for any of three reasons: (1) If the contents 208of the ports directory changes, (2) If anything the port depends on requires 209rebuilding so to will the port be rebuilt, (3) If there is no binary package 210already built for the port. 211.Pp 212If this option is specified, the first test is ignored. 213If this option is specified twice, the first and second tests are ignored. 214.It Fl y 215Automatically answer 'y'es to any questions. 216.It Fl D 217Turn on DEVELOPER mode when building ports. 218.It Fl P 219Include the check-plist stage. 220This is the default for the 221.Cm everything 222directive. 223.It Fl S[S] 224Turn off curses for script friendliness. 225The output will be log 00 and 226should be redirected to /dev/null or something similar. 227If you supply the options twice, color output escapes will also be 228turned off. 229You may also wish to use the 230.Fl y 231option for scripting dsynth. 232.It Fl N 233Normally 234.Nm 235nices its sub-processes to +10. 236This option disables the feature. 237.El 238.Sh DIRECTIVES 239Generally 240.Nm 241is run with a directive and some directives allow a list of ports to be 242specified. 243This list should be space-delimited in DIR/SUBDIR format, for example: 244.Ar www/chromium . 245For directives with an optional ports list, your current installed set 246of ports will be used if you do not specify a list. 247.Bl -tag -width indent 248.It Cm init 249Creates and initializes the 250.Pa /etc/dsynth 251directory if it does not exist. 252This directive will complain and exit if either 253.Pa /etc/dsynth 254or 255.Pa /usr/local/etc/dsynth 256exists. 257It will not create 258.Pa /etc/dsynth 259in this situation. 260.It Cm status 261This will do a dry-run of 262.Cm upgrade-system 263but not actually build anything. 264.It Cm cleanup 265This will clean up any left-over mounts from prior builds. 266.Nm 267attempts to clean up all processes and mounts when you interrupt 268a build but doesn't always succeed. 269.It Cm configure 270NOT CURRENTLY IMPLEMENTED 271.It Cm upgrade-system 272NOT CURRENTLY IMPLEMENTED. 273Incrementally build and upgrade your locally 274installed packages, then upgrade your local system with them. 275.It Cm prepare-system 276Incrementally build and upgrade your locally installed packages, but 277do not upgrade your system with them. 278.It Cm rebuild-repository 279Build or rebuild the database files for the configured repository. 280.It Cm purge-distfiles 281Delete any obsolete source distribution files. 282.It Cm reset-db 283Delete ports_crc.db from the build directory. 284This database is used to detect changes made to the dports tree. 285It will be regenerated on your next build without forcing any packages to be rebuilt. 286.It Cm status-everything 287This will do a dry-run of a full bulk build of everything, 288but not actually build anything. 289.It Cm everything 290This will build the entire dports tree and then rebuild the repository 291when it finishes. 292.It Cm version 293This is for synth compatibility. 294The version of 295.Nm 296will be printed and the program will exit. 297.It Cm help 298Output a synopsis of options and directives and exit. 299.It Cm status Op Ar ports 300Do a dry-run with 'build' of the given list. 301.It Cm build Op Ar ports 302Incrementally build dports based on the given list. 303When done, ask whether the repository should be rebuilt or not. 304.It Cm just-build Op Ar ports 305Incrementally build dports based on the given list, then 306exits. 307No post-build steps will be taken. 308.It Cm install Op Ar ports 309NOT CURRENTLY IMPLEMENTED. 'build' based on the supplied 310list (or using currently installed packages), then rebuild 311the repository and upgrade the system without asking any further 312questions. 313.It Cm force Op Ar ports 314This is the same as 'build' but will delete existing packages first. 315Dependencies are not deleted unless they are out of date. 316.It Cm test Op Ar ports 317This is the same as 'build' but sets the environment variable 318.Ev DEVELOPER 319to 320.Sq yes 321and pre-deletes specified packages. 322Dependencies are not deleted unless they are out of date. 323.It Cm debug Op Ar ports 324This is the same as 'build' but leaves the chroot mounts intact 325upon completion. 326.It Cm monitor Op Ar datfile 327Monitors a running dsynth instance. 328.El 329.Sh FILES 330.Bl -tag -width ".It Pa <fs>/abc/defghi/<name>" -compact 331.It Pa /etc/dsynth/dsynth.ini 332The primary configuration file. 333If not found, 334.Nm 335will also look in 336.Pa /usr/local/etc/dsynth/dsynth.ini . 337.Pp 338.It Pa /etc/dsynth/LiveSystem-make.conf 339Typically contains the environment variables that will be set in 340the workers. 341.Nm 342firewalls the environment it is run under from the environment it 343provides to the workers. 344.Pp 345.It Pa /build/synth/build 346Recommended setting for 347.Va Directory_buildbase , 348contains the build infrastructure... typically a template, mirrored 349system directories, and mount points for all the worker slots. 350The template will be [re]generated if 'pkg' needs to be built or 351if the 352.Pa .template.good 353file in this directory is deleted. 354.Pp 355.It Pa /build/synth/distfiles 356Recommended setting for 357.Va Directory_distfiles , 358ports to a directory into which 359.Nm 360will download any source distribution files required for building. 361.Pp 362.It Pa /build/synth/dports 363Recommended setting for 364.Va Directory_portsdir , 365points to a checked out dports repo. 366Note that 367.Nm 368does not automatically 'git pull' or otherwise synchronize the dports repo, 369you must do that yourself prior to starting a build. 370.Pp 371.It Pa /build/synth/live_packages 372Recommended setting for 373.Va Directory_packages , 374points to a directory which will contain the completed application 375packages. 376.Pp 377.It Pa /build/synth/logs 378Recommended setting for 379.Va Directory_logs , 380all log files will be placed in this directory. 381Special management logfiles begin with the numeral '0' for easily 382location. 383The logfiles for ports while and after building are stored in the 384form subdir____portname.log, with three underscores. 385.Pp 386.It Pa /build/synth/options 387Recommended setting for 388.Va Directory_options , 389where options overrides for specific ports may be located. 390.Pp 391.It Pa / 392Recommended setting for 393.Va Directory_system , 394which 395.Nm 396uses as a basis for creating the jails or chroots in each worker slot 397during building. 398No part of the system root is ever NULL-mounted read-write... it is always 399NULL-mounted read-only. 400Some elements from the system base will be mirrored in the build-base 401as an optimization. 402.Pp 403Note that the packages directory and the distfiles directory is mounted 404read-write in jails or chroots. 405All other r/w filesystems in the workers are 406.Xr tmpfs 5 407based filesystems and will be created and torn-down for each port. 408.Pp 409.It Pa .txz 410.It Pa .tgz 411.It Pa .tar 412.It Pa .tbz 413.It Pa .tzst 414The recommended setting for 415.Va Package_suffix 416is either 417.Pa .txz 418or 419.Pa .tgz . 420Use 421.Pa .txz 422for better compression at the cost of somewhat slower bulk builds due 423to the time overhead for compression and decompression, or 424use 425.Pa .tgz 426for modest compression and very fast compression and decompression. 427Due to the way the builder works, package dependencies are fresly 428installed into the chroot slot for each package being built, so 429decompression time matters. 430.El 431.Sh EXIT STATUS 432.Ex -std 433.Sh SEE ALSO 434.Xr synth 1 , 435.Xr dports 7 436.Sh HISTORY 437The 438.Nm 439utility first appeared in 440.Dx 5.7 . 441.Sh AUTHORS 442.An Matthew Dillon Aq Mt dillon@backplane.com 443