1.\" 2.\" Copyright (c) 2021 The DragonFly Project. All rights reserved. 3.\" 4.\" This code is derived from software contributed to The DragonFly Project 5.\" by Matthew Dillon <dillon@backplane.com> 6.\" This code is based on a concept originally developed by John R. Marino. 7.\" 8.\" Redistribution and use in source and binary forms, with or without 9.\" modification, are permitted provided that the following conditions 10.\" are met: 11.\" 12.\" 1. Redistributions of source code must retain the above copyright 13.\" notice, this list of conditions and the following disclaimer. 14.\" 2. Redistributions in binary form must reproduce the above copyright 15.\" notice, this list of conditions and the following disclaimer in 16.\" the documentation and/or other materials provided with the 17.\" distribution. 18.\" 3. Neither the name of The DragonFly Project nor the names of its 19.\" contributors may be used to endorse or promote products derived 20.\" from this software without specific, prior written permission. 21.\" 22.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 23.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 24.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 25.\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 26.\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 27.\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING, 28.\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 29.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED 30.\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 31.\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT 32.\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 33.\" SUCH DAMAGE. 34.\" 35.Dd August 21, 2021 36.Dt DSYNTH 1 37.Os 38.Sh NAME 39.Nm dsynth 40.Nd dsynth bulk dports builder utility 41.Sh SYNOPSIS 42.Nm 43.Op Fl dhvxyDNPS 44.Op Fl p Ar profile 45.Op Fl s Ar n 46.Op Fl m Ar gb 47.Op Fl M Ar scale 48.Ar directive 49.Op origins 50.Nm 51.Ar help 52.Sh DESCRIPTION 53The 54.Nm 55utility allows a user to build and maintain part or all of dports 56locally. 57.Nm 58figures out the dependency topology of the dport(s) for you and 59is capable of building any number of ports concurrently based 60on the configuration parameters you supply. 61.Pp 62also detects changes made to the ports tree and rebuilds packages 63and anything that depends on those packages as needed. 64.Pp 65.Nm 66is based on an application called 67.Xr synth 1 68which was written by John Marino in Ada and served as the conceptual base 69for this program. 70.Nm 71is written in C and designed to be as portable as possible given a 72ports-style infrastructure. 73.Pp 74Our recommended build topology is with a configuration as follows: 75.Bd -literal 76[Global Configuration] 77profile_selected= LiveSystem 78 79[LiveSystem] 80Operating_system= DragonFly 81Directory_packages= /build/synth/live_packages 82Directory_repository= /build/synth/live_packages/All 83Directory_portsdir= /build/synth/dports 84Directory_options= /build/synth/options 85Directory_distfiles= /build/synth/distfiles 86Directory_buildbase= /build/synth/build 87Directory_logs= /build/synth/logs 88Directory_ccache= disabled 89Directory_system= / 90Package_suffix= .txz 91; Meta_version= 2 (this is the default) 92; Check_plist= false (this is the default) 93; Numa_setsize= 0 (this is the default. 0, 2, or 4) 94Number_of_builders= 8 95Max_jobs_per_builder= 8 96Display_with_ncurses= true 97.Ed 98.Pp 99This places all major directories under 100.Pa /build/synth . 101If you want to use the same dports and the same distfiles as your base 102system, you can null-mount /usr/distfiles onto /build/synth/distfiles 103and /usr/dports onto /build/synth/dports with 104.Pa /etc/fstab 105entries as follows: 106.Bd -literal 107# Device Mountpoint FStype Options DumpPass# 108/usr/distfiles /build/synth/distfiles null rw 4 4 109/usr/dports /build/synth/dports null rw 4 4 110.Ed 111.Pp 112Please set the number of builders and the maximum number of jobs per 113builder according to available system resources. 114Remember that the total 115load on the system can be as high as (builders x jobs), and at least 4x 116that value in processes. 117Systems are typically restricted by memory and CPU horsepower. 118Start conservative and ramp up according to what your system can handle. 119A good rule of thumb is to set workers to the number of CPU threads your 120machine has or to 1/2 the number of gigabytes of memory your system has, 121whichever is lower. 122Then set the jobs per worker to no more than the 123number of CPU threads your machine has. 124.Pp 125.Nm 126has numerous features to manage machine load and swap usage to 127prevent a machine from being overloaded, allowing more workers 128to be configured than you might otherwise think is reasonable 129(which helps a lot when building the smaller ports). 130However, users running this program should be aware that very high loads 131and modest swap use are still likely to develop when building a large 132number of ports or when building very large ports like chromium. 133If the system is not dedicated to building packages you can reduce the 134impact to the rest of the system by running 135.Nm 136at nice +20 and also by reducing the number of workers and number of 137jobs per worker somewhat. 138.Pp 139We recommend that a minimum of 64GB of SSD-based swap be configured, 140or twice as much swap as main memory, whichever is the higher value. 141.Pp 142We recommend a minimum of 500GB of storage be configured in 143.Pa /build 144or wherever you have configured various directories. 145A full set of distfiles requires at least 120GB, a full dports including 146the git repo requires at least 1.5GB, and a full set of built packages 147requires at least 75GB. 148If using a filesystem such as HAMMER or HAMMER2 149which frees space overnight, double all of those numbers. 150.Pp 151The actual build infrastructure uses tmpfs... memory and swap, and does 152not use regular filesystem space. 153.Sh OPTIONS 154.Bl -tag -width indent 155.It Fl d[d...] 156Run in debug mode. 157If specified two or more times this will turn off 158ncurses and output the primary log (00_last_results.log) to the standard 159output, along with additional spew. 160.It Fl h 161Quickly output a synopsis of options and directives and exit. 162.It Fl m Ar gb 163Override the default package dependency memory target, in gigabytes. 164The default is 1/2 physical memory. 165The number of workers will be limited 166such that the aggregate size of package dependencies installed in each 167worker slot does not exceed this value. 168.Pp 169This handles a well-known effect where the sheer amount of data that has 170to be installed in tmpfs filesystems for large ports, when multiplied by 171the number of worker slots, can force excessive paging to occur and leave 172preciously little memory available to actually run compiles. 173Some paging 174is necessary to maintain maximum CPU utilization, but excessive paging 175can cause the whole machine to essentially become idle for extended 176periods of time. 177.It Fl M Ar scale 178Override dynamic workers calculation by a specific scale factor. 179Specifying 1.0 leaves it unchanged, 0.8 will reduce the number of jobs by 18020%, and 1.2 will increase the number of jobs by 20%. And so forth. 181.Pp 182This option may be specified in the range 0.01 to 99.0. Out of range values 183will simply be clipped. 184.It Fl p Ar profile 185Override the global profile default in 186.Pa /etc/dsynth/dsynth.ini , 187allowing you to trivially run whatever profile you like without having to 188edit the configuration file when switching. 189In addition, you can now run any number of dsynth's concurrently on the same 190machine without having to use a jail, each with a different profile, 191as long as the packages, repository, buildbase, and logs directories 192are different. 193.Pp 194Note that the distfiles directory can be shared and will not conflict 195or get confused with concurrent fetches. 196.It Fl s Ar n 197.Nm 198usually slow-starts the worker slots, beginning with one slot and increasing 199by one every 5 seconds until the maximum configured number of workers is 200reached. 201This gives 202.Nm 203a slower ramp that it can load manage against. 204Specifying 0 disables the slow-start feature and the maximum number of 205worker slots (limited by the dependency graph) will be loaded immediately. 206.It Fl v 207Quickly output the version and exit. 208.It Fl x 209.It Fl xx 210Normally dsynth builds a package for any of three reasons: (1) If the contents 211of the ports directory changes, (2) If anything the port depends on requires 212rebuilding so to will the port be rebuilt, (3) If there is no binary package 213already built for the port. 214.Pp 215If this option is specified, the first test is ignored. 216If this option is specified twice, the first and second tests are ignored. 217.It Fl y 218Automatically answer 'y'es to any questions. 219.It Fl D 220Turn on DEVELOPER mode when building ports. 221.It Fl P 222Include the check-plist stage. 223This is the default for the 224.Cm everything 225and 226.Cm test 227directives. 228This feature may also be turned on via the 229.Va Check_plist 230option in the configuration file. 231.It Fl S[S] 232Turn off curses for script friendliness. 233The output will be log 00 and 234should be redirected to /dev/null or something similar. 235If you supply the options twice, color output escapes will also be 236turned off. 237You may also wish to use the 238.Fl y 239option for scripting dsynth. 240.It Fl N 241Normally 242.Nm 243nices its sub-processes to +10. 244This option disables the feature. 245.El 246.Sh DIRECTIVES 247Generally 248.Nm 249is run with a directive and some directives allow a list of ports to be 250specified. 251This list should be space-delimited in DIR/SUBDIR format, for example: 252.Ar www/chromium . 253For directives with an optional ports list, your current installed set 254of ports will be used if you do not specify a list. You may also 255specify a filename instead of a port to have dsynth read the ports list 256from a text file. Ports are specified by dports directory and subdirectory. 257For example "www/chromium". 258.Bl -tag -width indent 259.It Cm init 260Creates and initializes the 261.Pa /etc/dsynth 262directory if it does not exist. 263This directive will complain and exit if either 264.Pa /etc/dsynth 265or 266.Pa /usr/local/etc/dsynth 267exists. 268It will not create 269.Pa /etc/dsynth 270in this situation. 271.It Cm status 272This will do a dry-run of 273.Cm upgrade-system 274but not actually build anything. 275.It Cm cleanup 276This will clean up any left-over mounts from prior builds. 277.Nm 278attempts to clean up all processes and mounts when you interrupt 279a build but doesn't always succeed. 280.It Cm configure 281NOT CURRENTLY IMPLEMENTED 282.It Cm fetch-only Op Ar ports/everything 283Fetch all source distributions required to build 284the specified target. Specifying 'everything' fetches 285all source distributions required to build the whole 286of dports. 287.Pp 288Any existing distfiles which do not match the expected 289signature will be re-fetched. 290.It Cm upgrade-system 291NOT CURRENTLY IMPLEMENTED. 292Incrementally build and upgrade your locally 293installed packages, then upgrade your local system with them. 294.It Cm list-system 295Write a build list to the file "build.txt". Do not build anything. 296This is typically used on your target system to generate a list for 297dsynth to use as a build list on another system. 298.It Cm prepare-system 299Incrementally build and upgrade your locally installed packages, but 300do not upgrade your system with them. 301.It Cm rebuild-repository 302Build or rebuild the database files for the configured repository. 303.It Cm purge-distfiles 304Delete any obsolete source distribution files. 305.It Cm reset-db 306Delete ports_crc.db from the build directory. 307This database is used to detect changes made to the dports tree. 308It will be regenerated on your next build without forcing any packages to be rebuilt. 309.It Cm status-everything 310This will do a dry-run of a full bulk build of everything, 311but not actually build anything. 312.It Cm everything 313This will build the entire dports tree and then rebuild the repository 314when it finishes. 315.It Cm version 316This is for synth compatibility. 317The version of 318.Nm 319will be printed and the program will exit. 320.It Cm help 321Output a synopsis of options and directives and exit. 322.It Cm status Op Ar ports 323Do a dry-run with 'build' of the given list. 324.It Cm add Op Ar ports 325This directive may be used when the user wishes to add additional 326ports to an existing dsynth run without interrupting and restarting 327the dsynth. 328It can be useful when the user intends to leave dsynth unattended for 329a long period of time and does not wish to interrupt potentially very 330long builds that are already in progress. 331.Pp 332When dsynth completes the current run it will re-exec itself with 333the same primary directive along with all ports specified by any 334.Cm add 335directives made in the interim. 336.Nm 337will still rebuild the repository after the initial run if it would 338normally have done so, but if so it will do it without asking first. 339Only the last rebuild request will potentially be interactive. 340.Pp 341Note that interrupting or killing the running dsynth cleans out any 342ports that might have been added while it was running. This directive 343also has numerous exit/exec lock-file races and is intended to only be used 344manually by the user. 345.It Cm build Op Ar ports 346Incrementally build dports based on the given list. 347When done, ask whether the repository should be rebuilt or not. 348.It Cm just-build Op Ar ports 349Incrementally build dports based on the given list, then 350exits. 351No post-build steps will be taken. 352.It Cm install Op Ar ports 353NOT CURRENTLY IMPLEMENTED. 'build' based on the supplied 354list (or using currently installed packages), then rebuild 355the repository and upgrade the system without asking any further 356questions. 357.It Cm force Op Ar ports 358This is the same as 'build' but will delete existing packages first. 359Dependencies are not deleted unless they are out of date. 360.It Cm test Op Ar ports 361This is the same as 'build' but sets the environment variable 362.Ev DEVELOPER 363to 364.Sq yes 365and pre-deletes specified packages. 366Dependencies are not deleted unless they are out of date. 367.It Cm debug Op Ar ports 368This is the same as 'build' but leaves the chroot mounts intact 369upon completion. 370.It Cm monitor Op Ar datfile 371Monitors a running dsynth instance. 372.El 373.Sh PER-PORT OPTIONS 374The 375.Va Directory_options 376configuration variable in 377.Pa /etc/dsynth/dsynth.ini 378points to the configured options directory tree. 379In the base system dports this would be 380.Pa /var/db/ports , 381but you can supply an independent set of ports options for your dsynth 382build if you like. 383The format of the structure in this directory is best described simply by 384CD'ing into a dport, say www/chromium, typing 'make config', and it will 385create a sub-directory and write out a file called 386.Pa /var/db/ports/www_chromium/options . 387.Pp 388For 389.Nm 390you can either point your configuration variable to the system default, 391or you can point it at a dsynth-specific directory and copy the options 392to or construct the options in your dsynth-specific directory tree. 393.Sh HOOKS 394.Nm 395provides several hooks that trigger at specific stages during the 396package building process. 397.Pp 398At the moment hooks are not configurable so the exact executable file is 399expected in the configuration directory with one of the names in the 400list below. 401Hooks are run via 402.Xr execve 2 . 403.Bl -tag -width indent 404.It Pa hook_run_start 405This hook triggers when the overall build process starts. 406.It Pa hook_run_end 407This hook is called when the overall build process ends. 408.It Pa hook_pkg_success 409For each successful port built this hook will trigger. 410.It Pa hook_pkg_failure 411This hook will trigger for each port that fails to build. 412.It Pa hook_pkg_ignored 413Each port that is marked as ignored will make this hook to trigger. 414.It Pa hook_pkg_skipped 415Each skipped port will trigger this hook. 416.El 417.Pp 418A number of environment variables are available for hooks, always in the context 419of an ongoing build and within a specific configuration profile, unless 420overridden from the command-line. 421Some are only available for a specific hook. 422.Bl -tag -width DIR_REPOSITORY 423.It Ev PROFILE 424The configuration profile. 425.It Ev DIR_PACKAGES 426The packages base directory, i.e where index files are generated. 427.It Ev DIR_REPOSITORY 428The packages repository, where the actual package files are stored. 429.It Ev DIR_PORTS 430The ports directory. 431.It Ev DIR_OPTIONS 432The options directory. 433.It Ev DIR_DISTFILES 434The distfiles directory, where the distribution files are stored. 435.It Ev DIR_LOGS 436The logs directory, which is also where the html Report is generated. 437.It Ev DIR_BUILDBASE 438The build base directory. 439.It Ev PORTS_QUEUED 440The number of ports queued to be built (only for 441.Pa hook_run_start ) . 442.It Ev PORTS_BUILT 443The number of successfully built ports (only for 444.Pa hook_run_end ) . 445.It Ev PORTS_FAILED 446The number of ports for which the build failed (only for 447.Pa hook_run_end ) . 448.It Ev PORTS_IGNORED 449The number of ports that where ignored and, hence, not built 450(only for 451.Pa hook_run_end ) . 452.It Ev PORTS_SKIPPED 453The number of ports that were skipped in the build (only for 454.Pa hook_run_end ) . 455.It Ev RESULT 456The result (success, failure, ignored, skipped) for the build of an individual 457port (only for 458.Pa hook_pkg_* ) . 459.It Ev ORIGIN 460The origin of a port (only for 461.Pa hook_pkg_* ) . 462.It Ev FLAVOR 463The flavor of a port (only for 464.Pa hook_pkg_* ) . 465.It Ev PKGNAME 466The port name (only for 467.Pa hook_pkg_* ) . 468.El 469.Sh MISC 470.Pp 471The default setting for 472.Va Meta_version 473is now 2. You can override it with this configuration variable. 474.Pp 475The default setting for 476.Va Check_plist 477is false. You can override it with the 478.Fl P 479option or by setting this configuration variable to true. 480.Pp 481The default setting for 482.Va Numa_setsize 483is 0, disabling any NUMA related CPU partitioning. This is the recommended 484setting as it allows the scheduler the freedom to maximize CPU utilization, 485particularly when doing incremental bulks. The only other values that have 486a decent chance of improving full bulk performance are 2 or 4. Any higher 487and CPU utilization will suffer from periods of high idle. 488.Sh FILES 489.Bl -tag -width ".It Pa <fs>/abc/defghi/<name>" -compact 490.It Pa /etc/dsynth/dsynth.ini 491The primary configuration file. 492If not found, 493.Nm 494will also look in 495.Pa /usr/local/etc/dsynth/dsynth.ini . 496.Pp 497.It Pa /etc/dsynth/LiveSystem-make.conf 498Typically contains the environment variables that will be set in 499the workers. 500.Nm 501firewalls the environment it is run under from the environment it 502provides to the workers. 503.Pp 504.It Pa /build/synth/build 505Recommended setting for 506.Va Directory_buildbase , 507contains the build infrastructure... typically a template, mirrored 508system directories, and mount points for all the worker slots. 509The template will be [re]generated if 'pkg' needs to be built or 510if the 511.Pa .template.good 512file in this directory is deleted. 513.Pp 514.It Pa /build/synth/distfiles 515Recommended setting for 516.Va Directory_distfiles , 517ports to a directory into which 518.Nm 519will download any source distribution files required for building. 520.Pp 521.It Pa /build/synth/dports 522Recommended setting for 523.Va Directory_portsdir , 524points to a checked out dports repo. 525Note that 526.Nm 527does not automatically 'git pull' or otherwise synchronize the dports repo, 528you must do that yourself prior to starting a build. 529.Pp 530.It Pa /build/synth/live_packages 531Recommended setting for 532.Va Directory_packages , 533points to a directory which will contain the completed application 534packages. 535.Pp 536.It Pa /build/synth/logs 537Recommended setting for 538.Va Directory_logs , 539all log files will be placed in this directory. 540Special management logfiles begin with the numeral '0' for easily 541location. 542The logfiles for ports while and after building are stored in the 543form subdir____portname.log, with three underscores. 544.Pp 545.It Pa /build/synth/options 546Recommended setting for 547.Va Directory_options , 548where options overrides for specific ports may be located. 549Then either null-mount the system /var/db/ports to this location, or 550construct your own dsynth-specific options. 551See the PER-PORT OPTIONS section above for more information. 552.Pp 553.It Pa / 554Recommended setting for 555.Va Directory_system , 556which 557.Nm 558uses as a basis for creating the jails or chroots in each worker slot 559during building. 560No part of the system root is ever NULL-mounted read-write... it is always 561NULL-mounted read-only. 562Some elements from the system base will be mirrored in the build-base 563as an optimization. 564.Pp 565Note that the packages directory and the distfiles directory is mounted 566read-write in jails or chroots. 567All other r/w filesystems in the workers are 568.Xr tmpfs 5 569based filesystems and will be created and torn-down for each port. 570.Pp 571.It Pa .txz 572.It Pa .tgz 573.It Pa .tar 574.It Pa .tbz 575.It Pa .tzst 576The recommended setting for 577.Va Package_suffix 578is either 579.Pa .txz 580or 581.Pa .tgz . 582Use 583.Pa .txz 584for better compression at the cost of somewhat slower bulk builds due 585to the time overhead for compression and decompression, or 586use 587.Pa .tgz 588for modest compression and very fast compression and decompression. 589Due to the way the builder works, package dependencies are fresly 590installed into the chroot slot for each package being built, so 591decompression time matters. 592.Pp 593.El 594.Sh EXIT STATUS 595.Ex -std 596.Sh SEE ALSO 597.Xr synth 1 , 598.Xr dports 7 599.Sh HISTORY 600The 601.Nm 602utility first appeared in 603.Dx 5.7 . 604.Sh AUTHORS 605.An Matthew Dillon Aq Mt dillon@backplane.com 606