1mk-files 2******** 3 4The term ``mk-files`` refers to a collection of ``*.mk`` files. 5 6You need bmake_ or a *recent* NetBSD_ make. 7If in doubt use bmake_. 8 9Introduction 10============ 11 12Many years ago, when building large software projects, I used GNU make 13(or my own patched version of it), and had developed a set of macros 14to simplify developing complex build trees. 15 16Since the early 90's my main development machines, run BSD 17(NetBSD_ to be precise, and more recently FreeBSD), and the BSD source 18tree is good example of a large software project. 19It quickly became clear that ``/usr/share/mk/*.mk`` were a great 20model, but at the time were quite tightly linked to building the BSD tree. 21 22Much as I liked using NetBSD, my customers were more likely to be 23using SunOS, HP-UX etc, so I started on bmake_ and a portable collection 24of mk-files (mk.tar.gz_). NetBSD provided much of the original structure. 25 26Since then I've added a lot of features to NetBSD's make and hence to 27bmake which is kept closely in sync. The mk-files however have 28diverged quite a bit, though ideas are still picked up from NetBSD 29and FreeBSD. 30 31Basics 32------ 33 34The BSD build model is very simple. A directory produces one 35component, which is generally either a library or a program. 36Library makefiles include ``lib.mk`` and programs include ``prog.mk`` 37and they *do the right thing*. 38 39A simple library makefile might look like:: 40 41 LIB = sig 42 43 SRCS = \ 44 sigaction.c \ 45 sigcompat.c \ 46 sighdl.c 47 48 .include <lib.mk> 49 50a simple program makefile:: 51 52 PROG = cat 53 54 SRCS = cat.c 55 56 .include <prog.mk> 57 58in such cases even the ``SRCS`` line is unnecessary as ``prog.mk`` 59will default it to ``${PROG}.c``. 60 61It is the sensible use of defaults and the plethora of macro modifiers 62provided by bmake_ that allow simple makefiles such as the above to 63*just work* on many different systems. 64 65 66mk-files 67======== 68 69This section provides a brief description of some of the ``*.mk`` 70files. 71 72sys.mk 73------ 74 75When bmake starts, it looks for ``sys.mk`` and reads it before doing 76anything else. Thus, this is the place to setup the environment for 77everyone else. 78 79In this distribution, ``sys.mk`` avoids doing anything platform or 80site dependent. 81It is quite short, and includes a number of other files (which may or 82may not exists) 83 84sys.env.mk 85 If it exists, is expected to do things like conditioning the 86 environment. Since it will only be included by the initial 87 instance of bmake, it should ``.export`` anything that 88 sub-makes might need. 89 90examples/sys.clean-env.mk 91 An example of how to clean the environment. 92 See the file for all the details:: 93 94 .if ${MAKE_VERSION} >= 20100606 && ${.MAKE.LEVEL} == 0 95 # we save any env var that starts with these 96 MAKE_SAVE_ENV_PREFIX += SB MK MAKE MACHINE NEED_ CCACHE DISTCC USE_ SSH 97 MAKE_SAVE_ENV_VARS += \ 98 PATH HOME USER LOGNAME \ 99 SRCTOP OBJTOP OBJROOT \ 100 ${_env_vars} 101 102 _env_vars != env | egrep '^(${MAKE_SAVE_ENV_PREFIX:ts|})' | sed 's,=.*,,'; echo 103 _export_list = 104 .for v in ${MAKE_SAVE_ENV_VARS:O:u} 105 .if !empty($v) 106 _export_list += $v 107 $v := ${$v} 108 .endif 109 .endfor 110 # now clobber the environment 111 .unexport-env 112 113 # list of vars that we handle specially below 114 _tricky_env_vars = MAKEOBJDIR 115 # export our selection - sans tricky ones 116 .export ${_export_list:${_tricky_env_vars:${M_ListToSkip}}} 117 118 # this next bit may need tweaking 119 .if defined(MAKEOBJDIR) 120 srctop := ${SRCTOP:U${SB_SRC:U${SB}/src}} 121 objroot := ${OBJROOT:U${SB_OBJROOT:U${SB}/${SB_OBJPREFIX}}} 122 # we'll take care of MACHINE below 123 objtop := ${OBJTOP:U${objroot}${MACHINE}} 124 .if !empty(objtop) 125 # we would normally want something like (/bin/sh): 126 # MAKEOBJDIR="\${.CURDIR:S,${SRCTOP},${OBJROOT}\${MACHINE},}" 127 # the $$ below is how we achieve the same result here. 128 # since everything saved from the environment above 129 # has run through := we need to compensate for ${MACHINE} 130 MAKEOBJDIR = $${.CURDIR:S,${srctop},${objtop:S,${MACHINE},\${MACHINE},},} 131 132 # export these as-is, and do not track... 133 .export-env ${_tricky_env_vars} 134 # now evaluate for ourselves 135 .for v in ${_tricky_env_vars} 136 $v := ${$v} 137 .endfor 138 139 .endif 140 .endif 141 .endif 142 143 144host-target.mk 145 Is used to set macros like ``HOST_TARGET``, ``HOST_OS`` and 146 ``host_os`` which are used to find the next step. 147 Note: since 20130303 bmake provides ``.MAKE.OS`` set to 148 the equivalent of ``HOST_OS``. 149 150sys/\*.mk 151 Platform specific additions, such as ``Darwin.mk`` or ``SunOS.mk`` 152 set things like ``HOST_LIBEXT = .dylib`` for Darwin or 153 ``SHLIB_FULLVERSION = ${SHLIB_MAJOR}`` for SunOS 5. 154 If there is no OS specific file, ``sys/Generic.mk`` is used. 155 156local.sys.mk 157 Any ``local.*.mk`` file is not part of the distribution. 158 This provides a hook for sites to do extra setup without 159 having to edit the distributed files. 160 161 162The above arrangement makes it easy for the mk files to be part of a 163src tree on an NFS volume and to allow building on multiple platforms. 164 165options.mk 166---------- 167 168Inspired by FreeBSD's ``bsd.own.mk`` but more flexible. 169FreeBSD now have similar functionality in ``bsd.mkopt.mk``. 170 171It allows users to express their intent with respect to options 172``MK_*`` by setting ``WITH_*`` or ``WITHOUT_*``. 173 174Note: ``WITHOUT_*`` wins if both are set, and makefiles can set 175``NO_*`` to say they cannot handle that option, or even ``MK_*`` if 176they really need to. 177 178lib.mk 179------ 180 181This file is used to build a number of different libraries from the 182same SRCS. 183 184``lib${LIB}.a`` 185 An archive lib of ``.o`` files, this is the default 186 187``lib${LIB}_p.a`` 188 A profiled lib of ``.po`` files. 189 Still an archive lib, but all the objects are built with 190 profiling in mind - hence the different extension. 191 It is skipped if ``MK_PROFILE`` is "no". 192 193``lib${LIB}_pic.a`` 194 An archive of ``.so`` objects compiled for relocation. 195 On NetBSD this is the input to ``lib${LIB}.${LD_so}``, it is 196 skipped if ``MK_PIC`` or ``MK_PICLIB`` are "no". 197 198``lib${LIB}.${LD_so}`` 199 A shared library. The value of ``LD_so`` is very platform 200 specific. For example:: 201 202 # SunOS 5 and most other ELF systems 203 libsslfd.so.1 204 205 # Darwin 206 libsslfd.1.dylib 207 208 This library will only be built if ``SHLIB_MAJOR`` has 209 a value, and ``MK_PIC`` is not set to "no". 210 211There is a lot of platform specific tweaking in ``lib.mk``, largely the 212result of the original distributions trying to avoid interfering with 213the system's ``sys.mk``. 214 215libnames.mk 216----------- 217 218This is included by both ``prog.mk`` and ``lib.mk`` and tries to 219include ``*.libnames.mk`` of which: 220 221``local.libnames.mk`` 222 does not exist unless you create it. It is a handy way for you 223 to customize without touching the distributed files. 224 For example, on a test machine I needed to build openssl but 225 not install it, so put the following in ``local.libnames.mk``:: 226 227 .if ${host_os} == "sunos" 228 LIBCRYPTO = ${OBJTOP}/openssl/lib/crypto/libcrypto${DLIBEXT} 229 LIBSSL = ${OBJTOP}/openssl/lib/ssl/libssl${DLIBEXT} 230 INCLUDES_libcrypto = -I${OBJ_libcrypto} 231 .endif 232 233 The makefile created an openssl dir in ``${OBJ_libcrypto}`` to 234 gather all the headers. dpadd.mk_ did the rest. 235 236``host.libnames.mk`` 237 contains logic to find any libs named in ``HOST_LIBS`` in 238 ``HOST_LIBDIRS``. 239 240Each file above gets an opportunity to define things like:: 241 242 LIBSSLFD ?= ${OBJTOP}/ssl/lib/sslfd/libsslfd${DLIBEXT} 243 INCLUDES_libsslfd = -I${SRC_libsslfd}/h -I${OBJ_libslfd} 244 245these are used by dpadd.mk_ and will be explained below. 246 247dpadd.mk 248-------- 249 250This file looks like line noise, and is best considered read-only. 251However it provides some very useful functionality, which simplifies the build. 252 253Makefiles can use the LIB* macros defined via libnames.mk_ or anywhere 254else in various ways:: 255 256 # indicate that we need to include headers from LIBCRYPTO 257 # this would result in ${INCLUDES_libcrypto} being added to CFLAGS. 258 SRC_LIBS += ${LIBCRYPTO} 259 260 # indicate that libsslfd must be built already. 261 # it also has the same effect as SRC_LIBS 262 DPADD += ${LIBSSLFD} 263 264 # indicate that not only must libsslfd be built, 265 # but that we need to link with it. 266 # this is almost exactly equivalent to 267 # DPADD += ${LIBSSLFD} 268 # LDADD += -L${LIBSSLFD:H} -lsslfd 269 # and mostly serves to ensure that DPADD and LDADD are in sync. 270 DPLIBS += ${LIBSSLFD} 271 272Any library (referenced by its full path) in any of the above, is 273added to ``DPMAGIC_LIBS`` with the following results, for each lib *foo*. 274 275``SRC_libfoo`` 276 Is set to indicate where the src for libfoo is. 277 By default it is derived from ``LIBFOO`` by replacing 278 ``${OBJTOP}`` with ``${SRCTOP}``. 279 280``OBJ_libfoo`` 281 Not very exciting, is just the dir where libfoo lives. 282 283``INCLUDES_libfoo`` 284 What to add to ``CFLAGS`` to find the public headers. 285 The default varies. If ``${SRC_libfoo}/h`` exists, it is assumed 286 to be the home of all public headers and thus the default is 287 ``-I${SRC_libfoo}/h`` 288 289 Otherwise we make no assumptions and the default is 290 ``-I${SRC_libfoo} -I${OBJ_libfoo}`` 291 292``LDADD_libfoo`` 293 This only applies to libs reference via ``DPLIBS``. 294 The default is ``-lfoo``, ``LDADD_*`` provides a hook to 295 instantiate other linker flags at the appropriate point 296 without losing the benfits of ``DPLIBS``. 297 298prog.mk 299------- 300 301Compiles the specified SRCS and links them and the nominated libraries 302into a program. Prog makefiles usually need to list the libraries 303that need to be linked. We prefer use of ``DPLIBS`` but the more 304traditional ``DPADD`` and ``LDADD`` work just as well. 305That is:: 306 307 DPLIBS += ${LIBCRYPTO} 308 309is equivalent to:: 310 311 DPADD += ${LIBCRYPTO} 312 LDADD += -lcrypto 313 314obj.mk 315------ 316 317One of the cool aspects of BSD make, is its support for separating 318object files from the src tree. This is also the source of much 319confusion for people unfamiliar with it. 320 321Traditionally one had to do a separate ``make obj`` pass through the 322tree. If ``MK_AUTO_OBJ`` is set we include auto.obj.mk_. 323 324In fact if ``MKOBJDIRS`` is set to "auto", `sys.mk`_ will set 325``MK_AUTO_OBJ=yes`` and include auto.obj.mk_ since it is best done early. 326 327auto.obj.mk 328----------- 329 330Creates object dirs and leverages the ``.OBJDIR`` target introduced 331some years ago to NetBSD make, to use them. 332 333 334subdir.mk 335--------- 336 337This is the traditional means of walking the tree. A makefile sets 338``SUBDIR`` to the list of sub-dirs to visit. 339 340If ``SUBDIR_MUST_EXIST`` is set, missing directories cause an error, 341otherwise a warning is issued. If you don't even want the warning, 342set ``MISSING_DIR=continue``. 343 344Traditionally, ``subdir.mk`` prints clues as it visits each subdir:: 345 346 ===> ssl 347 ===> ssl/lib 348 ===> ssl/lib/sslfd 349 350you can suppress that - or enhance it by setting ``ECHO_DIR``:: 351 352 # suppress subdir noise 353 ECHO_DIR=: 354 # print time stamps 355 ECHO_DIR=echo @ `date "+%s [%Y-%m-%d %T] "` 356 357I prefer to use `dirdeps.mk`_ which makes ``subdir.mk`` irrelevant. 358 359links.mk 360-------- 361 362Provides rules for processing lists of ``LINKS`` and ``SYMLINKS``. 363Each is expected to be a list of ``link`` and ``target`` pairs 364(``link`` -> ``target``). 365 366The logic is generally in a ``_*_SCRIPT`` which is referenced in a 367``_*_USE`` (``.USE``) target. 368 369The ``_BUILD_*`` forms are identical, but do not use ``${DESTDIR}`` 370and so are useful for creating symlinks during the build phase. 371For example:: 372 373 SYMLINKS += ${.CURDIR}/${MACHINE_ARCH}/include machine 374 header_links: _BUILD_SYMLINKS_USE 375 376 md.o: header_links 377 378would create a symlink called ``machine`` in ``${.OBJDIR}`` pointing to 379``${.CURDIR}/${MACHINE_ARCH}/include`` before compiling ``md.o`` 380 381 382autoconf.mk 383----------- 384 385Deals with running (or generating) GNU autoconf ``configure`` scripts. 386 387dep.mk 388------ 389 390Deals with collecting dependencies. Another useful feature of BSD 391make is the separation of this sort of information into a ``.depend`` 392file. ``MKDEP_CMD`` needs to point to a suitable tool (like mkdeps.sh_) 393 394If ``MK_AUTODEP`` is "yes" it sets ``MKDEP_MK`` to autodep.mk_ by default. 395 396``MKDEP_MK`` can also be set to `auto.dep.mk`_ which is more efficient 397but does not support an explicit ``depend`` target. 398 399autodep.mk 400---------- 401 402Leverages the ``-MD`` feature of recent GCC to collect dependency 403information as a side effect of compilation. With this GCC puts 404dependency info into a ``.d`` file. 405 406Unfortunately GCC bases the name of the ``.d`` file on the name of the 407input rather than the output file, which causes problems when the same 408source is compiled different ways. The latest GCC supports ``-MF`` to 409name the ``.d`` file and ``-MT`` to control the name to put as the 410dependent. 411 412Recent bmake allows dependencies for the ``.END`` target (run at the 413end if everything was successful), and ``autodep.mk`` uses this to 414post process the ``.d`` files into ``.depend``. 415 416auto.dep.mk 417----------- 418 419A much simpler implementation than autodep.mk_ it uses 420``-MF ${.TARGET:T}.d`` 421to avoid possible conflicts during parallel builds. 422This precludes the use of suffix rules to drive ``make depend``, so 423dep.mk_ handles that if specifically requested. 424 425If ``bmake`` is 20160218 or newer, ``auto.dep.mk`` uses ``.dinclude`` 426to includes the ``*.d`` files directly thus avoiding the need to 427create a ``.depend`` file from them. 428 429own.mk 430------ 431 432Normally included by ``init.mk`` (included by ``lib.mk`` and 433``prog.mk`` etc), sets macros for default ownership etc. 434 435It includes ``${MAKECONF}`` if it is defined and exists. 436 437ldorder.mk 438---------- 439 440Leverages ``bmake`` to compute optimal link order for libraries. 441This works nicely and makes refactoring a breeze - so long as you 442have no (or few) cicular dependencies between libraries. 443 444Consider this experimental. 445 446man.mk 447------ 448 449Deals with man pages. 450 451warnings.mk 452----------- 453 454This provides a means of fine grained control over warnings on a per 455``${MACHINE}`` or even file basis. 456 457A makefile sets ``WARNINGS_SET`` to name a list of warnings 458and individual ``W_*`` macros can be used to tweak them. 459For example:: 460 461 WARNINGS_SET = HIGH 462 W_unused_sparc = -Wno-unused 463 464would add all the warnings in ``${HIGH_WARNINGS}`` to CFLAGS, but 465on sparc, ``-Wno-unused`` would replace ``-Wunused``. 466 467You should never need to edit ``warnings.mk``, it will include 468``warnings-sets.mk`` and/or ``local.warnings.mk`` to pick up 469customizations. 470 471rst2htm.mk 472---------- 473 474Logic to simplify generating HTML (and PDF) documents from ReStructuredText. 475 476cython.mk 477--------- 478 479Logic to build Python C interface modules using Cython_ 480 481.. _Cython: http://www.cython.org/ 482 483cc-wrap.mk 484---------- 485 486This makefile leverages two new features in bmake 20220126 and later. 487 488First is the ablity to set target local variables (GNU make has done 489this for ages). 490 491The second (only intersting if using `meta mode`_) 492allows filtering commands before comparison with previous run to 493decide if a target is out-of-date. 494 495In the past, making use of compiler wrappers like ``ccache``, 496``distcc`` or the newer ``icecc`` could get quite ugly. 497Using ``cc-wrap.mk`` it could not be simpler. 498 499jobs.mk 500------- 501 502This should be included by the top-level makefile. 503If you do:: 504 505 make something-jobs 506 507then ``jobs.mk`` will run:: 508 509 make -j${JOB_MAX} someting > ${JOB_LOGDIR}/something.log 2>&1 510 511this ensures you get a build log and JOB_MAX is assumed to be set 512optimally for the host. 513 514META_MODE 515========= 516 517The 20110505 and later versions of ``mk-files`` include a number of 518makefiles contributed by Juniper Networks, Inc. 519These allow the latest version of bmake_ to run in `meta mode`_ 520see `dirdeps.mk`_ and DIRDEPS_BUILD_ below. 521 522.. _`dirdeps.mk`: /help/sjg/dirdeps.htm 523.. _`meta mode`: bmake-meta-mode.htm 524 525DIRDEPS_BUILD 526============= 527 528When the `meta mode`_ was originally done, there was no distinction 529between META_MODE_ and ``DIRDEPS_BUILD``, but as these were integrated 530into FreeBSD it became clear that META_MODE_ could be useful to many 531developers independently of ``DIRDEPS_BUILD``. 532 533Thus today we distinguish between the two. 534We have the following makefiles which are relevant to 535``DIRDEPS_BUILD`` or META_MODE_:: 536 537 share/mk/auto.obj.mk 538 share/mk/dirdeps-cache-update.mk 539 share/mk/dirdeps-only.mk 540 share/mk/dirdeps-options.mk 541 share/mk/dirdeps-targets.mk 542 share/mk/dirdeps.mk 543 share/mk/gendirdeps.mk 544 share/mk/host-target.mk 545 share/mk/install-new.mk 546 share/mk/meta.autodep.mk 547 share/mk/meta.stage.mk 548 share/mk/meta.sys.mk 549 share/mk/meta2deps.py 550 share/mk/meta2deps.sh 551 share/mk/sys.dependfile.mk 552 share/mk/sys.dirdeps.mk 553 554and the following are typically used for customization. 555See `freebsd-meta-mode`_ and `netbsd-meta-mode`_:: 556 557 share/mk/local.dirdeps-build.mk 558 share/mk/local.dirdeps-missing.mk 559 share/mk/local.dirdeps.mk 560 share/mk/local.meta.sys.mk 561 share/mk/local.sys.dirdeps.env.mk 562 share/mk/local.sys.dirdeps.mk 563 share/mk/local.sys.mk 564 565 566Install 567======= 568 569You can use the content of mk.tar.gz_ without installing at all. 570 571The script ``install-mk`` takes care of copying ``*.mk`` into a 572destination directory, and unless told not to, create ``bsd.*.mk`` links 573for ``lib.mk`` etc. 574 575If you just want to create the ``bsd.*.mk`` links in the directory 576where you unpacked the tar file, you can:: 577 578 ./mk/install-mk ./mk 579 580------ 581 582.. _bmake: bmake.htm 583.. _NetBSD: http://www.netbsd.org/ 584.. _mkdeps.sh: https://www.crufty.net/ftp/pub/sjg/mkdeps.sh 585.. _mk.tar.gz: https://www.crufty.net/ftp/pub/sjg/mk.tar.gz 586.. _`freebsd-meta-mode`: https://www.crufty.net/sjg/docs/freebsd-meta-mode.htm 587.. _`netbsd-meta-mode`: https://www.crufty.net/sjg/docs/netbsd-meta-mode.htm 588 589:Author: sjg@crufty.net 590:Revision: $Id: mk-files.txt,v 1.23 2023/05/11 22:55:08 sjg Exp $ 591:Copyright: Crufty.NET 592