1Design document for the unified scheme data 2=========================================== 3 4How are things connected? 5------------------------- 6 7The unified scheme takes all its data from the build.info files seen 8throughout the source tree. These files hold the minimum information 9needed to build end product files from diverse sources. See the 10section on build.info files below. 11 12From the information in build.info files, Configure builds up an 13information database as a hash table called %unified_info, which is 14stored in configdata.pm, found at the top of the build tree (which may 15or may not be the same as the source tree). 16 17Configurations/common.tmpl uses the data from %unified_info to 18generate the rules for building end product files as well as 19intermediary files with the help of a few functions found in the 20build-file templates. See the section on build-file templates further 21down for more information. 22 23build.info files 24---------------- 25 26As mentioned earlier, build.info files are meant to hold the minimum 27information needed to build output files, and therefore only (with a 28few possible exceptions [1]) have information about end products (such 29as scripts, library files and programs) and source files (such as C 30files, C header files, assembler files, etc). Intermediate files such 31as object files are rarely directly referred to in build.info files (and 32when they are, it's always with the file name extension .o), they are 33inferred by Configure. By the same rule of minimalism, end product 34file name extensions (such as .so, .a, .exe, etc) are never mentioned 35in build.info. Their file name extensions will be inferred by the 36build-file templates, adapted for the platform they are meant for (see 37sections on %unified_info and build-file templates further down). 38 39The variables PROGRAMS, LIBS, ENGINES and SCRIPTS are used to declare 40end products. There are variants for them with '_NO_INST' as suffix 41(PROGRAM_NO_INST etc) to specify end products that shouldn't get 42installed. 43 44The variables SOURCE, DEPEND and INCLUDE are indexed by a produced 45file, and their values are the source used to produce that particular 46produced file, extra dependencies, and include directories needed. 47 48All their values in all the build.info throughout the source tree are 49collected together and form a set of programs, libraries, engines and 50scripts to be produced, source files, dependencies, etc etc etc. 51 52Let's have a pretend example, a very limited contraption of OpenSSL, 53composed of the program 'apps/openssl', the libraries 'libssl' and 54'libcrypto', an engine 'engines/ossltest' and their sources and 55dependencies. 56 57 # build.info 58 LIBS=libcrypto libssl 59 INCLUDE[libcrypto]=include 60 INCLUDE[libssl]=include 61 DEPEND[libssl]=libcrypto 62 63This is the top directory build.info file, and it tells us that two 64libraries are to be built, the include directory 'include/' shall be 65used throughout when building anything that will end up in each 66library, and that the library 'libssl' depend on the library 67'libcrypto' to function properly. 68 69 # apps/build.info 70 PROGRAMS=openssl 71 SOURCE[openssl]=openssl.c 72 INCLUDE[openssl]=.. ../include 73 DEPEND[openssl]=../libssl 74 75This is the build.info file in 'apps/', one may notice that all file 76paths mentioned are relative to the directory the build.info file is 77located in. This one tells us that there's a program to be built 78called 'apps/openssl' (the file name extension will depend on the 79platform and is therefore not mentioned in the build.info file). It's 80built from one source file, 'apps/openssl.c', and building it requires 81the use of '.' and 'include' include directories (both are declared 82from the point of view of the 'apps/' directory), and that the program 83depends on the library 'libssl' to function properly. 84 85 # crypto/build.info 86 LIBS=../libcrypto 87 SOURCE[../libcrypto]=aes.c evp.c cversion.c 88 DEPEND[cversion.o]=buildinf.h 89 90 GENERATE[buildinf.h]=../util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)" 91 DEPEND[buildinf.h]=../Makefile 92 DEPEND[../util/mkbuildinf.pl]=../util/Foo.pm 93 94This is the build.info file in 'crypto', and it tells us a little more 95about what's needed to produce 'libcrypto'. LIBS is used again to 96declare that 'libcrypto' is to be produced. This declaration is 97really unnecessary as it's already mentioned in the top build.info 98file, but can make the info file easier to understand. This is to 99show that duplicate information isn't an issue. 100 101This build.info file informs us that 'libcrypto' is built from a few 102source files, 'crypto/aes.c', 'crypto/evp.c' and 'crypto/cversion.c'. 103It also shows us that building the object file inferred from 104'crypto/cversion.c' depends on 'crypto/buildinf.h'. Finally, it 105also shows the possibility to declare how some files are generated 106using some script, in this case a perl script, and how such scripts 107can be declared to depend on other files, in this case a perl module. 108 109Two things are worth an extra note: 110 111'DEPEND[cversion.o]' mentions an object file. DEPEND indexes is the 112only location where it's valid to mention them 113 114Lines in 'BEGINRAW'..'ENDRAW' sections must always mention files as 115seen from the top directory, no exception. 116 117 # ssl/build.info 118 LIBS=../libssl 119 SOURCE[../libssl]=tls.c 120 121This is the build.info file in 'ssl/', and it tells us that the 122library 'libssl' is built from the source file 'ssl/tls.c'. 123 124 # engines/build.info 125 ENGINES=dasync 126 SOURCE[dasync]=e_dasync.c 127 DEPEND[dasync]=../libcrypto 128 INCLUDE[dasync]=../include 129 130 ENGINES_NO_INST=ossltest 131 SOURCE[ossltest]=e_ossltest.c 132 DEPEND[ossltest]=../libcrypto.a 133 INCLUDE[ossltest]=../include 134 135This is the build.info file in 'engines/', telling us that two engines 136called 'engines/dasync' and 'engines/ossltest' shall be built, that 137dasync's source is 'engines/e_dasync.c' and ossltest's source is 138'engines/e_ossltest.c' and that the include directory 'include/' may 139be used when building anything that will be part of these engines. 140Also, both engines depend on the library 'libcrypto' to function 141properly. ossltest is explicitly linked with the static variant of 142the library 'libcrypto'. Finally, only dasync is being installed, as 143ossltest is only for internal testing. 144 145When Configure digests these build.info files, the accumulated 146information comes down to this: 147 148 LIBS=libcrypto libssl 149 SOURCE[libcrypto]=crypto/aes.c crypto/evp.c crypto/cversion.c 150 DEPEND[crypto/cversion.o]=crypto/buildinf.h 151 INCLUDE[libcrypto]=include 152 SOURCE[libssl]=ssl/tls.c 153 INCLUDE[libssl]=include 154 DEPEND[libssl]=libcrypto 155 156 PROGRAMS=apps/openssl 157 SOURCE[apps/openssl]=apps/openssl.c 158 INCLUDE[apps/openssl]=. include 159 DEPEND[apps/openssl]=libssl 160 161 ENGINES=engines/dasync 162 SOURCE[engines/dasync]=engines/e_dasync.c 163 DEPEND[engines/dasync]=libcrypto 164 INCLUDE[engines/dasync]=include 165 166 ENGINES_NO_INST=engines/ossltest 167 SOURCE[engines/ossltest]=engines/e_ossltest.c 168 DEPEND[engines/ossltest]=libcrypto.a 169 INCLUDE[engines/ossltest]=include 170 171 GENERATE[crypto/buildinf.h]=util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)" 172 DEPEND[crypto/buildinf.h]=Makefile 173 DEPEND[util/mkbuildinf.pl]=util/Foo.pm 174 175 176A few notes worth mentioning: 177 178LIBS may be used to declare routine libraries only. 179 180PROGRAMS may be used to declare programs only. 181 182ENGINES may be used to declare engines only. 183 184The indexes for SOURCE must only be end product files, such as 185libraries, programs or engines. The values of SOURCE variables must 186only be source files (possibly generated). 187 188INCLUDE and DEPEND shows a relationship between different files 189(usually produced files) or between files and directories, such as a 190program depending on a library, or between an object file and some 191extra source file. 192 193When Configure processes the build.info files, it will take it as 194truth without question, and will therefore perform very few checks. 195If the build tree is separate from the source tree, it will assume 196that all built files and up in the build directory and that all source 197files are to be found in the source tree, if they can be found there. 198Configure will assume that source files that can't be found in the 199source tree (such as 'crypto/bildinf.h' in the example above) are 200generated and will be found in the build tree. 201 202 203The %unified_info database 204-------------------------- 205 206The information in all the build.info get digested by Configure and 207collected into the %unified_info database, divided into the following 208indexes: 209 210 depends => a hash table containing 'file' => [ 'dependency' ... ] 211 pairs. These are directly inferred from the DEPEND 212 variables in build.info files. 213 214 engines => a list of engines. These are directly inferred from 215 the ENGINES variable in build.info files. 216 217 generate => a hash table containing 'file' => [ 'generator' ... ] 218 pairs. These are directly inferred from the GENERATE 219 variables in build.info files. 220 221 includes => a hash table containing 'file' => [ 'include' ... ] 222 pairs. These are directly inferred from the INCLUDE 223 variables in build.info files. 224 225 install => a hash table containing 'type' => [ 'file' ... ] pairs. 226 The types are 'programs', 'libraries', 'engines' and 227 'scripts', and the array of files list the files of 228 that type that should be installed. 229 230 libraries => a list of libraries. These are directly inferred from 231 the LIBS variable in build.info files. 232 233 programs => a list of programs. These are directly inferred from 234 the PROGRAMS variable in build.info files. 235 236 rawlines => a list of build-file lines. These are a direct copy of 237 the BEGINRAW..ENDRAW lines in build.info files. Note: 238 only the BEGINRAW..ENDRAW section for the current 239 platform are copied, the rest are ignored. 240 241 scripts => a list of scripts. There are directly inferred from 242 the SCRIPTS variable in build.info files. 243 244 sources => a hash table containing 'file' => [ 'sourcefile' ... ] 245 pairs. These are indirectly inferred from the SOURCE 246 variables in build.info files. Object files are 247 mentioned in this hash table, with source files from 248 SOURCE variables, and AS source files for programs and 249 libraries. 250 251 shared_sources => 252 a hash table just like 'sources', but only as source 253 files (object files) for building shared libraries. 254 255As an example, here is how the build.info files example from the 256section above would be digested into a %unified_info table: 257 258 our %unified_info = ( 259 "depends" => 260 { 261 "apps/openssl" => 262 [ 263 "libssl", 264 ], 265 "crypto/buildinf.h" => 266 [ 267 "Makefile", 268 ], 269 "crypto/cversion.o" => 270 [ 271 "crypto/buildinf.h", 272 ], 273 "engines/dasync" => 274 [ 275 "libcrypto", 276 ], 277 "engines/ossltest" => 278 [ 279 "libcrypto.a", 280 ], 281 "libssl" => 282 [ 283 "libcrypto", 284 ], 285 "util/mkbuildinf.pl" => 286 [ 287 "util/Foo.pm", 288 ], 289 }, 290 "engines" => 291 [ 292 "engines/dasync", 293 "engines/ossltest", 294 ], 295 "generate" => 296 { 297 "crypto/buildinf.h" => 298 [ 299 "util/mkbuildinf.pl", 300 "\"\$(CC)", 301 "\$(CFLAGS)\"", 302 "\"$(PLATFORM)\"", 303 ], 304 }, 305 "includes" => 306 { 307 "apps/openssl" => 308 [ 309 ".", 310 "include", 311 ], 312 "engines/ossltest" => 313 [ 314 "include" 315 ], 316 "libcrypto" => 317 [ 318 "include", 319 ], 320 "libssl" => 321 [ 322 "include", 323 ], 324 "util/mkbuildinf.pl" => 325 [ 326 "util", 327 ], 328 } 329 "install" => 330 { 331 "engines" => 332 [ 333 "engines/dasync", 334 ], 335 "libraries" => 336 [ 337 "libcrypto", 338 "libssl", 339 ], 340 "programs" => 341 [ 342 "apps/openssl", 343 ], 344 }, 345 "libraries" => 346 [ 347 "libcrypto", 348 "libssl", 349 ], 350 "programs" => 351 [ 352 "apps/openssl", 353 ], 354 "rawlines" => 355 [ 356 ], 357 "sources" => 358 { 359 "apps/openssl" => 360 [ 361 "apps/openssl.o", 362 ], 363 "apps/openssl.o" => 364 [ 365 "apps/openssl.c", 366 ], 367 "crypto/aes.o" => 368 [ 369 "crypto/aes.c", 370 ], 371 "crypto/cversion.o" => 372 [ 373 "crypto/cversion.c", 374 ], 375 "crypto/evp.o" => 376 [ 377 "crypto/evp.c", 378 ], 379 "engines/e_dasync.o" => 380 [ 381 "engines/e_dasync.c", 382 ], 383 "engines/dasync" => 384 [ 385 "engines/e_dasync.o", 386 ], 387 "engines/e_ossltest.o" => 388 [ 389 "engines/e_ossltest.c", 390 ], 391 "engines/ossltest" => 392 [ 393 "engines/e_ossltest.o", 394 ], 395 "libcrypto" => 396 [ 397 "crypto/aes.c", 398 "crypto/cversion.c", 399 "crypto/evp.c", 400 ], 401 "libssl" => 402 [ 403 "ssl/tls.c", 404 ], 405 "ssl/tls.o" => 406 [ 407 "ssl/tls.c", 408 ], 409 }, 410 ); 411 412As can be seen, everything in %unified_info is fairly simple suggest 413of information. Still, it tells us that to build all programs, we 414must build 'apps/openssl', and to build the latter, we will need to 415build all its sources ('apps/openssl.o' in this case) and all the 416other things it depends on (such as 'libssl'). All those dependencies 417need to be built as well, using the same logic, so to build 'libssl', 418we need to build 'ssl/tls.o' as well as 'libcrypto', and to build the 419latter... 420 421 422Build-file templates 423-------------------- 424 425Build-file templates are essentially build-files (such as Makefile on 426Unix) with perl code fragments mixed in. Those perl code fragment 427will generate all the configuration dependent data, including all the 428rules needed to build end product files and intermediary files alike. 429At a minimum, there must be a perl code fragment that defines a set of 430functions that are used to generates specific build-file rules, to 431build static libraries from object files, to build shared libraries 432from static libraries, to programs from object files and libraries, 433etc. 434 435 generatesrc - function that produces build file lines to generate 436 a source file from some input. 437 438 It's called like this: 439 440 generatesrc(src => "PATH/TO/tobegenerated", 441 generator => [ "generatingfile", ... ] 442 generator_incs => [ "INCL/PATH", ... ] 443 generator_deps => [ "dep1", ... ] 444 incs => [ "INCL/PATH", ... ], 445 deps => [ "dep1", ... ], 446 intent => one of "libs", "dso", "bin" ); 447 448 'src' has the name of the file to be generated. 449 'generator' is the command or part of command to 450 generate the file, of which the first item is 451 expected to be the file to generate from. 452 generatesrc() is expected to analyse and figure out 453 exactly how to apply that file and how to capture 454 the result. 'generator_incs' and 'generator_deps' 455 are include directories and files that the generator 456 file itself depends on. 'incs' and 'deps' are 457 include directories and files that are used if $(CC) 458 is used as an intermediary step when generating the 459 end product (the file indicated by 'src'). 'intent' 460 indicates what the generated file is going to be 461 used for. 462 463 src2obj - function that produces build file lines to build an 464 object file from source files and associated data. 465 466 It's called like this: 467 468 src2obj(obj => "PATH/TO/objectfile", 469 srcs => [ "PATH/TO/sourcefile", ... ], 470 deps => [ "dep1", ... ], 471 incs => [ "INCL/PATH", ... ] 472 intent => one of "lib", "dso", "bin" ); 473 474 'obj' has the intended object file *without* 475 extension, src2obj() is expected to add that. 476 'srcs' has the list of source files to build the 477 object file, with the first item being the source 478 file that directly corresponds to the object file. 479 'deps' is a list of explicit dependencies. 'incs' 480 is a list of include file directories. Finally, 481 'intent' indicates what this object file is going 482 to be used for. 483 484 obj2lib - function that produces build file lines to build a 485 static library file ("libfoo.a" in Unix terms) from 486 object files. 487 488 called like this: 489 490 obj2lib(lib => "PATH/TO/libfile", 491 objs => [ "PATH/TO/objectfile", ... ]); 492 493 'lib' has the intended library file name *without* 494 extension, obj2lib is expected to add that. 'objs' 495 has the list of object files (also *without* 496 extension) to build this library. 497 498 libobj2shlib - function that produces build file lines to build a 499 shareable object library file ("libfoo.so" in Unix 500 terms) from the corresponding static library file 501 or object files. 502 503 called like this: 504 505 libobj2shlib(shlib => "PATH/TO/shlibfile", 506 lib => "PATH/TO/libfile", 507 objs => [ "PATH/TO/objectfile", ... ], 508 deps => [ "PATH/TO/otherlibfile", ... ]); 509 510 'lib' has the intended library file name *without* 511 extension, libobj2shlib is expected to add that. 512 'shlib' has the corresponding shared library name 513 *without* extension. 'deps' has the list of other 514 libraries (also *without* extension) this library 515 needs to be linked with. 'objs' has the list of 516 object files (also *without* extension) to build 517 this library. 518 519 This function has a choice; it can use the 520 corresponding static library as input to make the 521 shared library, or the list of object files. 522 523 obj2dynlib - function that produces build file lines to build a 524 dynamically loadable library file ("libfoo.so" on 525 Unix) from object files. 526 527 called like this: 528 529 obj2dynlib(lib => "PATH/TO/libfile", 530 objs => [ "PATH/TO/objectfile", ... ], 531 deps => [ "PATH/TO/otherlibfile", 532 ... ]); 533 534 This is almost the same as libobj2shlib, but the 535 intent is to build a shareable library that can be 536 loaded in runtime (a "plugin"...). The differences 537 are subtle, one of the most visible ones is that the 538 resulting shareable library is produced from object 539 files only. 540 541 obj2bin - function that produces build file lines to build an 542 executable file from object files. 543 544 called like this: 545 546 obj2bin(bin => "PATH/TO/binfile", 547 objs => [ "PATH/TO/objectfile", ... ], 548 deps => [ "PATH/TO/libfile", ... ]); 549 550 'bin' has the intended executable file name 551 *without* extension, obj2bin is expected to add 552 that. 'objs' has the list of object files (also 553 *without* extension) to build this library. 'deps' 554 has the list of library files (also *without* 555 extension) that the programs needs to be linked 556 with. 557 558 in2script - function that produces build file lines to build a 559 script file from some input. 560 561 called like this: 562 563 in2script(script => "PATH/TO/scriptfile", 564 sources => [ "PATH/TO/infile", ... ]); 565 566 'script' has the intended script file name. 567 'sources' has the list of source files to build the 568 resulting script from. 569 570Along with the build-file templates is the driving engine 571Configurations/common.tmpl, which looks through all the information in 572%unified_info and generates all the rulesets to build libraries, 573programs and all intermediate files, using the rule generating 574functions defined in the build-file template. 575 576As an example with the smaller build.info set we've seen as an 577example, producing the rules to build 'libcrypto' would result in the 578following calls: 579 580 # Note: libobj2shlib will only be called if shared libraries are 581 # to be produced. 582 # Note 2: libobj2shlib gets both the name of the static library 583 # and the names of all the object files that go into it. It's up 584 # to the implementation to decide which to use as input. 585 # Note 3: common.tmpl peals off the ".o" extension from all object 586 # files, as the platform at hand may have a different one. 587 libobj2shlib(shlib => "libcrypto", 588 lib => "libcrypto", 589 objs => [ "crypto/aes", "crypto/evp", "crypto/cversion" ], 590 deps => [ ]); 591 592 obj2lib(lib => "libcrypto" 593 objs => [ "crypto/aes", "crypto/evp", "crypto/cversion" ]); 594 595 src2obj(obj => "crypto/aes" 596 srcs => [ "crypto/aes.c" ], 597 deps => [ ], 598 incs => [ "include" ], 599 intent => "lib"); 600 601 src2obj(obj => "crypto/evp" 602 srcs => [ "crypto/evp.c" ], 603 deps => [ ], 604 incs => [ "include" ], 605 intent => "lib"); 606 607 src2obj(obj => "crypto/cversion" 608 srcs => [ "crypto/cversion.c" ], 609 deps => [ "crypto/buildinf.h" ], 610 incs => [ "include" ], 611 intent => "lib"); 612 613 generatesrc(src => "crypto/buildinf.h", 614 generator => [ "util/mkbuildinf.pl", "\"$(CC)", 615 "$(CFLAGS)\"", "\"$(PLATFORM)\"" ], 616 generator_incs => [ "util" ], 617 generator_deps => [ "util/Foo.pm" ], 618 incs => [ ], 619 deps => [ ], 620 intent => "lib"); 621 622The returned strings from all those calls are then concatenated 623together and written to the resulting build-file. 624