1<!doctype refentry PUBLIC "-//OASIS//DTD DocBook V4.1//EN"> 2 3<refentry lang="en"> 4 <refentryinfo> 5 <address>jigdo atterer.net</address> 6 <author> 7 <firstname>Richard</firstname><surname>Atterer</surname> 8 </author> 9 <copyright> 10 <year>2001-2005</year><holder>Richard Atterer</holder> 11 </copyright> 12 <date>Jul 2, 2005</date> 13 </refentryinfo> 14 <refmeta> 15 <refentrytitle>jigdo-file</refentrytitle> 16 <manvolnum>1</manvolnum> 17 </refmeta> 18 <refnamediv> 19 <refname>jigdo-file</refname> 20 <refpurpose>Prepare files for Jigsaw Download (distribution of 21 huge files, e.g. CD images).</refpurpose> 22 </refnamediv> 23 <refsynopsisdiv> 24 <cmdsynopsis> 25 <command>jigdo-file</command> 26 <arg choice="req"> 27 <replaceable> COMMAND</replaceable> 28 </arg> 29 <arg><option>--image=<replaceable 30 >cdrom.iso</replaceable></option></arg> 31 <arg><option>--jigdo=<replaceable 32 >cdrom.jigdo</replaceable></option></arg> 33 <arg><option>--template=<replaceable 34 >cdrom.template</replaceable></option></arg> 35 <arg><option>--force</option></arg> 36 <arg>MORE OPTIONS</arg> 37 <group> 38 <arg rep="repeat"><replaceable>FILES</replaceable></arg> 39 <arg><option>--files-from=<replaceable 40 >f</replaceable></option></arg> 41 </group> 42 <sbr> 43 <arg choice="plain">Common COMMANDs: make-template, 44 make-image, verify</arg> 45 </cmdsynopsis> 46 </refsynopsisdiv> 47 <!-- ============================================================= --> 48 <refsect1 id="description"> 49 <title>DESCRIPTION</title> 50 51 <para>Jigsaw Download, or short jigdo, is a scheme developed 52 primarily to make it easy to distribute huge filesystem images 53 (e.g. CD (ISO9660) or DVD (UDF) images) over the internet, but it 54 could also be used for other data which is awkward to handle due 55 to its size, like audio/video files or large software 56 packages.</para> 57 58 <para>jigdo tries to ensure that the large file (always called 59 <firstterm>image</firstterm> from now on) is downloaded in small 60 parts which can be stored on different servers. People who want to 61 download the image do so by telling the <citerefentry> 62 <refentrytitle>jigdo</refentrytitle><manvolnum>1</manvolnum> 63 </citerefentry> <emphasis>(NOT IMPLEMENTED YET)</emphasis> 64 download tool to process one `<filename>.jigdo</filename>' file; 65 using it, <command>jigdo</command> downloads the parts and 66 reassembles the image. <command>jigdo-file</command> is used to 67 prepare the files for download.</para> 68 69 <para>What makes jigdo special is that the parts that are used to 70 reconstruct the image can have any size and content - they only 71 need to be contained in a contiguous region anywhere in the 72 image.</para> 73 74 <para>For example, if you wish to distribute an ISO9660 image 75 which contains a snapshot of an FTP server, you can instruct 76 <command>jigdo-file</command> to prepare the download data in such 77 a way that when people use <command>jigdo</command> to download 78 the image, <command>jigdo</command> actually fetches the 79 individual files from the FTP server and assembles them into an 80 exact copy of your image - during the download! (If the image is 81 not a filesystem dump, you can use 82 <citerefentry> 83 <refentrytitle>split</refentrytitle><manvolnum>1</manvolnum> 84 </citerefentry> to create the small parts that the image will be 85 reassembled from.)</para> 86 87 <para>You are completely free to choose where the individual parts 88 of the image are stored: They may be in entirely different 89 directories on different servers (e.g. because of 90 storage/bandwidth constraints), but this is invisible to the 91 people downloading your image. The information about available 92 servers only needs to be added to the 93 `<filename>.jigdo</filename>' file by you before distributing 94 it.</para> 95 96 <para>The `DETAILS' section below contains technical details on 97 how jigdo works. The `EXAMPLES' section lists a number of common 98 scenarios and may help you to get an idea of what jigdo is useful 99 for.</para> 100 101 </refsect1> 102 <!-- ============================================================= --> 103 <refsect1 id="options"> 104 <title>OPTIONS</title> 105 106 <para>Many options are specific to a particular 107 <replaceable>COMMAND</replaceable>; the ones below are general or 108 used by several commands. Further options are listed below with 109 the individual commands. All options are silently ignored if they 110 are not applicable to the current command. For any 111 <replaceable>BYTES</replaceable> parameters to options, you can 112 append one of the letters `k', `M' or `G' to the amount you 113 specify, to indicate kilobytes, megabytes or gigabytes.</para> 114 115 <variablelist> 116 117 <varlistentry> 118 <term><option>-h</option> <option>--help</option></term> 119 <listitem> 120 <para>Output short summary of commands and options.</para> 121 </listitem> 122 </varlistentry> 123 124 <varlistentry> 125 <term><option>-H</option> <option>--help-all</option></term> 126 <listitem> 127 <para>Output complete summary of commands and options.</para> 128 </listitem> 129 </varlistentry> 130 131 <varlistentry> 132 <term><option>-v</option> <option>--version</option></term> 133 <listitem> 134 <para>Output program version.</para> 135 </listitem> 136 </varlistentry> 137 138 <varlistentry> 139 <term><option>-i</option> <option>--image=<replaceable 140 >cdrom.iso</replaceable></option></term> 141 <listitem> 142 <para>Specify location of the file containing the image. The 143 image is the large file that you want to distribute.</para> 144 </listitem> 145 </varlistentry> 146 147 <varlistentry> 148 <term><option>-j</option> <option>--jigdo=<replaceable 149 >cdrom.jigdo</replaceable></option></term> 150 <listitem> 151 <para>Specify location of the Jigsaw Download description 152 file. The jigdo file is a human-readable file generated by 153 <command>jigdo-file</command>, to which you add information 154 about all the servers you are going to upload the files to. 155 <command>jigdo</command> will download this file as the 156 first step of retrieving the image.</para> 157 </listitem> 158 </varlistentry> 159 160 <varlistentry> 161 <term><option>-t</option> <option>--template=<replaceable 162 >cdrom.template</replaceable></option></term> 163 <listitem> 164 <para>Specify location of the image `template' file. The 165 template file is a binary file generated by 166 <command>jigdo-file</command>, it contains information on 167 how to reassemble the image and also (in compressed form) 168 all the data from the image which was not found in any of 169 the parts.</para> 170 171 <para>Depending on the command, each of these three files is 172 used sometimes for input, sometimes for output. If the file 173 is to be used for output for a particular command and the 174 output file already exists, <command>jigdo-file</command> 175 exits with an error, unless <option>--force</option> is 176 present.</para> 177 178 <para>In most cases, you will only need to specify one out 179 of <option>-i</option> <option>-j</option> 180 <option>-t</option>, because any missing filenames will be 181 deduced from the one you specify. This is done by first 182 stripping any extension from the supplied name and then 183 appending nothing (if deducing <option>--image</option>), 184 `<filename>.jigdo</filename>' or 185 `<filename>.template</filename>'.</para> 186 </listitem> 187 </varlistentry> 188 189 <varlistentry> 190 <term><option>-r</option> <option 191 >--report=default|noprogress|quiet|grep</option></term> 192 <listitem> 193 <para>Control how verbose the program is, and what format 194 the output has: <option>noprogress</option> is the same as 195 <option>default</option> except that no `<computeroutput>x% 196 done</computeroutput>' progress messages are printed. 197 <option>quiet</option> restricts the output to what is 198 absolutely necessary, mostly error messages. 199 <option>grep</option> is only different from 200 <option>default</option> for the 201 <command>make-template</command> command: It enables output 202 in a simple `<replaceable><offset> 203 <file></replaceable>' format which is useful when 204 searching for binary files in other binary files.</para> 205 </listitem> 206 </varlistentry> 207 208 <varlistentry> 209 <term><option>-f</option> <option>--force</option></term> 210 <listitem> 211 <para>Overwrite existent output files without 212 complaining.</para> 213 </listitem> 214 </varlistentry> 215 216 <varlistentry> 217 <term><option>--no-force</option></term> 218 <listitem> 219 <para><emphasis>This is the default.</emphasis> Refuse to 220 overwrite existent output files.</para> 221 </listitem> 222 </varlistentry> 223 224 <varlistentry> 225 <term><option>-c</option> <option>--cache=<replaceable 226 >jigdo-cache.db</replaceable></option></term> 227 <listitem> 228 <para><command>jigdo-file</command> usually needs to read 229 the entire contents of all the 230 <replaceable>FILES</replaceable> you specify. If you use it 231 repeatedly (e.g. because you make a new CD image available 232 daily), caching the file information will increase the 233 program's speed significantly. The cache file is 234 automatically created if it is not yet present. Data is 235 usually both read from and written to it.</para> 236 </listitem> 237 </varlistentry> 238 239 <varlistentry> 240 <term><option>--no-cache</option></term> 241 <listitem> 242 <para><emphasis>This is the default.</emphasis> Do not use a 243 cache.</para> 244 </listitem> 245 </varlistentry> 246 247 <varlistentry> 248 <term><option>--cache-expiry=<replaceable 249 >SECONDS</replaceable></option></term> 250 <listitem> 251 <para>Set maximum age of cache entries. Any entries older 252 than this will be removed from the cache. The default is 30 253 days. You can append one of the letters `h', `d', `w', `m', 254 `y' to denote hours, days, weeks, months or years, 255 respectively. A value of `0' or `off' disables expiry, so 256 that all entries will stay in the cache forever. See the 257 section `CACHE FILES' below for more information.</para> 258 </listitem> 259 </varlistentry> 260 261 <varlistentry> 262 <term><option>--readbuffer=<replaceable 263 >BYTES</replaceable></option></term> 264 <listitem> 265 <para>Set size of internal buffers. The default is 128k - if 266 you have a fast disc, increasing this value may make 267 <command>jigdo-file</command> faster, but in general, 268 changing it is not necessary.</para> 269 </listitem> 270 </varlistentry> 271 272 <varlistentry> 273 <term><option>--md5-block-size=<replaceable 274 >BYTES</replaceable></option></term> 275 <listitem> 276 <para><emphasis>Uninteresting internal parameter.</emphasis> 277 Set size of blocks into which files are subdivided. The 278 default is 128k. If you change it, any cache file will have 279 to be regenerated. Internally, <command>jigdo-file</command> 280 may choose to use a slightly larger or smaller value.</para> 281 </listitem> 282 </varlistentry> 283 284 <varlistentry> 285 <term><option>-T</option> <option>--files-from=<replaceable 286 >file</replaceable></option></term> 287 <listitem> 288 <para>Read file and directory names from the specified file. 289 If <replaceable>file</replaceable> is `-', read names from 290 standard input. Each line in the file is taken as a name, so 291 the names may contain spaces, but not newline characters. An 292 empty line causes <command>jigdo-file</command> to stop 293 reading from the file.</para> 294 295 <para><citerefentry> 296 <refentrytitle>find</refentrytitle><manvolnum>1</manvolnum> 297 </citerefentry> is a powerful tool for generating file 298 lists, but make sure to use `<command>find -type 299 f</command>' if possible - otherwise, if you instruct 300 <command>find</command> to output both a filename and a 301 symlink to that filename, <command>jigdo-file</command> will 302 read the file contents twice.</para> 303 </listitem> 304 </varlistentry> 305 306 <varlistentry> 307 <term><option>--hex</option></term> 308 <listitem> 309 <para>Output checksums in hexadecimal instead of Base64-like 310 format. This should not be used with the 311 <command>make-template</command> command, because the 312 resulting `<filename>.jigdo</filename>' file violates the 313 `<filename>.jigdo</filename>' file format. Its intended use 314 is to make <command>jigdo-file</command> more interoperable 315 with other Unix shell utilities like <citerefentry> 316 <refentrytitle>md5sum</refentrytitle><manvolnum>1</manvolnum> 317 </citerefentry>.</para> 318 </listitem> 319 </varlistentry> 320 321 <varlistentry> 322 <term><option>--no-hex</option></term> 323 <listitem> 324 <para><emphasis>This is the default.</emphasis> Use jigdo's 325 own Base64-like encoding of checksums.</para> 326 </listitem> 327 </varlistentry> 328 329 <varlistentry> 330 <term><option>--debug</option><optional>=<literal>help</literal 331 >|=<literal>all</literal>|=<replaceable>UNIT,~UNIT...</replaceable> 332 </optional></term> 333 <listitem> 334 <para>Switch on or off debugging output. Just `--debug' is 335 equivalent to `--debug=all'. The argument is a 336 comma-separated list of unit names for which debugging 337 output is to be enabled, or disabled if the name is preceded 338 by `~'. The special name `all' means all units. By default, 339 debugging output is switched off except for the units 340 `assert' and `general'. The exact list of available units 341 for which debugging can be switched on depends on whether 342 jigdo was compiled with debugging support - the list can be 343 printed with `--debug=help'.</para> 344 </listitem> 345 </varlistentry> 346 347 <varlistentry> 348 <term><replaceable>FILES</replaceable></term> 349 <listitem> 350 <para>Names of files or directories to use as input. These 351 are the parts that are contained in the image. In case one 352 of the names is a directory, the program recursively scans 353 the directory and adds all files contained in it. While 354 doing this, it follows symbolic links, but avoids symlink 355 loops.</para> 356 357 <para>If one of the filenames starts with the character `-', 358 you must precede the list of files with `--'. A value of `-' 359 has <emphasis>no</emphasis> special meaning in this list, it 360 stands for a file whose name is a single hyphen.</para> 361 </listitem> 362 </varlistentry> 363 364 </variablelist> 365 366 </refsect1> 367 <!-- ============================================================= --> 368 <refsect1 id="commands"> 369 <title>COMMANDS</title> 370 371 <para>The command name is the first non-option argument passed to 372 <command>jigdo-file</command>. Most commands have short 373 abbreviations as well as long names. <emphasis>The short command 374 names should not be used in scripts - there may be incompatible 375 changes to them in the future!</emphasis></para> 376 377 <refsect2 id="make-template"> 378 <title><command>make-template</command>, <command>mt</command></title> 379 380 <para>Reads <replaceable>image</replaceable> and 381 <replaceable>FILES</replaceable>, creates 382 `<filename>.jigdo</filename>' and 383 `<filename>.template</filename>'. This is the main functionality 384 of <command>jigdo-file</command>.</para> 385 386 <para>It is possible to specify both <option>--image=-</option> 387 and <option>--files-from=-</option>. In this case, first the 388 list of files is read from standard input until an empty line is 389 encountered. Everything following it is assumed to be the image 390 data. This can be useful if you use <citerefentry> 391 <refentrytitle>mkisofs</refentrytitle><manvolnum>1</manvolnum> 392 </citerefentry> or similar programs that can output the complete 393 image on their standard output, because there is no need to 394 store the image on disc temporarily.</para> 395 396 <para>If a <replaceable>FILES</replaceable> argument contains 397 the characters `<literal>//</literal>' (Unix) or 398 `<literal>\.\</literal>' (Windows), this has special meaning. In 399 the final jigdo file that users will download, each of the parts 400 is referenced in the `<literal>[Parts]</literal>' section with a 401 URI of the form `Label:some/filename'. (See `FORMAT OF .JIGDO 402 FILES' below for a detailed description.) The 403 `<literal>[Servers]</literal>' section gives a mapping of labels 404 to servers on the internet, with lines like 405 `Label=http://myserver.org/jigdofiles/'. Using this information, 406 <command>jigdo</command> will create the final download URI for 407 the part, `http://myserver.org/jigdofiles/some/filename'. 408 Specifying `<literal>//</literal>' (or `<literal>\.\</literal>') 409 in a file or directory name serves to `cut off' the names at the 410 right directory level. For example, if the Unix path of one of 411 your <replaceable>FILES</replaceable> is `/path/some/filename', 412 you can tell <command>jigdo-file</command> to cut off after the 413 `/path' by passing it the argument `/path//some/filename', or 414 `/path//' if you want the whole directory scanned. The path 415 names need not be absolute; `somedirectory//' is also 416 possible.</para> 417 418 <variablelist> 419 <varlistentry> 420 <term><option>--label <replaceable 421 >Label=/path</replaceable></option></term> 422 <listitem> 423 <para>Specify a name to use as the label name for a path 424 on disc. (Influences the output jigdo file.) If you used 425 `<literal>//</literal>' in the 426 <replaceable>FILES</replaceable> arguments as described 427 above, <command>jigdo-file</command> will by default pick 428 label names automatically (`A', `B' etc.). With this 429 option, you can give labels more meaningful names. Note 430 that the label name will only be used if one or more 431 <replaceable>FILES</replaceable> begin with 432 `/path//'.</para> 433 434 <para>Try to use label names that start with uppercase 435 characters, to disambiguate them clearly from protocol 436 names like `http', `ftp'.</para> 437 </listitem> 438 </varlistentry> 439 440 <varlistentry> 441 <term><option>--uri <replaceable 442 >Label=http://some.server.org/</replaceable></option></term> 443 <listitem> 444 <para>By default, using <option>--label</option> as 445 described above will cause lines of the form 446 `Label=file:/path/' to be written to the 447 `<literal>[Servers]</literal>' section of the output jigdo 448 file. If you want to override the `file:' URI so that the 449 line reads `Label=http://some.server.org/', you can do so 450 by specifying <option>--uri</option> along with 451 <option>--label</option>. Giving just <option>--uri 452 <replaceable>Label=...</replaceable></option> without the 453 corresponding <option>--label 454 <replaceable>Label=...</replaceable></option> has no 455 effect, and even if you specify both, an entry is only 456 added to the `<literal>[Servers]</literal>' section if the 457 label is referenced by at least one 458 `<literal>[Parts]</literal>' entry.</para> 459 460 <para>The supplied value is not quoted by the program; if 461 it contains characters such as space or any of the 462 characters <literal>#"'\</literal> then you must quote it. 463 (Under Unix, you may need to quote the value twice to also 464 protect it from the shell, e.g. <literal>\\\\</literal> or 465 <literal>'\\'</literal> to get a single backslash in the 466 URI.)</para> 467 468 <para>The mapping specified with an <option>--uri</option> 469 option is ignored if it is already present in the output 470 jigdo file.</para> 471 472 <para>Users of the Windows version may notice that the 473 `<literal>\</literal>' directory separators are converted 474 into `<literal>/</literal>' in the `file:' URIs that are 475 generated by default. This is done to increase 476 cross-platform compatibility of `file:' - the 477 <command>print-missing</command> command of the Windows 478 version will automatically re-convert the characters when 479 it prints the URIs. In case you supply your own `file:' 480 URIs under Windows using <option>--uri</option>, you must 481 also exchange `<literal>/</literal>' and 482 `<literal>\</literal>'.</para> 483 </listitem> 484 </varlistentry> 485 486 <varlistentry> 487 <term><option>-0</option> to <option>-9</option></term> 488 <listitem> 489 <para>Set amount of compression in the output template 490 file, from <option>-0</option> (no compression) to 491 <option>-9</option> (maximum compression). The default is 492 <option>-9</option>, which can make the template 493 generation quite slow. By default, the compression 494 algorithm used is the same as for <citerefentry> 495 <refentrytitle>gzip</refentrytitle><manvolnum>1</manvolnum> 496 </citerefentry>.</para> 497 </listitem> 498 </varlistentry> 499 500 <varlistentry> 501 <term><option>--gzip</option> and <option>--bzip2</option></term> 502 <listitem> 503 <para>Choose between the gzip and bzip2 compression 504 algorithms. The default is gzip. Bzip2 usually gives 505 a better compression ratio, but compression is 506 significantly slower than with gzip.</para> 507 </listitem> 508 </varlistentry> 509 510 <varlistentry> 511 <term><option>--min-length=<replaceable 512 >BYTES</replaceable></option></term> 513 <listitem> 514 <para>Set minimum length of a part for 515 <command>jigdo-file</command> to look for it in the image. 516 The default is 1k. Parts smaller than this will never be 517 found in the image, so their data will be included in the 518 template file. The search algorithm used requires such a 519 minimum length, otherwise template generation could become 520 extremely slow. If you know for sure that all your 521 <replaceable>FILES</replaceable> are larger than a certain 522 amount, you can increase <command>jigdo-file</command>'s 523 speed slightly by specifying the amount with this option. 524 There is a hard-wired absolute minimum of 256 bytes - 525 anything lower will silently be set to 256.</para> 526 </listitem> 527 </varlistentry> 528 529 <varlistentry> 530 <term><option>--merge=<replaceable>FILE</replaceable 531 ></option></term> 532 <listitem> 533 <para>Include the contents of 534 <replaceable>FILE</replaceable> in the output 535 `<filename>.jigdo</filename>' file. The file can contain 536 data which you want added to the output (for example, a 537 `<literal>[Servers]</literal>' section with a list of your 538 servers as entries), or it can be the jigdo file output by 539 an earlier run of <command>jigdo-file</command>.</para> 540 541 <para>It is possible to specify the same file for input 542 with <option>--merge</option> and for output with 543 <option>--jigdo</option>. However, you will also need to 544 use <option>--force</option> to make the program overwrite 545 the old version of the jigdo file with the new one. 546 <replaceable>FILE</replaceable> can be `-' for standard 547 input.</para> 548 549 <para>When <emphasis>adding</emphasis> new information to 550 the supplied file, <command>jigdo-file</command> will not 551 insert new lines into the `<literal>[Parts]</literal>' 552 section if an entry for the same MD5 checksum (but not 553 necessarily with the same URI!) already exists, and it 554 will not insert new lines into the 555 `<literal>[Servers]</literal>' section if a completely 556 identical entry already exists.</para> 557 558 <para>When <emphasis>reading in</emphasis> the existing 559 <replaceable>FILE</replaceable>, the behaviour is slightly 560 different: The program <emphasis>preserves</emphasis> 561 entries in the `<literal>[Parts]</literal>' section with 562 identical checksum, but different URIs. For completely 563 identical entries (same checksum and URI), only one entry 564 is preserved and the duplicates are removed. The 565 `<literal>[Servers]</literal>' section is left 566 untouched.</para> 567 </listitem> 568 </varlistentry> 569 570 <varlistentry> 571 <term><option>--image-section</option></term> 572 <listitem> 573 <para><emphasis>This is the default.</emphasis> Causes 574 <command>jigdo-file</command> to add an 575 `<literal>[Image]</literal>' section to the 576 `<filename>.jigdo</filename>' file.</para> 577 578 <para>As an exception, a new `<literal>[Image]</literal>' 579 section is <emphasis>not</emphasis> added if you use 580 <option>--merge</option> and the file to merge contains an 581 `<literal>[Image]</literal>' section with a line which 582 reads `<literal>Template-MD5Sum=</literal>' (end of line 583 after the `='). In this case, the generated template 584 data's MD5 checksum value is just added after the `=' of 585 the first line of this form in the file - no whole new 586 `<literal>[Image]</literal>' section is appended. This 587 behaviour is useful because it allows you to pass via 588 <option>--merge</option> an `<literal>[Image]</literal>' 589 section with arbitrary content and then have the MD5 590 checksum automatically added by 591 <command>jigdo-file</command>. The section `FORMAT OF 592 .JIGDO FILES' below explains the 593 `<literal>[Image]</literal>' section contents in greater 594 detail.</para> 595 </listitem> 596 </varlistentry> 597 598 <varlistentry> 599 <term><option>--no-image-section</option></term> 600 <listitem> 601 <para>Do <emphasis>not</emphasis> include an 602 `<literal>[Image]</literal>' section in the 603 `<filename>.jigdo</filename>' file. You need to add one 604 yourself if you use this option. However, doing that is 605 not easy (you also need to add a 606 `<literal>Template-MD5Sum</literal>' line with the correct 607 checksum, or <command>jigdo</command> will complain), so 608 use of this option is discouraged.</para> 609 </listitem> 610 </varlistentry> 611 612 <varlistentry> 613 <term><option>--servers-section</option></term> 614 <listitem> 615 <para><emphasis>This is the default.</emphasis> Causes 616 <command>jigdo-file</command> to add a 617 `<literal>[Servers]</literal>' section to the 618 `<filename>.jigdo</filename>' file. This default section 619 uses `file:' URIs, which allows for immediate reassembly 620 of the image from the local filesystem, and is also useful 621 if you want to edit the file manually and replace the 622 `file:' URIs with other URIs.</para> 623 </listitem> 624 </varlistentry> 625 626 <varlistentry> 627 <term><option>--no-servers-section</option></term> 628 <listitem> 629 <para>Do <emphasis>not</emphasis> add a 630 `<literal>[Servers]</literal>' section at the end of the 631 `<filename>.jigdo</filename>' file. Useful e.g. if you are 632 going to append the section with a script.</para> 633 </listitem> 634 </varlistentry> 635 636 <varlistentry> 637 <term><option>--match-exec=<replaceable>SHELLCOMMAND</replaceable 638 ></option></term> 639 <listitem> 640 <para>Whenever a file is found in the image, execute the 641 supplied command string by passing it to a shell. 642 <command>jigdo-file</command> sets up a number of 643 environment variables with information about the file 644 match. For example, if the file 645 `<filename>/path//a/b/file</filename>' was found in the 646 image and `Label:a/b/file' is going to be written to the 647 `<filename>.jigdo</filename>' file: 648 649 <itemizedlist> 650 <listitem><para><envar>LABEL</envar>="<literal>Label</literal 651 >" - Name of the label for the file. The example assumes 652 that `<option>--label</option> 653 <literal>Label=/path</literal>' was specified by you. 654 In the absence of such an option, <envar>LABEL</envar> 655 will be set but empty.</para></listitem> 656 657 <listitem><para><envar>LABELPATH</envar>="<literal 658 >/path/</literal>" - The path corresponding to the label, 659 or in other words, the prefix of the matched file's 660 path that will <emphasis>not</emphasis> appear in the 661 output `<filename>.jigdo</filename>' file. Is set even 662 without any `<option>--label</option>' option present. 663 Ends with a slash.</para></listitem> 664 665 <listitem><para><envar>MATCHPATH</envar>="<literal 666 >a/b/</literal>" - The rest of the path, without the 667 leafname of the matched file. Is either empty or ends 668 with a slash.</para></listitem> 669 670 <listitem><para><envar>LEAF</envar>="<literal>file</literal>" 671 - The leafname of the matched file.</para></listitem> 672 673 <listitem><para><envar>MD5SUM</envar>="<literal 674 >lNVdUSqbo2yqm33webrhnw</literal>" - The md5sum of the 675 matched file, in Base64-like format.</para></listitem> 676 677 <listitem><para><envar>FILE</envar>="<literal 678 >/path//a/b/file</literal>" - For convenience, the 679 complete path of the file. The variable is always set 680 to <literal>$LABELPATH$MATCHPATH$LEAF</literal 681 >.</para></listitem> 682 683 </itemizedlist> 684 </para> 685 686 <para>Please be careful to correctly quote the string 687 passed to this option, otherwise your supplied command 688 will not work with filenames that contain spaces. As an 689 example, to create a backup of hard links to the matched 690 files, use the following option: 691 <literal>--match-exec='mkdir -p "${LABEL:-.}/$MATCHPATH" 692 && ln -f "$FILE" "${LABEL:-.}/$MATCHPATH$LEAF"'</literal> 693 </para> 694 695 <para>By default, no command is executed. Use 696 --match-exec="" to remove a command string which was set 697 with an earlier use of this option.</para> 698 </listitem> 699 </varlistentry> 700 701 <varlistentry> 702 <term><option>--greedy-matching</option></term> 703 <listitem> 704 <para><emphasis>This is the default.</emphasis> Imagine 705 that your image contains a <filename>.tar</filename> file 706 which in turn contains another file 707 <filename>x</filename>, and that you provide both the 708 <filename>.tar</filename> and the files inside it on the 709 command line. When <command>jigdo-file</command> scans the 710 image, it encounters the beginning of the 711 <filename>.tar</filename> file, and then the file 712 <filename>x</filename>.</para> 713 714 <para>At this point, a decision must be made: Should the 715 smaller file <filename>x</filename> be recorded as 716 matched, or should it be ignored in favour of the larger 717 (and thus better) match of the <filename>.tar</filename> 718 file? Unfortunately, at this point it is not clear 719 whether there will actually be a full match of the 720 <filename>.tar</filename>, so by default, the program 721 prefers the small match.</para> 722 </listitem> 723 </varlistentry> 724 725 <varlistentry> 726 <term><option>--no-greedy-matching</option></term> 727 <listitem> 728 <para>In the case where a large partial match is present 729 and a shorter match has been confirmed, ignore the small 730 match. (See the option above.)</para> 731 </listitem> 732 </varlistentry> 733 734 </variablelist> 735 736 </refsect2> 737 <!-- ========================================= --> 738 <refsect2 id="make-image"> 739 <title><command>make-image</command>, <command>mi</command></title> 740 741 <para>Reads `<filename>.template</filename>' and 742 <replaceable>FILES</replaceable>, creates 743 <replaceable>image</replaceable> (or 744 `<filename>imagename.tmp</filename>'). Provides a rudimentary 745 way of reassembling images - <command>jigdo</command> is usually 746 better suited for this task. However, in contrast to 747 <command>jigdo</command>, no `<filename>.jigdo</filename>' file 748 is required.</para> 749 750 <para>If the image is to be written to a file (and not to 751 standard output), it is possible to create the image in several 752 steps, with several invocations of `<command>jigdo-file 753 make-image</command>', as follows: You first invoke 754 <command>jigdo-file</command>, specifying as many files as are 755 available at this time. The program scans the files, and those 756 that are contained in the image are copied to a temporary file, 757 whose name is formed by appending `<filename>.tmp</filename>' to 758 the image filename.</para> 759 760 <para>For all further files which could be parts of the image, 761 you repeat this process. As soon as all parts are present, the 762 temporary file will be truncated slightly (to delete some 763 administrative data that <command>jigdo-file</command> appends 764 at the end) and renamed to the final image name. The possibility 765 of reassembling the image in several steps is especially useful 766 for gathering files from removable media, e.g. several older 767 CDs.</para> 768 769 <para>Scripts using <command>make-image</command> can detect 770 whether image creation is complete by checking the exit status: 771 0 signals successful creation, whereas 1 means that more files 772 need to be supplied. Other errors result in an exit status of 2 773 (`recoverable', e.g. file not found) or 3 (non-recoverable, e.g. 774 write error).</para> 775 776 <variablelist> 777 <varlistentry> 778 <term><option>--check-files</option></term> 779 <listitem> 780 <para><emphasis>This is the default.</emphasis> Whenever 781 any part is copied to the image, re-check its checksum 782 against the checksum stored in the template. It is 783 recommended that you leave this switched on, even if it 784 slows down image creation a bit.</para> 785 </listitem> 786 </varlistentry> 787 788 <varlistentry> 789 <term><option>--no-check-files</option></term> 790 <listitem> 791 <para>Do not check files' checksums when copying them to 792 the image. This can be safely used when no cache file is 793 used (which means that files will be written to the image 794 immediately after being scanned) or the whole image is 795 checked later with the <command>verify</command> 796 command.</para> 797 </listitem> 798 </varlistentry> 799 </variablelist> 800 801 </refsect2> 802 <!-- ========================================= --> 803 <refsect2 id="print-missing"> 804 <title><command>print-missing</command>, <command>pm</command></title> 805 806 <para>Reads `<filename>.jigdo</filename>', 807 `<filename>.template</filename>' and (if present) 808 `<filename>imagename.tmp</filename>', outputs a list of URIs 809 still needed to completely reassemble the image.</para> 810 811 <para>Together with the <command>make-image</command> command, 812 this provides most of the functionality of 813 <command>jigdo</command> on the command line.</para> 814 815 <para>For each part that is not yet present in the temporary 816 image file, the file checksum is looked up in the 817 `<literal>[Parts]</literal>' section of the jigdo file. Any 818 label in the corresponding entry is then expanded according to 819 the label definitions in the `<literal>[Servers]</literal>' 820 section and printed on standard output. <command>jigdo</command> 821 allows you to specify several alternative locations for each 822 label in this section, but <command>print-missing</command> will 823 only output the first one for each missing part.</para> 824 825 <para>If the checksum cannot be found in the 826 `<literal>[Parts]</literal>' section (this Should Not Happen 827 unless you deleted that section), a lookup is instead made for 828 `MD5Sum:<replaceable><checksum></replaceable>', just like 829 with <command>jigdo</command>. (Thus, if you want to get rid of 830 the `<literal>[Parts]</literal>' section, you can do so if you 831 rename each part to its own checksum.)</para> 832 833 <variablelist> 834 <varlistentry> 835 <term><option>--uri <replaceable 836 >Label=http://some.server.org/</replaceable></option></term> 837 <listitem> 838 <para>Override the entries in the 839 `<filename>.jigdo</filename>' file for any label with a 840 URI of your choice. With the example above, a 841 `<literal>[Parts]</literal>' entry of 842 `Label:some/filename' will cause the line 843 `http://some.server.org/some/filename' to be 844 printed.</para> 845 846 <para>The supplied value is not quoted by the program; if 847 it contains characters such as space or any of the 848 characters <literal>#"'\</literal> then you must quote it. 849 (Under Unix, you may need to quote the value twice to also 850 protect it from the shell, e.g. <literal>\\\\</literal> or 851 <literal>'\\'</literal> to get a single backslash in the 852 URI.)</para> 853 </listitem> 854 </varlistentry> 855 </variablelist> 856 857 </refsect2> 858 <!-- ========================================= --> 859 <refsect2 id="print-missing-all"> 860 <title><command>print-missing-all</command>, 861 <command>pma</command></title> 862 863 <para>Just like <command>print-missing</command>, this command 864 outputs a list of URIs still needed to completely reassemble the 865 image. However, <emphasis>all</emphasis> alternative download 866 locations are printed instead of just one. In the output, the 867 URIs for a file are separated from other files' URIs with blank 868 lines. The <option>--uri</option> option has the same effect as 869 for <command>print-missing</command>.</para> 870 871 </refsect2> 872 <!-- ========================================= --> 873 <refsect2 id="verify"> 874 <title><command>verify</command>, <command>ver</command></title> 875 876 <para>Reads <replaceable>image</replaceable> (presumably 877 generated with <command>make-image</command>) and 878 `<filename>.template</filename>', checks for correct checksum of 879 image.</para> 880 881 <para>The template data does not only contain checksums of the 882 individual parts, but also of the image as a whole. 883 <command>make-image</command> already performs a number of 884 internal checks, but if you like, you can additionally check the 885 image with this command.</para> 886 887 </refsect2> 888 <!-- ========================================= --> 889 <refsect2 id="scan"> 890 <title><command>scan</command>, <command>sc</command></title> 891 892 <para>Reads all the <replaceable>FILES</replaceable> and enters 893 them into the cache, unless they are already cached. The 894 <option>--cache</option> option must be present for this 895 command.</para> 896 897 <variablelist> 898 <varlistentry> 899 <term><option>--no-scan-whole-file</option></term> 900 <listitem> 901 902 <para><emphasis>This is the default.</emphasis> This only 903 causes the first <option>--md5-block-size</option> bytes 904 of each file to be read. If the cache is used later by 905 <command>jigdo-file make-image</command>, the rest of the 906 file will be read once these first bytes are recognized in 907 the input image.</para> 908 </listitem> 909 </varlistentry> 910 911 <varlistentry> 912 <term><option>--scan-whole-file</option></term> 913 <listitem> 914 <para>Immediately read the entire file contents and store 915 them in the cache.</para> 916 </listitem> 917 </varlistentry> 918 </variablelist> 919 920 </refsect2> 921 <!-- ========================================= --> 922 <refsect2 id="md5sum"> 923 <title><command>md5sum</command>, <command>md5</command></title> 924 925 <para>Reads all the <replaceable>FILES</replaceable> and prints 926 out MD5 checksums of their contents. This command is quite 927 similar to <citerefentry> 928 <refentrytitle>md5sum</refentrytitle><manvolnum>1</manvolnum> 929 </citerefentry>, except that the checksum is output in the 930 Base64-like encoding which is also used elsewhere by 931 <command>jigdo-file</command>.</para> 932 933 <para>The <replaceable>FILES</replaceable> arguments are 934 processed in the same way as with the other commands, which 935 means that recursion automatically takes place for any arguments 936 that are directories, and that symbolic links are not listed 937 except when the file(s) they point to are not reachable 938 directly.</para> 939 940 <para>In the checksum list printed on standard output, only the 941 part of the filename following any `<literal>//</literal>' (or 942 `<literal>\.\</literal>' on Windows) is printed. Any 943 <option>--cache</option> will be used for querying files' MD5 944 checksums and/or writing the checksums of scanned files.</para> 945 946 </refsect2> 947 <!-- ========================================= --> 948 <refsect2 id="list-template"> 949 <title><command>list-template</command>, <command>ls</command></title> 950 951 <para>Reads a `<filename>.template</filename>' file and outputs 952 low-level information about the image and all parts contained in 953 it, including offset, length and checksum.</para> 954 955 <para>You can also use this command with temporary image files 956 (by specifying something like 957 <option>--template=imagename.tmp</option>) - in that case, the 958 output also distinguishes between parts that have been written 959 to the image and parts that haven't.</para> 960 961 <para>The exact output format may change incompatibly between 962 different jigdo releases. The following different types of lines 963 can be output. `have-file' only occurs for 964 `<filename>.tmp</filename>' files, indicating a file that has 965 already been successfully written to the temporary file:</para> 966 967 <screen 968>in-template <replaceable>offset-in-image length</replaceable> 969need-file <replaceable>offset-in-image length file-md5sum filestart-rsyncsum</replaceable> 970have-file <replaceable>offset-in-image length file-md5sum filestart-rsyncsum</replaceable> 971image-info <replaceable>image-length image-md5sum rsyncsum-size</replaceable> 972</screen> 973 974 </refsect2> 975 976 </refsect1> 977 <!-- ============================================================= --> 978 <refsect1 id="details"> 979 <title>DETAILS</title> 980 981 <para>Jigsaw Download was created with the format of ISO9660 CD 982 images in mind - however, the following also applies to many other 983 filesystem formats, as well as to `tar' archives and uncompressed 984 `zip' archives. A CD image contains both information for 985 organizing the filesystem (header with disc name etc., ISO9660 986 directory data, data of extensions such as Joliet or RockRidge, 987 zero padding) and the files contained on the CD. An important 988 property that jigdo relies on is that each file is stored in one 989 contiguous section of the image; it is not split into two or more 990 parts.</para> 991 992 <para>When <command>jigdo-file</command> is given a number of 993 files that might be contained in an image, it detects whether any 994 of the files are present using a `rolling checksum' inspired by 995 the one used by <citerefentry><refentrytitle>rsync</refentrytitle 996 ><manvolnum>1</manvolnum></citerefentry>. The resulting data is 997 written to the `<filename>.template</filename>' file: If a section 998 of the image could not be matched (e.g. it was directory 999 information), the data is compressed and written directly to the 1000 template. However, if a matching file was found, its data is 1001 omitted from the template. Instead, only a reference (an MD5 1002 checksum of the file) is inserted in the template.</para> 1003 1004 <para>Note that the template data only contains binary data, it 1005 does not contain any filenames or URIs, since it cannot be easily 1006 edited in case any of these values need to be changed. All that 1007 information is stored in the `<filename>.jigdo</filename>' file, a 1008 text file to which you can add URLs for your server(s). The jigdo 1009 file provides a mapping for each MD5 checksum to one or more 1010 alternative download locations for the corresponding part.</para> 1011 1012 <para>Apart from the mapping of MD5 sums to URIs, the jigdo file 1013 also contains an URI pointing to a download location for the 1014 template file. This way, the <command>jigdo</command> download 1015 tool only needs to be given one URI (that of the 1016 `<filename>.jigdo</filename>' file) to be able to download and 1017 reassemble the complete image.</para> 1018 1019 </refsect1> 1020 <!-- ============================================================= --> 1021 <refsect1 id="jigdo-format"> 1022 <title>FORMAT OF .JIGDO FILES</title> 1023 1024 <para>The overall format of `<filename>.jigdo</filename>' files 1025 follows that of `<filename>.ini</filename>' files, as also used by 1026 the Gnome and KDE projects for some data. The file is organized 1027 into sections, each of which is preceded by a line reading 1028 `<literal>[Sectionname]</literal>'. Within each section, lines 1029 have the form `Label=Value'. Such lines are also called `entries' 1030 below. All `<filename>.jigdo</filename>' files use UTF-8 as their 1031 character encoding.</para> 1032 1033 <para>Comments are introduced with the `<literal>#</literal>' 1034 character and extend to the end of the line. Whitespace is ignored 1035 at line start and end as well as to the left and right of section 1036 names and the `<literal>=</literal>' in entries. Furthermore, the 1037 jigdo utilities split up the text of the entry value (i.e. the 1038 part after the `<literal>=</literal>') into whitespace-separated 1039 words, much like the Unix shell. Single <literal>''</literal> and 1040 double <literal>""</literal> quotes can be used to prevent that 1041 e.g. URIs containing whitespace are split apart. Similarly, 1042 characters with special meaning (the characters 1043 <literal>'"#\</literal> and space/tab) must be quoted with 1044 <literal>\</literal> to appear in the value. As with the shell, 1045 there is a difference between <literal>' '</literal> and 1046 <literal>" "</literal>: Within <literal>' '</literal>, 1047 the characters <literal>"#\</literal> and whitespace lose their 1048 special meaning and become ordinary characters, whereas within 1049 <literal>" "</literal>, only the characters 1050 <literal>'#</literal> and whitespace lose their special meaning - 1051 in other words, backslash escapes still work inside 1052 <literal>" "</literal>, but not 1053 <literal>' '</literal>.</para> 1054 1055 <para>`<filename>.jigdo</filename>' files can optionally be 1056 compressed with <citerefentry> 1057 <refentrytitle>gzip</refentrytitle><manvolnum>1</manvolnum> 1058 </citerefentry>. <command>jigdo-file</command> always outputs 1059 uncompressed files, which you can compress yourself. 1060 <command>jigdo-lite</command> supports single uncompressed and 1061 compressed files.</para> 1062 1063 <para>(Behaviour which may change in the future and which should 1064 not be relied upon: <command>jigdo</command> additionally supports 1065 any number of concatenated plaintext and gzipped parts in the 1066 files - for example, you can compress a 1067 `<filename>.jigdo</filename>' file and then add a couple of lines 1068 of uncompressed data to the end.)</para> 1069 1070 <para>In all cases, the `<filename>.gz</filename>' extension 1071 should be removed from the filename - the tools will determine 1072 automatically from the file contents whether a file is compressed 1073 or not.</para> 1074 1075 <para>Below is a description of the individual section names used 1076 by jigdo.</para> 1077 1078 <refsect2 id="jigdo-section"> 1079 <title>Jigdo section</title> 1080 1081 <screen 1082>[Jigdo] 1083Version=1.1 1084Generator=jigdo-file/1.0.0</screen> 1085 1086 <para>Information about the version of the jigdo file format 1087 used, and the program that generated it. There should be one 1088 such section per `<filename>.jigdo</filename>' file.</para> 1089 </refsect2> 1090 1091 <refsect2 id="image-section"> 1092 <title>Image section</title> 1093 1094 <screen 1095>[Image] 1096Filename=<replaceable>"filename for saving on user's disc"</replaceable> 1097Template=<replaceable>"URI where to fetch template file"</replaceable> 1098Template-MD5Sum=OQ8riqT1BuyzsrT9964A7g 1099ShortInfo=<replaceable>single-line description of the image (200 characters max.)</replaceable> 1100Info=<replaceable>long description (5000 characters max.)</replaceable></screen> 1101 1102 <para>The value for the `Template' entry can be either an URL 1103 (absolute or relative to the URL of the jigdo file) or a string 1104 of the form `<replaceable>Label</replaceable>:<replaceable 1105 >pathname</replaceable>' (<emphasis>UNIMPLEMENTED</emphasis>), 1106 as described below.</para> 1107 1108 <para>The `Template-MD5Sum' entry is added by 1109 <command>jigdo-file</command> and specifies the MD5 checksum of 1110 the generated `<filename>.template</filename>' file. It is used 1111 by <command>jigdo</command> to detect cases where the downloaded 1112 template data is corrupted or belongs to a different 1113 image.</para> 1114 1115 <para>Unlike other entry values, the values of the 1116 `<literal>ShortInfo</literal>' and `<literal>Info</literal>' 1117 entries are <emphasis>not</emphasis> split up into words, 1118 instead all quoting is preserved.</para> 1119 1120 <para>The value of the `<literal>Info</literal>' entry is 1121 special in that <citerefentry> 1122 <refentrytitle>jigdo</refentrytitle><manvolnum>1</manvolnum> 1123 </citerefentry> can optionally parse XML markup it contains. If 1124 the markup has errors such as unbalanced/unsupported tags, the 1125 string is displayed literally, without XML parsing. Supported 1126 tags are <literal><b></b></literal> (bold), 1127 <literal><i></i></literal> (italic), 1128 <literal><tt></tt></literal> (typewriter font), 1129 <literal><u></u></literal> (underline), 1130 <literal><big></big></literal> (larger font), 1131 <literal><small></small></literal> (smaller font) 1132 and <literal><br/></literal> (linebreak). Supported 1133 entities include <literal>&lt;</literal> (`<'), 1134 <literal>&gt;</literal> (`>') and 1135 <literal>&amp;</literal> (`&'). Note that the whole 1136 `<literal>Info</literal>' entry must be on one line in the jigdo 1137 file.</para> 1138 1139 <para>This section may occur multiple times, but all except the 1140 first one will be ignored. This is useful e.g. when creating a 1141 `<filename>.jigdo</filename>' file for a DVD image when you 1142 already have `<filename>.jigdo</filename>' files for CDs with 1143 the same content: You can simply `<literal>[Include]</literal>' 1144 (see below) the CDs' jigdo files at the end of the DVD jigdo 1145 file, after its `<literal>[Image]</literal>' section.</para> 1146 </refsect2> 1147 1148 <refsect2 id="parts-section"> 1149 <title>Parts section</title> 1150 1151 <screen 1152>[Parts] 1153xJNkjrq8NYMraeGavUpllw=LabelA:part0 1154GoTResP2EC6Lb_2wTsqOoQ=LabelA:part1 1155kyfebwu6clbYqqWUdFIyaw=LabelB:some/path/part2 1156-J9UAimo0Bqg9c0oOXI1mQ=http://some.where.com/part3</screen> 1157 1158 <para>All lines in the section, which provides the mapping from 1159 MD5 checksums to URIs, have the same format: On the left side of 1160 the `<literal>=</literal>' the checksum (encoded with a 1161 Base64-like encoding) is given, and on the right a string 1162 corresponding to the part with this checksum; either a complete 1163 URI or a string of the form `<replaceable>Label</replaceable 1164 >:<replaceable>pathname</replaceable>', which is expanded into 1165 one or more URIs by looking up the definition(s) for the 1166 <replaceable>Label</replaceable> in the 1167 `<literal>[Servers]</literal>' section.</para> 1168 1169 <para>In case a particular MD5 checksum cannot be found in any 1170 `<literal>[Parts]</literal>' section by 1171 <command>jigdo</command>, the program will perform a lookup for 1172 `<literal>MD5Sum:</literal><replaceable 1173 ><checksum></replaceable>', e.g. for 1174 `<literal>MD5Sum:xJNkjrq8NYMraeGavUpllw</literal>' if you 1175 deleted the line for `part0' above.</para> 1176 1177 <para>A checksum appearing multiple times in this section 1178 indicates alternative download locations for the part.</para> 1179 1180 <para>There may be any number of `<literal>[Parts]</literal>' 1181 sections in the file; they are all considered when looking up 1182 MD5 checksums.</para> 1183 1184 <para><command>jigdo-file</command> always puts the 1185 `<literal>[Parts]</literal>' section at the end of the file, and 1186 it even rearranges any file specified with 1187 <option>--merge</option> to have only one such section at the 1188 end. This is done to allow <command>jigdo</command> to display 1189 the information from the `<literal>[Image]</literal>' section 1190 while the rest of that file is still being downloaded.</para> 1191 </refsect2> 1192 1193 <refsect2 id="servers-section"> 1194 <title>Servers section</title> 1195 1196 <screen 1197>[Servers] 1198LabelA=http://myserver.org/ 1199LabelA=ftp://mirror.myserver.org/ 1200LabelB=LabelC:subdirectory/ 1201LabelC=http://some.where.com/jigdo/</screen> 1202 1203 <para>All lines in the section, which provides the mapping from 1204 server labels to server locations, have the same format: On the 1205 left side of the `<literal>=</literal>' the label name is given, 1206 and on the right the value to expand the label name to.</para> 1207 1208 <para>A label name appearing multiple times in this section 1209 indicates alternative download locations for the parts that use 1210 the label in the `<literal>[Parts]</literal>' section. This 1211 notation makes it very easy to add mirrors to the jigdo 1212 file.</para> 1213 1214 <para>As shown by the example above, the label values may 1215 themselves reference other labels. In this case, the entry 1216 `LabelB:some/path/part2' in the `<literal>[Parts]</literal>' 1217 section will expand to 1218 `http://some.where.com/jigdo/subdirectory/some/path/part2'. 1219 Loops in the label definitions result in undefined behaviour and 1220 must be avoided.</para> 1221 1222 <para>There may be any number of `<literal>[Servers]</literal>' 1223 sections in the file; they are all considered when looking up 1224 labels. Either of `<literal>[Parts]</literal>' or 1225 `<literal>[Servers]</literal>', but not both, can be omitted 1226 from the jigdo file.</para> 1227 </refsect2> 1228 1229 <refsect2 id="comment-section"> 1230 <title>Comment section</title> 1231 1232 <screen 1233>[Comment] 1234Any text, except that lines must not begin with `['.</screen> 1235 1236 <para>All text following a `<literal>[Comment]</literal>' or 1237 `<literal>[comment]</literal>' line is ignored, up to the next 1238 line with a section label.</para> 1239 </refsect2> 1240 1241 <refsect2 id="include"> 1242 <title>Include directive</title> 1243 1244 <screen>[Include http://some.url/file.jigdo]</screen> 1245 1246 <para>Lines of this form cause the content of the specified 1247 jigdo file to be downloaded and parsed just like the main jigdo 1248 file. The effect will be the same as copying the included file's 1249 contents into the file which contains the include 1250 directive. (Exception: Any relative URLs are always resolved 1251 using the URL of the `<filename>.jigdo</filename>' file that 1252 contains that relative URL.)</para> 1253 1254 <para>The URL argument can be an absolute or relative URL. 1255 Relative URLs are assumed to be relative to the URL of the jigdo 1256 file which contains the include directive. Includes can be 1257 nested, but it is an error to create a loop of include 1258 directives. It is <emphasis>not</emphasis> possible to use URLs 1259 of the form `<replaceable>Label</replaceable>:<replaceable 1260 >pathname</replaceable>'.</para> 1261 1262 <para>The URL cannot be quoted with "". Any 1263 `<literal>]</literal>' characters in the argument must be 1264 escaped as `<literal>%5D</literal>', and any spaces as 1265 `<literal>%20</literal>'.</para> 1266 1267 <para>Include directives are only supported by 1268 <command>jigdo</command>, they are ignored by 1269 <command>jigdo-lite</command>.</para> 1270 1271 <para>An include directive terminates any previous section, but 1272 it does not start a new one. In other words, a new section must 1273 always be started after the include line, 1274 <command>jigdo</command> does not allow normal entries to appear 1275 below the `<literal>[Include]</literal>'.</para> 1276 </refsect2> 1277 1278 </refsect1> 1279 <!-- ============================================================= --> 1280 <refsect1 id="cache-files"> 1281 <title>CACHE FILES</title> 1282 1283 <para>Any file specified with the <option>--cache</option> option 1284 is used to store information about the 1285 <replaceable>FILES</replaceable> presented to 1286 <command>jigdo-file</command>. When querying the cache, a file is 1287 considered unchanged (and the cached data is used) only if 1288 filename, file size and last modification time (mtime) match 1289 exactly. For the filename match, not the entire file name is used, 1290 but only the part following any `<literal>//</literal>', so that 1291 any changes to the part before the `<literal>//</literal>' will 1292 not invalidate the cache.</para> 1293 1294 <para>Old cache entries are removed from the cache if they have 1295 not been read from or written to for the amount of time specified 1296 with <option>--cache-expiry</option>. Entries are 1297 <emphasis>not</emphasis> immediately removed from the cache if the 1298 file they refer to no longer exists - this makes it possible to 1299 cache information about files on removable media.</para> 1300 1301 <para>Cache expiry only takes place <emphasis>after</emphasis> 1302 <command>jigdo-file</command> has done its main work - if any old 1303 entries are accessed before expiry takes place, they will be kept. 1304 For example, if the program is run using the default expiry time 1305 of 30 days, but accesses a cache file with entries generated 2 1306 months ago, then entries in that cache <emphasis>will</emphasis> 1307 be considered, and only those cache entries that were not needed 1308 during the program run will be expired.</para> 1309 1310 <para>Due to a peculiarity of the underlying database library 1311 (libdb3), cache files never shrink, they only grow. If a large 1312 number of entries was expired from your cache file and you want it 1313 to shrink, you can either just delete it (of course then 1314 everything will have to be regenerated) or use the utilities 1315 accompanying libdb3 to dump and restore the database, with a 1316 command like `<command>db3_dump 1317 <replaceable>old-cache.db</replaceable> | db3_load 1318 <replaceable>new-cache.db</replaceable></command>'. <phrase 1319 vendor="Debian">For Debian, these programs are supplied in the 1320 package `libdb3-util'.</phrase></para> 1321 1322 <para>If a different <option>--md5-block-size</option> is 1323 specified, the entire file needs to be re-read to update its cache 1324 entry. If a different <option>--min-length</option> is specified, 1325 only the first `md5-block-size' bytes of the file need to be 1326 re-read.</para> 1327 1328 </refsect1> 1329 <!-- ============================================================= --> 1330 <refsect1 id="examples"> 1331 <title>EXAMPLES</title> 1332 1333 <refsect2 id="ex-cdimage"> 1334 <title>Preparing your CD image for distribution</title> 1335 1336 <para>You have created a CD image 1337 `<filename>image.iso</filename>' from some of the files stored 1338 in the directory `<filename>/home/ftp</filename>' on your 1339 harddisc, which is also available online as `ftp://mysite.org'. 1340 As you don't want to waste space by effectively hosting the same 1341 data twice (once as files on the FTP server, once inside the 1342 image), and you are fed up with users' downloads aborting after 1343 200MB and their restarting the download dozens of times, you 1344 decide to use jigdo. How do you prepare the image for 1345 download?</para> 1346 1347 <para>In fact, only one command is necessary: 1348 1349 <blockquote><para><command>jigdo-file make-template 1350 --image=image.iso --jigdo=/home/ftp/image.jigdo 1351 --template=/home/ftp/image.template /home/ftp// --label 1352 Mysite=/home/ftp --uri 1353 Mysite=ftp://mysite.org/</command></para></blockquote> 1354 </para> 1355 1356 <para>People can now point <command>jigdo</command> at 1357 `ftp://mysite.org/image.jigdo' to download your image. The 1358 template file needs to be accessible as 1359 `ftp://mysite.org/image.template'.</para> 1360 1361 <para>Note that nothing prevents you from doing the same for an 1362 FTP server that isn't administrated by you - in that case, you 1363 only need to host the `<filename>.jigdo</filename>' and 1364 `<filename>.template</filename>' files on your own 1365 server/homepage.</para> 1366 </refsect2> 1367 1368 <refsect2 id="ex-largefile"> 1369 <title>Preparing an arbitrary large file for 1370 distribution</title> 1371 1372 <para>We assume that you have a large file that is not a 1373 filesystem, e.g. `<filename>movie.mpeg</filename>'. Because of 1374 space problems, you want to distribute the data on two 1375 servers.</para> 1376 1377 <para>In this case, the parts of the image need to be generated 1378 artificially with the <command>split</command> command. For 1379 example, to create chunks of 4MB each, use `<command>split -b 4m 1380 movie.mpeg part</command>'. Copy the resulting files 1381 `<filename>part<replaceable>XX</replaceable></filename>' into 1382 two directories `<filename>1</filename>' and 1383 `<filename>2</filename>' that you create, according to how you 1384 want the files distributed between the servers. Next, create the 1385 jigdo and template files with `<command>jigdo-file make-template 1386 --image=movie.mpeg 1// 2//</command>'. You will need to edit the 1387 `<filename>.jigdo</filename>' file and provide the right URIs 1388 for the two servers that you are going to upload the 1389 `<filename>part<replaceable>XX</replaceable></filename>' files 1390 to.</para> 1391 </refsect2> 1392 1393 <refsect2 id="ex-customize"> 1394 <title>Customized versions of images</title> 1395 1396 <para>Because it is possible to assign a different URI for each 1397 part of an image if necessary, jigdo is very flexible. Only one 1398 example is the possibility of customized versions of images: 1399 Suppose that someone is distributing a CD image, and that you 1400 want to make a few small changes to it and redistribute your own 1401 version. You download the `<filename>official.iso</filename>' CD 1402 image with <command>jigdo</command> (passing it the URL of 1403 `<filename>official.jigdo</filename>'), write it to CD-R, make 1404 your changes (say, adding files from the 1405 `<filename>myfiles</filename>' directory on your harddisc) and 1406 produce your own version, `<filename>myversion.iso</filename>'. 1407 Next, you instruct <command>jigdo-file</command> to create the 1408 jigdo and template files for your modified image, using the 1409 command 1410 1411 <blockquote><para><command>jigdo-file make-template 1412 --image=myversion.iso /mnt/cdrom/ myfiles// --label 1413 My=myfiles/ --uri My=http://my.homepage.net/ 1414 --merge=official.jigdo</command></para></blockquote> 1415 1416 while `<filename>official.iso</filename>' is mounted under 1417 `<filename>/mnt/cdrom</filename>'. By using 1418 <option>--merge</option>, you have told 1419 <command>jigdo-file</command> to take the contents of 1420 `<filename>official.jigdo</filename>', add to it a new 1421 `<literal>[Image]</literal>' section for 1422 `<filename>myversion.iso</filename>' and write the resulting 1423 jigdo file to `<filename>myversion.jigdo</filename>' - so now 1424 `<filename>myversion.jigdo</filename>' offers two images for 1425 download, the original version and your modified version. (If 1426 you do not want it to offer the official version, edit it and 1427 remove the `<literal>[Image]</literal>' section that lists 1428 `<filename>official.iso</filename>'.)</para> 1429 1430 <para>Now you can upload the `<filename>.jigdo</filename>' file, 1431 the `<filename>.template</filename>' file and also the files in 1432 `<filename>myfiles</filename>' to `http://my.homepage.net/'. 1433 Thus, for people to download your modified image, you do 1434 <emphasis>not</emphasis> need to upload the complete image 1435 contents to your web space, but only the changes you 1436 made!</para> 1437 1438 <para>(In case you only made very few changes, you could also 1439 omit the `myfiles' parameter in the command above, then all your 1440 changes end up in the new template file.)</para> 1441 </refsect2> 1442 1443 <refsect2 id="ex-combine"> 1444 <title>Combining many jigdo-managed images into one</title> 1445 1446 <para>It is also no problem to combine data from several sources 1447 that use jigdo. For example, if of five different and unrelated 1448 servers each one distributes a different CD image via jigdo, you 1449 can create a customized DVD image that contains the data from 1450 all these CDs. When people use <command>jigdo</command> to 1451 download your image, the individual files on the DVD are fetched 1452 from the same sources as the original CDs.</para> 1453 1454 <para>Consequently, even though you will be distributing a 3.2GB 1455 file via your web space, the actual amount of data that is 1456 stored on your server will only be in the order of several 1457 MBs.</para> 1458 </refsect2> 1459 1460 </refsect1> 1461 <!-- ============================================================= --> 1462 <refsect1 id="bugs"> 1463 <title>BUGS</title> 1464 1465 <para>For certain contents of one of the input files, most notably 1466 a sequence of zero bytes longer than <option>--min-length</option> 1467 at the start of the file and an area of zeros preceding the file 1468 data in the image, <command>jigdo-file make-template</command> may 1469 fail to find the file in the image. Unfortunately, this 1470 restriction cannot be avoided because the program could become 1471 very slow otherwise. If you use the <option>--debug</option> 1472 option, all instances of <command>jigdo-file</command> discarding 1473 possible matches are indicated by lines containing the word 1474 `<literal>DROPPED</literal>'.</para> 1475 1476 <para>In fact, not only all-zeroes files trigger this behaviour, 1477 but also files which contain at their start a long sequence of 1478 short identical strings. For example, both a file containing only 1479 `<literal>a</literal>' characters and one containing 1480 `<literal>abcabcabcabc</literal>...' are problematic.</para> 1481 1482 </refsect1> 1483 <!-- ============================================================= --> 1484 <refsect1 id="seealso"> 1485 <title>SEE ALSO</title> 1486 1487 <para> 1488 <citerefentry> 1489 <refentrytitle>jigdo</refentrytitle><manvolnum>1</manvolnum> 1490 </citerefentry> (NOT YET IMPLEMENTED), 1491 <citerefentry> 1492 <refentrytitle>jigdo-lite</refentrytitle><manvolnum>1</manvolnum> 1493 </citerefentry>, 1494 <citerefentry> 1495 <refentrytitle>jigdo-mirror</refentrytitle><manvolnum>1</manvolnum> 1496 </citerefentry>, 1497 <citerefentry> 1498 <refentrytitle>split</refentrytitle><manvolnum>1</manvolnum> 1499 </citerefentry> (or `<command>info split</command>'), 1500 <citerefentry> 1501 <refentrytitle>find</refentrytitle><manvolnum>1</manvolnum> 1502 </citerefentry> (or `<command>info find</command>'), 1503 <citerefentry> 1504 <refentrytitle>mkisofs</refentrytitle><manvolnum>1</manvolnum> 1505 </citerefentry>, 1506 <citerefentry> 1507 <refentrytitle>md5sum</refentrytitle><manvolnum>1</manvolnum> 1508 </citerefentry> 1509 </para> 1510 1511 </refsect1> 1512 <!-- ============================================================= --> 1513 <refsect1 id="author"> 1514 <title>AUTHOR</title> 1515 1516 <para><ulink url="http://atterer.net/jigdo/">Jigsaw 1517 Download</ulink> was written by Richard Atterer 1518 <email>jigdo atterer.net</email>, to make downloading of CD ROM 1519 images for the Debian Linux distribution more convenient.</para> 1520 1521 </refsect1> 1522</refentry> 1523 1524<!-- Keep this comment at the end of the file 1525Local variables: 1526mode: sgml 1527sgml-omittag:t 1528sgml-shorttag:t 1529sgml-minimize-attributes:nil 1530sgml-always-quote-attributes:t 1531sgml-indent-step:2 1532sgml-indent-data:t 1533sgml-parent-document:nil 1534sgml-default-dtd-file:nil 1535sgml-exposed-tags:nil 1536sgml-local-catalogs:nil 1537sgml-local-ecat-files:nil 1538End: 1539--> 1540