1.\" Copyright (c) 2003-2009 Tim Kientzle 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions 6.\" are met: 7.\" 1. Redistributions of source code must retain the above copyright 8.\" notice, this list of conditions and the following disclaimer. 9.\" 2. Redistributions in binary form must reproduce the above copyright 10.\" notice, this list of conditions and the following disclaimer in the 11.\" documentation and/or other materials provided with the distribution. 12.\" 13.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND 14.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 15.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 16.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE 17.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 18.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 19.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 20.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 21.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 22.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 23.\" SUCH DAMAGE. 24.\" 25.\" $FreeBSD$ 26.\" 27.Dd March 18, 2012 28.Dt LIBARCHIVE-FORMATS 5 29.Os 30.Sh NAME 31.Nm libarchive-formats 32.Nd archive formats supported by the libarchive library 33.Sh DESCRIPTION 34The 35.Xr libarchive 3 36library reads and writes a variety of streaming archive formats. 37Generally speaking, all of these archive formats consist of a series of 38.Dq entries . 39Each entry stores a single file system object, such as a file, directory, 40or symbolic link. 41.Pp 42The following provides a brief description of each format supported 43by libarchive, with some information about recognized extensions or 44limitations of the current library support. 45Note that just because a format is supported by libarchive does not 46imply that a program that uses libarchive will support that format. 47Applications that use libarchive specify which formats they wish 48to support, though many programs do use libarchive convenience 49functions to enable all supported formats. 50.Ss Tar Formats 51The 52.Xr libarchive 3 53library can read most tar archives. 54It can write POSIX-standard 55.Dq ustar 56and 57.Dq pax interchange 58formats and a subset of the legacy GNU tar format. 59.Pp 60All tar formats store each entry in one or more 512-byte records. 61The first record is used for file metadata, including filename, 62timestamp, and mode information, and the file data is stored in 63subsequent records. 64Later variants have extended this by either appropriating undefined 65areas of the header record, extending the header to multiple records, 66or by storing special entries that modify the interpretation of 67subsequent entries. 68.Pp 69.Bl -tag -width indent 70.It Cm gnutar 71The 72.Xr libarchive 3 73library can read most GNU-format tar archives. 74It currently supports the most popular GNU extensions, including 75modern long filename and linkname support, as well as atime and ctime data. 76The libarchive library does not support multi-volume 77archives, nor the old GNU long filename format. 78It can read GNU sparse file entries, including the new POSIX-based 79formats. 80.Pp 81The 82.Xr libarchive 3 83library can write GNU tar format, including long filename 84and linkname support, as well as atime and ctime data. 85.It Cm pax 86The 87.Xr libarchive 3 88library can read and write POSIX-compliant pax interchange format 89archives. 90Pax interchange format archives are an extension of the older ustar 91format that adds a separate entry with additional attributes stored 92as key/value pairs immediately before each regular entry. 93The presence of these additional entries is the only difference between 94pax interchange format and the older ustar format. 95The extended attributes are of unlimited length and are stored 96as UTF-8 Unicode strings. 97Keywords defined in the standard are in all lowercase; vendors are allowed 98to define custom keys by preceding them with the vendor name in all uppercase. 99When writing pax archives, libarchive uses many of the SCHILY keys 100defined by Joerg Schilling's 101.Dq star 102archiver and a few LIBARCHIVE keys. 103The libarchive library can read most of the SCHILY keys 104and most of the GNU keys introduced by GNU tar. 105It silently ignores any keywords that it does not understand. 106.Pp 107The pax interchange format converts filenames to Unicode 108and stores them using the UTF-8 encoding. 109Prior to libarchive 3.0, libarchive erroneously assumed 110that the system wide-character routines natively supported 111Unicode. 112This caused it to mis-handle non-ASCII filenames on systems 113that did not satisfy this assumption. 114.It Cm restricted pax 115The libarchive library can also write pax archives in which it 116attempts to suppress the extended attributes entry whenever 117possible. 118The result will be identical to a ustar archive unless the 119extended attributes entry is required to store a long file 120name, long linkname, extended ACL, file flags, or if any of the standard 121ustar data (user name, group name, UID, GID, etc) cannot be fully 122represented in the ustar header. 123In all cases, the result can be dearchived by any program that 124can read POSIX-compliant pax interchange format archives. 125Programs that correctly read ustar format (see below) will also be 126able to read this format; any extended attributes will be extracted as 127separate files stored in 128.Pa PaxHeader 129directories. 130.It Cm ustar 131The libarchive library can both read and write this format. 132This format has the following limitations: 133.Bl -bullet -compact 134.It 135Device major and minor numbers are limited to 21 bits. 136Nodes with larger numbers will not be added to the archive. 137.It 138Path names in the archive are limited to 255 bytes. 139(Shorter if there is no / character in exactly the right place.) 140.It 141Symbolic links and hard links are stored in the archive with 142the name of the referenced file. 143This name is limited to 100 bytes. 144.It 145Extended attributes, file flags, and other extended 146security information cannot be stored. 147.It 148Archive entries are limited to 8 gigabytes in size. 149.El 150Note that the pax interchange format has none of these restrictions. 151The ustar format is old and widely supported. 152It is recommended when compatibility is the primary concern. 153.El 154.Pp 155The libarchive library also reads a variety of commonly-used extensions to 156the basic tar format. 157These extensions are recognized automatically whenever they appear. 158.Bl -tag -width indent 159.It Numeric extensions. 160The POSIX standards require fixed-length numeric fields to be written with 161some character position reserved for terminators. 162Libarchive allows these fields to be written without terminator characters. 163This extends the allowable range; in particular, ustar archives with this 164extension can support entries up to 64 gigabytes in size. 165Libarchive also recognizes base-256 values in most numeric fields. 166This essentially removes all limitations on file size, modification time, 167and device numbers. 168.It Solaris extensions 169Libarchive recognizes ACL and extended attribute records written 170by Solaris tar. 171Currently, libarchive only has support for old-style ACLs; the 172newer NFSv4 ACLs are recognized but discarded. 173.El 174.Pp 175The first tar program appeared in Seventh Edition Unix in 1979. 176The first official standard for the tar file format was the 177.Dq ustar 178(Unix Standard Tar) format defined by POSIX in 1988. 179POSIX.1-2001 extended the ustar format to create the 180.Dq pax interchange 181format. 182.Ss Cpio Formats 183The libarchive library can read a number of common cpio variants and can write 184.Dq odc 185and 186.Dq newc 187format archives. 188A cpio archive stores each entry as a fixed-size header followed 189by a variable-length filename and variable-length data. 190Unlike the tar format, the cpio format does only minimal padding 191of the header or file data. 192There are several cpio variants, which differ primarily in 193how they store the initial header: some store the values as 194octal or hexadecimal numbers in ASCII, others as binary values of 195varying byte order and length. 196.Bl -tag -width indent 197.It Cm binary 198The libarchive library transparently reads both big-endian and little-endian 199variants of the original binary cpio format. 200This format used 32-bit binary values for file size and mtime, 201and 16-bit binary values for the other fields. 202.It Cm odc 203The libarchive library can both read and write this 204POSIX-standard format, which is officially known as the 205.Dq cpio interchange format 206or the 207.Dq octet-oriented cpio archive format 208and sometimes unofficially referred to as the 209.Dq old character format . 210This format stores the header contents as octal values in ASCII. 211It is standard, portable, and immune from byte-order confusion. 212File sizes and mtime are limited to 33 bits (8GB file size), 213other fields are limited to 18 bits. 214.It Cm SVR4 215The libarchive library can read both CRC and non-CRC variants of 216this format. 217The SVR4 format uses eight-digit hexadecimal values for 218all header fields. 219This limits file size to 4GB, and also limits the mtime and 220other fields to 32 bits. 221The SVR4 format can optionally include a CRC of the file 222contents, although libarchive does not currently verify this CRC. 223.El 224.Pp 225Cpio first appeared in PWB/UNIX 1.0, which was released within 226AT&T in 1977. 227PWB/UNIX 1.0 formed the basis of System III Unix, released outside 228of AT&T in 1981. 229This makes cpio older than tar, although cpio was not included 230in Version 7 AT&T Unix. 231As a result, the tar command became much better known in universities 232and research groups that used Version 7. 233The combination of the 234.Nm find 235and 236.Nm cpio 237utilities provided very precise control over file selection. 238Unfortunately, the format has many limitations that make it unsuitable 239for widespread use. 240Only the POSIX format permits files over 4GB, and its 18-bit 241limit for most other fields makes it unsuitable for modern systems. 242In addition, cpio formats only store numeric UID/GID values (not 243usernames and group names), which can make it very difficult to correctly 244transfer archives across systems with dissimilar user numbering. 245.Ss Shar Formats 246A 247.Dq shell archive 248is a shell script that, when executed on a POSIX-compliant 249system, will recreate a collection of file system objects. 250The libarchive library can write two different kinds of shar archives: 251.Bl -tag -width indent 252.It Cm shar 253The traditional shar format uses a limited set of POSIX 254commands, including 255.Xr echo 1 , 256.Xr mkdir 1 , 257and 258.Xr sed 1 . 259It is suitable for portably archiving small collections of plain text files. 260However, it is not generally well-suited for large archives 261(many implementations of 262.Xr sh 1 263have limits on the size of a script) nor should it be used with non-text files. 264.It Cm shardump 265This format is similar to shar but encodes files using 266.Xr uuencode 1 267so that the result will be a plain text file regardless of the file contents. 268It also includes additional shell commands that attempt to reproduce as 269many file attributes as possible, including owner, mode, and flags. 270The additional commands used to restore file attributes make 271shardump archives less portable than plain shar archives. 272.El 273.Ss ISO9660 format 274Libarchive can read and extract from files containing ISO9660-compliant 275CDROM images. 276In many cases, this can remove the need to burn a physical CDROM 277just in order to read the files contained in an ISO9660 image. 278It also avoids security and complexity issues that come with 279virtual mounts and loopback devices. 280Libarchive supports the most common Rockridge extensions and has partial 281support for Joliet extensions. 282If both extensions are present, the Joliet extensions will be 283used and the Rockridge extensions will be ignored. 284In particular, this can create problems with hardlinks and symlinks, 285which are supported by Rockridge but not by Joliet. 286.Pp 287Libarchive reads ISO9660 images using a streaming strategy. 288This allows it to read compressed images directly 289(decompressing on the fly) and allows it to read images 290directly from network sockets, pipes, and other non-seekable 291data sources. 292This strategy works well for optimized ISO9660 images created 293by many popular programs. 294Such programs collect all directory information at the beginning 295of the ISO9660 image so it can be read from a physical disk 296with a minimum of seeking. 297However, not all ISO9660 images can be read in this fashion. 298.Pp 299Libarchive can also write ISO9660 images. 300Such images are fully optimized with the directory information 301preceding all file data. 302This is done by storing all file data to a temporary file 303while collecting directory information in memory. 304When the image is finished, libarchive writes out the 305directory structure followed by the file data. 306The location used for the temporary file can be changed 307by the usual environment variables. 308.Ss Zip format 309Libarchive can read and write zip format archives that have 310uncompressed entries and entries compressed with the 311.Dq deflate 312algorithm. 313Other zip compression algorithms are not supported. 314It can extract jar archives, archives that use Zip64 extensions and 315self-extracting zip archives. 316Libarchive can use either of two different strategies for 317reading Zip archives: 318a streaming strategy which is fast and can handle extremely 319large archives, and a seeking strategy which can correctly 320process self-extracting Zip archives and archives with 321deleted members or other in-place modifications. 322.Pp 323The streaming reader processes Zip archives as they are read. 324It can read archives of arbitrary size from tape or 325network sockets, and can decode Zip archives that have 326been separately compressed or encoded. 327However, self-extracting Zip archives and archives with 328certain types of modifications cannot be correctly 329handled. 330Such archives require that the reader first process the 331Central Directory, which is ordinarily located 332at the end of a Zip archive and is thus inaccessible 333to the streaming reader. 334If the program using libarchive has enabled seek support, then 335libarchive will use this to processes the central directory first. 336.Pp 337In particular, the seeking reader must be used to 338correctly handle self-extracting archives. 339Such archives consist of a program followed by a regular 340Zip archive. 341The streaming reader cannot parse the initial program 342portion, but the seeking reader starts by reading the 343Central Directory from the end of the archive. 344Similarly, Zip archives that have been modified in-place 345can have deleted entries or other garbage data that 346can only be accurately detected by first reading the 347Central Directory. 348.Ss Archive (library) file format 349The Unix archive format (commonly created by the 350.Xr ar 1 351archiver) is a general-purpose format which is 352used almost exclusively for object files to be 353read by the link editor 354.Xr ld 1 . 355The ar format has never been standardised. 356There are two common variants: 357the GNU format derived from SVR4, 358and the BSD format, which first appeared in 4.4BSD. 359The two differ primarily in their handling of filenames 360longer than 15 characters: 361the GNU/SVR4 variant writes a filename table at the beginning of the archive; 362the BSD format stores each long filename in an extension 363area adjacent to the entry. 364Libarchive can read both extensions, 365including archives that may include both types of long filenames. 366Programs using libarchive can write GNU/SVR4 format 367if they provide a filename table to be written into 368the archive before any of the entries. 369Any entries whose names are not in the filename table 370will be written using BSD-style long filenames. 371This can cause problems for programs such as 372GNU ld that do not support the BSD-style long filenames. 373.Ss mtree 374Libarchive can read and write files in 375.Xr mtree 5 376format. 377This format is not a true archive format, but rather a textual description 378of a file hierarchy in which each line specifies the name of a file and 379provides specific metadata about that file. 380Libarchive can read all of the keywords supported by both 381the NetBSD and FreeBSD versions of 382.Xr mtree 8 , 383although many of the keywords cannot currently be stored in an 384.Tn archive_entry 385object. 386When writing, libarchive supports use of the 387.Xr archive_write_set_options 3 388interface to specify which keywords should be included in the 389output. 390If libarchive was compiled with access to suitable 391cryptographic libraries (such as the OpenSSL libraries), 392it can compute hash entries such as 393.Cm sha512 394or 395.Cm md5 396from file data being written to the mtree writer. 397.Pp 398When reading an mtree file, libarchive will locate the corresponding 399files on disk using the 400.Cm contents 401keyword if present or the regular filename. 402If it can locate and open the file on disk, it will use that 403to fill in any metadata that is missing from the mtree file 404and will read the file contents and return those to the program 405using libarchive. 406If it cannot locate and open the file on disk, libarchive 407will return an error for any attempt to read the entry 408body. 409.Ss LHA 410XXX Information about libarchive's LHA support XXX 411.Ss CAB 412XXX Information about libarchive's CAB support XXX 413.Ss XAR 414XXX Information about libarchive's XAR support XXX 415.Ss RAR 416Libarchive has limited support for reading RAR format archives. 417Currently, libarchive can read RARv3 format archives 418which have been either created uncompressed, or compressed using 419any of the compression methods supported by the RARv3 format. 420Libarchive can also read self-extracting RAR archives. 421.Sh SEE ALSO 422.Xr ar 1 , 423.Xr cpio 1 , 424.Xr mkisofs 1 , 425.Xr shar 1 , 426.Xr tar 1 , 427.Xr zip 1 , 428.Xr zlib 3 , 429.Xr cpio 5 , 430.Xr mtree 5 , 431.Xr tar 5 432