doc/html/libarchive-formats.5.html

<!-- Creator     : groff version 1.22.4 -->
<!-- CreationDate: Sun Aug 22 23:03:27 2021 -->
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta name="generator" content="groff -Thtml, see www.gnu.org">
<meta http-equiv="Content-Type" content="text/html; charset=US-ASCII">
<meta name="Content-Style" content="text/css">
<style type="text/css">
       p       { margin-top: 0; margin-bottom: 0; vertical-align: top }
       pre     { margin-top: 0; margin-bottom: 0; vertical-align: top }
       table   { margin-top: 0; margin-bottom: 0; vertical-align: top }
       h1      { text-align: center }
</style>
<title></title>
</head>
<body>

<hr>


<p>LIBARCHIVE-FORMATS(5) BSD File Formats Manual
LIBARCHIVE-FORMATS(5)</p>

<p style="margin-top: 1em"><b>NAME</b></p>

<p style="margin-left:6%;"><b>libarchive-formats</b>
&mdash; archive formats supported by the libarchive
library</p>

<p style="margin-top: 1em"><b>DESCRIPTION</b></p>

<p style="margin-left:6%;">The libarchive(3) library reads
and writes a variety of streaming archive formats. Generally
speaking, all of these archive formats consist of a series
of &ldquo;entries&rdquo;. Each entry stores a single file
system object, such as a file, directory, or symbolic
link.</p>

<p style="margin-left:6%; margin-top: 1em">The following
provides a brief description of each format supported by
libarchive, with some information about recognized
extensions or limitations of the current library support.
Note that just because a format is supported by libarchive
does not imply that a program that uses libarchive will
support that format. Applications that use libarchive
specify which formats they wish to support, though many
programs do use libarchive convenience functions to enable
all supported formats.</p>

<p style="margin-left:6%; margin-top: 1em"><b>Tar
Formats</b> <br>
The libarchive(3) library can read most tar archives. It can
write POSIX-standard &ldquo;ustar&rdquo; and &ldquo;pax
interchange&rdquo; formats as well as v7 tar format and a
subset of the legacy GNU tar format.</p>

<p style="margin-left:6%; margin-top: 1em">All tar formats
store each entry in one or more 512-byte records. The first
record is used for file metadata, including filename,
timestamp, and mode information, and the file data is stored
in subsequent records. Later variants have extended this by
either appropriating undefined areas of the header record,
extending the header to multiple records, or by storing
special entries that modify the interpretation of subsequent
entries.</p>

<p style="margin-top: 1em"><b>gnutar</b></p>

<p style="margin-left:17%; margin-top: 1em">The
libarchive(3) library can read most GNU-format tar archives.
It currently supports the most popular GNU extensions,
including modern long filename and linkname support, as well
as atime and ctime data. The libarchive library does not
support multi-volume archives, nor the old GNU long filename
format. It can read GNU sparse file entries, including the
new POSIX-based formats.</p>

<p style="margin-left:17%; margin-top: 1em">The
libarchive(3) library can write GNU tar format, including
long filename and linkname support, as well as atime and
ctime data.</p>

<p style="margin-top: 1em"><b>pax</b></p>

<p style="margin-left:17%; margin-top: 1em">The
libarchive(3) library can read and write POSIX-compliant pax
interchange format archives. Pax interchange format archives
are an extension of the older ustar format that adds a
separate entry with additional attributes stored as
key/value pairs immediately before each regular entry. The
presence of these additional entries is the only difference
between pax interchange format and the older ustar format.
The extended attributes are of unlimited length and are
stored as UTF-8 Unicode strings. Keywords defined in the
standard are in all lowercase; vendors are allowed to define
custom keys by preceding them with the vendor name in all
uppercase. When writing pax archives, libarchive uses many
of the SCHILY keys defined by Joerg Schilling&rsquo;s
&ldquo;star&rdquo; archiver and a few LIBARCHIVE keys. The
libarchive library can read most of the SCHILY keys and most
of the GNU keys introduced by GNU tar. It silently ignores
any keywords that it does not understand.</p>

<p style="margin-left:17%; margin-top: 1em">The pax
interchange format converts filenames to Unicode and stores
them using the UTF-8 encoding. Prior to libarchive 3.0,
libarchive erroneously assumed that the system
wide-character routines natively supported Unicode. This
caused it to mis-handle non-ASCII filenames on systems that
did not satisfy this assumption.</p>

<p style="margin-top: 1em"><b>restricted pax</b></p>

<p style="margin-left:17%;">The libarchive library can also
write pax archives in which it attempts to suppress the
extended attributes entry whenever possible. The result will
be identical to a ustar archive unless the extended
attributes entry is required to store a long file name, long
linkname, extended ACL, file flags, or if any of the
standard ustar data (user name, group name, UID, GID, etc)
cannot be fully represented in the ustar header. In all
cases, the result can be dearchived by any program that can
read POSIX-compliant pax interchange format archives.
Programs that correctly read ustar format (see below) will
also be able to read this format; any extended attributes
will be extracted as separate files stored in
<i>PaxHeader</i> directories.</p>

<p style="margin-top: 1em"><b>ustar</b></p>

<p style="margin-left:17%; margin-top: 1em">The libarchive
library can both read and write this format. This format has
the following limitations:</p>

<p><b>&bull;</b></p>

<p style="margin-left:22%;">Device major and minor numbers
are limited to 21 bits. Nodes with larger numbers will not
be added to the archive.</p>

<p><b>&bull;</b></p>

<p style="margin-left:22%;">Path names in the archive are
limited to 255 bytes. (Shorter if there is no / character in
exactly the right place.)</p>

<p><b>&bull;</b></p>

<p style="margin-left:22%;">Symbolic links and hard links
are stored in the archive with the name of the referenced
file. This name is limited to 100 bytes.</p>

<p><b>&bull;</b></p>

<p style="margin-left:22%;">Extended attributes, file
flags, and other extended security information cannot be
stored.</p>

<p><b>&bull;</b></p>

<p style="margin-left:22%;">Archive entries are limited to
8 gigabytes in size.</p>

<p style="margin-left:17%;">Note that the pax interchange
format has none of these restrictions. The ustar format is
old and widely supported. It is recommended when
compatibility is the primary concern.</p>

<p style="margin-top: 1em"><b>v7</b></p>

<p style="margin-left:17%; margin-top: 1em">The libarchive
library can read and write the legacy v7 tar format. This
format has the following limitations:</p>

<p><b>&bull;</b></p>

<p style="margin-left:22%;">Only regular files,
directories, and symbolic links can be archived. Block and
character device nodes, FIFOs, and sockets cannot be
archived.</p>

<p><b>&bull;</b></p>

<p style="margin-left:22%;">Path names in the archive are
limited to 100 bytes.</p>

<p><b>&bull;</b></p>

<p style="margin-left:22%;">Symbolic links and hard links
are stored in the archive with the name of the referenced
file. This name is limited to 100 bytes.</p>

<p><b>&bull;</b></p>

<p style="margin-left:22%;">User and group information are
stored as numeric IDs; there is no provision for storing
user or group names.</p>

<p><b>&bull;</b></p>

<p style="margin-left:22%;">Extended attributes, file
flags, and other extended security information cannot be
stored.</p>

<p><b>&bull;</b></p>

<p style="margin-left:22%;">Archive entries are limited to
8 gigabytes in size.</p>

<p style="margin-left:17%;">Generally, users should prefer
the ustar format for portability as the v7 tar format is
both less useful and less portable.</p>

<p style="margin-left:6%; margin-top: 1em">The libarchive
library also reads a variety of commonly-used extensions to
the basic tar format. These extensions are recognized
automatically whenever they appear.</p>

<p style="margin-top: 1em">Numeric extensions.</p>

<p style="margin-left:17%;">The POSIX standards require
fixed-length numeric fields to be written with some
character position reserved for terminators. Libarchive
allows these fields to be written without terminator
characters. This extends the allowable range; in particular,
ustar archives with this extension can support entries up to
64 gigabytes in size. Libarchive also recognizes base-256
values in most numeric fields. This essentially removes all
limitations on file size, modification time, and device
numbers.</p>

<p style="margin-top: 1em">Solaris extensions</p>

<p style="margin-left:17%;">Libarchive recognizes ACL and
extended attribute records written by Solaris tar.</p>

<p style="margin-left:6%; margin-top: 1em">The first tar
program appeared in Seventh Edition Unix in 1979. The first
official standard for the tar file format was the
&ldquo;ustar&rdquo; (Unix Standard Tar) format defined by
POSIX in 1988. POSIX.1-2001 extended the ustar format to
create the &ldquo;pax interchange&rdquo; format.</p>

<p style="margin-left:6%; margin-top: 1em"><b>Cpio
Formats</b> <br>
The libarchive library can read and write a number of common
cpio variants. A cpio archive stores each entry as a
fixed-size header followed by a variable-length filename and
variable-length data. Unlike the tar format, the cpio format
does only minimal padding of the header or file data. There
are several cpio variants, which differ primarily in how
they store the initial header: some store the values as
octal or hexadecimal numbers in ASCII, others as binary
values of varying byte order and length.</p>

<p style="margin-top: 1em"><b>binary</b></p>

<p style="margin-left:17%; margin-top: 1em">The libarchive
library transparently reads both big-endian and
little-endian variants of the the two binary cpio formats;
the original one from PWB/UNIX, and the later, more widely
used, variant. This format used 32-bit binary values for
file size and mtime, and 16-bit binary values for the other
fields. The formats support only the file types present in
UNIX at the time of their creation. File sizes are limited
to 24 bits in the PWB format, because of the limits of the
file system, and to 31 bits in the newer binary format,
where signed 32 bit longs were used.</p>

<p style="margin-top: 1em"><b>odc</b></p>

<p style="margin-left:17%; margin-top: 1em">This is the
POSIX standardized format, which is officially known as the
&ldquo;cpio interchange format&rdquo; or the
&ldquo;octet-oriented cpio archive format&rdquo; and
sometimes unofficially referred to as the &ldquo;old
character format&rdquo;. This format stores the header
contents as octal values in ASCII. It is standard, portable,
and immune from byte-order confusion. File sizes and mtime
are limited to 33 bits (8GB file size), other fields are
limited to 18 bits.</p>

<p style="margin-top: 1em"><b>SVR4/newc</b></p>

<p style="margin-left:17%;">The libarchive library can read
both CRC and non-CRC variants of this format. The SVR4
format uses eight-digit hexadecimal values for all header
fields. This limits file size to 4GB, and also limits the
mtime and other fields to 32 bits. The SVR4 format can
optionally include a CRC of the file contents, although
libarchive does not currently verify this CRC.</p>

<p style="margin-left:6%; margin-top: 1em">Cpio first
appeared in PWB/UNIX 1.0, which was released within AT&amp;T
in 1977. PWB/UNIX 1.0 formed the basis of System III Unix,
released outside of AT&amp;T in 1981. This makes cpio older
than tar, although cpio was not included in Version 7
AT&amp;T Unix. As a result, the tar command became much
better known in universities and research groups that used
Version 7. The combination of the <b>find</b> and
<b>cpio</b> utilities provided very precise control over
file selection. Unfortunately, the format has many
limitations that make it unsuitable for widespread use. Only
the POSIX format permits files over 4GB, and its 18-bit
limit for most other fields makes it unsuitable for modern
systems. In addition, cpio formats only store numeric
UID/GID values (not usernames and group names), which can
make it very difficult to correctly transfer archives across
systems with dissimilar user numbering.</p>

<p style="margin-left:6%; margin-top: 1em"><b>Shar
Formats</b> <br>
A &ldquo;shell archive&rdquo; is a shell script that, when
executed on a POSIX-compliant system, will recreate a
collection of file system objects. The libarchive library
can write two different kinds of shar archives:</p>

<p style="margin-top: 1em"><b>shar</b></p>

<p style="margin-left:17%; margin-top: 1em">The traditional
shar format uses a limited set of POSIX commands, including
echo(1), mkdir(1), and sed(1). It is suitable for portably
archiving small collections of plain text files. However, it
is not generally well-suited for large archives (many
implementations of sh(1) have limits on the size of a
script) nor should it be used with non-text files.</p>

<p style="margin-top: 1em"><b>shardump</b></p>

<p style="margin-left:17%;">This format is similar to shar
but encodes files using uuencode(1) so that the result will
be a plain text file regardless of the file contents. It
also includes additional shell commands that attempt to
reproduce as many file attributes as possible, including
owner, mode, and flags. The additional commands used to
restore file attributes make shardump archives less portable
than plain shar archives.</p>

<p style="margin-left:6%; margin-top: 1em"><b>ISO9660
format</b> <br>
Libarchive can read and extract from files containing
ISO9660-compliant CDROM images. In many cases, this can
remove the need to burn a physical CDROM just in order to
read the files contained in an ISO9660 image. It also avoids
security and complexity issues that come with virtual mounts
and loopback devices. Libarchive supports the most common
Rockridge extensions and has partial support for Joliet
extensions. If both extensions are present, the Joliet
extensions will be used and the Rockridge extensions will be
ignored. In particular, this can create problems with
hardlinks and symlinks, which are supported by Rockridge but
not by Joliet.</p>

<p style="margin-left:6%; margin-top: 1em">Libarchive reads
ISO9660 images using a streaming strategy. This allows it to
read compressed images directly (decompressing on the fly)
and allows it to read images directly from network sockets,
pipes, and other non-seekable data sources. This strategy
works well for optimized ISO9660 images created by many
popular programs. Such programs collect all directory
information at the beginning of the ISO9660 image so it can
be read from a physical disk with a minimum of seeking.
However, not all ISO9660 images can be read in this
fashion.</p>

<p style="margin-left:6%; margin-top: 1em">Libarchive can
also write ISO9660 images. Such images are fully optimized
with the directory information preceding all file data. This
is done by storing all file data to a temporary file while
collecting directory information in memory. When the image
is finished, libarchive writes out the directory structure
followed by the file data. The location used for the
temporary file can be changed by the usual environment
variables.</p>

<p style="margin-left:6%; margin-top: 1em"><b>Zip
format</b> <br>
Libarchive can read and write zip format archives that have
uncompressed entries and entries compressed with the
&ldquo;deflate&rdquo; algorithm. Other zip compression
algorithms are not supported. It can extract jar archives,
archives that use Zip64 extensions and self-extracting zip
archives. Libarchive can use either of two different
strategies for reading Zip archives: a streaming strategy
which is fast and can handle extremely large archives, and a
seeking strategy which can correctly process self-extracting
Zip archives and archives with deleted members or other
in-place modifications.</p>

<p style="margin-left:6%; margin-top: 1em">The streaming
reader processes Zip archives as they are read. It can read
archives of arbitrary size from tape or network sockets, and
can decode Zip archives that have been separately compressed
or encoded. However, self-extracting Zip archives and
archives with certain types of modifications cannot be
correctly handled. Such archives require that the reader
first process the Central Directory, which is ordinarily
located at the end of a Zip archive and is thus inaccessible
to the streaming reader. If the program using libarchive has
enabled seek support, then libarchive will use this to
processes the central directory first.</p>

<p style="margin-left:6%; margin-top: 1em">In particular,
the seeking reader must be used to correctly handle
self-extracting archives. Such archives consist of a program
followed by a regular Zip archive. The streaming reader
cannot parse the initial program portion, but the seeking
reader starts by reading the Central Directory from the end
of the archive. Similarly, Zip archives that have been
modified in-place can have deleted entries or other garbage
data that can only be accurately detected by first reading
the Central Directory.</p>

<p style="margin-left:6%; margin-top: 1em"><b>Archive
(library) file format</b> <br>
The Unix archive format (commonly created by the ar(1)
archiver) is a general-purpose format which is used almost
exclusively for object files to be read by the link editor
ld(1). The ar format has never been standardised. There are
two common variants: the GNU format derived from SVR4, and
the BSD format, which first appeared in 4.4BSD. The two
differ primarily in their handling of filenames longer than
15 characters: the GNU/SVR4 variant writes a filename table
at the beginning of the archive; the BSD format stores each
long filename in an extension area adjacent to the entry.
Libarchive can read both extensions, including archives that
may include both types of long filenames. Programs using
libarchive can write GNU/SVR4 format if they provide an
entry called <i>//</i> containing a filename table to be
written into the archive before any of the entries. Any
entries whose names are not in the filename table will be
written using BSD-style long filenames. This can cause
problems for programs such as GNU ld that do not support the
BSD-style long filenames.</p>

<p style="margin-left:6%; margin-top: 1em"><b>mtree</b>
<br>
Libarchive can read and write files in mtree(5) format. This
format is not a true archive format, but rather a textual
description of a file hierarchy in which each line specifies
the name of a file and provides specific metadata about that
file. Libarchive can read all of the keywords supported by
both the NetBSD and FreeBSD versions of mtree(8), although
many of the keywords cannot currently be stored in an
archive_entry object. When writing, libarchive supports use
of the archive_write_set_options(3) interface to specify
which keywords should be included in the output. If
libarchive was compiled with access to suitable
cryptographic libraries (such as the OpenSSL libraries), it
can compute hash entries such as <b>sha512</b> or <b>md5</b>
from file data being written to the mtree writer.</p>

<p style="margin-left:6%; margin-top: 1em">When reading an
mtree file, libarchive will locate the corresponding files
on disk using the <b>contents</b> keyword if present or the
regular filename. If it can locate and open the file on
disk, it will use that to fill in any metadata that is
missing from the mtree file and will read the file contents
and return those to the program using libarchive. If it
cannot locate and open the file on disk, libarchive will
return an error for any attempt to read the entry body.</p>

<p style="margin-left:6%; margin-top: 1em"><b>7-Zip</b>
<br>
Libarchive can read and write 7-Zip format archives. TODO:
Need more information</p>

<p style="margin-left:6%; margin-top: 1em"><b>CAB</b> <br>
Libarchive can read Microsoft Cabinet ( &ldquo;CAB&rdquo;)
format archives. TODO: Need more information.</p>

<p style="margin-left:6%; margin-top: 1em"><b>LHA</b> <br>
TODO: Information about libarchive&rsquo;s LHA support</p>

<p style="margin-left:6%; margin-top: 1em"><b>RAR</b> <br>
Libarchive has limited support for reading RAR format
archives. Currently, libarchive can read RARv3 format
archives which have been either created uncompressed, or
compressed using any of the compression methods supported by
the RARv3 format. Libarchive can also read self-extracting
RAR archives.</p>

<p style="margin-left:6%; margin-top: 1em"><b>Warc</b> <br>
Libarchive can read and write &ldquo;web archives&rdquo;.
TODO: Need more information</p>

<p style="margin-left:6%; margin-top: 1em"><b>XAR</b> <br>
Libarchive can read and write the XAR format used by many
Apple tools. TODO: Need more information</p>

<p style="margin-top: 1em"><b>SEE ALSO</b></p>

<p style="margin-left:6%;">ar(1), cpio(1), mkisofs(1),
shar(1), tar(1), zip(1), zlib(3), cpio(5), mtree(5),
tar(5)</p>

<p style="margin-left:6%; margin-top: 1em">BSD
December&nbsp;27, 2016 BSD</p>
<hr>
</body>
</html>