Dec 18, 2015

Parallel BZIP2 v1.1.13 - by: Jeff Gilchrist <pbzip2@compression.ca>
Available at:  http://compression.ca/

This is the README for pbzip2, a parallel implementation of the
bzip2 block-sorting file compressor.  The output of this version
should be fully compatible with bzip2 v1.0.2 or newer (ie: anything
compressed with pbzip2 can be decompressed with bzip2).

pbzip2 is distributed under a BSD-style license.  For details,
see the file COPYING.


1. HOW TO BUILD -- UNIX

Type `make'.  This builds the pbzip2 program and dynamically
links to the libbzip2 library.  You should ensure that you have
at least libbzip2 1.0.5 or newer installed as it contains some
important security bug fixes.

If you do not have libbzip2 installed on your system, you should
first go to http://www.bzip.org/ and install it.

Debian users need the package "libbz2-dev".  If you want to
install a pre-built package on Debian, run the following command:
'apt-get update; apt-get install pbzip2'

If you would like to build pbzip2 with a statically linked
libbzip2 library, download the bzip2 source from the above site,
compile it, and copy the libbz2.a and bzlib.h files into the
pbzip2 source directory.  Then type `make pbzip2-static'.

Note: This software has been tested on Linux (Intel, Alpha),
Solaris (Sparc), HP-UX, Irix (SGI), and Tru64/OSF1 (Alpha).


2. HOW TO BUILD -- Windows

On Windows, pbzip2 can be compiled using Cygwin.

If you do not have libbzip2 installed on your system, you should
first go to http://www.bzip.org/ and install it.

Cygwin can be found at:  http://www.cygwin.com/
From a Cygwin shell, go to the directory where the pbzip2 source
files are located and type `make'.  This builds the pbzip2
program and dynamically links to the libbzip2 library.

If you would like to build pbzip2 with a statically linked
libbzip2 library, download the bzip2 source from the above site,
compile it, and copy the libbz2.a file into the pbzip2 source
directory.  Then type `make pbzip2-static'.


3. DISCLAIMER

   I TAKE NO RESPONSIBILITY FOR ANY LOSS OF DATA ARISING FROM THE
   USE OF THIS PROGRAM, HOWSOEVER CAUSED.

   DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE
   PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE
   DATA WILL NOT BE RECOVERABLE.

* Portions of this README were copied directly from the bzip2 README
  written by Julian Seward.


4. PBZIP2 DATA FORMAT

You should be able to compress files larger than 4GB with pbzip2.

Files that are compressed with pbzip2 are broken up into pieces and
each individual piece is compressed.  This is how pbzip2 runs faster
on multiple CPUs since the pieces can be compressed simultaneously.
The final .bz2 file may be slightly larger than if it was compressed
with the regular bzip2 program due to this file splitting (usually
less than 0.2% larger).  Files that are compressed with pbzip2 will
also gain considerable speedup when decompressed using pbzip2.

Files that were compressed using bzip2 will not see speedup since
bzip2 pacakages the data into a single chunk that cannot be split
between processors.  pbzip2 will still be able to decompress these
files, but it will be slower than if the .bz2 file was created
with pbzip2.

A file compressed with bzip2 will contain one compressed stream of
data that looks like this:
[-----------------------------------------------------]

Data compressed with pbzip2 is broken into multiple streams and each
stream is bzip2 compressed looking like this:
[-----|-----|-----|-----|-----|-----|-----|-----|-----]

If you are writing software with libbzip2 to decompress data created
with pbzip2, you must take into account that the data contains multiple
bzip2 streams so you will encounter end-of-stream markers from libbzip2
after each stream and must look-ahead to see if there are any more
streams to process before quitting.  The bzip2 program itself will
automatically handle this condition.


5. USAGE

The pbzip2 program is a parallel version of bzip2 for use on shared
memory machines.  It provides near-linear speedup when used on true
multi-processor machines and 5-10% speedup on Hyperthreaded machines.
The output is fully compatible with the regular bzip2 data so any
files created with pbzip2 can be uncompressed by bzip2 and vice-versa.
The default settings for pbzip2 will work well in most cases.  The
only switch you will likely need to use is -d to decompress files and
-p to set the # of processors for pbzip2 to use if autodetect is not
supported on your system, or you want to use a specific # of CPUs.
Note, that if you are using a large number of CPUs you may wish to
lower your default stack size setting (with the -S switch or ulimit)
to reduce the amount of memory each thread uses.

Usage:  pbzip2 [-1 .. -9] [-b#cdfhklm#p#qrS#tvVz] <filename> <filename2> <filenameN>

Switches:
 -b#    		  Where # is block size in 100k steps (default 9 = 900k)
 -c, --stdout	  Output to standard out (stdout)
 -d,--decompress  Decompress file
 -f,--force		  Force, overwrite existing output file
 -h,--help		  Print this help message
 -k,--keep		  Keep input file, do not delete
 -l,--loadavg 	  Load average determines max number processors to use
 -m#			  Where # is max memory usage in 1MB steps (default 100 = 100MB)
 -p#    		  Where # is the number of processors (default: autodetect)
 -q,--quiet		  Quiet mode (default)
 -r,--read		  Read entire input file into RAM and split between processors
 -S#              Child thread stack size in 1KB steps (default stack size if unspecified)
 -t,--test		  Test compressed file integrity
 -v,--verbose	  Verbose mode
 -V     		  Display version info for pbzip2 then exit
 -z,--compress	  Compress file (default)
 -1,--fast ... -9,--best 	Set BWT block size to 100k .. 900k (default 900k).
 --ignore-trailing-garbage=# Ignore trailing garbage flag (1 - ignored; 0 - forbidden)


Example:  pbzip2 myfile.tar

This example will compress the file "myfile.tar" into the compressed
file "myfile.tar.bz2".  It will use the autodetected # of processors
(or 2 processors if autodetect not supported) with the default file
block size of 900k and default BWT block size of 900k.


Example:  pbzip2 -b15vk myfile.tar

This example will compress the file "myfile.tar" into the compressed
file "myfile.tar.bz2".  It will use the autodetected # of processors
(or 2 processors if autodetect not supported) with a file block
size of 1500k and a BWT block size of 900k.  Verbose mode will be
enabled so progress and other messages will be output to the display
and the file myfile.tar will not be deleted after compression is
finished.


Example:  pbzip2 -p4 -r -5 myfile.tar second*.txt

This example will compress the file "myfile.tar" into the compressed
file "myfile.tar.bz2".  It will use 4 processors with a BWT block
size of 500k.  The file block size will be the size of "myfile.tar"
divided by 4 (# of processors) so that the data will be split
evenly among each processor.  This requires you have enough RAM for
pbzip2 to read the entire file into memory for compression.  pbzip2
will then use the same options to compress all other files that
match the wildcard "second*.txt" in that directory.


Example:  pbzip2 -l myfile.tar

This example will compress the file "myfile.tar" into the compressed
file "myfile.tar.bz2".  It will use the autodetected # of processors
(or 2 processors if autodetect not supported) if the 1 minute load
average is less than 0.5, otherwise it will select the maximum # of
processors so that only idle processors are used for pbzip2.  If the
system has 4 processors and the load average is 2.00, then pbzip2
will use 2 processors to compress the data.


Example: tar cf myfile.tar.bz2 --use-compress-prog=pbzip2 dir_to_compress/
Example: tar -c directory_to_compress/ | pbzip2 -c > myfile.tar.bz2

These examples will compress the data being given to pbzip2 via pipe
from TAR into the compressed file "myfile.tar.bz2".  It will use the
autodetected # of processors (or 2 processors if autodetect not
supported) with the default file block size of 900k and default BWT
block size of 900k.  TAR is collecting all of the files from the
"directory_to_compress/" directory and passing the data to pbzip2 as
it works.


Example:  pbzip2 -d -m500 myfile.tar.bz2

This example will decompress the file "myfile.tar.bz2" into the
decompressed file "myfile.tar".  It will use the autodetected # of
processors (or 2 processors if autodetect not supported).  It will use
a maximum of 500MB of memory for decompression.
The switches -b, -r, -t, and -1..-9 are not valid for decompression.