diff options
Diffstat (limited to 'winsup/bz2lib/manual.texi')
| -rw-r--r-- | winsup/bz2lib/manual.texi | 2215 |
1 files changed, 2215 insertions, 0 deletions
diff --git a/winsup/bz2lib/manual.texi b/winsup/bz2lib/manual.texi new file mode 100644 index 000000000..336776ab8 --- /dev/null +++ b/winsup/bz2lib/manual.texi @@ -0,0 +1,2215 @@ +\input texinfo @c -*- Texinfo -*- +@setfilename bzip2.info + +@ignore +This file documents bzip2 version 1.0, and associated library +libbzip2, written by Julian Seward (jseward@acm.org). + +Copyright (C) 1996-2000 Julian R Seward + +Permission is granted to make and distribute verbatim copies of +this manual provided the copyright notice and this permission notice +are preserved on all copies. + +Permission is granted to copy and distribute translations of this manual +into another language, under the above conditions for verbatim copies. +@end ignore + +@ifinfo +@format +START-INFO-DIR-ENTRY +* Bzip2: (bzip2). A program and library for data compression. +END-INFO-DIR-ENTRY +@end format + +@end ifinfo + +@iftex +@c @finalout +@settitle bzip2 and libbzip2 +@titlepage +@title bzip2 and libbzip2 +@subtitle a program and library for data compression +@subtitle copyright (C) 1996-2000 Julian Seward +@subtitle version 1.0 of 21 March 2000 +@author Julian Seward + +@end titlepage + +@parindent 0mm +@parskip 2mm + +@end iftex +@node Top, Overview, (dir), (dir) + +This program, @code{bzip2}, +and associated library @code{libbzip2}, are +Copyright (C) 1996-2000 Julian R Seward. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: +@itemize @bullet +@item + Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. +@item + The origin of this software must not be misrepresented; you must + not claim that you wrote the original software. If you use this + software in a product, an acknowledgment in the product + documentation would be appreciated but is not required. +@item + Altered source versions must be plainly marked as such, and must + not be misrepresented as being the original software. +@item + The name of the author may not be used to endorse or promote + products derived from this software without specific prior written + permission. +@end itemize +THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS +OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE +GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, +WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Julian Seward, Cambridge, UK. + +@code{jseward@@acm.org} + +@code{http://sourceware.cygnus.com/bzip2} + +@code{http://www.cacheprof.org} + +@code{http://www.muraroa.demon.co.uk} + +@code{bzip2}/@code{libbzip2} version 1.0 of 21 March 2000. + +PATENTS: To the best of my knowledge, @code{bzip2} does not use any patented +algorithms. However, I do not have the resources available to carry out +a full patent search. Therefore I cannot give any guarantee of the +above statement. + + + + + + + +@node Overview, Implementation, Top, Top +@chapter Introduction + +@code{bzip2} compresses files using the Burrows-Wheeler +block-sorting text compression algorithm, and Huffman coding. +Compression is generally considerably better than that +achieved by more conventional LZ77/LZ78-based compressors, +and approaches the performance of the PPM family of statistical compressors. + +@code{bzip2} is built on top of @code{libbzip2}, a flexible library +for handling compressed data in the @code{bzip2} format. This manual +describes both how to use the program and +how to work with the library interface. Most of the +manual is devoted to this library, not the program, +which is good news if your interest is only in the program. + +Chapter 2 describes how to use @code{bzip2}; this is the only part +you need to read if you just want to know how to operate the program. +Chapter 3 describes the programming interfaces in detail, and +Chapter 4 records some miscellaneous notes which I thought +ought to be recorded somewhere. + + +@chapter How to use @code{bzip2} + +This chapter contains a copy of the @code{bzip2} man page, +and nothing else. + +@quotation + +@unnumberedsubsubsec NAME +@itemize +@item @code{bzip2}, @code{bunzip2} +- a block-sorting file compressor, v1.0 +@item @code{bzcat} +- decompresses files to stdout +@item @code{bzip2recover} +- recovers data from damaged bzip2 files +@end itemize + +@unnumberedsubsubsec SYNOPSIS +@itemize +@item @code{bzip2} [ -cdfkqstvzVL123456789 ] [ filenames ... ] +@item @code{bunzip2} [ -fkvsVL ] [ filenames ... ] +@item @code{bzcat} [ -s ] [ filenames ... ] +@item @code{bzip2recover} filename +@end itemize + +@unnumberedsubsubsec DESCRIPTION + +@code{bzip2} compresses files using the Burrows-Wheeler block sorting +text compression algorithm, and Huffman coding. Compression is +generally considerably better than that achieved by more conventional +LZ77/LZ78-based compressors, and approaches the performance of the PPM +family of statistical compressors. + +The command-line options are deliberately very similar to those of GNU +@code{gzip}, but they are not identical. + +@code{bzip2} expects a list of file names to accompany the command-line +flags. Each file is replaced by a compressed version of itself, with +the name @code{original_name.bz2}. Each compressed file has the same +modification date, permissions, and, when possible, ownership as the +corresponding original, so that these properties can be correctly +restored at decompression time. File name handling is naive in the +sense that there is no mechanism for preserving original file names, +permissions, ownerships or dates in filesystems which lack these +concepts, or have serious file name length restrictions, such as MS-DOS. + +@code{bzip2} and @code{bunzip2} will by default not overwrite existing +files. If you want this to happen, specify the @code{-f} flag. + +If no file names are specified, @code{bzip2} compresses from standard +input to standard output. In this case, @code{bzip2} will decline to +write compressed output to a terminal, as this would be entirely +incomprehensible and therefore pointless. + +@code{bunzip2} (or @code{bzip2 -d}) decompresses all +specified files. Files which were not created by @code{bzip2} +will be detected and ignored, and a warning issued. +@code{bzip2} attempts to guess the filename for the decompressed file +from that of the compressed file as follows: +@itemize +@item @code{filename.bz2 } becomes @code{filename} +@item @code{filename.bz } becomes @code{filename} +@item @code{filename.tbz2} becomes @code{filename.tar} +@item @code{filename.tbz } becomes @code{filename.tar} +@item @code{anyothername } becomes @code{anyothername.out} +@end itemize +If the file does not end in one of the recognised endings, +@code{.bz2}, @code{.bz}, +@code{.tbz2} or @code{.tbz}, @code{bzip2} complains that it cannot +guess the name of the original file, and uses the original name +with @code{.out} appended. + +As with compression, supplying no +filenames causes decompression from standard input to standard output. + +@code{bunzip2} will correctly decompress a file which is the +concatenation of two or more compressed files. The result is the +concatenation of the corresponding uncompressed files. Integrity +testing (@code{-t}) of concatenated compressed files is also supported. + +You can also compress or decompress files to the standard output by +giving the @code{-c} flag. Multiple files may be compressed and +decompressed like this. The resulting outputs are fed sequentially to +stdout. Compression of multiple files in this manner generates a stream +containing multiple compressed file representations. Such a stream +can be decompressed correctly only by @code{bzip2} version 0.9.0 or +later. Earlier versions of @code{bzip2} will stop after decompressing +the first file in the stream. + +@code{bzcat} (or @code{bzip2 -dc}) decompresses all specified files to +the standard output. + +@code{bzip2} will read arguments from the environment variables +@code{BZIP2} and @code{BZIP}, in that order, and will process them +before any arguments read from the command line. This gives a +convenient way to supply default arguments. + +Compression is always performed, even if the compressed file is slightly +larger than the original. Files of less than about one hundred bytes +tend to get larger, since the compression mechanism has a constant +overhead in the region of 50 bytes. Random data (including the output +of most file compressors) is coded at about 8.05 bits per byte, giving +an expansion of around 0.5%. + +As a self-check for your protection, @code{bzip2} uses 32-bit CRCs to +make sure that the decompressed version of a file is identical to the +original. This guards against corruption of the compressed data, and +against undetected bugs in @code{bzip2} (hopefully very unlikely). The +chances of data corruption going undetected is microscopic, about one +chance in four billion for each file processed. Be aware, though, that +the check occurs upon decompression, so it can only tell you that +something is wrong. It can't help you recover the original uncompressed +data. You can use @code{bzip2recover} to try to recover data from +damaged files. + +Return values: 0 for a normal exit, 1 for environmental problems (file +not found, invalid flags, I/O errors, &c), 2 to indicate a corrupt +compressed file, 3 for an internal consistency error (eg, bug) which +caused @code{bzip2} to panic. + + +@unnumberedsubsubsec OPTIONS +@table @code +@item -c --stdout +Compress or decompress to standard output. +@item -d --decompress +Force decompression. @code{bzip2}, @code{bunzip2} and @code{bzcat} are +really the same program, and the decision about what actions to take is +done on the basis of which name is used. This flag overrides that +mechanism, and forces bzip2 to decompress. +@item -z --compress +The complement to @code{-d}: forces compression, regardless of the +invokation name. +@item -t --test +Check integrity of the specified file(s), but don't decompress them. +This really performs a trial decompression and throws away the result. +@item -f --force +Force overwrite of output files. Normally, @code{bzip2} will not overwrite +existing output files. Also forces @code{bzip2} to break hard links +to files, which it otherwise wouldn't do. +@item -k --keep +Keep (don't delete) input files during compression +or decompression. +@item -s --small +Reduce memory usage, for compression, decompression and testing. Files +are decompressed and tested using a modified algorithm which only +requires 2.5 bytes per block byte. This means any file can be +decompressed in 2300k of memory, albeit at about half the normal speed. + +During compression, @code{-s} selects a block size of 200k, which limits +memory use to around the same figure, at the expense of your compression +ratio. In short, if your machine is low on memory (8 megabytes or +less), use -s for everything. See MEMORY MANAGEMENT below. +@item -q --quiet +Suppress non-essential warning messages. Messages pertaining to +I/O errors and other critical events will not be suppressed. +@item -v --verbose +Verbose mode -- show the compression ratio for each file processed. +Further @code{-v}'s increase the verbosity level, spewing out lots of +information which is primarily of interest for diagnostic purposes. +@item -L --license -V --version +Display the software version, license terms and conditions. +@item -1 to -9 +Set the block size to 100 k, 200 k .. 900 k when compressing. Has no +effect when decompressing. See MEMORY MANAGEMENT below. +@item -- +Treats all subsequent arguments as file names, even if they start +with a dash. This is so you can handle files with names beginning +with a dash, for example: @code{bzip2 -- -myfilename}. +@item --repetitive-fast +@item --repetitive-best +These flags are redundant in versions 0.9.5 and above. They provided +some coarse control over the behaviour of the sorting algorithm in +earlier versions, which was sometimes useful. 0.9.5 and above have an +improved algorithm which renders these flags irrelevant. +@end table + + +@unnumberedsubsubsec MEMORY MANAGEMENT + +@code{bzip2} compresses large files in blocks. The block size affects +both the compression ratio achieved, and the amount of memory needed for +compression and decompression. The flags @code{-1} through @code{-9} +specify the block size to be 100,000 bytes through 900,000 bytes (the +default) respectively. At decompression time, the block size used for +compression is read from the header of the compressed file, and +@code{bunzip2} then allocates itself just enough memory to decompress +the file. Since block sizes are stored in compressed files, it follows +that the flags @code{-1} to @code{-9} are irrelevant to and so ignored +during decompression. + +Compression and decompression requirements, in bytes, can be estimated +as: +@example + Compression: 400k + ( 8 x block size ) + + Decompression: 100k + ( 4 x block size ), or + 100k + ( 2.5 x block size ) +@end example +Larger block sizes give rapidly diminishing marginal returns. Most of +the compression comes from the first two or three hundred k of block +size, a fact worth bearing in mind when using @code{bzip2} on small machines. +It is also important to appreciate that the decompression memory +requirement is set at compression time by the choice of block size. + +For files compressed with the default 900k block size, @code{bunzip2} +will require about 3700 kbytes to decompress. To support decompression +of any file on a 4 megabyte machine, @code{bunzip2} has an option to +decompress using approximately half this amount of memory, about 2300 +kbytes. Decompression speed is also halved, so you should use this +option only where necessary. The relevant flag is @code{-s}. + +In general, try and use the largest block size memory constraints allow, +since that maximises the compression achieved. Compression and +decompression speed are virtually unaffected by block size. + +Another significant point applies to files which fit in a single block +-- that means most files you'd encounter using a large block size. The +amount of real memory touched is proportional to the size of the file, +since the file is smaller than a block. For example, compressing a file +20,000 bytes long with the flag @code{-9} will cause the compressor to +allocate around 7600k of memory, but only touch 400k + 20000 * 8 = 560 +kbytes of it. Similarly, the decompressor will allocate 3700k but only +touch 100k + 20000 * 4 = 180 kbytes. + +Here is a table which summarises the maximum memory usage for different +block sizes. Also recorded is the total compressed size for 14 files of +the Calgary Text Compression Corpus totalling 3,141,622 bytes. This +column gives some feel for how compression varies with block size. +These figures tend to understate the advantage of larger block sizes for +larger files, since the Corpus is dominated by smaller files. +@example + Compress Decompress Decompress Corpus + Flag usage usage -s usage Size + + -1 1200k 500k 350k 914704 + -2 2000k 900k 600k 877703 + -3 2800k 1300k 850k 860338 + -4 3600k 1700k 1100k 846899 + -5 4400k 2100k 1350k 845160 + -6 5200k 2500k 1600k 838626 + -7 6100k 2900k 1850k 834096 + -8 6800k 3300k 2100k 828642 + -9 7600k 3700k 2350k 828642 +@end example + +@unnumberedsubsubsec RECOVERING DATA FROM DAMAGED FILES + +@code{bzip2} compresses files in blocks, usually 900kbytes long. Each +block is handled independently. If a media or transmission error causes +a multi-block @code{.bz2} file to become damaged, it may be possible to +recover data from the undamaged blocks in the file. + +The compressed representation of each block is delimited by a 48-bit +pattern, which makes it possible to find the block boundaries with +reasonable certainty. Each block also carries its own 32-bit CRC, so +damaged blocks can be distinguished from undamaged ones. + +@code{bzip2recover} is a simple program whose purpose is to search for +blocks in @code{.bz2} files, and write each block out into its own +@code{.bz2} file. You can then use @code{bzip2 -t} to test the +integrity of the resulting files, and decompress those which are +undamaged. + +@code{bzip2recover} +takes a single argument, the name of the damaged file, +and writes a number of files @code{rec0001file.bz2}, + @code{rec0002file.bz2}, etc, containing the extracted blocks. + The output filenames are designed so that the use of + wildcards in subsequent processing -- for example, +@code{bzip2 -dc rec*file.bz2 > recovered_data} -- lists the files in + the correct order. + +@code{bzip2recover} should be of most use dealing with large @code{.bz2} + files, as these will contain many blocks. It is clearly + futile to use it on damaged single-block files, since a + damaged block cannot be recovered. If you wish to minimise +any potential data loss through media or transmission errors, +you might consider compressing with a smaller + block size. + + +@unnumberedsubsubsec PERFORMANCE NOTES + +The sorting phase of compression gathers together similar strings in the +file. Because of this, files containing very long runs of repeated +symbols, like "aabaabaabaab ..." (repeated several hundred times) may +compress more slowly than normal. Versions 0.9.5 and above fare much +better than previous versions in this respect. The ratio between +worst-case and average-case compression time is in the region of 10:1. +For previous versions, this figure was more like 100:1. You can use the +@code{-vvvv} option to monitor progress in great detail, if you want. + +Decompression speed is unaffected by these phenomena. + +@code{bzip2} usually allocates several megabytes of memory to operate +in, and then charges all over it in a fairly random fashion. This means +that performance, both for compressing and decompressing, is largely +determined by the speed at which your machine can service cache misses. +Because of this, small changes to the code to reduce the miss rate have +been observed to give disproportionately large performance improvements. +I imagine @code{bzip2} will perform best on machines with very large +caches. + + +@unnumberedsubsubsec CAVEATS + +I/O error messages are not as helpful as they could be. @code{bzip2} +tries hard to detect I/O errors and exit cleanly, but the details of +what the problem is sometimes seem rather misleading. + +This manual page pertains to version 1.0 of @code{bzip2}. Compressed +data created by this version is entirely forwards and backwards +compatible with the previous public releases, versions 0.1pl2, 0.9.0 and +0.9.5, but with the following exception: 0.9.0 and above can correctly +decompress multiple concatenated compressed files. 0.1pl2 cannot do +this; it will stop after decompressing just the first file in the +stream. + +@code{bzip2recover} uses 32-bit integers to represent bit positions in +compressed files, so it cannot handle compressed files more than 512 +megabytes long. This could easily be fixed. + + +@unnumberedsubsubsec AUTHOR +Julian Seward, @code{jseward@@acm.org}. + +The ideas embodied in @code{bzip2} are due to (at least) the following +people: Michael Burrows and David Wheeler (for the block sorting +transformation), David Wheeler (again, for the Huffman coder), Peter +Fenwick (for the structured coding model in the original @code{bzip}, +and many refinements), and Alistair Moffat, Radford Neal and Ian Witten +(for the arithmetic coder in the original @code{bzip}). I am much +indebted for their help, support and advice. See the manual in the +source distribution for pointers to sources of documentation. Christian +von Roques encouraged me to look for faster sorting algorithms, so as to +speed up compression. Bela Lubkin encouraged me to improve the +worst-case compression performance. Many people sent patches, helped +with portability problems, lent machines, gave advice and were generally +helpful. + +@end quotation + + + + +@chapter Programming with @code{libbzip2} + +This chapter describes the programming interface to @code{libbzip2}. + +For general background information, particularly about memory +use and performance aspects, you'd be well advised to read Chapter 2 +as well. + +@section Top-level structure + +@code{libbzip2} is a flexible library for compressing and decompressing +data in the @code{bzip2} data format. Although packaged as a single +entity, it helps to regard the library as three separate parts: the low +level interface, and the high level interface, and some utility +functions. + +The structure of @code{libbzip2}'s interfaces is similar to +that of Jean-loup Gailly's and Mark Adler's excellent @code{zlib} +library. + +All externally visible symbols have names beginning @code{BZ2_}. +This is new in version 1.0. The intention is to minimise pollution +of the namespaces of library clients. + +@subsection Low-level summary + +This interface provides services for compressing and decompressing +data in memory. There's no provision for dealing with files, streams +or any other I/O mechanisms, just straight memory-to-memory work. +In fact, this part of the library can be compiled without inclusion +of @code{stdio.h}, which may be helpful for embedded applications. + +The low-level part of the library has no global variables and +is therefore thread-safe. + +Six routines make up the low level interface: +@code{BZ2_bzCompressInit}, @code{BZ2_bzCompress}, and @* @code{BZ2_bzCompressEnd} +for compression, +and a corresponding trio @code{BZ2_bzDecompressInit}, @* @code{BZ2_bzDecompress} +and @code{BZ2_bzDecompressEnd} for decompression. +The @code{*Init} functions allocate +memory for compression/decompression and do other +initialisations, whilst the @code{*End} functions close down operations +and release memory. + +The real work is done by @code{BZ2_bzCompress} and @code{BZ2_bzDecompress}. +These compress and decompress data from a user-supplied input buffer +to a user-supplied output buffer. These buffers can be any size; +arbitrary quantities of data are handled by making repeated calls +to these functions. This is a flexible mechanism allowing a +consumer-pull style of activity, or producer-push, or a mixture of +both. + + + +@subsection High-level summary + +This interface provides some handy wrappers around the low-level +interface to facilitate reading and writing @code{bzip2} format +files (@code{.bz2} files). The routines provide hooks to facilitate +reading files in which the @code{bzip2} data stream is embedded +within some larger-scale file structure, or where there are +multiple @code{bzip2} data streams concatenated end-to-end. + +For reading files, @code{BZ2_bzReadOpen}, @code{BZ2_bzRead}, +@code{BZ2_bzReadClose} and @* @code{BZ2_bzReadGetUnused} are supplied. For +writing files, @code{BZ2_bzWriteOpen}, @code{BZ2_bzWrite} and +@code{BZ2_bzWriteFinish} are available. + +As with the low-level library, no global variables are used +so the library is per se thread-safe. However, if I/O errors +occur whilst reading or writing the underlying compressed files, +you may have to consult @code{errno} to determine the cause of +the error. In that case, you'd need a C library which correctly +supports @code{errno} in a multithreaded environment. + +To make the library a little simpler and more portable, +@code{BZ2_bzReadOpen} and @code{BZ2_bzWriteOpen} require you to pass them file +handles (@code{FILE*}s) which have previously been opened for reading or +writing respectively. That avoids portability problems associated with +file operations and file attributes, whilst not being much of an +imposition on the programmer. + + + +@subsection Utility functions summary +For very simple needs, @code{BZ2_bzBuffToBuffCompress} and +@code{BZ2_bzBuffToBuffDecompress} are provided. These compress +data in memory from one buffer to another buffer in a single +function call. You should assess whether these functions +fulfill your memory-to-memory compression/decompression +requirements before investing effort in understanding the more +general but more complex low-level interface. + +Yoshioka Tsuneo (@code{QWF00133@@niftyserve.or.jp} / +@code{tsuneo-y@@is.aist-nara.ac.jp}) has contributed some functions to +give better @code{zlib} compatibility. These functions are +@code{BZ2_bzopen}, @code{BZ2_bzread}, @code{BZ2_bzwrite}, @code{BZ2_bzflush}, +@code{BZ2_bzclose}, +@code{BZ2_bzerror} and @code{BZ2_bzlibVersion}. You may find these functions +more convenient for simple file reading and writing, than those in the +high-level interface. These functions are not (yet) officially part of +the library, and are minimally documented here. If they break, you +get to keep all the pieces. I hope to document them properly when time +permits. + +Yoshioka also contributed modifications to allow the library to be +built as a Windows DLL. + + +@section Error handling + +The library is designed to recover cleanly in all situations, including +the worst-case situation of decompressing random data. I'm not +100% sure that it can always do this, so you might want to add +a signal handler to catch segmentation violations during decompression +if you are feeling especially paranoid. I would be interested in +hearing more about the robustness of the library to corrupted +compressed data. + +Version 1.0 is much more robust in this respect than +0.9.0 or 0.9.5. Investigations with Checker (a tool for +detecting problems with memory management, similar to Purify) +indicate that, at least for the few files I tested, all single-bit +errors in the decompressed data are caught properly, with no +segmentation faults, no reads of uninitialised data and no +out of range reads or writes. So it's certainly much improved, +although I wouldn't claim it to be totally bombproof. + +The file @code{bzlib.h} contains all definitions needed to use +the library. In particular, you should definitely not include +@code{bzlib_private.h}. + +In @code{bzlib.h}, the various return values are defined. The following +list is not intended as an exhaustive description of the circumstances +in which a given value may be returned -- those descriptions are given +later. Rather, it is intended to convey the rough meaning of each +return value. The first five actions are normal and not intended to +denote an error situation. +@table @code +@item BZ_OK +The requested action was completed successfully. +@item BZ_RUN_OK +@itemx BZ_FLUSH_OK +@itemx BZ_FINISH_OK +In @code{BZ2_bzCompress}, the requested flush/finish/nothing-special action +was completed successfully. +@item BZ_STREAM_END +Compression of data was completed, or the logical stream end was +detected during decompression. +@end table + +The following return values indicate an error of some kind. +@table @code +@item BZ_CONFIG_ERROR +Indicates that the library has been improperly compiled on your +platform -- a major configuration error. Specifically, it means +that @code{sizeof(char)}, @code{sizeof(short)} and @code{sizeof(int)} +are not 1, 2 and 4 respectively, as they should be. Note that the +library should still work properly on 64-bit platforms which follow +the LP64 programming model -- that is, where @code{sizeof(long)} +and @code{sizeof(void*)} are 8. Under LP64, @code{sizeof(int)} is +still 4, so @code{libbzip2}, which doesn't use the @code{long} type, +is OK. +@item BZ_SEQUENCE_ERROR +When using the library, it is important to call the functions in the +correct sequence and with data structures (buffers etc) in the correct +states. @code{libbzip2} checks as much as it can to ensure this is +happening, and returns @code{BZ_SEQUENCE_ERROR} if not. Code which +complies precisely with the function semantics, as detailed below, +should never receive this value; such an event denotes buggy code +which you should investigate. +@item BZ_PARAM_ERROR +Returned when a parameter to a function call is out of range +or otherwise manifestly incorrect. As with @code{BZ_SEQUENCE_ERROR}, +this denotes a bug in the client code. The distinction between +@code{BZ_PARAM_ERROR} and @code{BZ_SEQUENCE_ERROR} is a bit hazy, but still worth +making. +@item BZ_MEM_ERROR +Returned when a request to allocate memory failed. Note that the +quantity of memory needed to decompress a stream cannot be determined +until the stream's header has been read. So @code{BZ2_bzDecompress} and +@code{BZ2_bzRead} may return @code{BZ_MEM_ERROR} even though some of +the compressed data has been read. The same is not true for +compression; once @code{BZ2_bzCompressInit} or @code{BZ2_bzWriteOpen} have +successfully completed, @code{BZ_MEM_ERROR} cannot occur. +@item BZ_DATA_ERROR +Returned when a data integrity error is detected during decompression. +Most importantly, this means when stored and computed CRCs for the +data do not match. This value is also returned upon detection of any +other anomaly in the compressed data. +@item BZ_DATA_ERROR_MAGIC +As a special case of @code{BZ_DATA_ERROR}, it is sometimes useful to +know when the compressed stream does not start with the correct +magic bytes (@code{'B' 'Z' 'h'}). +@item BZ_IO_ERROR +Returned by @code{BZ2_bzRead} and @code{BZ2_bzWrite} when there is an error +reading or writing in the compressed file, and by @code{BZ2_bzReadOpen} +and @code{BZ2_bzWriteOpen} for attempts to use a file for which the +error indicator (viz, @code{ferror(f)}) is set. +On receipt of @code{BZ_IO_ERROR}, the caller should consult +@code{errno} and/or @code{perror} to acquire operating-system +specific information about the problem. +@item BZ_UNEXPECTED_EOF +Returned by @code{BZ2_bzRead} when the compressed file finishes +before the logical end of stream is detected. +@item BZ_OUTBUFF_FULL +Returned by @code{BZ2_bzBuffToBuffCompress} and +@code{BZ2_bzBuffToBuffDecompress} to indicate that the output data +will not fit into the output buffer provided. +@end table + + + +@section Low-level interface + +@subsection @code{BZ2_bzCompressInit} +@example +typedef + struct @{ + char *next_in; + unsigned int avail_in; + unsigned int total_in_lo32; + unsigned int total_in_hi32; + + char *next_out; + unsigned int avail_out; + unsigned int total_out_lo32; + unsigned int total_out_hi32; + + void *state; + + void *(*bzalloc)(void *,int,int); + void (*bzfree)(void *,void *); + void *opaque; + @} + bz_stream; + +int BZ2_bzCompressInit ( bz_stream *strm, + int blockSize100k, + int verbosity, + int workFactor ); + +@end example + +Prepares for compression. The @code{bz_stream} structure +holds all data pertaining to the compression activity. +A @code{bz_stream} structure should be allocated and initialised +prior to the call. +The fields of @code{bz_stream} +comprise the entirety of the user-visible data. @code{state} +is a pointer to the private data structures required for compression. + +Custom memory allocators are supported, via fields @code{bzalloc}, +@code{bzfree}, +and @code{opaque}. The value +@code{opaque} is passed to as the first argument to +all calls to @code{bzalloc} and @code{bzfree}, but is +otherwise ignored by the library. +The call @code{bzalloc ( opaque, n, m )} is expected to return a +pointer @code{p} to +@code{n * m} bytes of memory, and @code{bzfree ( opaque, p )} +should free +that memory. + +If you don't want to use a custom memory allocator, set @code{bzalloc}, +@code{bzfree} and +@code{opaque} to @code{NULL}, +and the library will then use the standard @code{malloc}/@code{free} +routines. + +Before calling @code{BZ2_bzCompressInit}, fields @code{bzalloc}, +@code{bzfree} and @code{opaque} should +be filled appropriately, as just described. Upon return, the internal +state will have been allocated and initialised, and @code{total_in_lo32}, +@code{total_in_hi32}, @code{total_out_lo32} and +@code{total_out_hi32} will have been set to zero. +These four fields are used by the library +to inform the caller of the total amount of data passed into and out of +the library, respectively. You should not try to change them. +As of version 1.0, 64-bit counts are maintained, even on 32-bit +platforms, using the @code{_hi32} fields to store the upper 32 bits +of the count. So, for example, the total amount of data in +is @code{(total_in_hi32 << 32) + total_in_lo32}. + +Parameter @code{blockSize100k} specifies the block size to be used for +compression. It should be a value between 1 and 9 inclusive, and the +actual block size used is 100000 x this figure. 9 gives the best +compression but takes most memory. + +Parameter @code{verbosity} should be set to a number between 0 and 4 +inclusive. 0 is silent, and greater numbers give increasingly verbose +monitoring/debugging output. If the library has been compiled with +@code{-DBZ_NO_STDIO}, no such output will appear for any verbosity +setting. + +Parameter @code{workFactor} controls how the compression phase behaves +when presented with worst case, highly repetitive, input data. If +compression runs into difficulties caused by repetitive data, the +library switches from the standard sorting algorithm to a fallback +algorithm. The fallback is slower than the standard algorithm by +perhaps a factor of three, but always behaves reasonably, no matter how +bad the input. + +Lower values of @code{workFactor} reduce the amount of effort the +standard algorithm will expend before resorting to the fallback. You +should set this parameter carefully; too low, and many inputs will be +handled by the fallback algorithm and so compress rather slowly, too +high, and your average-to-worst case compression times can become very +large. The default value of 30 gives reasonable behaviour over a wide +range of circumstances. + +Allowable values range from 0 to 250 inclusive. 0 is a special case, +equivalent to using the default value of 30. + +Note that the compressed output generated is the same regardless of +whether or not the fallback algorithm is used. + +Be aware also that this parameter may disappear entirely in future +versions of the library. In principle it should be possible to devise a +good way to automatically choose which algorithm to use. Such a +mechanism would render the parameter obsolete. + +Possible return values: +@display + @code{BZ_CONFIG_ERROR} + if the library has been mis-compiled + @code{BZ_PARAM_ERROR} + if @code{strm} is @code{NULL} + or @code{blockSize} < 1 or @code{blockSize} > 9 + or @code{verbosity} < 0 or @code{verbosity} > 4 + or @code{workFactor} < 0 or @code{workFactor} > 250 + @code{BZ_MEM_ERROR} + if not enough memory is available + @code{BZ_OK} + otherwise +@end display +Allowable next actions: +@display + @code{BZ2_bzCompress} + if @code{BZ_OK} is returned + no specific action needed in case of error +@end display + +@subsection @code{BZ2_bzCompress} +@example + int BZ2_bzCompress ( bz_stream *strm, int action ); +@end example +Provides more input and/or output buffer space for the library. The +caller maintains input and output buffers, and calls @code{BZ2_bzCompress} to +transfer data between them. + +Before each call to @code{BZ2_bzCompress}, @code{next_in} should point at +the data to be compressed, and @code{avail_in} should indicate how many +bytes the library may read. @code{BZ2_bzCompress} updates @code{next_in}, +@code{avail_in} and @code{total_in} to reflect the number of bytes it +has read. + +Similarly, @code{next_out} should point to a buffer in which the +compressed data is to be placed, with @code{avail_out} indicating how +much output space is available. @code{BZ2_bzCompress} updates +@code{next_out}, @code{avail_out} and @code{total_out} to reflect the +number of bytes output. + +You may provide and remove as little or as much data as you like on each +call of @code{BZ2_bzCompress}. In the limit, it is acceptable to supply and +remove data one byte at a time, although this would be terribly +inefficient. You should always ensure that at least one byte of output +space is available at each call. + +A second purpose of @code{BZ2_bzCompress} is to request a change of mode of the +compressed stream. + +Conceptually, a compressed stream can be in one of four states: IDLE, +RUNNING, FLUSHING and FINISHING. Before initialisation +(@code{BZ2_bzCompressInit}) and after termination (@code{BZ2_bzCompressEnd}), a +stream is regarded as IDLE. + +Upon initialisation (@code{BZ2_bzCompressInit}), the stream is placed in the +RUNNING state. Subsequent calls to @code{BZ2_bzCompress} should pass +@code{BZ_RUN} as the requested action; other actions are illegal and +will result in @code{BZ_SEQUENCE_ERROR}. + +At some point, the calling program will have provided all the input data +it wants to. It will then want to finish up -- in effect, asking the +library to process any data it might have buffered internally. In this +state, @code{BZ2_bzCompress} will no longer attempt to read data from +@code{next_in}, but it will want to write data to @code{next_out}. +Because the output buffer supplied by the user can be arbitrarily small, +the finishing-up operation cannot necessarily be done with a single call +of @code{BZ2_bzCompress}. + +Instead, the calling program passes @code{BZ_FINISH} as an action to +@code{BZ2_bzCompress}. This changes the stream's state to FINISHING. Any +remaining input (ie, @code{next_in[0 .. avail_in-1]}) is compressed and +transferred to the output buffer. To do this, @code{BZ2_bzCompress} must be +called repeatedly until all the output has been consumed. At that +point, @code{BZ2_bzCompress} returns @code{BZ_STREAM_END}, and the stream's +state is set back to IDLE. @code{BZ2_bzCompressEnd} should then be +called. + +Just to make sure the calling program does not cheat, the library makes +a note of @code{avail_in} at the time of the first call to +@code{BZ2_bzCompress} which has @code{BZ_FINISH} as an action (ie, at the +time the program has announced its intention to not supply any more +input). By comparing this value with that of @code{avail_in} over +subsequent calls to @code{BZ2_bzCompress}, the library can detect any +attempts to slip in more data to compress. Any calls for which this is +detected will return @code{BZ_SEQUENCE_ERROR}. This indicates a +programming mistake which should be corrected. + +Instead of asking to finish, the calling program may ask +@code{BZ2_bzCompress} to take all the remaining input, compress it and +terminate the current (Burrows-Wheeler) compression block. This could +be useful for error control purposes. The mechanism is analogous to +that for finishing: call @code{BZ2_bzCompress} with an action of +@code{BZ_FLUSH}, remove output data, and persist with the +@code{BZ_FLUSH} action until the value @code{BZ_RUN} is returned. As +with finishing, @code{BZ2_bzCompress} detects any attempt to provide more +input data once the flush has begun. + +Once the flush is complete, the stream returns to the normal RUNNING +state. + +This all sounds pretty complex, but isn't really. Here's a table +which shows which actions are allowable in each state, what action +will be taken, what the next state is, and what the non-error return +values are. Note that you can't explicitly ask what state the +stream is in, but nor do you need to -- it can be inferred from the +values returned by @code{BZ2_bzCompress}. +@display +IDLE/@code{any} + Illegal. IDLE state only exists after @code{BZ2_bzCompressEnd} or + before @code{BZ2_bzCompressInit}. + Return value = @code{BZ_SEQUENCE_ERROR} + +RUNNING/@code{BZ_RUN} + Compress from @code{next_in} to @code{next_out} as much as possible. + Next state = RUNNING + Return value = @code{BZ_RUN_OK} + +RUNNING/@code{BZ_FLUSH} + Remember current value of @code{next_in}. Compress from @code{next_in} + to @code{next_out} as much as possible, but do not accept any more input. + Next state = FLUSHING + Return value = @code{BZ_FLUSH_OK} + +RUNNING/@code{BZ_FINISH} + Remember current value of @code{next_in}. Compress from @code{next_in} + to @code{next_out} as much as possible, but do not accept any more input. + Next state = FINISHING + Return value = @code{BZ_FINISH_OK} + +FLUSHING/@code{BZ_FLUSH} + Compress from @code{next_in} to @code{next_out} as much as possible, + but do not accept any more input. + If all the existing input has been used up and all compressed + output has been removed + Next state = RUNNING; Return value = @code{BZ_RUN_OK} + else + Next state = FLUSHING; Return value = @code{BZ_FLUSH_OK} + +FLUSHING/other + Illegal. + Return value = @code{BZ_SEQUENCE_ERROR} + +FINISHING/@code{BZ_FINISH} + Compress from @code{next_in} to @code{next_out} as much as possible, + but to not accept any more input. + If all the existing input has been used up and all compressed + output has been removed + Next state = IDLE; Return value = @code{BZ_STREAM_END} + else + Next state = FINISHING; Return value = @code{BZ_FINISHING} + +FINISHING/other + Illegal. + Return value = @code{BZ_SEQUENCE_ERROR} +@end display + +That still looks complicated? Well, fair enough. The usual sequence +of calls for compressing a load of data is: +@itemize @bullet +@item Get started with @code{BZ2_bzCompressInit}. +@item Shovel data in and shlurp out its compressed form using zero or more +calls of @code{BZ2_bzCompress} with action = @code{BZ_RUN}. +@item Finish up. +Repeatedly call @code{BZ2_bzCompress} with action = @code{BZ_FINISH}, +copying out the compressed output, until @code{BZ_STREAM_END} is returned. +@item Close up and go home. Call @code{BZ2_bzCompressEnd}. +@end itemize +If the data you want to compress fits into your input buffer all +at once, you can skip the calls of @code{BZ2_bzCompress ( ..., BZ_RUN )} and +just do the @code{BZ2_bzCompress ( ..., BZ_FINISH )} calls. + +All required memory is allocated by @code{BZ2_bzCompressInit}. The +compression library can accept any data at all (obviously). So you +shouldn't get any error return values from the @code{BZ2_bzCompress} calls. +If you do, they will be @code{BZ_SEQUENCE_ERROR}, and indicate a bug in +your programming. + +Trivial other possible return values: +@display + @code{BZ_PARAM_ERROR} + if @code{strm} is @code{NULL}, or @code{strm->s} is @code{NULL} +@end display + +@subsection @code{BZ2_bzCompressEnd} +@example +int BZ2_bzCompressEnd ( bz_stream *strm ); +@end example +Releases all memory associated with a compression stream. + +Possible return values: +@display + @code{BZ_PARAM_ERROR} if @code{strm} is @code{NULL} or @code{strm->s} is @code{NULL} + @code{BZ_OK} otherwise +@end display + + +@subsection @code{BZ2_bzDecompressInit} +@example +int BZ2_bzDecompressInit ( bz_stream *strm, int verbosity, int small ); +@end example +Prepares for decompression. As with @code{BZ2_bzCompressInit}, a +@code{bz_stream} record should be allocated and initialised before the +call. Fields @code{bzalloc}, @code{bzfree} and @code{opaque} should be +set if a custom memory allocator is required, or made @code{NULL} for +the normal @code{malloc}/@code{free} routines. Upon return, the internal +state will have been initialised, and @code{total_in} and +@code{total_out} will be zero. + +For the meaning of parameter @code{verbosity}, see @code{BZ2_bzCompressInit}. + +If @code{small} is nonzero, the library will use an alternative +decompression algorithm which uses less memory but at the cost of +decompressing more slowly (roughly speaking, half the speed, but the +maximum memory requirement drops to around 2300k). See Chapter 2 for +more information on memory management. + +Note that the amount of memory needed to decompress +a stream cannot be determined until the stream's header has been read, +so even if @code{BZ2_bzDecompressInit} succeeds, a subsequent +@code{BZ2_bzDecompress} could fail with @code{BZ_MEM_ERROR}. + +Possible return values: +@display + @code{BZ_CONFIG_ERROR} + if the library has been mis-compiled + @code{BZ_PARAM_ERROR} + if @code{(small != 0 && small != 1)} + or @code{(verbosity < 0 || verbosity > 4)} + @code{BZ_MEM_ERROR} + if insufficient memory is available +@end display + +Allowable next actions: +@display + @code{BZ2_bzDecompress} + if @code{BZ_OK} was returned + no specific action required in case of error +@end display + + + +@subsection @code{BZ2_bzDecompress} +@example +int BZ2_bzDecompress ( bz_stream *strm ); +@end example +Provides more input and/out output buffer space for the library. The +caller maintains input and output buffers, and uses @code{BZ2_bzDecompress} +to transfer data between them. + +Before each call to @code{BZ2_bzDecompress}, @code{next_in} +should point at the compressed data, +and @code{avail_in} should indicate how many bytes the library +may read. @code{BZ2_bzDecompress} updates @code{next_in}, @code{avail_in} +and @code{total_in} +to reflect the number of bytes it has read. + +Similarly, @code{next_out} should point to a buffer in which the uncompressed +output is to be placed, with @code{avail_out} indicating how much output space +is available. @code{BZ2_bzCompress} updates @code{next_out}, +@code{avail_out} and @code{total_out} to reflect +the number of bytes output. + +You may provide and remove as little or as much data as you like on +each call of @code{BZ2_bzDecompress}. +In the limit, it is acceptable to +supply and remove data one byte at a time, although this would be +terribly inefficient. You should always ensure that at least one +byte of output space is available at each call. + +Use of @code{BZ2_bzDecompress} is simpler than @code{BZ2_bzCompress}. + +You should provide input and remove output as described above, and +repeatedly call @code{BZ2_bzDecompress} until @code{BZ_STREAM_END} is +returned. Appearance of @code{BZ_STREAM_END} denotes that +@code{BZ2_bzDecompress} has detected the logical end of the compressed +stream. @code{BZ2_bzDecompress} will not produce @code{BZ_STREAM_END} until +all output data has been placed into the output buffer, so once +@code{BZ_STREAM_END} appears, you are guaranteed to have available all +the decompressed output, and @code{BZ2_bzDecompressEnd} can safely be +called. + +If case of an error return value, you should call @code{BZ2_bzDecompressEnd} +to clean up and release memory. + +Possible return values: +@display + @code{BZ_PARAM_ERROR} + if @code{strm} is @code{NULL} or @code{strm->s} is @code{NULL} + or @code{strm->avail_out < 1} + @code{BZ_DATA_ERROR} + if a data integrity error is detected in the compressed stream + @code{BZ_DATA_ERROR_MAGIC} + if the compressed stream doesn't begin with the right magic bytes + @code{BZ_MEM_ERROR} + if there wasn't enough memory available + @code{BZ_STREAM_END} + if the logical end of the data stream was detected and all + output in has been consumed, eg @code{s->avail_out > 0} + @code{BZ_OK} + otherwise +@end display +Allowable next actions: +@display + @code{BZ2_bzDecompress} + if @code{BZ_OK} was returned + @code{BZ2_bzDecompressEnd} + otherwise +@end display + + +@subsection @code{BZ2_bzDecompressEnd} +@example +int BZ2_bzDecompressEnd ( bz_stream *strm ); +@end example +Releases all memory associated with a decompression stream. + +Possible return values: +@display + @code{BZ_PARAM_ERROR} + if @code{strm} is @code{NULL} or @code{strm->s} is @code{NULL} + @code{BZ_OK} + otherwise +@end display + +Allowable next actions: +@display + None. +@end display + + +@section High-level interface + +This interface provides functions for reading and writing +@code{bzip2} format files. First, some general points. + +@itemize @bullet +@item All of the functions take an @code{int*} first argument, + @code{bzerror}. + After each call, @code{bzerror} should be consulted first to determine + the outcome of the call. If @code{bzerror} is @code{BZ_OK}, + the call completed + successfully, and only then should the return value of the function + (if any) be consulted. If @code{bzerror} is @code{BZ_IO_ERROR}, + there was an error + reading/writing the underlying compressed file, and you should + then consult @code{errno}/@code{perror} to determine the + cause of the difficulty. + @code{bzerror} may also be set to various other values; precise details are + given on a per-function basis below. +@item If @code{bzerror} indicates an error + (ie, anything except @code{BZ_OK} and @code{BZ_STREAM_END}), + you should immediately call @code{BZ2_bzReadClose} (or @code{BZ2_bzWriteClose}, + depending on whether you are attempting to read or to write) + to free up all resources associated + with the stream. Once an error has been indicated, behaviour of all calls + except @code{BZ2_bzReadClose} (@code{BZ2_bzWriteClose}) is undefined. + The implication is that (1) @code{bzerror} should + be checked after each call, and (2) if @code{bzerror} indicates an error, + @code{BZ2_bzReadClose} (@code{BZ2_bzWriteClose}) should then be called to clean up. +@item The @code{FILE*} arguments passed to + @code{BZ2_bzReadOpen}/@code{BZ2_bzWriteOpen} + should be set to binary mode. + Most Unix systems will do this by default, but other platforms, + including Windows and Mac, will not. If you omit this, you may + encounter problems when moving code to new platforms. +@item Memory allocation requests are handled by + @code{malloc}/@code{free}. + At present + there is no facility for user-defined memory allocators in the file I/O + functions (could easily be added, though). +@end itemize + + + +@subsection @code{BZ2_bzReadOpen} +@example + typedef void BZFILE; + + BZFILE *BZ2_bzReadOpen ( int *bzerror, FILE *f, + int small, int verbosity, + void *unused, int nUnused ); +@end example +Prepare to read compressed data from file handle @code{f}. @code{f} +should refer to a file which has been opened for reading, and for which +the error indicator (@code{ferror(f)})is not set. If @code{small} is 1, +the library will try to decompress using less memory, at the expense of +speed. + +For reasons explained below, @code{BZ2_bzRead} will decompress the +@code{nUnused} bytes starting at @code{unused}, before starting to read +from the file @code{f}. At most @code{BZ_MAX_UNUSED} bytes may be +supplied like this. If this facility is not required, you should pass +@code{NULL} and @code{0} for @code{unused} and n@code{Unused} +respectively. + +For the meaning of parameters @code{small} and @code{verbosity}, +see @code{BZ2_bzDecompressInit}. + +The amount of memory needed to decompress a file cannot be determined +until the file's header has been read. So it is possible that +@code{BZ2_bzReadOpen} returns @code{BZ_OK} but a subsequent call of +@code{BZ2_bzRead} will return @code{BZ_MEM_ERROR}. + +Possible assignments to @code{bzerror}: +@display + @code{BZ_CONFIG_ERROR} + if the library has been mis-compiled + @code{BZ_PARAM_ERROR} + if @code{f} is @code{NULL} + or @code{small} is neither @code{0} nor @code{1} + or @code{(unused == NULL && nUnused != 0)} + or @code{(unused != NULL && !(0 <= nUnused <= BZ_MAX_UNUSED))} + @code{BZ_IO_ERROR} + if @code{ferror(f)} is nonzero + @code{BZ_MEM_ERROR} + if insufficient memory is available + @code{BZ_OK} + otherwise. +@end display + +Possible return values: +@display + Pointer to an abstract @code{BZFILE} + if @code{bzerror} is @code{BZ_OK} + @code{NULL} + otherwise +@end display + +Allowable next actions: +@display + @code{BZ2_bzRead} + if @code{bzerror} is @code{BZ_OK} + @code{BZ2_bzClose} + otherwise +@end display + + +@subsection @code{BZ2_bzRead} +@example + int BZ2_bzRead ( int *bzerror, BZFILE *b, void *buf, int len ); +@end example +Reads up to @code{len} (uncompressed) bytes from the compressed file +@code{b} into +the buffer @code{buf}. If the read was successful, +@code{bzerror} is set to @code{BZ_OK} +and the number of bytes read is returned. If the logical end-of-stream +was detected, @code{bzerror} will be set to @code{BZ_STREAM_END}, +and the number +of bytes read is returned. All other @code{bzerror} values denote an error. + +@code{BZ2_bzRead} will supply @code{len} bytes, +unless the logical stream end is detected +or an error occurs. Because of this, it is possible to detect the +stream end by observing when the number of bytes returned is +less than the number +requested. Nevertheless, this is regarded as inadvisable; you should +instead check @code{bzerror} after every call and watch out for +@code{BZ_STREAM_END}. + +Internally, @code{BZ2_bzRead} copies data from the compressed file in chunks +of size @code{BZ_MAX_UNUSED} bytes +before decompressing it. If the file contains more bytes than strictly +needed to reach the logical end-of-stream, @code{BZ2_bzRead} will almost certainly +read some of the trailing data before signalling @code{BZ_SEQUENCE_END}. +To collect the read but unused data once @code{BZ_SEQUENCE_END} has +appeared, call @code{BZ2_bzReadGetUnused} immediately before @code{BZ2_bzReadClose}. + +Possible assignments to @code{bzerror}: +@display + @code{BZ_PARAM_ERROR} + if @code{b} is @code{NULL} or @code{buf} is @code{NULL} or @code{len < 0} + @code{BZ_SEQUENCE_ERROR} + if @code{b} was opened with @code{BZ2_bzWriteOpen} + @code{BZ_IO_ERROR} + if there is an error reading from the compressed file + @code{BZ_UNEXPECTED_EOF} + if the compressed file ended before the logical end-of-stream was detected + @code{BZ_DATA_ERROR} + if a data integrity error was detected in the compressed stream + @code{BZ_DATA_ERROR_MAGIC} + if the stream does not begin with the requisite header bytes (ie, is not + a @code{bzip2} data file). This is really a special case of @code{BZ_DATA_ERROR}. + @code{BZ_MEM_ERROR} + if insufficient memory was available + @code{BZ_STREAM_END} + if the logical end of stream was detected. + @code{BZ_OK} + otherwise. +@end display + +Possible return values: +@display + number of bytes read + if @code{bzerror} is @code{BZ_OK} or @code{BZ_STREAM_END} + undefined + otherwise +@end display + +Allowable next actions: +@display + collect data from @code{buf}, then @code{BZ2_bzRead} or @code{BZ2_bzReadClose} + if @code{bzerror} is @code{BZ_OK} + collect data from @code{buf}, then @code{BZ2_bzReadClose} or @code{BZ2_bzReadGetUnused} + if @code{bzerror} is @code{BZ_SEQUENCE_END} + @code{BZ2_bzReadClose} + otherwise +@end display + + + +@subsection @code{BZ2_bzReadGetUnused} +@example + void BZ2_bzReadGetUnused ( int* bzerror, BZFILE *b, + void** unused, int* nUnused ); +@end example +Returns data which was read from the compressed file but was not needed +to get to the logical end-of-stream. @code{*unused} is set to the address +of the data, and @code{*nUnused} to the number of bytes. @code{*nUnused} will +be set to a value between @code{0} and @code{BZ_MAX_UNUSED} inclusive. + +This function may only be called once @code{BZ2_bzRead} has signalled +@code{BZ_STREAM_END} but before @code{BZ2_bzReadClose}. + +Possible assignments to @code{bzerror}: +@display + @code{BZ_PARAM_ERROR} + if @code{b} is @code{NULL} + or @code{unused} is @code{NULL} or @code{nUnused} is @code{NULL} + @code{BZ_SEQUENCE_ERROR} + if @code{BZ_STREAM_END} has not been signalled + or if @code{b} was opened with @code{BZ2_bzWriteOpen} + @code{BZ_OK} + otherwise +@end display + +Allowable next actions: +@display + @code{BZ2_bzReadClose} +@end display + + +@subsection @code{BZ2_bzReadClose} +@example + void BZ2_bzReadClose ( int *bzerror, BZFILE *b ); +@end example +Releases all memory pertaining to the compressed file @code{b}. +@code{BZ2_bzReadClose} does not call @code{fclose} on the underlying file +handle, so you should do that yourself if appropriate. +@code{BZ2_bzReadClose} should be called to clean up after all error +situations. + +Possible assignments to @code{bzerror}: +@display + @code{BZ_SEQUENCE_ERROR} + if @code{b} was opened with @code{BZ2_bzOpenWrite} + @code{BZ_OK} + otherwise +@end display + +Allowable next actions: +@display + none +@end display + + + +@subsection @code{BZ2_bzWriteOpen} +@example + BZFILE *BZ2_bzWriteOpen ( int *bzerror, FILE *f, + int blockSize100k, int verbosity, + int workFactor ); +@end example +Prepare to write compressed data to file handle @code{f}. +@code{f} should refer to +a file which has been opened for writing, and for which the error +indicator (@code{ferror(f)})is not set. + +For the meaning of parameters @code{blockSize100k}, +@code{verbosity} and @code{workFactor}, see +@* @code{BZ2_bzCompressInit}. + +All required memory is allocated at this stage, so if the call +completes successfully, @code{BZ_MEM_ERROR} cannot be signalled by a +subsequent call to @code{BZ2_bzWrite}. + +Possible assignments to @code{bzerror}: +@display + @code{BZ_CONFIG_ERROR} + if the library has been mis-compiled + @code{BZ_PARAM_ERROR} + if @code{f} is @code{NULL} + or @code{blockSize100k < 1} or @code{blockSize100k > 9} + @code{BZ_IO_ERROR} + if @code{ferror(f)} is nonzero + @code{BZ_MEM_ERROR} + if insufficient memory is available + @code{BZ_OK} + otherwise +@end display + +Possible return values: +@display + Pointer to an abstract @code{BZFILE} + if @code{bzerror} is @code{BZ_OK} + @code{NULL} + otherwise +@end display + +Allowable next actions: +@display + @code{BZ2_bzWrite} + if @code{bzerror} is @code{BZ_OK} + (you could go directly to @code{BZ2_bzWriteClose}, but this would be pretty pointless) + @code{BZ2_bzWriteClose} + otherwise +@end display + + + +@subsection @code{BZ2_bzWrite} +@example + void BZ2_bzWrite ( int *bzerror, BZFILE *b, void *buf, int len ); +@end example +Absorbs @code{len} bytes from the buffer @code{buf}, eventually to be +compressed and written to the file. + +Possible assignments to @code{bzerror}: +@display + @code{BZ_PARAM_ERROR} + if @code{b} is @code{NULL} or @code{buf} is @code{NULL} or @code{len < 0} + @code{BZ_SEQUENCE_ERROR} + if b was opened with @code{BZ2_bzReadOpen} + @code{BZ_IO_ERROR} + if there is an error writing the compressed file. + @code{BZ_OK} + otherwise +@end display + + + + +@subsection @code{BZ2_bzWriteClose} +@example + void BZ2_bzWriteClose ( int *bzerror, BZFILE* f, + int abandon, + unsigned int* nbytes_in, + unsigned int* nbytes_out ); + + void BZ2_bzWriteClose64 ( int *bzerror, BZFILE* f, + int abandon, + unsigned int* nbytes_in_lo32, + unsigned int* nbytes_in_hi32, + unsigned int* nbytes_out_lo32, + unsigned int* nbytes_out_hi32 ); +@end example + +Compresses and flushes to the compressed file all data so far supplied +by @code{BZ2_bzWrite}. The logical end-of-stream markers are also written, so +subsequent calls to @code{BZ2_bzWrite} are illegal. All memory associated +with the compressed file @code{b} is released. +@code{fflush} is called on the +compressed file, but it is not @code{fclose}'d. + +If @code{BZ2_bzWriteClose} is called to clean up after an error, the only +action is to release the memory. The library records the error codes +issued by previous calls, so this situation will be detected +automatically. There is no attempt to complete the compression +operation, nor to @code{fflush} the compressed file. You can force this +behaviour to happen even in the case of no error, by passing a nonzero +value to @code{abandon}. + +If @code{nbytes_in} is non-null, @code{*nbytes_in} will be set to be the +total volume of uncompressed data handled. Similarly, @code{nbytes_out} +will be set to the total volume of compressed data written. For +compatibility with older versions of the library, @code{BZ2_bzWriteClose} +only yields the lower 32 bits of these counts. Use +@code{BZ2_bzWriteClose64} if you want the full 64 bit counts. These +two functions are otherwise absolutely identical. + + +Possible assignments to @code{bzerror}: +@display + @code{BZ_SEQUENCE_ERROR} + if @code{b} was opened with @code{BZ2_bzReadOpen} + @code{BZ_IO_ERROR} + if there is an error writing the compressed file + @code{BZ_OK} + otherwise +@end display + +@subsection Handling embedded compressed data streams + +The high-level library facilitates use of +@code{bzip2} data streams which form some part of a surrounding, larger +data stream. +@itemize @bullet +@item For writing, the library takes an open file handle, writes +compressed data to it, @code{fflush}es it but does not @code{fclose} it. +The calling application can write its own data before and after the +compressed data stream, using that same file handle. +@item Reading is more complex, and the facilities are not as general +as they could be since generality is hard to reconcile with efficiency. +@code{BZ2_bzRead} reads from the compressed file in blocks of size +@code{BZ_MAX_UNUSED} bytes, and in doing so probably will overshoot +the logical end of compressed stream. +To recover this data once decompression has +ended, call @code{BZ2_bzReadGetUnused} after the last call of @code{BZ2_bzRead} +(the one returning @code{BZ_STREAM_END}) but before calling +@code{BZ2_bzReadClose}. +@end itemize + +This mechanism makes it easy to decompress multiple @code{bzip2} +streams placed end-to-end. As the end of one stream, when @code{BZ2_bzRead} +returns @code{BZ_STREAM_END}, call @code{BZ2_bzReadGetUnused} to collect the +unused data (copy it into your own buffer somewhere). +That data forms the start of the next compressed stream. +To start uncompressing that next stream, call @code{BZ2_bzReadOpen} again, +feeding in the unused data via the @code{unused}/@code{nUnused} +parameters. +Keep doing this until @code{BZ_STREAM_END} return coincides with the +physical end of file (@code{feof(f)}). In this situation +@code{BZ2_bzReadGetUnused} +will of course return no data. + +This should give some feel for how the high-level interface can be used. +If you require extra flexibility, you'll have to bite the bullet and get +to grips with the low-level interface. + +@subsection Standard file-reading/writing code +Here's how you'd write data to a compressed file: +@example @code +FILE* f; +BZFILE* b; +int nBuf; +char buf[ /* whatever size you like */ ]; +int bzerror; +int nWritten; + +f = fopen ( "myfile.bz2", "w" ); +if (!f) @{ + /* handle error */ +@} +b = BZ2_bzWriteOpen ( &bzerror, f, 9 ); +if (bzerror != BZ_OK) @{ + BZ2_bzWriteClose ( b ); + /* handle error */ +@} + +while ( /* condition */ ) @{ + /* get data to write into buf, and set nBuf appropriately */ + nWritten = BZ2_bzWrite ( &bzerror, b, buf, nBuf ); + if (bzerror == BZ_IO_ERROR) @{ + BZ2_bzWriteClose ( &bzerror, b ); + /* handle error */ + @} +@} + +BZ2_bzWriteClose ( &bzerror, b ); +if (bzerror == BZ_IO_ERROR) @{ + /* handle error */ +@} +@end example +And to read from a compressed file: +@example +FILE* f; +BZFILE* b; +int nBuf; +char buf[ /* whatever size you like */ ]; +int bzerror; +int nWritten; + +f = fopen ( "myfile.bz2", "r" ); +if (!f) @{ + /* handle error */ +@} +b = BZ2_bzReadOpen ( &bzerror, f, 0, NULL, 0 ); +if (bzerror != BZ_OK) @{ + BZ2_bzReadClose ( &bzerror, b ); + /* handle error */ +@} + +bzerror = BZ_OK; +while (bzerror == BZ_OK && /* arbitrary other conditions */) @{ + nBuf = BZ2_bzRead ( &bzerror, b, buf, /* size of buf */ ); + if (bzerror == BZ_OK) @{ + /* do something with buf[0 .. nBuf-1] */ + @} +@} +if (bzerror != BZ_STREAM_END) @{ + BZ2_bzReadClose ( &bzerror, b ); + /* handle error */ +@} else @{ + BZ2_bzReadClose ( &bzerror ); +@} +@end example + + + +@section Utility functions +@subsection @code{BZ2_bzBuffToBuffCompress} +@example + int BZ2_bzBuffToBuffCompress( char* dest, + unsigned int* destLen, + char* source, + unsigned int sourceLen, + int blockSize100k, + int verbosity, + int workFactor ); +@end example +Attempts to compress the data in @code{source[0 .. sourceLen-1]} +into the destination buffer, @code{dest[0 .. *destLen-1]}. +If the destination buffer is big enough, @code{*destLen} is +set to the size of the compressed data, and @code{BZ_OK} is +returned. If the compressed data won't fit, @code{*destLen} +is unchanged, and @code{BZ_OUTBUFF_FULL} is returned. + +Compression in this manner is a one-shot event, done with a single call +to this function. The resulting compressed data is a complete +@code{bzip2} format data stream. There is no mechanism for making +additional calls to provide extra input data. If you want that kind of +mechanism, use the low-level interface. + +For the meaning of parameters @code{blockSize100k}, @code{verbosity} +and @code{workFactor}, @* see @code{BZ2_bzCompressInit}. + +To guarantee that the compressed data will fit in its buffer, allocate +an output buffer of size 1% larger than the uncompressed data, plus +six hundred extra bytes. + +@code{BZ2_bzBuffToBuffDecompress} will not write data at or +beyond @code{dest[*destLen]}, even in case of buffer overflow. + +Possible return values: +@display + @code{BZ_CONFIG_ERROR} + if the library has been mis-compiled + @code{BZ_PARAM_ERROR} + if @code{dest} is @code{NULL} or @code{destLen} is @code{NULL} + or @code{blockSize100k < 1} or @code{blockSize100k > 9} + or @code{verbosity < 0} or @code{verbosity > 4} + or @code{workFactor < 0} or @code{workFactor > 250} + @code{BZ_MEM_ERROR} + if insufficient memory is available + @code{BZ_OUTBUFF_FULL} + if the size of the compressed data exceeds @code{*destLen} + @code{BZ_OK} + otherwise +@end display + + + +@subsection @code{BZ2_bzBuffToBuffDecompress} +@example + int BZ2_bzBuffToBuffDecompress ( char* dest, + unsigned int* destLen, + char* source, + unsigned int sourceLen, + int small, + int verbosity ); +@end example +Attempts to decompress the data in @code{source[0 .. sourceLen-1]} +into the destination buffer, @code{dest[0 .. *destLen-1]}. +If the destination buffer is big enough, @code{*destLen} is +set to the size of the uncompressed data, and @code{BZ_OK} is +returned. If the compressed data won't fit, @code{*destLen} +is unchanged, and @code{BZ_OUTBUFF_FULL} is returned. + +@code{source} is assumed to hold a complete @code{bzip2} format +data stream. @* @code{BZ2_bzBuffToBuffDecompress} tries to decompress +the entirety of the stream into the output buffer. + +For the meaning of parameters @code{small} and @code{verbosity}, +see @code{BZ2_bzDecompressInit}. + +Because the compression ratio of the compressed data cannot be known in +advance, there is no easy way to guarantee that the output buffer will +be big enough. You may of course make arrangements in your code to +record the size of the uncompressed data, but such a mechanism is beyond +the scope of this library. + +@code{BZ2_bzBuffToBuffDecompress} will not write data at or +beyond @code{dest[*destLen]}, even in case of buffer overflow. + +Possible return values: +@display + @code{BZ_CONFIG_ERROR} + if the library has been mis-compiled + @code{BZ_PARAM_ERROR} + if @code{dest} is @code{NULL} or @code{destLen} is @code{NULL} + or @code{small != 0 && small != 1} + or @code{verbosity < 0} or @code{verbosity > 4} + @code{BZ_MEM_ERROR} + if insufficient memory is available + @code{BZ_OUTBUFF_FULL} + if the size of the compressed data exceeds @code{*destLen} + @code{BZ_DATA_ERROR} + if a data integrity error was detected in the compressed data + @code{BZ_DATA_ERROR_MAGIC} + if the compressed data doesn't begin with the right magic bytes + @code{BZ_UNEXPECTED_EOF} + if the compressed data ends unexpectedly + @code{BZ_OK} + otherwise +@end display + + + +@section @code{zlib} compatibility functions +Yoshioka Tsuneo has contributed some functions to +give better @code{zlib} compatibility. These functions are +@code{BZ2_bzopen}, @code{BZ2_bzread}, @code{BZ2_bzwrite}, @code{BZ2_bzflush}, +@code{BZ2_bzclose}, +@code{BZ2_bzerror} and @code{BZ2_bzlibVersion}. +These functions are not (yet) officially part of +the library. If they break, you get to keep all the pieces. +Nevertheless, I think they work ok. +@example +typedef void BZFILE; + +const char * BZ2_bzlibVersion ( void ); +@end example +Returns a string indicating the library version. +@example +BZFILE * BZ2_bzopen ( const char *path, const char *mode ); +BZFILE * BZ2_bzdopen ( int fd, const char *mode ); +@end example +Opens a @code{.bz2} file for reading or writing, using either its name +or a pre-existing file descriptor. +Analogous to @code{fopen} and @code{fdopen}. +@example +int BZ2_bzread ( BZFILE* b, void* buf, int len ); +int BZ2_bzwrite ( BZFILE* b, void* buf, int len ); +@end example +Reads/writes data from/to a previously opened @code{BZFILE}. +Analogous to @code{fread} and @code{fwrite}. +@example +int BZ2_bzflush ( BZFILE* b ); +void BZ2_bzclose ( BZFILE* b ); +@end example +Flushes/closes a @code{BZFILE}. @code{BZ2_bzflush} doesn't actually do +anything. Analogous to @code{fflush} and @code{fclose}. + +@example +const char * BZ2_bzerror ( BZFILE *b, int *errnum ) +@end example +Returns a string describing the more recent error status of +@code{b}, and also sets @code{*errnum} to its numerical value. + + +@section Using the library in a @code{stdio}-free environment + +@subsection Getting rid of @code{stdio} + +In a deeply embedded application, you might want to use just +the memory-to-memory functions. You can do this conveniently +by compiling the library with preprocessor symbol @code{BZ_NO_STDIO} +defined. Doing this gives you a library containing only the following +eight functions: + +@code{BZ2_bzCompressInit}, @code{BZ2_bzCompress}, @code{BZ2_bzCompressEnd} @* +@code{BZ2_bzDecompressInit}, @code{BZ2_bzDecompress}, @code{BZ2_bzDecompressEnd} @* +@code{BZ2_bzBuffToBuffCompress}, @code{BZ2_bzBuffToBuffDecompress} + +When compiled like this, all functions will ignore @code{verbosity} +settings. + +@subsection Critical error handling +@code{libbzip2} contains a number of internal assertion checks which +should, needless to say, never be activated. Nevertheless, if an +assertion should fail, behaviour depends on whether or not the library +was compiled with @code{BZ_NO_STDIO} set. + +For a normal compile, an assertion failure yields the message +@example + bzip2/libbzip2: internal error number N. + This is a bug in bzip2/libbzip2, 1.0 of 21-Mar-2000. + Please report it to me at: jseward@@acm.org. If this happened + when you were using some program which uses libbzip2 as a + component, you should also report this bug to the author(s) + of that program. Please make an effort to report this bug; + timely and accurate bug reports eventually lead to higher + quality software. Thanks. Julian Seward, 21 March 2000. +@end example +where @code{N} is some error code number. @code{exit(3)} +is then called. + +For a @code{stdio}-free library, assertion failures result +in a call to a function declared as: +@example + extern void bz_internal_error ( int errcode ); +@end example +The relevant code is passed as a parameter. You should supply +such a function. + +In either case, once an assertion failure has occurred, any +@code{bz_stream} records involved can be regarded as invalid. +You should not attempt to resume normal operation with them. + +You may, of course, change critical error handling to suit +your needs. As I said above, critical errors indicate bugs +in the library and should not occur. All "normal" error +situations are indicated via error return codes from functions, +and can be recovered from. + + +@section Making a Windows DLL +Everything related to Windows has been contributed by Yoshioka Tsuneo +@* (@code{QWF00133@@niftyserve.or.jp} / +@code{tsuneo-y@@is.aist-nara.ac.jp}), so you should send your queries to +him (but perhaps Cc: me, @code{jseward@@acm.org}). + +My vague understanding of what to do is: using Visual C++ 5.0, +open the project file @code{libbz2.dsp}, and build. That's all. + +If you can't +open the project file for some reason, make a new one, naming these files: +@code{blocksort.c}, @code{bzlib.c}, @code{compress.c}, +@code{crctable.c}, @code{decompress.c}, @code{huffman.c}, @* +@code{randtable.c} and @code{libbz2.def}. You will also need +to name the header files @code{bzlib.h} and @code{bzlib_private.h}. + +If you don't use VC++, you may need to define the proprocessor symbol +@code{_WIN32}. + +Finally, @code{dlltest.c} is a sample program using the DLL. It has a +project file, @code{dlltest.dsp}. + +If you just want a makefile for Visual C, have a look at +@code{makefile.msc}. + +Be aware that if you compile @code{bzip2} itself on Win32, you must set +@code{BZ_UNIX} to 0 and @code{BZ_LCCWIN32} to 1, in the file +@code{bzip2.c}, before compiling. Otherwise the resulting binary won't +work correctly. + +I haven't tried any of this stuff myself, but it all looks plausible. + + + +@chapter Miscellanea + +These are just some random thoughts of mine. Your mileage may +vary. + +@section Limitations of the compressed file format +@code{bzip2-1.0}, @code{0.9.5} and @code{0.9.0} +use exactly the same file format as the previous +version, @code{bzip2-0.1}. This decision was made in the interests of +stability. Creating yet another incompatible compressed file format +would create further confusion and disruption for users. + +Nevertheless, this is not a painless decision. Development +work since the release of @code{bzip2-0.1} in August 1997 +has shown complexities in the file format which slow down +decompression and, in retrospect, are unnecessary. These are: +@itemize @bullet +@item The run-length encoder, which is the first of the + compression transformations, is entirely irrelevant. + The original purpose was to protect the sorting algorithm + from the very worst case input: a string of repeated + symbols. But algorithm steps Q6a and Q6b in the original + Burrows-Wheeler technical report (SRC-124) show how + repeats can be handled without difficulty in block + sorting. +@item The randomisation mechanism doesn't really need to be + there. Udi Manber and Gene Myers published a suffix + array construction algorithm a few years back, which + can be employed to sort any block, no matter how + repetitive, in O(N log N) time. Subsequent work by + Kunihiko Sadakane has produced a derivative O(N (log N)^2) + algorithm which usually outperforms the Manber-Myers + algorithm. + + I could have changed to Sadakane's algorithm, but I find + it to be slower than @code{bzip2}'s existing algorithm for + most inputs, and the randomisation mechanism protects + adequately against bad cases. I didn't think it was + a good tradeoff to make. Partly this is due to the fact + that I was not flooded with email complaints about + @code{bzip2-0.1}'s performance on repetitive data, so + perhaps it isn't a problem for real inputs. + + Probably the best long-term solution, + and the one I have incorporated into 0.9.5 and above, + is to use the existing sorting + algorithm initially, and fall back to a O(N (log N)^2) + algorithm if the standard algorithm gets into difficulties. +@item The compressed file format was never designed to be + handled by a library, and I have had to jump though + some hoops to produce an efficient implementation of + decompression. It's a bit hairy. Try passing + @code{decompress.c} through the C preprocessor + and you'll see what I mean. Much of this complexity + could have been avoided if the compressed size of + each block of data was recorded in the data stream. +@item An Adler-32 checksum, rather than a CRC32 checksum, + would be faster to compute. +@end itemize +It would be fair to say that the @code{bzip2} format was frozen +before I properly and fully understood the performance +consequences of doing so. + +Improvements which I was able to incorporate into +0.9.0, despite using the same file format, are: +@itemize @bullet +@item Single array implementation of the inverse BWT. This + significantly speeds up decompression, presumably + because it reduces the number of cache misses. +@item Faster inverse MTF transform for large MTF values. The + new implementation is based on the notion of sliding blocks + of values. +@item @code{bzip2-0.9.0} now reads and writes files with @code{fread} + and @code{fwrite}; version 0.1 used @code{putc} and @code{getc}. + Duh! Well, you live and learn. + +@end itemize +Further ahead, it would be nice +to be able to do random access into files. This will +require some careful design of compressed file formats. + + + +@section Portability issues +After some consideration, I have decided not to use +GNU @code{autoconf} to configure 0.9.5 or 1.0. + +@code{autoconf}, admirable and wonderful though it is, +mainly assists with portability problems between Unix-like +platforms. But @code{bzip2} doesn't have much in the way +of portability problems on Unix; most of the difficulties appear +when porting to the Mac, or to Microsoft's operating systems. +@code{autoconf} doesn't help in those cases, and brings in a +whole load of new complexity. + +Most people should be able to compile the library and program +under Unix straight out-of-the-box, so to speak, especially +if you have a version of GNU C available. + +There are a couple of @code{__inline__} directives in the code. GNU C +(@code{gcc}) should be able to handle them. If you're not using +GNU C, your C compiler shouldn't see them at all. +If your compiler does, for some reason, see them and doesn't +like them, just @code{#define} @code{__inline__} to be @code{/* */}. One +easy way to do this is to compile with the flag @code{-D__inline__=}, +which should be understood by most Unix compilers. + +If you still have difficulties, try compiling with the macro +@code{BZ_STRICT_ANSI} defined. This should enable you to build the +library in a strictly ANSI compliant environment. Building the program +itself like this is dangerous and not supported, since you remove +@code{bzip2}'s checks against compressing directories, symbolic links, +devices, and other not-really-a-file entities. This could cause +filesystem corruption! + +One other thing: if you create a @code{bzip2} binary for public +distribution, please try and link it statically (@code{gcc -s}). This +avoids all sorts of library-version issues that others may encounter +later on. + +If you build @code{bzip2} on Win32, you must set @code{BZ_UNIX} to 0 and +@code{BZ_LCCWIN32} to 1, in the file @code{bzip2.c}, before compiling. +Otherwise the resulting binary won't work correctly. + + + +@section Reporting bugs +I tried pretty hard to make sure @code{bzip2} is +bug free, both by design and by testing. Hopefully +you'll never need to read this section for real. + +Nevertheless, if @code{bzip2} dies with a segmentation +fault, a bus error or an internal assertion failure, it +will ask you to email me a bug report. Experience with +version 0.1 shows that almost all these problems can +be traced to either compiler bugs or hardware problems. +@itemize @bullet +@item +Recompile the program with no optimisation, and see if it +works. And/or try a different compiler. +I heard all sorts of stories about various flavours +of GNU C (and other compilers) generating bad code for +@code{bzip2}, and I've run across two such examples myself. + +2.7.X versions of GNU C are known to generate bad code from +time to time, at high optimisation levels. +If you get problems, try using the flags +@code{-O2} @code{-fomit-frame-pointer} @code{-fno-strength-reduce}. +You should specifically @emph{not} use @code{-funroll-loops}. + +You may notice that the Makefile runs six tests as part of +the build process. If the program passes all of these, it's +a pretty good (but not 100%) indication that the compiler has +done its job correctly. +@item +If @code{bzip2} crashes randomly, and the crashes are not +repeatable, you may have a flaky memory subsystem. @code{bzip2} +really hammers your memory hierarchy, and if it's a bit marginal, +you may get these problems. Ditto if your disk or I/O subsystem +is slowly failing. Yup, this really does happen. + +Try using a different machine of the same type, and see if +you can repeat the problem. +@item This isn't really a bug, but ... If @code{bzip2} tells +you your file is corrupted on decompression, and you +obtained the file via FTP, there is a possibility that you +forgot to tell FTP to do a binary mode transfer. That absolutely +will cause the file to be non-decompressible. You'll have to transfer +it again. +@end itemize + +If you've incorporated @code{libbzip2} into your own program +and are getting problems, please, please, please, check that the +parameters you are passing in calls to the library, are +correct, and in accordance with what the documentation says +is allowable. I have tried to make the library robust against +such problems, but I'm sure I haven't succeeded. + +Finally, if the above comments don't help, you'll have to send +me a bug report. Now, it's just amazing how many people will +send me a bug report saying something like +@display + bzip2 crashed with segmentation fault on my machine +@end display +and absolutely nothing else. Needless to say, a such a report +is @emph{totally, utterly, completely and comprehensively 100% useless; +a waste of your time, my time, and net bandwidth}. +With no details at all, there's no way I can possibly begin +to figure out what the problem is. + +The rules of the game are: facts, facts, facts. Don't omit +them because "oh, they won't be relevant". At the bare +minimum: +@display + Machine type. Operating system version. + Exact version of @code{bzip2} (do @code{bzip2 -V}). + Exact version of the compiler used. + Flags passed to the compiler. +@end display +However, the most important single thing that will help me is +the file that you were trying to compress or decompress at the +time the problem happened. Without that, my ability to do anything +more than speculate about the cause, is limited. + +Please remember that I connect to the Internet with a modem, so +you should contact me before mailing me huge files. + + +@section Did you get the right package? + +@code{bzip2} is a resource hog. It soaks up large amounts of CPU cycles +and memory. Also, it gives very large latencies. In the worst case, you +can feed many megabytes of uncompressed data into the library before +getting any compressed output, so this probably rules out applications +requiring interactive behaviour. + +These aren't faults of my implementation, I hope, but more +an intrinsic property of the Burrows-Wheeler transform (unfortunately). +Maybe this isn't what you want. + +If you want a compressor and/or library which is faster, uses less +memory but gets pretty good compression, and has minimal latency, +consider Jean-loup +Gailly's and Mark Adler's work, @code{zlib-1.1.2} and +@code{gzip-1.2.4}. Look for them at + +@code{http://www.cdrom.com/pub/infozip/zlib} and +@code{http://www.gzip.org} respectively. + +For something faster and lighter still, you might try Markus F X J +Oberhumer's @code{LZO} real-time compression/decompression library, at +@* @code{http://wildsau.idv.uni-linz.ac.at/mfx/lzo.html}. + +If you want to use the @code{bzip2} algorithms to compress small blocks +of data, 64k bytes or smaller, for example on an on-the-fly disk +compressor, you'd be well advised not to use this library. Instead, +I've made a special library tuned for that kind of use. It's part of +@code{e2compr-0.40}, an on-the-fly disk compressor for the Linux +@code{ext2} filesystem. Look at +@code{http://www.netspace.net.au/~reiter/e2compr}. + + + +@section Testing + +A record of the tests I've done. + +First, some data sets: +@itemize @bullet +@item B: a directory containing 6001 files, one for every length in the + range 0 to 6000 bytes. The files contain random lowercase + letters. 18.7 megabytes. +@item H: my home directory tree. Documents, source code, mail files, + compressed data. H contains B, and also a directory of + files designed as boundary cases for the sorting; mostly very + repetitive, nasty files. 565 megabytes. +@item A: directory tree holding various applications built from source: + @code{egcs}, @code{gcc-2.8.1}, KDE, GTK, Octave, etc. + 2200 megabytes. +@end itemize +The tests conducted are as follows. Each test means compressing +(a copy of) each file in the data set, decompressing it and +comparing it against the original. + +First, a bunch of tests with block sizes and internal buffer +sizes set very small, +to detect any problems with the +blocking and buffering mechanisms. +This required modifying the source code so as to try to +break it. +@enumerate +@item Data set H, with + buffer size of 1 byte, and block size of 23 bytes. +@item Data set B, buffer sizes 1 byte, block size 1 byte. +@item As (2) but small-mode decompression. +@item As (2) with block size 2 bytes. +@item As (2) with block size 3 bytes. +@item As (2) with block size 4 bytes. +@item As (2) with block size 5 bytes. +@item As (2) with block size 6 bytes and small-mode decompression. +@item H with buffer size of 1 byte, but normal block + size (up to 900000 bytes). +@end enumerate +Then some tests with unmodified source code. +@enumerate +@item H, all settings normal. +@item As (1), with small-mode decompress. +@item H, compress with flag @code{-1}. +@item H, compress with flag @code{-s}, decompress with flag @code{-s}. +@item Forwards compatibility: H, @code{bzip2-0.1pl2} compressing, + @code{bzip2-0.9.5} decompressing, all settings normal. +@item Backwards compatibility: H, @code{bzip2-0.9.5} compressing, + @code{bzip2-0.1pl2} decompressing, all settings normal. +@item Bigger tests: A, all settings normal. +@item As (7), using the fallback (Sadakane-like) sorting algorithm. +@item As (8), compress with flag @code{-1}, decompress with flag + @code{-s}. +@item H, using the fallback sorting algorithm. +@item Forwards compatibility: A, @code{bzip2-0.1pl2} compressing, + @code{bzip2-0.9.5} decompressing, all settings normal. +@item Backwards compatibility: A, @code{bzip2-0.9.5} compressing, + @code{bzip2-0.1pl2} decompressing, all settings normal. +@item Misc test: about 400 megabytes of @code{.tar} files with + @code{bzip2} compiled with Checker (a memory access error + detector, like Purify). +@item Misc tests to make sure it builds and runs ok on non-Linux/x86 + platforms. +@end enumerate +These tests were conducted on a 225 MHz IDT WinChip machine, running +Linux 2.0.36. They represent nearly a week of continuous computation. +All tests completed successfully. + + +@section Further reading +@code{bzip2} is not research work, in the sense that it doesn't present +any new ideas. Rather, it's an engineering exercise based on existing +ideas. + +Four documents describe essentially all the ideas behind @code{bzip2}: +@example +Michael Burrows and D. J. Wheeler: + "A block-sorting lossless data compression algorithm" + 10th May 1994. + Digital SRC Research Report 124. + ftp://ftp.digital.com/pub/DEC/SRC/research-reports/SRC-124.ps.gz + If you have trouble finding it, try searching at the + New Zealand Digital Library, http://www.nzdl.org. + +Daniel S. Hirschberg and Debra A. LeLewer + "Efficient Decoding of Prefix Codes" + Communications of the ACM, April 1990, Vol 33, Number 4. + You might be able to get an electronic copy of this + from the ACM Digital Library. + +David J. Wheeler + Program bred3.c and accompanying document bred3.ps. + This contains the idea behind the multi-table Huffman + coding scheme. + ftp://ftp.cl.cam.ac.uk/users/djw3/ + +Jon L. Bentley and Robert Sedgewick + "Fast Algorithms for Sorting and Searching Strings" + Available from Sedgewick's web page, + www.cs.princeton.edu/~rs +@end example +The following paper gives valuable additional insights into the +algorithm, but is not immediately the basis of any code +used in bzip2. +@example +Peter Fenwick: + Block Sorting Text Compression + Proceedings of the 19th Australasian Computer Science Conference, + Melbourne, Australia. Jan 31 - Feb 2, 1996. + ftp://ftp.cs.auckland.ac.nz/pub/peter-f/ACSC96paper.ps +@end example +Kunihiko Sadakane's sorting algorithm, mentioned above, +is available from: +@example +http://naomi.is.s.u-tokyo.ac.jp/~sada/papers/Sada98b.ps.gz +@end example +The Manber-Myers suffix array construction +algorithm is described in a paper +available from: +@example +http://www.cs.arizona.edu/people/gene/PAPERS/suffix.ps +@end example +Finally, the following paper documents some recent investigations +I made into the performance of sorting algorithms: +@example +Julian Seward: + On the Performance of BWT Sorting Algorithms + Proceedings of the IEEE Data Compression Conference 2000 + Snowbird, Utah. 28-30 March 2000. +@end example + + +@contents + +@bye + |