| blob | f59427f7f07d1c4f9aee4bcefd64daa6b2e0a33e |
1 <html>
2 <head>
7 #74240f dark brown h1, h2, h3, h4
8 #336699 medium blue links
9 #339999 turquoise link hover colour
10 #202020 almost black general text
11 #761596 purple md5sum text
12 #626262 dark gray pre border
13 #eeeeee very light gray pre background
14 #f2f2f9 very light blue nav table background
15 #3366cc medium blue nav table border
16 */
18 a, a:link, a:visited, a:active { color: #336699; }
19 a:hover { color: #339999; }
22 h1, h2, h3, h4 { color: #74240f; }
24 dt { color: #336699; font-weight: bold }
25 dd {
26 margin-left: 1.5em;
27 padding-bottom: 0.8em;
28 }
30 /* -- ruler -- */
31 div.hr_blue {
32 height: 3px;
33 background:#ffffff url("/images/hr_blue.png") repeat-x; }
34 div.hr_blue hr { display:none; }
36 /* release styles */
37 #release p { margin-top: 0.4em; }
38 #release .md5sum { color: #761596; }
41 /* ------ styles for docs|manuals|howto ------ */
42 /* -- lists -- */
43 ul {
45 padding: 0px;
46 list-style: url("/images/li-blue.png");
47 }
48 ul li {
49 margin-bottom: 10px;
50 }
51 ul ul {
52 list-style-type: none;
53 list-style-image: none;
54 margin-left: 0px;
55 }
57 /* header / footer nav tables */
58 table.nav {
60 background: #f2f2f9;
61 background-color: #f2f2f9;
62 margin-bottom: 0.5em;
63 }
64 /* don't have underlined links in chunked nav menus */
65 table.nav a { text-decoration: none; }
66 table.nav a:hover { text-decoration: underline; }
67 table.nav td { font-size: 85%; }
69 code, tt, pre { font-size: 120%; }
70 code, tt { color: #761596; }
72 div.literallayout, pre.programlisting, pre.screen {
73 color: #000000;
74 padding: 0.5em;
75 background: #eeeeee;
77 background-color: #eeeeee;
79 }
80 </style>
81 </head>
82 <body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div lang="en" class="book" title="bzip2 and libbzip2, version 1.0.6">
84 <div>
91 </h3>
93 </div></div></div>
100 All rights reserved.</p>
101 <p>Redistribution and use in source and binary forms, with
102 or without modification, are permitted provided that the
103 following conditions are met:</p>
105 <li class="listitem" style="list-style-type: disc"><p>Redistributions of source code must retain the
106 above copyright notice, this list of conditions and the
107 following disclaimer.</p></li>
109 misrepresented; you must not claim that you wrote the original
110 software. If you use this software in a product, an
111 acknowledgment in the product documentation would be
112 appreciated but is not required.</p></li>
113 <li class="listitem" style="list-style-type: disc"><p>Altered source versions must be plainly marked
114 as such, and must not be misrepresented as being the original
115 software.</p></li>
117 endorse or promote products derived from this software without
118 specific prior written permission.</p></li>
119 </ul></div>
121 EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
122 THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
123 PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
124 AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
125 EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
126 TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
127 DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
128 ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
129 LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
130 IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
131 THE POSSIBILITY OF SUCH DAMAGE.</p>
132 <p>PATENTS: To the best of my knowledge,
135 algorithms. However, I do not have the resources to carry
136 out a patent search. Therefore I cannot give any guarantee of
137 the above statement.
138 </p>
139 </div></div>
140 </div>
141 <hr>
142 </div>
145 <dl>
148 <dd><dl>
154 <dt><span class="sect1"><a href="#recovering">2.6. RECOVERING DATA FROM DAMAGED FILES</a></span></dt>
158 </dl></dd>
161 </a></span></dt>
162 <dd><dl>
164 <dd><dl>
167 <dt><span class="sect2"><a href="#util-fns-summary">3.1.3. Utility functions summary</a></span></dt>
168 </dl></dd>
171 <dd><dl>
175 <dt><span class="sect2"><a href="#bzDecompress-init">3.3.4. BZ2_bzDecompressInit</a></span></dt>
178 </dl></dd>
180 <dd><dl>
188 <dt><span class="sect2"><a href="#embed">3.4.8. Handling embedded compressed data streams</a></span></dt>
189 <dt><span class="sect2"><a href="#std-rdwr">3.4.9. Standard file-reading/writing code</a></span></dt>
190 </dl></dd>
192 <dd><dl>
193 <dt><span class="sect2"><a href="#bzbufftobuffcompress">3.5.1. BZ2_bzBuffToBuffCompress</a></span></dt>
194 <dt><span class="sect2"><a href="#bzbufftobuffdecompress">3.5.2. BZ2_bzBuffToBuffDecompress</a></span></dt>
195 </dl></dd>
196 <dt><span class="sect1"><a href="#zlib-compat">3.6. zlib compatibility functions</a></span></dt>
197 <dt><span class="sect1"><a href="#stdio-free">3.7. Using the library in a stdio-free environment</a></span></dt>
198 <dd><dl>
200 <dt><span class="sect2"><a href="#critical-error">3.7.2. Critical error handling</a></span></dt>
201 </dl></dd>
203 </dl></dd>
205 <dd><dl>
206 <dt><span class="sect1"><a href="#limits">4.1. Limitations of the compressed file format</a></span></dt>
211 </dl></dd>
212 </dl>
213 </div>
218 using the Burrows-Wheeler block-sorting text compression
219 algorithm, and Huffman coding. Compression is generally
220 considerably better than that achieved by more conventional
221 LZ77/LZ78-based compressors, and approaches the performance of
222 the PPM family of statistical compressors.</p>
225 handling compressed data in the
227 describes both how to use the program and how to work with the
228 library interface. Most of the manual is devoted to this
229 library, not the program, which is good news if your interest is
230 only in the program.</p>
232 <li class="listitem" style="list-style-type: disc"><p><a class="xref" href="#using" title="2. How to use bzip2">How to use bzip2</a> describes how to use
234 you need to read if you just want to know how to operate the
235 program.</p></li>
236 <li class="listitem" style="list-style-type: disc"><p><a class="xref" href="#libprog" title="3. Programming with libbzip2">Programming with libbzip2</a> describes the
237 programming interfaces in detail, and</p></li>
238 <li class="listitem" style="list-style-type: disc"><p><a class="xref" href="#misc" title="4. Miscellanea">Miscellanea</a> records some
239 miscellaneous notes which I thought ought to be recorded
240 somewhere.</p></li>
241 </ul></div>
242 </div>
248 <dl>
254 <dt><span class="sect1"><a href="#recovering">2.6. RECOVERING DATA FROM DAMAGED FILES</a></span></dt>
258 </dl>
259 </div>
260 <p>This chapter contains a copy of the
262 else.</p>
267 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">bzip2</code>,
270 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">bzcat</code> -
271 decompresses files to stdout</p></li>
272 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">bzip2recover</code> -
273 recovers data from damaged bzip2 files</p></li>
274 </ul></div>
275 </div>
280 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">bzip2</code> [
281 -cdfkqstvzVL123456789 ] [ filenames ... ]</p></li>
282 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">bunzip2</code> [
283 -fkvsVL ] [ filenames ... ]</p></li>
284 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">bzcat</code> [ -s ] [
285 filenames ... ]</p></li>
286 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">bzip2recover</code>
287 filename</p></li>
288 </ul></div>
289 </div>
294 using the Burrows-Wheeler block sorting text compression
295 algorithm, and Huffman coding. Compression is generally
296 considerably better than that achieved by more conventional
297 LZ77/LZ78-based compressors, and approaches the performance of
298 the PPM family of statistical compressors.</p>
299 <p>The command-line options are deliberately very similar to
301 not identical.</p>
303 file names to accompany the command-line flags. Each file is
304 replaced by a compressed version of itself, with the name
306 compressed file has the same modification date, permissions, and,
307 when possible, ownership as the corresponding original, so that
308 these properties can be correctly restored at decompression time.
309 File name handling is naive in the sense that there is no
310 mechanism for preserving original file names, permissions,
311 ownerships or dates in filesystems which lack these concepts, or
312 have serious file name length restrictions, such as
313 MS-DOS.</p>
316 overwrite existing files. If you want this to happen, specify
318 <p>If no file names are specified,
320 input to standard output. In this case,
322 compressed output to a terminal, as this would be entirely
323 incomprehensible and therefore pointless.</p>
326 specified files. Files which were not created by
328 ignored, and a warning issued.
330 filename for the decompressed file from that of the compressed
331 file as follows:</p>
333 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">filename.bz2 </code>
334 becomes
336 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">filename.bz </code>
337 becomes
339 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">filename.tbz2</code>
340 becomes
342 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">filename.tbz </code>
343 becomes
345 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">anyothername </code>
346 becomes
348 </ul></div>
349 <p>If the file does not end in one of the recognised endings,
355 guess the name of the original file, and uses the original name
357 <p>As with compression, supplying no filenames causes
358 decompression from standard input to standard output.</p>
360 decompress a file which is the concatenation of two or more
361 compressed files. The result is the concatenation of the
362 corresponding uncompressed files. Integrity testing
364 files is also supported.</p>
365 <p>You can also compress or decompress files to the standard
367 Multiple files may be compressed and decompressed like this. The
368 resulting outputs are fed sequentially to stdout. Compression of
369 multiple files in this manner generates a stream containing
370 multiple compressed file representations. Such a stream can be
371 decompressed correctly only by
374 stop after decompressing the first file in the stream.</p>
377 specified files to the standard output.</p>
379 from the environment variables
382 process them before any arguments read from the command line.
383 This gives a convenient way to supply default arguments.</p>
384 <p>Compression is always performed, even if the compressed
385 file is slightly larger than the original. Files of less than
386 about one hundred bytes tend to get larger, since the compression
387 mechanism has a constant overhead in the region of 50 bytes.
388 Random data (including the output of most file compressors) is
389 coded at about 8.05 bits per byte, giving an expansion of around
391 <p>As a self-check for your protection,
393 sure that the decompressed version of a file is identical to the
394 original. This guards against corruption of the compressed data,
395 and against undetected bugs in
397 The chances of data corruption going undetected is microscopic,
398 about one chance in four billion for each file processed. Be
399 aware, though, that the check occurs upon decompression, so it
400 can only tell you that something is wrong. It can't help you
401 recover the original uncompressed data. You can use
403 data from damaged files.</p>
405 problems (file not found, invalid flags, I/O errors, etc.), 2
406 to indicate a corrupt compressed file, 3 for an internal
407 consistency error (eg, bug) which caused
409 </div>
415 <dd><p>Compress or decompress to standard
416 output.</p></dd>
418 <dd><p>Force decompression.
422 program, and the decision about what actions to take is done on
423 the basis of which name is used. This flag overrides that
424 mechanism, and forces bzip2 to decompress.</p></dd>
426 <dd><p>The complement to
428 regardless of the invokation name.</p></dd>
430 <dd><p>Check integrity of the specified file(s), but
431 don't decompress them. This really performs a trial
432 decompression and throws away the result.</p></dd>
434 <dd>
435 <p>Force overwrite of output files. Normally,
437 existing output files. Also forces
439 files, which it otherwise wouldn't do.</p>
441 to decompress files which don't have the correct magic header
443 however, it will pass such files through unmodified. This is
445 </dd>
447 <dd><p>Keep (don't delete) input files during
448 compression or decompression.</p></dd>
450 <dd>
451 <p>Reduce memory usage, for compression,
452 decompression and testing. Files are decompressed and tested
453 using a modified algorithm which only requires 2.5 bytes per
454 block byte. This means any file can be decompressed in 2300k
455 of memory, albeit at about half the normal speed.</p>
457 selects a block size of 200k, which limits memory use to around
458 the same figure, at the expense of your compression ratio. In
459 short, if your machine is low on memory (8 megabytes or less),
461 <a class="xref" href="#memory-management" title="2.5. MEMORY MANAGEMENT">MEMORY MANAGEMENT</a> below.</p>
462 </dd>
464 <dd><p>Suppress non-essential warning messages.
465 Messages pertaining to I/O errors and other critical events
466 will not be suppressed.</p></dd>
468 <dd><p>Verbose mode -- show the compression ratio for
469 each file processed. Further
471 level, spewing out lots of information which is primarily of
472 interest for diagnostic purposes.</p></dd>
473 <dt><span class="term"><code class="computeroutput">-L --license -V --version</code></span></dt>
474 <dd><p>Display the software version, license terms and
475 conditions.</p></dd>
481 when compressing. Has no effect when decompressing. See <a class="xref" href="#memory-management" title="2.5. MEMORY MANAGEMENT">MEMORY MANAGEMENT</a> below. The
486 make things significantly faster. And
488 default behaviour.</p></dd>
490 <dd><p>Treats all subsequent arguments as file names,
491 even if they start with a dash. This is so you can handle
492 files with names beginning with a dash, for example:
495 <dt>
496 <span class="term"><code class="computeroutput">--repetitive-fast</code>, </span><span class="term"><code class="computeroutput">--repetitive-best</code></span>
497 </dt>
499 above. They provided some coarse control over the behaviour of
500 the sorting algorithm in earlier versions, which was sometimes
501 useful. 0.9.5 and above have an improved algorithm which
502 renders these flags irrelevant.</p></dd>
503 </dl></div>
504 </div>
509 files in blocks. The block size affects both the compression
510 ratio achieved, and the amount of memory needed for compression
514 respectively. At decompression time, the block size used for
515 compression is read from the header of the compressed file, and
517 just enough memory to decompress the file. Since block sizes are
518 stored in compressed files, it follows that the flags
521 ignored during decompression.</p>
522 <p>Compression and decompression requirements, in bytes, can be
523 estimated as:</p>
528 <p>Larger block sizes give rapidly diminishing marginal
529 returns. Most of the compression comes from the first two or
530 three hundred k of block size, a fact worth bearing in mind when
532 It is also important to appreciate that the decompression memory
533 requirement is set at compression time by the choice of block
534 size.</p>
537 kbytes to decompress. To support decompression of any file on a
539 an option to decompress using approximately half this amount of
540 memory, about 2300 kbytes. Decompression speed is also halved,
541 so you should use this option only where necessary. The relevant
543 <p>In general, try and use the largest block size memory
544 constraints allow, since that maximises the compression achieved.
545 Compression and decompression speed are virtually unaffected by
546 block size.</p>
547 <p>Another significant point applies to files which fit in a
548 single block -- that means most files you'd encounter using a
549 large block size. The amount of real memory touched is
550 proportional to the size of the file, since the file is smaller
553 compressor to allocate around 7600k of memory, but only touch
556 kbytes.</p>
557 <p>Here is a table which summarises the maximum memory usage
558 for different block sizes. Also recorded is the total compressed
559 size for 14 files of the Calgary Text Compression Corpus
561 compression varies with block size. These figures tend to
562 understate the advantage of larger block sizes for larger files,
563 since the Corpus is dominated by smaller files.</p>
565 Flag usage usage -s usage Size
576 </div>
581 blocks, usually 900kbytes long. Each block is handled
582 independently. If a media or transmission error causes a
584 damaged, it may be possible to recover data from the undamaged
585 blocks in the file.</p>
586 <p>The compressed representation of each block is delimited by
587 a 48-bit pattern, which makes it possible to find the block
588 boundaries with reasonable certainty. Each block also carries
589 its own 32-bit CRC, so damaged blocks can be distinguished from
590 undamaged ones.</p>
592 program whose purpose is to search for blocks in
596 the integrity of the resulting files, and decompress those which
597 are undamaged.</p>
599 single argument, the name of the damaged file, and writes a
602 the extracted blocks. The output filenames are designed so that
603 the use of wildcards in subsequent processing -- for example,
605 recovered_data</code> -- lists the files in the correct
606 order.</p>
609 files, as these will contain many blocks. It is clearly futile
610 to use it on damaged single-block files, since a damaged block
611 cannot be recovered. If you wish to minimise any potential data
612 loss through media or transmission errors, you might consider
613 compressing with a smaller block size.</p>
614 </div>
618 <p>The sorting phase of compression gathers together similar
619 strings in the file. Because of this, files containing very long
620 runs of repeated symbols, like "aabaabaabaab ..." (repeated
621 several hundred times) may compress more slowly than normal.
622 Versions 0.9.5 and above fare much better than previous versions
623 in this respect. The ratio between worst-case and average-case
627 in great detail, if you want.</p>
628 <p>Decompression speed is unaffected by these
629 phenomena.</p>
631 several megabytes of memory to operate in, and then charges all
632 over it in a fairly random fashion. This means that performance,
633 both for compressing and decompressing, is largely determined by
634 the speed at which your machine can service cache misses.
635 Because of this, small changes to the code to reduce the miss
636 rate have been observed to give disproportionately large
637 performance improvements. I imagine
639 machines with very large caches.</p>
640 </div>
644 <p>I/O error messages are not as helpful as they could be.
646 errors and exit cleanly, but the details of what the problem is
647 sometimes seem rather misleading.</p>
650 this version is entirely forwards and backwards compatible with the
653 above can correctly decompress multiple concatenated compressed files.
654 0.1pl2 cannot do this; it will stop after decompressing just the first
655 file in the stream.</p>
658 compressed files, so it could not handle compressed files more
660 on some platforms which support them (GNU supported targets, and
661 Windows). To establish whether or not
663 a limitation, run it without arguments. In any event you can
664 build yourself an unlimited version if you can recompile it with
667 </div>
671 <p>Julian Seward,
673 <p>The ideas embodied in
675 following people: Michael Burrows and David Wheeler (for the
676 block sorting transformation), David Wheeler (again, for the
677 Huffman coder), Peter Fenwick (for the structured coding model in
679 refinements), and Alistair Moffat, Radford Neal and Ian Witten
680 (for the arithmetic coder in the original
682 their help, support and advice. See the manual in the source
683 distribution for pointers to sources of documentation. Christian
684 von Roques encouraged me to look for faster sorting algorithms,
685 so as to speed up compression. Bela Lubkin encouraged me to
686 improve the worst-case compression performance.
687 Donna Robinson XMLised the documentation.
688 Many people sent
689 patches, helped with portability problems, lent machines, gave
690 advice and were generally helpful.</p>
691 </div>
692 </div>
697 </h2></div></div></div>
700 <dl>
702 <dd><dl>
705 <dt><span class="sect2"><a href="#util-fns-summary">3.1.3. Utility functions summary</a></span></dt>
706 </dl></dd>
709 <dd><dl>
713 <dt><span class="sect2"><a href="#bzDecompress-init">3.3.4. BZ2_bzDecompressInit</a></span></dt>
716 </dl></dd>
718 <dd><dl>
726 <dt><span class="sect2"><a href="#embed">3.4.8. Handling embedded compressed data streams</a></span></dt>
727 <dt><span class="sect2"><a href="#std-rdwr">3.4.9. Standard file-reading/writing code</a></span></dt>
728 </dl></dd>
730 <dd><dl>
731 <dt><span class="sect2"><a href="#bzbufftobuffcompress">3.5.1. BZ2_bzBuffToBuffCompress</a></span></dt>
732 <dt><span class="sect2"><a href="#bzbufftobuffdecompress">3.5.2. BZ2_bzBuffToBuffDecompress</a></span></dt>
733 </dl></dd>
734 <dt><span class="sect1"><a href="#zlib-compat">3.6. zlib compatibility functions</a></span></dt>
735 <dt><span class="sect1"><a href="#stdio-free">3.7. Using the library in a stdio-free environment</a></span></dt>
736 <dd><dl>
738 <dt><span class="sect2"><a href="#critical-error">3.7.2. Critical error handling</a></span></dt>
739 </dl></dd>
741 </dl>
742 </div>
743 <p>This chapter describes the programming interface to
745 <p>For general background information, particularly about
746 memory use and performance aspects, you'd be well advised to read
752 library for compressing and decompressing data in the
754 packaged as a single entity, it helps to regard the library as
755 three separate parts: the low level interface, and the high level
756 interface, and some utility functions.</p>
757 <p>The structure of
759 to that of Jean-loup Gailly's and Mark Adler's excellent
761 <p>All externally visible symbols have names beginning
763 1.0. The intention is to minimise pollution of the namespaces of
764 library clients.</p>
765 <p>To use any part of the library, you need to
767 into your sources.</p>
771 <p>This interface provides services for compressing and
772 decompressing data in memory. There's no provision for dealing
773 with files, streams or any other I/O mechanisms, just straight
774 memory-to-memory work. In fact, this part of the library can be
775 compiled without inclusion of
777 for embedded applications.</p>
778 <p>The low-level part of the library has no global variables
779 and is therefore thread-safe.</p>
780 <p>Six routines make up the low level interface:
784 compression, and a corresponding trio
789 functions allocate memory for compression/decompression and do
790 other initialisations, whilst the
792 operations and release memory.</p>
793 <p>The real work is done by
796 compress and decompress data from a user-supplied input buffer to
797 a user-supplied output buffer. These buffers can be any size;
798 arbitrary quantities of data are handled by making repeated calls
799 to these functions. This is a flexible mechanism allowing a
800 consumer-pull style of activity, or producer-push, or a mixture
801 of both.</p>
802 </div>
806 <p>This interface provides some handy wrappers around the
807 low-level interface to facilitate reading and writing
810 provide hooks to facilitate reading files in which the
812 within some larger-scale file structure, or where there are
814 concatenated end-to-end.</p>
815 <p>For reading files,
820 supplied. For writing files,
824 available.</p>
825 <p>As with the low-level library, no global variables are used
826 so the library is per se thread-safe. However, if I/O errors
827 occur whilst reading or writing the underlying compressed files,
829 determine the cause of the error. In that case, you'd need a C
830 library which correctly supports
832 environment.</p>
833 <p>To make the library a little simpler and more portable,
837 which have previously been opened for reading or writing
838 respectively. That avoids portability problems associated with
839 file operations and file attributes, whilst not being much of an
840 imposition on the programmer.</p>
841 </div>
845 <p>For very simple needs,
848 provided. These compress data in memory from one buffer to
849 another buffer in a single function call. You should assess
850 whether these functions fulfill your memory-to-memory
851 compression/decompression requirements before investing effort in
852 understanding the more general but more complex low-level
853 interface.</p>
854 <p>Yoshioka Tsuneo
856 contributed some functions to give better
865 these functions more convenient for simple file reading and
866 writing, than those in the high-level interface. These functions
867 are not (yet) officially part of the library, and are minimally
868 documented here. If they break, you get to keep all the pieces.
869 I hope to document them properly when time permits.</p>
870 <p>Yoshioka also contributed modifications to allow the
871 library to be built as a Windows DLL.</p>
872 </div>
873 </div>
877 <p>The library is designed to recover cleanly in all
878 situations, including the worst-case situation of decompressing
879 random data. I'm not 100% sure that it can always do this, so
880 you might want to add a signal handler to catch segmentation
881 violations during decompression if you are feeling especially
882 paranoid. I would be interested in hearing more about the
883 robustness of the library to corrupted compressed data.</p>
885 previous version. Investigations with Valgrind (a tool for detecting
886 problems with memory management) indicate
887 that, at least for the few files I tested, all single-bit errors
888 in the decompressed data are caught properly, with no
889 segmentation faults, no uses of uninitialised data, no out of
890 range reads or writes, and no infinite looping in the decompressor.
891 So it's certainly pretty robust, although
892 I wouldn't claim it to be totally bombproof.</p>
894 all definitions needed to use the library. In particular, you
895 should definitely not include
898 return values are defined. The following list is not intended as
899 an exhaustive description of the circumstances in which a given
900 value may be returned -- those descriptions are given later.
901 Rather, it is intended to convey the rough meaning of each return
902 value. The first five actions are normal and not intended to
903 denote an error situation.</p>
906 <dd><p>The requested action was completed
907 successfully.</p></dd>
909 BZ_FINISH_OK</code></span></dt>
910 <dd><p>In
912 flush/finish/nothing-special action was completed
913 successfully.</p></dd>
915 <dd><p>Compression of data was completed, or the
916 logical stream end was detected during
917 decompression.</p></dd>
918 </dl></div>
919 <p>The following return values indicate an error of some
920 kind.</p>
923 <dd><p>Indicates that the library has been improperly
924 compiled on your platform -- a major configuration error.
925 Specifically, it means that
929 4 respectively, as they should be. Note that the library
930 should still work properly on 64-bit platforms which follow
931 the LP64 programming model -- that is, where
937 OK.</p></dd>
939 <dd><p>When using the library, it is important to call
940 the functions in the correct sequence and with data structures
941 (buffers etc) in the correct states.
943 can to ensure this is happening, and returns
945 Code which complies precisely with the function semantics, as
946 detailed below, should never receive this value; such an event
947 denotes buggy code which you should
948 investigate.</p></dd>
950 <dd><p>Returned when a parameter to a function call is
951 out of range or otherwise manifestly incorrect. As with
953 denotes a bug in the client code. The distinction between
956 hazy, but still worth making.</p></dd>
958 <dd><p>Returned when a request to allocate memory
959 failed. Note that the quantity of memory needed to decompress
960 a stream cannot be determined until the stream's header has
961 been read. So
965 of the compressed data has been read. The same is not true
966 for compression; once
969 successfully completed,
971 occur.</p></dd>
973 <dd><p>Returned when a data integrity error is
974 detected during decompression. Most importantly, this means
975 when stored and computed CRCs for the data do not match. This
976 value is also returned upon detection of any other anomaly in
977 the compressed data.</p></dd>
979 <dd><p>As a special case of
981 sometimes useful to know when the compressed stream does not
985 <dd><p>Returned by
988 error reading or writing in the compressed file, and by
991 to use a file for which the error indicator (viz,
996 operating-system specific information about the
997 problem.</p></dd>
999 <dd><p>Returned by
1001 compressed file finishes before the logical end of stream is
1002 detected.</p></dd>
1004 <dd><p>Returned by
1007 indicate that the output data will not fit into the output
1008 buffer provided.</p></dd>
1009 </dl></div>
1010 </div>
1018 char *next_in;
1019 unsigned int avail_in;
1020 unsigned int total_in_lo32;
1021 unsigned int total_in_hi32;
1023 char *next_out;
1024 unsigned int avail_out;
1025 unsigned int total_out_lo32;
1026 unsigned int total_out_hi32;
1028 void *state;
1030 void *(*bzalloc)(void *,int,int);
1031 void (*bzfree)(void *,void *);
1032 void *opaque;
1033 } bz_stream;
1035 int BZ2_bzCompressInit ( bz_stream *strm,
1036 int blockSize100k,
1037 int verbosity,
1038 int workFactor );</pre>
1039 <p>Prepares for compression. The
1041 data pertaining to the compression activity. A
1043 allocated and initialised prior to the call. The fields of
1046 is a pointer to the private data structures required for
1047 compression.</p>
1048 <p>Custom memory allocators are supported, via fields
1056 opaque, n, m )</code> is expected to return a pointer
1060 <p>If you don't want to use a custom memory allocator, set
1067 <p>Before calling
1072 appropriately, as just described. Upon return, the internal
1073 state will have been allocated and initialised, and
1078 set to zero. These four fields are used by the library to inform
1079 the caller of the total amount of data passed into and out of the
1080 library, respectively. You should not try to change them. As of
1083 fields to store the upper 32 bits of the count. So, for example,
1087 specifies the block size to be used for compression. It should
1090 takes most memory.</p>
1093 greater numbers give increasingly verbose monitoring/debugging
1094 output. If the library has been compiled with
1096 will appear for any verbosity setting.</p>
1098 controls how the compression phase behaves when presented with
1099 worst case, highly repetitive, input data. If compression runs
1100 into difficulties caused by repetitive data, the library switches
1101 from the standard sorting algorithm to a fallback algorithm. The
1102 fallback is slower than the standard algorithm by perhaps a
1103 factor of three, but always behaves reasonably, no matter how bad
1104 the input.</p>
1106 reduce the amount of effort the standard algorithm will expend
1107 before resorting to the fallback. You should set this parameter
1108 carefully; too low, and many inputs will be handled by the
1109 fallback algorithm and so compress rather slowly, too high, and
1110 your average-to-worst case compression times can become very
1111 large. The default value of 30 gives reasonable behaviour over a
1112 wide range of circumstances.</p>
1115 <p>Note that the compressed output generated is the same
1116 regardless of whether or not the fallback algorithm is
1117 used.</p>
1118 <p>Be aware also that this parameter may disappear entirely in
1119 future versions of the library. In principle it should be
1120 possible to devise a good way to automatically choose which
1121 algorithm to use. Such a mechanism would render the parameter
1122 obsolete.</p>
1125 if the library has been mis-compiled
1126 BZ_PARAM_ERROR
1127 if strm is NULL
1131 BZ_MEM_ERROR
1132 if not enough memory is available
1133 BZ_OK
1134 otherwise</pre>
1137 if BZ_OK is returned
1138 no specific action needed in case of error</pre>
1139 </div>
1144 <p>Provides more input and/or output buffer space for the
1145 library. The caller maintains input and output buffers, and
1147 data between them.</p>
1148 <p>Before each call to
1152 should indicate how many bytes the library may read.
1157 of bytes it has read.</p>
1159 point to a buffer in which the compressed data is to be placed,
1161 much output space is available.
1166 of bytes output.</p>
1167 <p>You may provide and remove as little or as much data as you
1168 like on each call of
1170 it is acceptable to supply and remove data one byte at a time,
1171 although this would be terribly inefficient. You should always
1172 ensure that at least one byte of output space is available at
1173 each call.</p>
1174 <p>A second purpose of
1176 change of mode of the compressed stream.</p>
1177 <p>Conceptually, a compressed stream can be in one of four
1178 states: IDLE, RUNNING, FLUSHING and FINISHING. Before
1179 initialisation
1182 a stream is regarded as IDLE.</p>
1183 <p>Upon initialisation
1185 is placed in the RUNNING state. Subsequent calls to
1188 other actions are illegal and will result in
1190 <p>At some point, the calling program will have provided all
1191 the input data it wants to. It will then want to finish up -- in
1192 effect, asking the library to process any data it might have
1193 buffered internally. In this state,
1195 attempt to read data from
1198 the output buffer supplied by the user can be arbitrarily small,
1199 the finishing-up operation cannot necessarily be done with a
1200 single call of
1202 <p>Instead, the calling program passes
1205 the stream's state to FINISHING. Any remaining input (ie,
1207 compressed and transferred to the output buffer. To do this,
1209 repeatedly until all the output has been consumed. At that
1212 state is set back to IDLE.
1214 called.</p>
1215 <p>Just to make sure the calling program does not cheat, the
1217 at the time of the first call to
1220 the time the program has announced its intention to not supply
1221 any more input). By comparing this value with that of
1224 can detect any attempts to slip in more data to compress. Any
1225 calls for which this is detected will return
1227 indicates a programming mistake which should be corrected.</p>
1228 <p>Instead of asking to finish, the calling program may ask
1230 remaining input, compress it and terminate the current
1231 (Burrows-Wheeler) compression block. This could be useful for
1232 error control purposes. The mechanism is analogous to that for
1235 remove output data, and persist with the
1239 detects any attempt to provide more input data once the flush has
1240 begun.</p>
1241 <p>Once the flush is complete, the stream returns to the
1242 normal RUNNING state.</p>
1243 <p>This all sounds pretty complex, but isn't really. Here's a
1244 table which shows which actions are allowable in each state, what
1245 action will be taken, what the next state is, and what the
1246 non-error return values are. Note that you can't explicitly ask
1247 what state the stream is in, but nor do you need to -- it can be
1248 inferred from the values returned by
1251 Illegal. IDLE state only exists after BZ2_bzCompressEnd or
1252 before BZ2_bzCompressInit.
1253 Return value = BZ_SEQUENCE_ERROR
1255 RUNNING/BZ_RUN
1256 Compress from next_in to next_out as much as possible.
1257 Next state = RUNNING
1258 Return value = BZ_RUN_OK
1260 RUNNING/BZ_FLUSH
1261 Remember current value of next_in. Compress from next_in
1262 to next_out as much as possible, but do not accept any more input.
1263 Next state = FLUSHING
1264 Return value = BZ_FLUSH_OK
1266 RUNNING/BZ_FINISH
1267 Remember current value of next_in. Compress from next_in
1268 to next_out as much as possible, but do not accept any more input.
1269 Next state = FINISHING
1270 Return value = BZ_FINISH_OK
1272 FLUSHING/BZ_FLUSH
1273 Compress from next_in to next_out as much as possible,
1274 but do not accept any more input.
1275 If all the existing input has been used up and all compressed
1276 output has been removed
1277 Next state = RUNNING; Return value = BZ_RUN_OK
1278 else
1279 Next state = FLUSHING; Return value = BZ_FLUSH_OK
1281 FLUSHING/other
1282 Illegal.
1283 Return value = BZ_SEQUENCE_ERROR
1285 FINISHING/BZ_FINISH
1286 Compress from next_in to next_out as much as possible,
1287 but to not accept any more input.
1288 If all the existing input has been used up and all compressed
1289 output has been removed
1290 Next state = IDLE; Return value = BZ_STREAM_END
1291 else
1292 Next state = FINISHING; Return value = BZ_FINISH_OK
1294 FINISHING/other
1295 Illegal.
1296 Return value = BZ_SEQUENCE_ERROR</pre>
1297 <p>That still looks complicated? Well, fair enough. The
1298 usual sequence of calls for compressing a load of data is:</p>
1303 using zero or more calls of
1309 compressed output, until
1311 returned.</p></li>
1314 </ol></div>
1315 <p>If the data you want to compress fits into your input
1316 buffer all at once, you can skip the calls of
1320 <p>All required memory is allocated by
1322 compression library can accept any data at all (obviously). So
1323 you shouldn't get any error return values from the
1325 do, they will be
1327 a bug in your programming.</p>
1331 </div>
1336 <p>Releases all memory associated with a compression
1337 stream.</p>
1340 BZ_OK otherwise</pre>
1341 </div>
1345 <pre class="programlisting">int BZ2_bzDecompressInit ( bz_stream *strm, int verbosity, int small );</pre>
1346 <p>Prepares for decompression. As with
1349 allocated and initialised before the call. Fields
1353 memory allocator is required, or made
1357 internal state will have been initialised, and
1360 <p>For the meaning of parameter
1364 library will use an alternative decompression algorithm which
1365 uses less memory but at the cost of decompressing more slowly
1366 (roughly speaking, half the speed, but the maximum memory
1367 requirement drops to around 2300k). See <a class="xref" href="#using" title="2. How to use bzip2">How to use bzip2</a>
1368 for more information on memory management.</p>
1369 <p>Note that the amount of memory needed to decompress a
1370 stream cannot be determined until the stream's header has been
1371 read, so even if
1374 could fail with
1378 if the library has been mis-compiled
1379 BZ_PARAM_ERROR
1382 BZ_MEM_ERROR
1383 if insufficient memory is available</pre>
1386 if BZ_OK was returned
1387 no specific action required in case of error</pre>
1388 </div>
1393 <p>Provides more input and/out output buffer space for the
1394 library. The caller maintains input and output buffers, and uses
1396 data between them.</p>
1397 <p>Before each call to
1401 should indicate how many bytes the library may read.
1406 of bytes it has read.</p>
1408 point to a buffer in which the uncompressed output is to be
1410 indicating how much output space is available.
1415 of bytes output.</p>
1416 <p>You may provide and remove as little or as much data as you
1417 like on each call of
1419 it is acceptable to supply and remove data one byte at a time,
1420 although this would be terribly inefficient. You should always
1421 ensure that at least one byte of output space is available at
1422 each call.</p>
1424 simpler than
1426 <p>You should provide input and remove output as described
1427 above, and repeatedly call
1432 has detected the logical end of the compressed stream.
1435 output data has been placed into the output buffer, so once
1437 guaranteed to have available all the decompressed output, and
1439 be called.</p>
1440 <p>If case of an error return value, you should call
1442 and release memory.</p>
1445 if strm is NULL or strm->s is NULL
1447 BZ_DATA_ERROR
1448 if a data integrity error is detected in the compressed stream
1449 BZ_DATA_ERROR_MAGIC
1450 if the compressed stream doesn't begin with the right magic bytes
1451 BZ_MEM_ERROR
1452 if there wasn't enough memory available
1453 BZ_STREAM_END
1454 if the logical end of the data stream was detected and all
1456 BZ_OK
1457 otherwise</pre>
1460 if BZ_OK was returned
1461 BZ2_bzDecompressEnd
1462 otherwise</pre>
1463 </div>
1468 <p>Releases all memory associated with a decompression
1469 stream.</p>
1472 if strm is NULL or strm->s is NULL
1473 BZ_OK
1474 otherwise</pre>
1477 </div>
1478 </div>
1482 <p>This interface provides functions for reading and writing
1484 general points.</p>
1490 first to determine the outcome of the call. If
1493 successfully, and only then should the return value of the
1494 function (if any) be consulted. If
1497 error reading/writing the underlying compressed file, and you
1501 may also be set to various other values; precise details are
1502 given on a per-function basis below.</p></li>
1503 <li class="listitem" style="list-style-type: disc"><p>If <code class="computeroutput">bzerror</code> indicates
1504 an error (ie, anything except
1507 immediately call
1510 whether you are attempting to read or to write) to free up all
1511 resources associated with the stream. Once an error has been
1512 indicated, behaviour of all calls except
1515 undefined. The implication is that (1)
1517 after each call, and (2) if
1521 be called to clean up.</p></li>
1522 <li class="listitem" style="list-style-type: disc"><p>The <code class="computeroutput">FILE*</code> arguments
1525 to binary mode. Most Unix systems will do this by default, but
1526 other platforms, including Windows and Mac, will not. If you
1527 omit this, you may encounter problems when moving code to new
1528 platforms.</p></li>
1529 <li class="listitem" style="list-style-type: disc"><p>Memory allocation requests are handled by
1532 facility for user-defined memory allocators in the file I/O
1533 functions (could easily be added, though).</p></li>
1534 </ul></div>
1540 BZFILE *BZ2_bzReadOpen( int *bzerror, FILE *f,
1541 int verbosity, int small,
1542 void *unused, int nUnused );</pre>
1543 <p>Prepare to read compressed data from file handle
1546 has been opened for reading, and for which the error indicator
1549 to decompress using less memory, at the expense of speed.</p>
1550 <p>For reasons explained below,
1556 supplied like this. If this facility is not required, you should
1561 <p>For the meaning of parameters
1565 <p>The amount of memory needed to decompress a file cannot be
1566 determined until the file's header has been read. So it is
1571 <p>Possible assignments to
1574 if the library has been mis-compiled
1575 BZ_PARAM_ERROR
1576 if f is NULL
1580 BZ_IO_ERROR
1581 if ferror(f) is nonzero
1582 BZ_MEM_ERROR
1583 if insufficient memory is available
1584 BZ_OK
1585 otherwise.</pre>
1588 if bzerror is BZ_OK
1589 NULL
1590 otherwise</pre>
1593 if bzerror is BZ_OK
1594 BZ2_bzClose
1595 otherwise</pre>
1596 </div>
1600 <pre class="programlisting">int BZ2_bzRead ( int *bzerror, BZFILE *b, void *buf, int len );</pre>
1602 (uncompressed) bytes from the compressed file
1607 read is returned. If the logical end-of-stream was detected,
1610 bytes read is returned. All other
1612 error.</p>
1615 stream end is detected or an error occurs. Because of this, it
1616 is possible to detect the stream end by observing when the number
1617 of bytes returned is less than the number requested.
1618 Nevertheless, this is regarded as inadvisable; you should instead
1620 and watch out for
1623 copies data from the compressed file in chunks of size
1625 decompressing it. If the file contains more bytes than strictly
1626 needed to reach the logical end-of-stream,
1628 read some of the trailing data before signalling
1630 read but unused data once
1633 immediately before
1635 <p>Possible assignments to
1639 BZ_SEQUENCE_ERROR
1640 if b was opened with BZ2_bzWriteOpen
1641 BZ_IO_ERROR
1642 if there is an error reading from the compressed file
1643 BZ_UNEXPECTED_EOF
1644 if the compressed file ended before
1645 the logical end-of-stream was detected
1646 BZ_DATA_ERROR
1647 if a data integrity error was detected in the compressed stream
1648 BZ_DATA_ERROR_MAGIC
1649 if the stream does not begin with the requisite header bytes
1650 (ie, is not a bzip2 data file). This is really
1651 a special case of BZ_DATA_ERROR.
1652 BZ_MEM_ERROR
1653 if insufficient memory was available
1654 BZ_STREAM_END
1655 if the logical end of stream was detected.
1656 BZ_OK
1657 otherwise.</pre>
1660 if bzerror is BZ_OK or BZ_STREAM_END
1661 undefined
1662 otherwise</pre>
1665 if bzerror is BZ_OK
1666 collect data from buf, then BZ2_bzReadClose or BZ2_bzReadGetUnused
1667 if bzerror is BZ_SEQUENCE_END
1668 BZ2_bzReadClose
1669 otherwise</pre>
1670 </div>
1675 void** unused, int* nUnused );</pre>
1676 <p>Returns data which was read from the compressed file but
1677 was not needed to get to the logical end-of-stream.
1683 <p>This function may only be called once
1687 <p>Possible assignments to
1690 if b is NULL
1691 or unused is NULL or nUnused is NULL
1692 BZ_SEQUENCE_ERROR
1693 if BZ_STREAM_END has not been signalled
1694 or if b was opened with BZ2_bzWriteOpen
1695 BZ_OK
1696 otherwise</pre>
1699 </div>
1704 <p>Releases all memory pertaining to the compressed file
1708 handle, so you should do that yourself if appropriate.
1710 to clean up after all error situations.</p>
1711 <p>Possible assignments to
1714 if b was opened with BZ2_bzOpenWrite
1715 BZ_OK
1716 otherwise</pre>
1719 </div>
1724 int blockSize100k, int verbosity,
1725 int workFactor );</pre>
1726 <p>Prepare to write compressed data to file handle
1729 has been opened for writing, and for which the error indicator
1731 <p>For the meaning of parameters
1736 <p>All required memory is allocated at this stage, so if the
1737 call completes successfully,
1739 by a subsequent call to
1741 <p>Possible assignments to
1744 if the library has been mis-compiled
1745 BZ_PARAM_ERROR
1746 if f is NULL
1748 BZ_IO_ERROR
1749 if ferror(f) is nonzero
1750 BZ_MEM_ERROR
1751 if insufficient memory is available
1752 BZ_OK
1753 otherwise</pre>
1756 if bzerror is BZ_OK
1757 NULL
1758 otherwise</pre>
1761 if bzerror is BZ_OK
1762 (you could go directly to BZ2_bzWriteClose, but this would be pretty pointless)
1763 BZ2_bzWriteClose
1764 otherwise</pre>
1765 </div>
1769 <pre class="programlisting">void BZ2_bzWrite ( int *bzerror, BZFILE *b, void *buf, int len );</pre>
1772 compressed and written to the file.</p>
1773 <p>Possible assignments to
1777 BZ_SEQUENCE_ERROR
1778 if b was opened with BZ2_bzReadOpen
1779 BZ_IO_ERROR
1780 if there is an error writing the compressed file.
1781 BZ_OK
1782 otherwise</pre>
1783 </div>
1788 int abandon,
1789 unsigned int* nbytes_in,
1790 unsigned int* nbytes_out );
1792 void BZ2_bzWriteClose64( int *bzerror, BZFILE* f,
1793 int abandon,
1794 unsigned int* nbytes_in_lo32,
1795 unsigned int* nbytes_in_hi32,
1796 unsigned int* nbytes_out_lo32,
1797 unsigned int* nbytes_out_hi32 );</pre>
1798 <p>Compresses and flushes to the compressed file all data so
1800 The logical end-of-stream markers are also written, so subsequent
1802 illegal. All memory associated with the compressed file
1805 compressed file, but it is not
1808 called to clean up after an error, the only action is to release
1809 the memory. The library records the error codes issued by
1810 previous calls, so this situation will be detected automatically.
1811 There is no attempt to complete the compression operation, nor to
1813 can force this behaviour to happen even in the case of no error,
1814 by passing a nonzero value to
1818 total volume of uncompressed data handled. Similarly,
1820 total volume of compressed data written. For compatibility with
1821 older versions of the library,
1823 lower 32 bits of these counts. Use
1825 the full 64 bit counts. These two functions are otherwise
1826 absolutely identical.</p>
1827 <p>Possible assignments to
1830 if b was opened with BZ2_bzReadOpen
1831 BZ_IO_ERROR
1832 if there is an error writing the compressed file
1833 BZ_OK
1834 otherwise</pre>
1835 </div>
1839 <p>The high-level library facilitates use of
1841 some part of a surrounding, larger data stream.</p>
1843 <li class="listitem" style="list-style-type: disc"><p>For writing, the library takes an open file handle,
1844 writes compressed data to it,
1847 application can write its own data before and after the
1848 compressed data stream, using that same file handle.</p></li>
1849 <li class="listitem" style="list-style-type: disc"><p>Reading is more complex, and the facilities are not as
1850 general as they could be since generality is hard to reconcile
1852 reads from the compressed file in blocks of size
1854 doing so probably will overshoot the logical end of compressed
1855 stream. To recover this data once decompression has ended,
1858 (the one returning
1860 calling
1862 </ul></div>
1863 <p>This mechanism makes it easy to decompress multiple
1865 As the end of one stream, when
1869 the unused data (copy it into your own buffer somewhere). That
1870 data forms the start of the next compressed stream. To start
1871 uncompressing that next stream, call
1876 coincides with the physical end of file
1879 course return no data.</p>
1880 <p>This should give some feel for how the high-level interface
1881 can be used. If you require extra flexibility, you'll have to
1882 bite the bullet and get to grips with the low-level
1883 interface.</p>
1884 </div>
1890 BZFILE* b;
1891 int nBuf;
1892 char buf[ /* whatever size you like */ ];
1893 int bzerror;
1894 int nWritten;
1897 if ( !f ) {
1898 /* handle error */
1899 }
1901 if (bzerror != BZ_OK) {
1902 BZ2_bzWriteClose ( b );
1903 /* handle error */
1904 }
1906 while ( /* condition */ ) {
1907 /* get data to write into buf, and set nBuf appropriately */
1908 nWritten = BZ2_bzWrite ( &bzerror, b, buf, nBuf );
1909 if (bzerror == BZ_IO_ERROR) {
1910 BZ2_bzWriteClose ( &bzerror, b );
1911 /* handle error */
1912 }
1913 }
1915 BZ2_bzWriteClose( &bzerror, b );
1916 if (bzerror == BZ_IO_ERROR) {
1917 /* handle error */
1918 }</pre>
1921 BZFILE* b;
1922 int nBuf;
1923 char buf[ /* whatever size you like */ ];
1924 int bzerror;
1925 int nWritten;
1928 if ( !f ) {
1929 /* handle error */
1930 }
1932 if ( bzerror != BZ_OK ) {
1933 BZ2_bzReadClose ( &bzerror, b );
1934 /* handle error */
1935 }
1937 bzerror = BZ_OK;
1938 while ( bzerror == BZ_OK && /* arbitrary other conditions */) {
1939 nBuf = BZ2_bzRead ( &bzerror, b, buf, /* size of buf */ );
1940 if ( bzerror == BZ_OK ) {
1942 }
1943 }
1944 if ( bzerror != BZ_STREAM_END ) {
1945 BZ2_bzReadClose ( &bzerror, b );
1946 /* handle error */
1947 } else {
1948 BZ2_bzReadClose ( &bzerror, b );
1949 }</pre>
1950 </div>
1951 </div>
1959 unsigned int* destLen,
1960 char* source,
1961 unsigned int sourceLen,
1962 int blockSize100k,
1963 int verbosity,
1964 int workFactor );</pre>
1968 destination buffer is big enough,
1971 is returned. If the compressed data won't fit,
1974 returned.</p>
1975 <p>Compression in this manner is a one-shot event, done with a
1976 single call to this function. The resulting compressed data is a
1978 stream. There is no mechanism for making additional calls to
1979 provide extra input data. If you want that kind of mechanism,
1980 use the low-level interface.</p>
1981 <p>For the meaning of parameters
1986 <p>To guarantee that the compressed data will fit in its
1987 buffer, allocate an output buffer of size 1% larger than the
1988 uncompressed data, plus six hundred extra bytes.</p>
1990 will not write data at or beyond
1992 buffer overflow.</p>
1995 if the library has been mis-compiled
1996 BZ_PARAM_ERROR
1997 if dest is NULL or destLen is NULL
2001 BZ_MEM_ERROR
2002 if insufficient memory is available
2003 BZ_OUTBUFF_FULL
2004 if the size of the compressed data exceeds *destLen
2005 BZ_OK
2006 otherwise</pre>
2007 </div>
2012 unsigned int* destLen,
2013 char* source,
2014 unsigned int sourceLen,
2015 int small,
2016 int verbosity );</pre>
2020 destination buffer is big enough,
2023 is returned. If the compressed data won't fit,
2026 returned.</p>
2029 stream.
2031 to decompress the entirety of the stream into the output
2032 buffer.</p>
2033 <p>For the meaning of parameters
2037 <p>Because the compression ratio of the compressed data cannot
2038 be known in advance, there is no easy way to guarantee that the
2039 output buffer will be big enough. You may of course make
2040 arrangements in your code to record the size of the uncompressed
2041 data, but such a mechanism is beyond the scope of this
2042 library.</p>
2044 will not write data at or beyond
2046 buffer overflow.</p>
2049 if the library has been mis-compiled
2050 BZ_PARAM_ERROR
2051 if dest is NULL or destLen is NULL
2054 BZ_MEM_ERROR
2055 if insufficient memory is available
2056 BZ_OUTBUFF_FULL
2057 if the size of the compressed data exceeds *destLen
2058 BZ_DATA_ERROR
2059 if a data integrity error was detected in the compressed data
2060 BZ_DATA_ERROR_MAGIC
2061 if the compressed data doesn't begin with the right magic bytes
2062 BZ_UNEXPECTED_EOF
2063 if the compressed data ends unexpectedly
2064 BZ_OK
2065 otherwise</pre>
2066 </div>
2067 </div>
2071 <p>Yoshioka Tsuneo has contributed some functions to give
2080 functions are not (yet) officially part of the library. If they
2081 break, you get to keep all the pieces. Nevertheless, I think
2082 they work ok.</p>
2085 const char * BZ2_bzlibVersion ( void );</pre>
2088 BZFILE * BZ2_bzdopen ( int fd, const char *mode );</pre>
2090 reading or writing, using either its name or a pre-existing file
2094 int BZ2_bzwrite ( BZFILE* b, void* buf, int len );</pre>
2095 <p>Reads/writes data from/to a previously opened
2100 void BZ2_bzclose ( BZFILE* b );</pre>
2106 <p>Returns a string describing the more recent error status of
2109 value.</p>
2110 </div>
2113 <a name="stdio-free"></a>3.7. Using the library in a stdio-free environment</h2></div></div></div>
2117 <p>In a deeply embedded application, you might want to use
2118 just the memory-to-memory functions. You can do this
2119 conveniently by compiling the library with preprocessor symbol
2121 gives you a library containing only the following eight
2122 functions:</p>
2131 <p>When compiled like this, all functions will ignore
2133 </div>
2138 of internal assertion checks which should, needless to say, never
2139 be activated. Nevertheless, if an assertion should fail,
2140 behaviour depends on whether or not the library was compiled with
2142 <p>For a normal compile, an assertion failure yields the
2143 message:</p>
2147 Please report it to me at: jseward@bzip.org. If this happened
2148 when you were using some program which uses libbzip2 as a
2149 component, you should also report this bug to the author(s)
2150 of that program. Please make an effort to report this bug;
2151 timely and accurate bug reports eventually lead to higher
2153 </p>
2154 </blockquote></div>
2157 prints some extra text advising the reader that unreliable memory
2158 is often associated with internal error 1007. (This is a
2161 called.</p>
2163 assertion failures result in a call to a function declared
2164 as:</p>
2166 <p>The relevant code is passed as a parameter. You should
2167 supply such a function.</p>
2168 <p>In either case, once an assertion failure has occurred, any
2170 be regarded as invalid. You should not attempt to resume normal
2171 operation with them.</p>
2172 <p>You may, of course, change critical error handling to suit
2173 your needs. As I said above, critical errors indicate bugs in
2174 the library and should not occur. All "normal" error situations
2175 are indicated via error return codes from functions, and can be
2176 recovered from.</p>
2177 </div>
2178 </div>
2182 <p>Everything related to Windows has been contributed by
2183 Yoshioka Tsuneo
2185 you should send your queries to him (but perhaps Cc: me,
2187 <p>My vague understanding of what to do is: using Visual C++
2188 5.0, open the project file
2190 all.</p>
2191 <p>If you can't open the project file for some reason, make a
2192 new one, naming these files:
2203 <p>If you don't use VC++, you may need to define the
2204 proprocessor symbol
2207 sample program using the DLL. It has a project file,
2209 <p>If you just want a makefile for Visual C, have a look at
2211 <p>Be aware that if you compile
2216 Otherwise the resulting binary won't work correctly.</p>
2217 <p>I haven't tried any of this stuff myself, but it all looks
2218 plausible.</p>
2219 </div>
2220 </div>
2226 <dl>
2227 <dt><span class="sect1"><a href="#limits">4.1. Limitations of the compressed file format</a></span></dt>
2232 </dl>
2233 </div>
2234 <p>These are just some random thoughts of mine. Your mileage
2235 may vary.</p>
2242 format as the original version,
2244 made in the interests of stability. Creating yet another
2245 incompatible compressed file format would create further
2246 confusion and disruption for users.</p>
2247 <p>Nevertheless, this is not a painless decision. Development
2248 work since the release of
2250 shown complexities in the file format which slow down
2251 decompression and, in retrospect, are unnecessary. These
2252 are:</p>
2254 <li class="listitem" style="list-style-type: disc"><p>The run-length encoder, which is the first of the
2255 compression transformations, is entirely irrelevant. The
2256 original purpose was to protect the sorting algorithm from the
2257 very worst case input: a string of repeated symbols. But
2258 algorithm steps Q6a and Q6b in the original Burrows-Wheeler
2259 technical report (SRC-124) show how repeats can be handled
2260 without difficulty in block sorting.</p></li>
2262 <p>The randomisation mechanism doesn't really need to be
2263 there. Udi Manber and Gene Myers published a suffix array
2264 construction algorithm a few years back, which can be employed
2265 to sort any block, no matter how repetitive, in O(N log N)
2266 time. Subsequent work by Kunihiko Sadakane has produced a
2267 derivative O(N (log N)^2) algorithm which usually outperforms
2268 the Manber-Myers algorithm.</p>
2269 <p>I could have changed to Sadakane's algorithm, but I find
2271 existing algorithm for most inputs, and the randomisation
2272 mechanism protects adequately against bad cases. I didn't
2273 think it was a good tradeoff to make. Partly this is due to
2274 the fact that I was not flooded with email complaints about
2276 repetitive data, so perhaps it isn't a problem for real
2277 inputs.</p>
2278 <p>Probably the best long-term solution, and the one I have
2279 incorporated into 0.9.5 and above, is to use the existing
2280 sorting algorithm initially, and fall back to a O(N (log N)^2)
2281 algorithm if the standard algorithm gets into
2282 difficulties.</p>
2283 </li>
2284 <li class="listitem" style="list-style-type: disc"><p>The compressed file format was never designed to be
2285 handled by a library, and I have had to jump though some hoops
2286 to produce an efficient implementation of decompression. It's
2287 a bit hairy. Try passing
2289 preprocessor and you'll see what I mean. Much of this
2290 complexity could have been avoided if the compressed size of
2291 each block of data was recorded in the data stream.</p></li>
2292 <li class="listitem" style="list-style-type: disc"><p>An Adler-32 checksum, rather than a CRC32 checksum,
2293 would be faster to compute.</p></li>
2294 </ul></div>
2295 <p>It would be fair to say that the
2297 properly and fully understood the performance consequences of
2298 doing so.</p>
2300 despite using the same file format, are:</p>
2302 <li class="listitem" style="list-style-type: disc"><p>Single array implementation of the inverse BWT. This
2303 significantly speeds up decompression, presumably because it
2304 reduces the number of cache misses.</p></li>
2305 <li class="listitem" style="list-style-type: disc"><p>Faster inverse MTF transform for large MTF values.
2306 The new implementation is based on the notion of sliding blocks
2307 of values.</p></li>
2308 <li class="listitem" style="list-style-type: disc"><p><code class="computeroutput">bzip2-0.9.0</code> now reads
2313 and learn.</p></li>
2314 </ul></div>
2315 <p>Further ahead, it would be nice to be able to do random
2316 access into files. This will require some careful design of
2317 compressed file formats.</p>
2318 </div>
2322 <p>After some consideration, I have decided not to use GNU
2326 wonderful though it is, mainly assists with portability problems
2327 between Unix-like platforms. But
2329 way of portability problems on Unix; most of the difficulties
2330 appear when porting to the Mac, or to Microsoft's operating
2332 in those cases, and brings in a whole load of new
2333 complexity.</p>
2334 <p>Most people should be able to compile the library and
2335 program under Unix straight out-of-the-box, so to speak,
2336 especially if you have a version of GNU C available.</p>
2337 <p>There are a couple of
2340 able to handle them. If you're not using GNU C, your C compiler
2341 shouldn't see them at all. If your compiler does, for some
2342 reason, see them and doesn't like them, just
2346 is to compile with the flag
2348 understood by most Unix compilers.</p>
2349 <p>If you still have difficulties, try compiling with the
2351 This should enable you to build the library in a strictly ANSI
2352 compliant environment. Building the program itself like this is
2353 dangerous and not supported, since you remove
2355 compressing directories, symbolic links, devices, and other
2356 not-really-a-file entities. This could cause filesystem
2357 corruption!</p>
2358 <p>One other thing: if you create a
2361 -static</code>). This avoids all sorts of library-version
2362 issues that others may encounter later on.</p>
2367 Otherwise the resulting binary won't work correctly.</p>
2368 </div>
2372 <p>I tried pretty hard to make sure
2374 design and by testing. Hopefully you'll never need to read this
2375 section for real.</p>
2377 with a segmentation fault, a bus error or an internal assertion
2378 failure, it will ask you to email me a bug report. Experience from
2379 years of feedback of bzip2 users indicates that almost all these
2380 problems can be traced to either compiler bugs or hardware
2381 problems.</p>
2384 <p>Recompile the program with no optimisation, and
2385 see if it works. And/or try a different compiler. I heard all
2386 sorts of stories about various flavours of GNU C (and other
2387 compilers) generating bad code for
2389 such examples myself.</p>
2391 from time to time, at high optimisation levels. If you get
2392 problems, try using the flags
2398 <p>You may notice that the Makefile runs six tests as part
2399 of the build process. If the program passes all of these, it's
2400 a pretty good (but not 100%) indication that the compiler has
2401 done its job correctly.</p>
2402 </li>
2405 crashes randomly, and the crashes are not repeatable, you may
2406 have a flaky memory subsystem.
2408 memory hierarchy, and if it's a bit marginal, you may get these
2409 problems. Ditto if your disk or I/O subsystem is slowly
2410 failing. Yup, this really does happen.</p>
2411 <p>Try using a different machine of the same type, and see
2412 if you can repeat the problem.</p>
2413 </li>
2416 corrupted on decompression, and you obtained the file via FTP,
2417 there is a possibility that you forgot to tell FTP to do a
2418 binary mode transfer. That absolutely will cause the file to
2419 be non-decompressible. You'll have to transfer it
2420 again.</p></li>
2421 </ul></div>
2422 <p>If you've incorporated
2424 and are getting problems, please, please, please, check that the
2425 parameters you are passing in calls to the library, are correct,
2426 and in accordance with what the documentation says is allowable.
2427 I have tried to make the library robust against such problems,
2428 but I'm sure I haven't succeeded.</p>
2429 <p>Finally, if the above comments don't help, you'll have to
2430 send me a bug report. Now, it's just amazing how many people
2431 will send me a bug report saying something like:</p>
2433 <p>and absolutely nothing else. Needless to say, a such a
2435 comprehensively 100% useless; a waste of your time, my time, and
2436 net bandwidth</em></span>. With no details at all, there's no way
2437 I can possibly begin to figure out what the problem is.</p>
2438 <p>The rules of the game are: facts, facts, facts. Don't omit
2439 them because "oh, they won't be relevant". At the bare
2440 minimum:</p>
2442 Exact version of bzip2 (do bzip2 -V).
2443 Exact version of the compiler used.
2444 Flags passed to the compiler.</pre>
2445 <p>However, the most important single thing that will help me
2446 is the file that you were trying to compress or decompress at the
2447 time the problem happened. Without that, my ability to do
2448 anything more than speculate about the cause, is limited.</p>
2449 </div>
2454 It soaks up large amounts of CPU cycles and memory. Also, it
2455 gives very large latencies. In the worst case, you can feed many
2456 megabytes of uncompressed data into the library before getting
2457 any compressed output, so this probably rules out applications
2458 requiring interactive behaviour.</p>
2459 <p>These aren't faults of my implementation, I hope, but more
2460 an intrinsic property of the Burrows-Wheeler transform
2461 (unfortunately). Maybe this isn't what you want.</p>
2462 <p>If you want a compressor and/or library which is faster,
2463 uses less memory but gets pretty good compression, and has
2464 minimal latency, consider Jean-loup Gailly's and Mark Adler's
2469 respectively.</p>
2470 <p>For something faster and lighter still, you might try Markus F
2472 compression/decompression library, at
2473 <a class="ulink" href="http://www.oberhumer.com/opensource" target="_top">http://www.oberhumer.com/opensource</a>.</p>
2474 </div>
2479 work, in the sense that it doesn't present any new ideas.
2480 Rather, it's an engineering exercise based on existing
2481 ideas.</p>
2482 <p>Four documents describe essentially all the ideas behind
2489 If you have trouble finding it, try searching at the<br>
2490 New Zealand Digital Library, http://www.nzdl.org.<br>
2491 <br>
2492 Daniel S. Hirschberg and Debra A. LeLewer<br>
2495 You might be able to get an electronic copy of this<br>
2496 from the ACM Digital Library.<br>
2497 <br>
2498 David J. Wheeler<br>
2499 Program bred3.c and accompanying document bred3.ps.<br>
2500 This contains the idea behind the multi-table Huffman coding scheme.<br>
2501 ftp://ftp.cl.cam.ac.uk/users/djw3/<br>
2502 <br>
2503 Jon L. Bentley and Robert Sedgewick<br>
2505 Available from Sedgewick's web page,<br>
2506 www.cs.princeton.edu/~rs<br>
2507 </p></div>
2508 <p>The following paper gives valuable additional insights into
2509 the algorithm, but is not immediately the basis of any code used
2510 in bzip2.</p>
2512 Block Sorting Text Compression<br>
2515 ftp://ftp.cs.auckland.ac.nz/pub/peter-f/ACSC96paper.ps</p></div>
2516 <p>Kunihiko Sadakane's sorting algorithm, mentioned above, is
2517 available from:</p>
2519 </p></div>
2520 <p>The Manber-Myers suffix array construction algorithm is
2521 described in a paper available from:</p>
2523 </p></div>
2524 <p>Finally, the following papers document some
2525 investigations I made into the performance of sorting
2526 and decompression algorithms:</p>
2528 On the Performance of BWT Sorting Algorithms<br>
2531 <br>
2532 Julian Seward<br>
2533 Space-time Tradeoffs in the Inverse B-W Transform<br>
2536 </p></div>
2537 </div>
2538 </div>
2539 </div></body>
2540 </html>