1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
|
LZMA SDK 4.46
-------------
LZMA SDK Copyright (C) 1999-2007 Igor Pavlov
LZMA SDK provides the documentation, samples, header files, libraries,
and tools you need to develop applications that use LZMA compression.
LZMA is default and general compression method of 7z format
in 7-Zip compression program (www.7-zip.org). LZMA provides high
compression ratio and very fast decompression.
LZMA is an improved version of famous LZ77 compression algorithm.
It was improved in way of maximum increasing of compression ratio,
keeping high decompression speed and low memory requirements for
decompressing.
LICENSE
-------
LZMA SDK is available under any of the following licenses:
1) GNU Lesser General Public License (GNU LGPL)
2) Common Public License (CPL)
3) Simplified license for unmodified code (read SPECIAL EXCEPTION)
4) Proprietary license
It means that you can select one of these four options and follow rules of that license.
1,2) GNU LGPL and CPL licenses are pretty similar and both these
licenses are classified as
- "Free software licenses" at http://www.gnu.org/
- "OSI-approved" at http://www.opensource.org/
3) SPECIAL EXCEPTION
Igor Pavlov, as the author of this code, expressly permits you
to statically or dynamically link your code (or bind by name)
to the files from LZMA SDK without subjecting your linked
code to the terms of the CPL or GNU LGPL.
Any modifications or additions to files from LZMA SDK, however,
are subject to the GNU LGPL or CPL terms.
SPECIAL EXCEPTION allows you to use LZMA SDK in applications with closed code,
while you keep LZMA SDK code unmodified.
SPECIAL EXCEPTION #2: Igor Pavlov, as the author of this code, expressly permits
you to use this code under the same terms and conditions contained in the License
Agreement you have for any previous version of LZMA SDK developed by Igor Pavlov.
SPECIAL EXCEPTION #2 allows owners of proprietary licenses to use latest version
of LZMA SDK as update for previous versions.
SPECIAL EXCEPTION #3: Igor Pavlov, as the author of this code, expressly permits
you to use code of the following files:
BranchTypes.h, LzmaTypes.h, LzmaTest.c, LzmaStateTest.c, LzmaAlone.cpp,
LzmaAlone.cs, LzmaAlone.java
as public domain code.
4) Proprietary license
LZMA SDK also can be available under a proprietary license which
can include:
1) Right to modify code without subjecting modified code to the
terms of the CPL or GNU LGPL
2) Technical support for code
To request such proprietary license or any additional consultations,
send email message from that page:
http://www.7-zip.org/support.html
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
You should have received a copy of the Common Public License
along with this library.
LZMA SDK Contents
-----------------
LZMA SDK includes:
- C++ source code of LZMA compressing and decompressing
- ANSI-C compatible source code for LZMA decompressing
- C# source code for LZMA compressing and decompressing
- Java source code for LZMA compressing and decompressing
- Compiled file->file LZMA compressing/decompressing program for Windows system
ANSI-C LZMA decompression code was ported from original C++ sources to C.
Also it was simplified and optimized for code size.
But it is fully compatible with LZMA from 7-Zip.
UNIX/Linux version
------------------
To compile C++ version of file->file LZMA, go to directory
C/7zip/Compress/LZMA_Alone
and type "make" or "make clean all" to recompile all.
In some UNIX/Linux versions you must compile LZMA with static libraries.
To compile with static libraries, change string in makefile
LIB = -lm
to string
LIB = -lm -static
Files
---------------------
C - C source code
CPP - CPP source code
CS - C# source code
Java - Java source code
lzma.txt - LZMA SDK description (this file)
7zFormat.txt - 7z Format description
7zC.txt - 7z ANSI-C Decoder description (this file)
methods.txt - Compression method IDs for .7z
LGPL.txt - GNU Lesser General Public License
CPL.html - Common Public License
lzma.exe - Compiled file->file LZMA encoder/decoder for Windows
history.txt - history of the LZMA SDK
Source code structure
---------------------
C - C files
Compress - files related to compression/decompression
Lz - files related to LZ (Lempel-Ziv) compression algorithm
Lzma - ANSI-C compatible LZMA decompressor
LzmaDecode.h - interface for LZMA decoding on ANSI-C
LzmaDecode.c - LZMA decoding on ANSI-C (new fastest version)
LzmaDecodeSize.c - LZMA decoding on ANSI-C (old size-optimized version)
LzmaTest.c - test application that decodes LZMA encoded file
LzmaTypes.h - basic types for LZMA Decoder
LzmaStateDecode.h - interface for LZMA decoding (State version)
LzmaStateDecode.c - LZMA decoding on ANSI-C (State version)
LzmaStateTest.c - test application (State version)
Branch - Filters for x86, IA-64, ARM, ARM-Thumb, PowerPC and SPARC code
Archive - files related to archiving
7z_C - 7z ANSI-C Decoder
CPP -- CPP files
Common - common files for C++ projects
Windows - common files for Windows related code
7zip - files related to 7-Zip Project
Common - common files for 7-Zip
Compress - files related to compression/decompression
LZ - files related to LZ (Lempel-Ziv) compression algorithm
Copy - Copy coder
RangeCoder - Range Coder (special code of compression/decompression)
LZMA - LZMA compression/decompression on C++
LZMA_Alone - file->file LZMA compression/decompression
Branch - Filters for x86, IA-64, ARM, ARM-Thumb, PowerPC and SPARC code
Archive - files related to archiving
Common - common files for archive handling
7z - 7z C++ Encoder/Decoder
Bundle - Modules that are bundles of other modules
Alone7z - 7zr.exe: Standalone version of 7z.exe that supports only 7z/LZMA/BCJ/BCJ2
Format7zR - 7zr.dll: Reduced version of 7za.dll: extracting/compressing to 7z/LZMA/BCJ/BCJ2
Format7zExtractR - 7zxr.dll: Reduced version of 7zxa.dll: extracting from 7z/LZMA/BCJ/BCJ2.
UI - User Interface files
Client7z - Test application for 7za.dll, 7zr.dll, 7zxr.dll
Common - Common UI files
Console - Code for console archiver
CS - C# files
7zip
Common - some common files for 7-Zip
Compress - files related to compression/decompression
LZ - files related to LZ (Lempel-Ziv) compression algorithm
LZMA - LZMA compression/decompression
LzmaAlone - file->file LZMA compression/decompression
RangeCoder - Range Coder (special code of compression/decompression)
Java - Java files
SevenZip
Compression - files related to compression/decompression
LZ - files related to LZ (Lempel-Ziv) compression algorithm
LZMA - LZMA compression/decompression
RangeCoder - Range Coder (special code of compression/decompression)
C/C++ source code of LZMA SDK is part of 7-Zip project.
You can find ANSI-C LZMA decompressing code at folder
C/7zip/Compress/Lzma
7-Zip doesn't use that ANSI-C LZMA code and that code was developed
specially for this SDK. And files from C/7zip/Compress/Lzma do not need
files from other directories of SDK for compiling.
7-Zip source code can be downloaded from 7-Zip's SourceForge page:
http://sourceforge.net/projects/sevenzip/
LZMA features
-------------
- Variable dictionary size (up to 1 GB)
- Estimated compressing speed: about 1 MB/s on 1 GHz CPU
- Estimated decompressing speed:
- 8-12 MB/s on 1 GHz Intel Pentium 3 or AMD Athlon
- 500-1000 KB/s on 100 MHz ARM, MIPS, PowerPC or other simple RISC
- Small memory requirements for decompressing (8-32 KB + DictionarySize)
- Small code size for decompressing: 2-8 KB (depending from
speed optimizations)
LZMA decoder uses only integer operations and can be
implemented in any modern 32-bit CPU (or on 16-bit CPU with some conditions).
Some critical operations that affect to speed of LZMA decompression:
1) 32*16 bit integer multiply
2) Misspredicted branches (penalty mostly depends from pipeline length)
3) 32-bit shift and arithmetic operations
Speed of LZMA decompressing mostly depends from CPU speed.
Memory speed has no big meaning. But if your CPU has small data cache,
overall weight of memory speed will slightly increase.
How To Use
----------
Using LZMA encoder/decoder executable
--------------------------------------
Usage: LZMA <e|d> inputFile outputFile [<switches>...]
e: encode file
d: decode file
b: Benchmark. There are two tests: compressing and decompressing
with LZMA method. Benchmark shows rating in MIPS (million
instructions per second). Rating value is calculated from
measured speed and it is normalized with AMD Athlon 64 X2 CPU
results. Also Benchmark checks possible hardware errors (RAM
errors in most cases). Benchmark uses these settings:
(-a1, -d21, -fb32, -mfbt4). You can change only -d. Also you
can change number of iterations. Example for 30 iterations:
LZMA b 30
Default number of iterations is 10.
<Switches>
-a{N}: set compression mode 0 = fast, 1 = normal
default: 1 (normal)
d{N}: Sets Dictionary size - [0, 30], default: 23 (8MB)
The maximum value for dictionary size is 1 GB = 2^30 bytes.
Dictionary size is calculated as DictionarySize = 2^N bytes.
For decompressing file compressed by LZMA method with dictionary
size D = 2^N you need about D bytes of memory (RAM).
-fb{N}: set number of fast bytes - [5, 273], default: 128
Usually big number gives a little bit better compression ratio
and slower compression process.
-lc{N}: set number of literal context bits - [0, 8], default: 3
Sometimes lc=4 gives gain for big files.
-lp{N}: set number of literal pos bits - [0, 4], default: 0
lp switch is intended for periodical data when period is
equal 2^N. For example, for 32-bit (4 bytes)
periodical data you can use lp=2. Often it's better to set lc0,
if you change lp switch.
-pb{N}: set number of pos bits - [0, 4], default: 2
pb switch is intended for periodical data
when period is equal 2^N.
-mf{MF_ID}: set Match Finder. Default: bt4.
Algorithms from hc* group doesn't provide good compression
ratio, but they often works pretty fast in combination with
fast mode (-a0).
Memory requirements depend from dictionary size
(parameter "d" in table below).
MF_ID Memory Description
bt2 d * 9.5 + 4MB Binary Tree with 2 bytes hashing.
bt3 d * 11.5 + 4MB Binary Tree with 3 bytes hashing.
bt4 d * 11.5 + 4MB Binary Tree with 4 bytes hashing.
hc4 d * 7.5 + 4MB Hash Chain with 4 bytes hashing.
-eos: write End Of Stream marker. By default LZMA doesn't write
eos marker, since LZMA decoder knows uncompressed size
stored in .lzma file header.
-si: Read data from stdin (it will write End Of Stream marker).
-so: Write data to stdout
Examples:
1) LZMA e file.bin file.lzma -d16 -lc0
compresses file.bin to file.lzma with 64 KB dictionary (2^16=64K)
and 0 literal context bits. -lc0 allows to reduce memory requirements
for decompression.
2) LZMA e file.bin file.lzma -lc0 -lp2
compresses file.bin to file.lzma with settings suitable
for 32-bit periodical data (for example, ARM or MIPS code).
3) LZMA d file.lzma file.bin
decompresses file.lzma to file.bin.
Compression ratio hints
-----------------------
Recommendations
---------------
To increase compression ratio for LZMA compressing it's desirable
to have aligned data (if it's possible) and also it's desirable to locate
data in such order, where code is grouped in one place and data is
grouped in other place (it's better than such mixing: code, data, code,
data, ...).
Using Filters
-------------
You can increase compression ratio for some data types, using
special filters before compressing. For example, it's possible to
increase compression ratio on 5-10% for code for those CPU ISAs:
x86, IA-64, ARM, ARM-Thumb, PowerPC, SPARC.
You can find C/C++ source code of such filters in folder "7zip/Compress/Branch"
You can check compression ratio gain of these filters with such
7-Zip commands (example for ARM code):
No filter:
7z a a1.7z a.bin -m0=lzma
With filter for little-endian ARM code:
7z a a2.7z a.bin -m0=bc_arm -m1=lzma
With filter for big-endian ARM code (using additional Swap4 filter):
7z a a3.7z a.bin -m0=swap4 -m1=bc_arm -m2=lzma
It works in such manner:
Compressing = Filter_encoding + LZMA_encoding
Decompressing = LZMA_decoding + Filter_decoding
Compressing and decompressing speed of such filters is very high,
so it will not increase decompressing time too much.
Moreover, it reduces decompression time for LZMA_decoding,
since compression ratio with filtering is higher.
These filters convert CALL (calling procedure) instructions
from relative offsets to absolute addresses, so such data becomes more
compressible. Source code of these CALL filters is pretty simple
(about 20 lines of C++), so you can convert it from C++ version yourself.
For some ISAs (for example, for MIPS) it's impossible to get gain from such filter.
LZMA compressed file format
---------------------------
Offset Size Description
0 1 Special LZMA properties for compressed data
1 4 Dictionary size (little endian)
5 8 Uncompressed size (little endian). -1 means unknown size
13 Compressed data
ANSI-C LZMA Decoder
~~~~~~~~~~~~~~~~~~~
To compile ANSI-C LZMA Decoder you can use one of the following files sets:
1) LzmaDecode.h + LzmaDecode.c + LzmaTest.c (fastest version)
2) LzmaDecode.h + LzmaDecodeSize.c + LzmaTest.c (old size-optimized version)
3) LzmaStateDecode.h + LzmaStateDecode.c + LzmaStateTest.c (zlib-like interface)
Memory requirements for LZMA decoding
-------------------------------------
LZMA decoder doesn't allocate memory itself, so you must
allocate memory and send it to LZMA.
Stack usage of LZMA decoding function for local variables is not
larger than 200 bytes.
How To decompress data
----------------------
LZMA Decoder (ANSI-C version) now supports 5 interfaces:
1) Single-call Decompressing
2) Single-call Decompressing with input stream callback
3) Multi-call Decompressing with output buffer
4) Multi-call Decompressing with input callback and output buffer
5) Multi-call State Decompressing (zlib-like interface)
Variant-5 is similar to Variant-4, but Variant-5 doesn't use callback functions.
Decompressing steps
-------------------
1) read LZMA properties (5 bytes):
unsigned char properties[LZMA_PROPERTIES_SIZE];
2) read uncompressed size (8 bytes, little-endian)
3) Decode properties:
CLzmaDecoderState state; /* it's 24-140 bytes structure, if int is 32-bit */
if (LzmaDecodeProperties(&state.Properties, properties, LZMA_PROPERTIES_SIZE) != LZMA_RESULT_OK)
return PrintError(rs, "Incorrect stream properties");
4) Allocate memory block for internal Structures:
state.Probs = (CProb *)malloc(LzmaGetNumProbs(&state.Properties) * sizeof(CProb));
if (state.Probs == 0)
return PrintError(rs, kCantAllocateMessage);
LZMA decoder uses array of CProb variables as internal structure.
By default, CProb is unsigned_short. But you can define _LZMA_PROB32 to make
it unsigned_int. It can increase speed on some 32-bit CPUs, but memory
usage will be doubled in that case.
5) Main Decompressing
You must use one of the following interfaces:
5.1 Single-call Decompressing
-----------------------------
When to use: RAM->RAM decompressing
Compile files: LzmaDecode.h, LzmaDecode.c
Compile defines: no defines
Memory Requirements:
- Input buffer: compressed size
- Output buffer: uncompressed size
- LZMA Internal Structures (~16 KB for default settings)
Interface:
int res = LzmaDecode(&state,
inStream, compressedSize, &inProcessed,
outStream, outSize, &outProcessed);
5.2 Single-call Decompressing with input stream callback
--------------------------------------------------------
When to use: File->RAM or Flash->RAM decompressing.
Compile files: LzmaDecode.h, LzmaDecode.c
Compile defines: _LZMA_IN_CB
Memory Requirements:
- Buffer for input stream: any size (for example, 16 KB)
- Output buffer: uncompressed size
- LZMA Internal Structures (~16 KB for default settings)
Interface:
typedef struct _CBuffer
{
ILzmaInCallback InCallback;
FILE *File;
unsigned char Buffer[kInBufferSize];
} CBuffer;
int LzmaReadCompressed(void *object, const unsigned char **buffer, SizeT *size)
{
CBuffer *bo = (CBuffer *)object;
*buffer = bo->Buffer;
*size = MyReadFile(bo->File, bo->Buffer, kInBufferSize);
return LZMA_RESULT_OK;
}
CBuffer g_InBuffer;
g_InBuffer.File = inFile;
g_InBuffer.InCallback.Read = LzmaReadCompressed;
int res = LzmaDecode(&state,
&g_InBuffer.InCallback,
outStream, outSize, &outProcessed);
5.3 Multi-call decompressing with output buffer
-----------------------------------------------
When to use: RAM->File decompressing
Compile files: LzmaDecode.h, LzmaDecode.c
Compile defines: _LZMA_OUT_READ
Memory Requirements:
- Input buffer: compressed size
- Buffer for output stream: any size (for example, 16 KB)
- LZMA Internal Structures (~16 KB for default settings)
- LZMA dictionary (dictionary size is encoded in stream properties)
Interface:
state.Dictionary = (unsigned char *)malloc(state.Properties.DictionarySize);
LzmaDecoderInit(&state);
do
{
LzmaDecode(&state,
inBuffer, inAvail, &inProcessed,
g_OutBuffer, outAvail, &outProcessed);
inAvail -= inProcessed;
inBuffer += inProcessed;
}
while you need more bytes
see LzmaTest.c for more details.
5.4 Multi-call decompressing with input callback and output buffer
------------------------------------------------------------------
When to use: File->File decompressing
Compile files: LzmaDecode.h, LzmaDecode.c
Compile defines: _LZMA_IN_CB, _LZMA_OUT_READ
Memory Requirements:
- Buffer for input stream: any size (for example, 16 KB)
- Buffer for output stream: any size (for example, 16 KB)
- LZMA Internal Structures (~16 KB for default settings)
- LZMA dictionary (dictionary size is encoded in stream properties)
Interface:
state.Dictionary = (unsigned char *)malloc(state.Properties.DictionarySize);
LzmaDecoderInit(&state);
do
{
LzmaDecode(&state,
&bo.InCallback,
g_OutBuffer, outAvail, &outProcessed);
}
while you need more bytes
see LzmaTest.c for more details:
5.5 Multi-call State Decompressing (zlib-like interface)
------------------------------------------------------------------
When to use: file->file decompressing
Compile files: LzmaStateDecode.h, LzmaStateDecode.c
Compile defines:
Memory Requirements:
- Buffer for input stream: any size (for example, 16 KB)
- Buffer for output stream: any size (for example, 16 KB)
- LZMA Internal Structures (~16 KB for default settings)
- LZMA dictionary (dictionary size is encoded in stream properties)
Interface:
state.Dictionary = (unsigned char *)malloc(state.Properties.DictionarySize);
LzmaDecoderInit(&state);
do
{
res = LzmaDecode(&state,
inBuffer, inAvail, &inProcessed,
g_OutBuffer, outAvail, &outProcessed,
finishDecoding);
inAvail -= inProcessed;
inBuffer += inProcessed;
}
while you need more bytes
see LzmaStateTest.c for more details:
6) Free all allocated blocks
Note
----
LzmaDecodeSize.c is size-optimized version of LzmaDecode.c.
But compiled code of LzmaDecodeSize.c can be larger than
compiled code of LzmaDecode.c. So it's better to use
LzmaDecode.c in most cases.
EXIT codes
-----------
LZMA decoder can return one of the following codes:
#define LZMA_RESULT_OK 0
#define LZMA_RESULT_DATA_ERROR 1
If you use callback function for input data and you return some
error code, LZMA Decoder also returns that code.
LZMA Defines
------------
_LZMA_IN_CB - Use callback for input data
_LZMA_OUT_READ - Use read function for output data
_LZMA_LOC_OPT - Enable local speed optimizations inside code.
_LZMA_LOC_OPT is only for LzmaDecodeSize.c (size-optimized version).
_LZMA_LOC_OPT doesn't affect LzmaDecode.c (speed-optimized version)
and LzmaStateDecode.c
_LZMA_PROB32 - It can increase speed on some 32-bit CPUs,
but memory usage will be doubled in that case
_LZMA_UINT32_IS_ULONG - Define it if int is 16-bit on your compiler
and long is 32-bit.
_LZMA_SYSTEM_SIZE_T - Define it if you want to use system's size_t.
You can use it to enable 64-bit sizes supporting
C++ LZMA Encoder/Decoder
~~~~~~~~~~~~~~~~~~~~~~~~
C++ LZMA code use COM-like interfaces. So if you want to use it,
you can study basics of COM/OLE.
By default, LZMA Encoder contains all Match Finders.
But for compressing it's enough to have just one of them.
So for reducing size of compressing code you can define:
#define COMPRESS_MF_BT
#define COMPRESS_MF_BT4
and it will use only bt4 match finder.
---
http://www.7-zip.org
http://www.7-zip.org/support.html
|