LZMA SDK - How to Use

Warning! Some information on this page is older than 3 years now. I keep it for reference, but it probably doesn't reflect my current knowledge and beliefs.

May 2010

LZMA SDK - How to Use

What do you think about when I tell a word "compression"? If you currently study computer science, you probably think about some details of algorithms like RLE, Huffman coding or Burrows-Wheeler transform. If not, then you surely associate compression with archive file formats such as ZIP and RAR. But there is something in between - a kind of libraries that let you compress some data - implement a compression algorithm but do not provide ready file format to pack multiple files and directories into one archive. Such library is useful in gamedev for creating VFS (Virtual File System). Probably the most popular one is zlib - a free C library that implements Deflate algorithm. I've recently discovered another one - LZMA. Its SDK is also free (public domain) and the basic version is a small C library (C++, C# and Java API-s are also available, as well as some additional tools). The library uses LZMA algorithm (Lempel–Ziv–Markov chain algorithm, same as in 7z archive format), which has better compression ratio than Deflate AFAIK. So I've decided to start using it. Here is what I've learned:

If you decide to use only the C API, it's enough to add some C and H files to your project - the ones from LZMASDK\C directory (without subdirectories). Alternatively you can compile them as a static library.

There is a little bit of theory behind the LZMA SDK API. First, the term props means a 5-byte header where the library stores some settings. It must be saved with compressed data to be given to the library before decompression.

Next, the dictionary size. It is the size of a temporary memory block used during compression and decompression. Dictionary size can be set during compression and is then saved inside props. Library uses dictionary of same size during decompression. Default dictionary size is 16 MB so IMHO it's worth changing, especially as I haven't noticed any meaninful drop in compression rate when set it to 64 KB.

And finally, end mark can be saved at the end of compressed data. You can use it so decompression code can determine the end of data. Alternatively you can decide not to use the end mark, but you must remember the exact size of uncompressed data somewhere. I prefer the second method, because remembering data size takes only 4 bytes (for the 4 GB limit) and can be useful anyway, while compressed data finished with end mark are about 6 bytes longer than without it.

Compressing full block of data with single call is simple. You can find appropriate functions in LzmaLib.h header. Here is how you can compress a vector of bytes using LzmaCompress function:

void Compress1(
  std::vector<unsigned char> &outBuf,
  const std::vector<unsigned char> &inBuf)
  unsigned propsSize = LZMA_PROPS_SIZE;
  unsigned destLen = inBuf.size() + inBuf.size() / 3 + 128;
  outBuf.resize(propsSize + destLen);
  int res = LzmaCompress(
    &outBuf[LZMA_PROPS_SIZE], &destLen,
    &inBuf[0], inBuf.size(),
    &outBuf[0], &propsSize,
    -1, 0, -1, -1, -1, -1, -1);
  assert(propsSize == LZMA_PROPS_SIZE);
  assert(res == SZ_OK);
  outBuf.resize(propsSize + destLen);

Destination buffer - outBuf - will contain props in its first 5 bytes (LZMA_PROPS_SIZE) and compressed data in the remaining part (it's just my idea to store it this way). Starting size of outBuf (destLen) is set to some arbitrary quantity big enough to fit all compressed data. The buffer is then trimmed to its real size at the end. LzmaCompress function always saves compressed data without the end mark. Its last 7 parameters are some compression settings (including dictionary size), here set to special values meaning defaults.

A little more advanced example is also a compression of all data with one call, but with props structure explicitly declared and filled. This example uses LzmaEnc.h header and LzmaEncode function.

void Compress2(
  std::vector<unsigned char> &outBuf,
  const std::vector<unsigned char> &inBuf)
  unsigned propsSize = LZMA_PROPS_SIZE;
  unsigned destLen = inBuf.size() + inBuf.size() / 3 + 128;
  outBuf.resize(propsSize + destLen);

  CLzmaEncProps props;
  props.dictSize = 1 << 16; // 64 KB
  props.writeEndMark = 1; // 0 or 1

  int res = LzmaEncode(
    &outBuf[LZMA_PROPS_SIZE], &destLen,
    &inBuf[0], inBuf.size(),
    &props, &outBuf[0], &propsSize, props.writeEndMark,
    &g_ProgressCallback, &SzAllocForLzma, &SzAllocForLzma);
  assert(res == SZ_OK && propsSize == LZMA_PROPS_SIZE);
  outBuf.resize(propsSize + destLen);

Here we first define props structure of type CLzmaEncProps, initialize it with default values and then change some fields. We can change here not only dictionary size, but also a flag telling whether compressed data should end with the end mark. The props are encoded to &inBuf[0] and data are compressed to the location 5 bytes further (&outBuf[LZMA_PROPS_SIZE]) by the LzmaEncode function.

g_ProgressCallback is a pointer to a C "interface" (let's call it this way ;P) where you can place your callback with progress notification.

SRes OnProgress(void *p, UInt64 inSize, UInt64 outSize)
  // Update progress bar.
  return SZ_OK;
static ICompressProgress g_ProgressCallback = { &OnProgress };

SzAllocForLzma is another interface which gives LZMA library pointers to the memory allocation and deallocation functions. To just use standard malloc and free functions, you can copy this code:

static void * AllocForLzma(void *p, size_t size) { return malloc(size); }
static void FreeForLzma(void *p, void *address) { free(address); }
static ISzAlloc SzAllocForLzma = { &AllocForLzma, &FreeForLzma };

Decompression with single call is also quite simple. You #include "LzmaLib.h" and use LzmaUncompress function.

static void Uncompress1(
  std::vector<unsigned char> &outBuf,
  const std::vector<unsigned char> &inBuf)
  unsigned dstLen = outBuf.size();
  unsigned srcLen = inBuf.size() - LZMA_PROPS_SIZE;
  SRes res = LzmaUncompress(
    &outBuf[0], &dstLen,
    &inBuf[LZMA_PROPS_SIZE], &srcLen,
    &inBuf[0], LZMA_PROPS_SIZE);
  assert(res == SZ_OK);
  outBuf.resize(dstLen); // If uncompressed data can be smaller

You can request decompression of the exact number of bytes if you know the uncompressed data size or alternatively you can try to uncompress more bytes when compressed data are finished with the end mark. In the second case, you should check for new value of dstLen and trim the output buffer to that size. Another function for single-call decompression is LzmaDecode from LzmaDec.h file.

Incremental compression and decompression is a little more complicated. Unfortunately there is no push-like interface for incremental compression. To do that, you have to make a single call to LzmaEnc_Encode function from LzmaEnc.h and pass your implementation of ISeqInStream and ISeqOutStream "interfaces" that read uncompressed and save compressed data. Here is an example:

typedef struct
  ISeqInStream SeqInStream;
  const std::vector<unsigned char> *Buf;
  unsigned BufPos;
} VectorInStream;

SRes VectorInStream_Read(void *p, void *buf, size_t *size)
  VectorInStream *ctx = (VectorInStream*)p;
  *size = min(*size, ctx->Buf->size() - ctx->BufPos);
  if (*size)
    memcpy(buf, &(*ctx->Buf)[ctx->BufPos], *size);
  ctx->BufPos += *size;
  return SZ_OK;

typedef struct
  ISeqOutStream SeqOutStream;
  std::vector<unsigned char> *Buf;
} VectorOutStream;

size_t VectorOutStream_Write(void *p, const void *buf, size_t size)
  VectorOutStream *ctx = (VectorOutStream*)p;
  if (size)
    unsigned oldSize = ctx->Buf->size();
    ctx->Buf->resize(oldSize + size);
    memcpy(&(*ctx->Buf)[oldSize], buf, size);
  return size;

static void CompressInc(
  std::vector<unsigned char> &outBuf,
  const std::vector<unsigned char> &inBuf)
  CLzmaEncHandle enc = LzmaEnc_Create(&SzAllocForLzma);

  CLzmaEncProps props;
  props.writeEndMark = 1; // 0 or 1
  SRes res = LzmaEnc_SetProps(enc, &props);
  assert(res == SZ_OK);

  unsigned propsSize = LZMA_PROPS_SIZE;

  res = LzmaEnc_WriteProperties(enc, &outBuf[0], &propsSize);
  assert(res == SZ_OK && propsSize == LZMA_PROPS_SIZE);

  VectorInStream inStream = { &VectorInStream_Read, &inBuf, 0 };
  VectorOutStream outStream = { &VectorOutStream_Write, &outBuf };

  res = LzmaEnc_Encode(enc,
    (ISeqOutStream*)&outStream, (ISeqInStream*)&inStream,
    0, &SzAllocForLzma, &SzAllocForLzma);
  assert(res == SZ_OK);

  LzmaEnc_Destroy(enc, &SzAllocForLzma, &SzAllocForLzma);

Incremental decompression is even more complicated, although here we have an interface to just process a piece of data. We use LzmaDec_DecodeToBuf function from LzmaDec.h. For example:

static void UncompressInc(
  std::vector<unsigned char> &outBuf,
  const std::vector<unsigned char> &inBuf)
  CLzmaDec dec;  
  SRes res = LzmaDec_Allocate(&dec, &inBuf[0], LZMA_PROPS_SIZE, &SzAllocForLzma);
  assert(res == SZ_OK);


  unsigned outPos = 0, inPos = LZMA_PROPS_SIZE;
  ELzmaStatus status;
  const unsigned BUF_SIZE = 10240;
  while (outPos < outBuf.size())
    unsigned destLen = min(BUF_SIZE, outBuf.size() - outPos);
    unsigned srcLen  = min(BUF_SIZE, inBuf.size() - inPos);
    unsigned srcLenOld = srcLen, destLenOld = destLen;
    res = LzmaDec_DecodeToBuf(&dec,
      &outBuf[outPos], &destLen,
      &inBuf[inPos], &srcLen,
      (outPos + destLen == outBuf.size())
      ? LZMA_FINISH_END : LZMA_FINISH_ANY, &status);
    assert(res == SZ_OK);
    inPos += srcLen;
    outPos += destLen;

  LzmaDec_Free(&dec, &SzAllocForLzma);

LzmaDec_DecodeToBuf has several possible status codes returned via last parameter. LZMA_STATUS_FINISHED_WITH_MARK means decompression finished with end mark met inside compressed data. Look for it if you use use end mark. LZMA_STATUS_MAYBE_FINISHED_WITHOUT_MARK means it's possible that compressed data finished in this place but it's your task to be sure whether you already have as much uncompressed data as you expect. Check for this code when you don't use end mark. Finally LZMA_STATUS_NOT_FINISHED and LZMA_STATUS_NEEDS_MORE_INPUT mean you are in the middle of decompression and need to call the function again to process more data. Of course you should check if res == SZ_OK first to ensure there is no error.

That was just some test code from my learning and now it's time for the real one. It's not a standalone code snippet, but it's finished, well tested and will be included in the next version of my CommonLib libary. You can preview it (and use it however you want) here:


Comments (3) | Tags: commonlib libraries algorithms | Author: Adam Sawicki | Share


Chanel Handbags
2014-08-19 10:11:24
http://www.coachoutletstoreinuk.com/ Coach Outlet Online
http://www.coachfactoryoutletanus.com/ Coach Outlet
http://www.coachoutletonlineunsius.com/ Coach Factory Outlet
http://www.coachoutletonlinetur.com/ Coach Outlet Online 
http://www.coachfactoryoutletnbsa.com/ Coach Factory Outlet
http://www.coachfactoryoutletuisa.com/ Coach Outlet Store Online http://www.coachoutletstorenie.com/ Coach Outlet Store Online
http://www.coachfactorystoreuin.com/ Coach Factory Outlet
http://www.coachfactoryoutletreba.com/ Coach Outlet 
http://www.coachfactoryoutletonlinesius.com/ Coach Handbags Outlet  http://www.coachoutletonlinetnse.com/ Coach Outlet http://www.coachoutletnoia.com/ Coach Outlet
http://www.chaneloutletinaus.com/ Chanel Outlet Online
http://www.louisvuittonoutletina.com/ Louis Vuitton
http://www.louisvuittonoutletine.com/ Louis Vuitton Outlet
http://www.guccibeltsoutletbcus.com/ Gucci Belts
http://www.guccibelststco.com/ Gucci Belt
http://www.coachoutletstoreinuk.com/ Coach Outlet Online
http://www.coachfactoryoutletanus.com/ Coach Outlet
http://www.coachoutletonlineunsius.com/ Coach Factory Outlet http://www.coachoutletonlinetur.com/ Coach Outlet Online
http://www.coachfactoryoutletnbsa.com/ Coach Outlet Store
http://www.coachfactoryoutletuisa.com/ Coach Outlet Store Online
http://www.coachoutletstorenie.com/ Coach Outlet
http://www.coachfactorystoreuin.com/ Coach Factory Online
http://www.coachfactoryoutletreba.com/ Coach Factory Outlet http://www.coachfactoryoutletonlinesius.com/ Coach Factory Outlet
http://www.coachoutletonlinetnse.com/ Coach Outlet http://www.coachoutletnoia.com/ Coach Outlet Store Online
http://www.chaneloutletinaus.com/ Chanel Outlet Online
http://www.louisvuittonoutletina.com/ Louis Vuitton Outlet
http://www.louisvuittonoutletine.com/ Louis Vuitton Handbags
http://www.guccibeltsoutletbcus.com/ Gucci Belt
http://www.guccibelststco.com/ Gucci Belts
http://www.louisvuittonoutletsnib.com/ Louis Vuitton Outlet Online
http://www.oakleysunglassescnc.name/ Oakley Sunglasses Outlet
http://www.louisvuittonoutletmoba.com/ Louis Vuitton Outlet Online
http://www.oakleyssunglassesmobo.com/ Oakley Sunglasses Outlet
http://www.oakleysunglassesmaba.com/ Cheap Oakley Sunglasses
http://www.oakleysunglassinsc.com/ Oakley Sunglasses
http://www.louisvuittonoutletsnib.com/ Louis Vuitton Outlet
http://www.oakleysunglassescnc.name/ Oakley Sunglasses Outlet
http://www.louisvuittonoutletmoba.com/ Louis Vuitton Outlet
http://www.oakleyssunglassesmobo.com/ Oakley Sunglasses Outlet
http://www.oakleysunglassesmaba.com/ Oakley Sunglasses
http://www.oakleysunglassinsc.com/ Oakley Sunglasses
2014-09-21 05:43:45
2015-04-10 09:25:55
chanel bags, http://www.chanelhandbags.net.co/
flat iron, http://www.chiflatiron.net.co/
converse shoes, http://www.converse.net.co/
ferragamo outlet, http://www.ferragamo.com.co/
gucci belt, http://www.guccihandbags.com.co/
hollister kids, http://www.hollister.us.org/
insanity workout calendar, http://www.insanityworkout.net.co/
jimmy choo shoes, http://www.jimmychoo.net.co/
air jordan shoes, http://www.jordan-shoes.com.co/
juicy couture, http://www.juicycouture.com.co/
juicy couture outlet, http://www.juicycoutureoutlet.net.co/
kate spade, http://www.kate-spade.com.co/
katespade, http://www.katespade-outlet.com.co/
louis vuitton outlet, http://www.louisvuitton-outlet.com.co/
michael kors outlet online, http://www.michael-kors.us.org/
michael kors, http://www.michael-kors-handbags.us.com/
michael kors purses, http://www.michael-kors-outlet.us.org/
michael kors, http://www.michael-kors-outlet-online.us.org/
michael kors outlet online sale, http://www.michaelkorsoutletonline-sale.us.com/
north face jackets, http://www.northface.us.org/
oakley outlet, http://www.oakley-sunglasses.us.org/
pandora charms, http://www.pandorajewelry.com.co/
prada outlet, http://www.pradahandbags.com.co/
prada shoes for men, http://www.pradashoes.com.co/
ray-ban, http://www.ray-ban-outlet.us.com/
ray ban sunglasses outlet, http://www.rayban-sunglasses.us.org/
rolex watches, http://www.replicawatches.us.com/
swarovski outlet, http://www.swarovskicrystal.com.co/
the north face outlet, http://www.thenorthface.us.org/
tiffany jewelry, http://www.tiffanyandco.net.co/
tiffany co, http://www.tiffanyjewelry.us.org/
true religion jeans outlet, http://www.true-religion.com.co/
true religion outlet, http://www.truereligionjeans.net.co/
ugg boots, http://www.uggaustralia.net.co/
uggs outlet, http://www.uggboots.net.co/
ugg australia, http://www.uggbootsclearance.com.co/
ugg, http://www.uggsonsale.com.co/
ugg boots clearance, http://www.uggsoutlet.com.co/
cheap oakley, http://www.cheap-oakleysunglasses.in.net/
oakley vault, http://www.oakleysunglassescheap.in.net/
ralph lauren outlet online, http://www.ralphlaurenoutletonline.in.net/
ralph lauren outlet, http://www.ralphlauren--outlet.in.net/
burberry.com, http://www.burberryoutlet--online.in.net/
toms shoes, http://www.tomshoes.in.net/
toms outlet, http://www.toms-outlet.in.net/
christian louboutin shoes, http://www.christianlouboutin.in.net/
tory burch handbags, http://www.tory-burch-outlet.in.net/
gucci belt, http://www.gucci-outlet.in.net/
longchamp bags, http://www.longchamphandbagsoutlet.in.net/
abercrombie.com, http://www.abercrombie-kids.in.net/
hollister, http://www.hollisterclothing-store.in.net/
hermes birkin bag, http://www.hermes.in.net/
cheap nfl jerseys, http://www.cheapjerseys.in.net/
beats headphones, http://www.beats-by-dre.in.net/
beats by dr dre, http://www.beatsheadphones.in.net/
hair straightener, http://www.ghdhairstraightener.in.net/
ray ban sunglasses, http://www.rayban-sunglasses.org.uk/
michael kors bags, http://www.michaelkors-uk.org.uk/
michael kors uk, http://www.michaelkorshandbags.org.uk/
louis vuitton outlet, http://www.louis--vuitton.org.uk/
ralph lauren outlet, http://www.ralphlauren-outlet.org.uk/
nike running, http://www.nikefree-run.org.uk/
omega watches, http://www.rolex-watches.me.uk/
dresses for weddings, http://www.weddingdressesuk.org.uk/
pandora sale, http://www.pandora-charms.org.uk/
thomas sabo, http://www.thomassabo-uk.org.uk/
swarovski, http://www.swarovski-uk.org.uk/
christian louboutin, http://www.christianlouboutin.org.uk/
air huarache, http://www.air-huarache.co.uk/
roshe run, http://www.rosherun.org.uk/
north face uk, http://www.thenorth-face.org.uk/
louis vuitton canada, http://www.louis--vuitton.ca/
jordan shoes, http://www.airjordans.us/
air jordan, http://www.jordanretro.org/
air jordan retro, http://www.cheap-jordans.net/
nike outlet, http://www.nikefactory.org/
cheap nike shoes, http://www.nikestore.us/
nike sneakers, http://www.cheap-nikeshoes.com/
air max 2015, http://www.nike-air-max.us/
nike air max 2014, http://www.airmax-2015.org/
nike air max 2015, http://www.airmax-90.org/
nike free, http://www.nikefree-run.net/
nike running shoes, http://www.nikefree5.net/
supra shoes, http://www.suprashoe.net/
new balance, http://www.newbalance-outlet.org/
mont blanc, http://www.montblanc--pens.net/
salvatore ferragamo outlet, http://www.ferragamoshoes.net/
mcm bags, http://www.mcm-bags.net/
bottega, http://www.bottega.us/
nike mercurial, http://www.nikemercurial.net/
celine bags, http://www.celinebags.org/
roshe runs, http://www.rosheruns.us/
roshe run, http://www.nikerosherun.us/
vans shoes, http://www.vansshoes.us/
timberland outlet, http://www.timberland-shoes.com/
iphone 4s cases, http://www.iphone-cases.us/
softball bats, http://www.softball-bats.us/
babyliss pro, http://www.babyliss-pro.net/
louis vuitton handbags, http://www.louisvuitton-outlets.us/
burberry outlet, http://www.burberry-outlet.jp.net/
louboutin, http://www.louboutin.jp.net/
christian louboutin shoes, http://www.christianlouboutinshoes.jp.net/
louis vuitton outlet stores, http://www.louisvuitton.jp.net/
prada outlet, http://www.official-pradaoutlet.com/
michael kors outlet online sale, http://www.michaelkorsclance.com/
michael kors handbags, http://www.official-michaelkorsonline.com/
tommy hilfiger polos, http://www.tommyhilfiger.in.net/
mcm handbags, http://www.mcmworldwide.ca/
moncler outlet, http://www.mmoncler-outlet.com/
kate spade sale, http://www.kate-spades.com/
barbour quilted jacket, http://www.barbour-factory.com/
salvatore ferragamo shoes, http://www.ferragamos.in.net/
canada goose outlet, http://www.canada-gooser.com/
louis vuitton outlet online, http://www.louisvuittonas.com/
burberry outlet, http://www.burberry-outlet2015.com/
hollister clothes, http://www.fashion-clothing.us.com/
north face outlet online, http://www.thenorthface.so/
barbour jackets, http://www.barbour-jacketsoutlet.com/
north face jackets outlet, http://www.northfaceoutlet.in.net/
north face jackets women, http://www.northclearance.com/
burberry, http://www.burberrysfactory.com/
polo ralph lauren outlet online, http://www.ralph-laurenuk.com/
ralph lauren outlet, http://www.polo-outlets.com/
ralph lauren outlet online, http://www.ralphlaurenr.com/
beats by dre outlet, http://www.monsterbeatsbydres.net/
louis vuitton outlet online, http://www.louis-vuittonblackfriday.com/
gucci outlet, http://www.lv-guccishoesfactory.com/
longchamp outlet, http://www.longchampsoutlet.com/
moncler clearance, http://www.moncler-clearance.com/
ralph lauren shirts, http://www.ralphlaurentshirts.com/
hermes handbags, http://www.usahanbags.com/
beats by dre outlet, http://www.beatsbydreoutlet.net/
coach factory outlet online, http://www.coach-blackfriday2014.com/
coachfactory.com/shop, http://www.coachccoachoutlet.com/
coachoutlet.com, http://www.coach-clearance.com/
michael kors outlet, http://www.michaelkors.in.net/
coachfactory.com, http://www.coach-factories.net/
oakley vault, http://www.oakley.so/
hermes bags, http://www.hermes-outletonline.com/
marc jacobs bags sale, http://www.marcjacobsonsale.com/
nike air max, http://www.nike-jordanshoes.com/
woolrich clearance, http://www.woolrich-clearance.com/
coach outlet store, http://www.coachoutletew.com/
coach bags outlet, http://www.coachoutlet.so/
coach handbags outlet, http://www.coachoutletstates.com/
gucci sale, http://www.guccishoes-uk.com/
coach outlet online, http://www.official-coachoutlets.com/
michael kors outlet store, http://www.michaelkors.so/
gucci handbags, http://www.guccifactorys.com/
the north face outlet, http://www.thenorthface.in.net/
ugg boots clearance, http://www.cheapuggboots.eu.com/
tommy hilfiger coupons, http://www.tommy-hilfigeroutlet.com/
uggs outlet, http://www.uggboots.so/
louis vuitton outlet, http://www.louisvuitton.so/
calvin kleins outlet, http://www.calvin-kleins.com/
christian louboutin, http://www.cheap-christianlouboutinshoes.com/
jordan retro, http://www.airjordanshoes2015.com/
burberry outlet, http://www.burberry.eu.com/
abercrombie fitch, http://www.abercrombiee.com/

Post comment

Nick *
Your name or nickname
Your contact information (optional, will not be shown)
Text *
Content of your comment
Calculate *
(* - required field)
STAT NO AD [Stat] [Admin] [STAT NO AD] [pub] [Mirror] Copyright © 2004-2015 Adam Sawicki
Copyright © 2004-2015 Adam Sawicki