LZ4 Decompress Binary Memory Block Of Unknown Size

Question

I'm trying to decompress a block of memory but I don't know it's original uncompressed size, however, I do know the maximum size this original block of memory could ever be.

Is there any way to decompress with LZ4 without knowing the original uncompressed size?

Short-Story-Long: I'm serializing a bunch of variables into a stringstream using a third party library called Cereal. Cereal will serialize your data into a stringstream using a 'portable binary format' which means the endianness is preserved and even converted to that of the host machines during deserialization if needed. The stringstream is then compressed using LZ4 and transmitted to a remote machine for decompression and deserialization.

My issue is that LZ4 will output a memory block of compressed data and an integer specifying the compressed data's size. To decompress you need the compressed size and original size and I'm only sending over the compressed data block to the remote machines.

SO, is there any way to decompress a block of data with LZ4 without knowing it's original compressed size? Essentially 'start decompression, when you run out of data you're done'

Answer 1

The function LZ4_decompress_safe (see https://github.com/Cyan4973/lz4/blob/master/lib/lz4.c#L1288) seems to require only the maximal decompressed size. In this case, you can allocate a buffer large enough for decompression and use it.

So either that, or transfer the original uncompressed size to the remote machines as well.

EDIT: In your case, you can also use LZ4 streaming decompression, see this code sample for more information.