Potential bug in compress/flate?

Using compress/flate to compress a stream of packets, the decompressor fails after 32767 (2^15) iterations and claims that there was an unexpected EOF.

Here’s a sample that demonstrates the problem
https://play.golang.org/p/iKvVWBlch8Y

I don’t think resetting the reader underneath the decompressor is a supported operation. The decompressor has state of its own. There is a method to reset the decompressor to work with a new underlying reader which you should use instead.

1 Like

Perhaps, but how would it know? All the decompressor knows is that it implements io.Reader and that it can call Read to get more bytes.

This Reset on the decompressor? https://golang.org/pkg/compress/flate/#Resetter It discards the current state so it would not longer be able to decompress the stream from the sending side.

That said, the underlying reader does seem to be at fault. I tried modifying the synchronization example on compress/flate to be more like the example https://play.golang.org/p/IAgu8wkwqIC and it sails 32768 decodes. I tried 10 million and it didn’t blink.

I’ll try replacing bytes.Reader in the original example with something else.

It can’t know that, but it can notice that what you’re doing is unexpected. The number 32768 isn’t random. Perhaps the flate stream expects an extra chunk header or whatever at that interval (I have no idea, but it’s not hard to imagine), which you inadvertently put into the next “message”. Breaking up a single stream into separate messages with EOF in between is clearly not the way it’s supposed to be used.

If you want separate messages, both the compressor and decompressor should be reset per message.

Your comment that maybe extra data was being added to the stream made me wonder if there was residual data after the decompression step. Sure enough there was and it just needs to be fed back in before the next message, effectively making it a sliding bytes buffer.

https://play.golang.org/p/TysU48fSMZQ

I want to avoid resetting the compressor/decompressor because that would kill the compression ratio, especially on smaller messages and compress/flate looks like it was designed to support that case.

Thanks for taking a look.

Which then lead to your initial comment about the use of reset.

I switched the bytes.Reader to a bytes.Buffer and just append the new data using Write
https://play.golang.org/p/gzzmc-2KA5A

I was initially nervous about using Write to append data because I don’t want to keep the entire network stream in memory, but looking at bytes.Buffer.Read, it internally calls Reset if it’s empty. https://github.com/golang/go/blob/master/src/bytes/buffer.go#L297

As is often the case with Go, the simple solution was the correct one.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.