Paul of librempeg fame informs me about his achievements occasionally (and in my turn I try to remind the world time from time that this project exists and may provide you functionality hardly found elsewhere, like various filters or console formats support). And his recent work was implementing inflate routine specifically for the multimedia needs. This made me think if it makes sense to have a custom implementation for deflate decompressor and packer in multimedia project when zlib exists and I think it makes perfect sense. Of course in NihAV I NIHed it because the project concept demands it and it was a nice exercise, but in more serious projects it makes sense too and below I’ll try to explain the reasons.
Zeroth of all, a quick reminder about flexibility. RFC 1951 merely specifies the format, so the implementations can output varying bitstreams that will correctly decompress the data. Back in the day when I worked on it I mentioned how you can compress data in different ways, dedicating more time to achieve better compression. And there are more tricks not mentioned there like parallel compression.
Now, the first reason why to have your own implementation is that you can adapt it to your custom needs. As Paul demonstrated, inflating data into a custom format might be beneficial. If all you need in 99% cases is unpacking data from one buffer into another or to a frame buffer (which has padding so you have to output, say, 27 bytes, then skip 5 bytes, output 27 bytes and so on) you can do without all additional functions and not bother with a sequence of calls to partially inflate data.
Then, there’s a question of consistency. You cannot be sure that a new version of zlib will produce the same output as its previous version (I vaguely remember that there was a small scandal when GitHub releases were re-generated using different deflate implementation and resulted in a lot of headache because of old archive hashes being no longer valid; or there’s a story of some Linux distros replacing zlib with zlib-ng resulting in failed tests; and even “no compression” format may change apparently). The case with liblzma is probably a good demonstration of other reasons why it’s not always wise to rely on third-party components.
And finally, you can not merely adapt interface to your needs, you can tune it to handle your data better too. There’s a reason why there exist compressors targeting e.g. genome data. So when you compress image data, it may be beneficial to search for matches around the position right above the current line first, and the presets for the compression levels may be tuned to the different sets of trade-offs. After all, deflate is often used in screen capture codecs where real-time performance is more important than the compression ratio. But who can imagine people tinkering with the encoder trying to improve its performance in a multimedia project?
I hope this helped to convince you that there are circumstances where NIHing something may prove worthy. As a rule of thumb, if it’s easier to implement yourself than to re-use an existing library then maybe you should do so. That it the right way, and the other way lies left-pad.
Now that I think about inflate NIH more I know why I never attempted to write at least native zlib decompression. Partially it was because of ignorance/laziness and partially of unawareness of algorithm and format simplicity and not so good performance and code quality in those legacy projects implementations. Zlib-ng does not help as its source code is not that much better than original Zlib. Source code should be written in straight-forward way and so that does not hurt performance. Anything else that is over-engineered will not help. If zlib/zlib-ng was not that bad and if it had alternative API I would probably not NIH it and would just reuse code and adapt to internal libavcodec API. So my rule of thumb when to NIH something is when existing solutions are not good enough for librempeg/certain usecase.
I’d attribute it to the curse of legacy.
zlibwas written back in the days when 16-bit OSes were widespread and quite incompatible with one another (and C compilers were not much better either), hence all this state management (if you don’t know what it takes to handle over 64kB of data in DOS, be thankful), custom I/O and memory allocation functions and all those functions for e.g. providing a semblance of random access.It’s like the difference between implementing a standard codec with all features that somebody pushed into the standard and just a subset you will use. That’s why I have my own AAC and H.264 decoders that are good enough to play videos I have—if I had to implement all AAC coding tools (like all those additional codecs, scalable mode, TWIN-VQ and so on so on) or H.264 OMGfeatures (like interlaced and scalable modes) I’d probably not attempt such task at all.