Kostya's Boring Codec World

Now that I’ve finished remembering various developers it’s time to evaluate their impact and how it would be without certain them.

Nowadays FFmpeg is mostly used for playing back videos from Internet. If you play, say, MP4 via HLS then it’s HLS protocol handler (mostly a work of Martin Storsjö) feeding data to MP4 demuxer (a work of Baptiste Coudurier) which feeds data to H.264 decoder (initial version by Michael Niedermayer, optimisations by many other people, current design with multithreading and hardware acceleration support is from Alexander Strange and Anton Khirnov) and AAC decoder (mostly by Alex Converse with significant optimisations by Måns Rullgård and others). If you prefer WebM then you’ll get data passed via Matroska demuxer (written by Ronald Bultje) to VP8 or VP9 decoder (written by Ronald in cooperation with other people) or even to AV1 decoder (an external library written by many FFmpeg and x264 developers) and Opus decoder (written by Anton Khirnov). As you can see the two most known names barely have a mention—that’s because Fabrice stopped working on it in time with the different set of popular formats. So in order to use his work you need to play DivX 4 rip with MP3 track.

What would have changed if Fabrice Bellard haven’t started his project? To him—not much, he’s done enough awesome stuff before and after. To the multimedia world—it would be a serious loss but probably not critical. There had been various libraries for supporting separate families of multimedia: libquicktime for MOV and QT-specific codecs, libmpeg2 for MPEG video decoding, MPEG audio decoders are dime a dozen, there was even avifile for loading binary decoders. If Árpi had an idea to use them all in his player to support all possible formats, somebody else could’ve come with an idea of making a universal decoding framework out of existing libraries too. MPEG-4 ASP decoding could be done with OpenDivX or XviD (the latter also offers a better encoder that libavcodec because it does not function like Soviet machinery where you need to be an extremely skilled user to bring its potential to the fullest). AAC could be still supported via FAAC/FAAD2. Maybe somebody would have to rewrite it all to bring it under LGPL to make it more popular but you can’t say that opensource multimedia was a desert before Fabrice came. Still, he created a project that gave a significant impulse for the whole open-source multimedia and you should not underestimate it.

How would FFmpeg fare without MPlayer developers? I’d say poorly as they created the environment where it could crystallise into something bigger (of course Xine developers played a significant role but since the player was more centred on DVD playback instead of playing everything I expect much smaller outcome).

And how would FFmpeg fare without Michael Niedermayer specifically? I’m inclined to say that the same or even better. For all good he has done to the project in some areas he compensated it by harming it in different areas. He was the one trying to keep the project in relative obscurity by not having releases or simpler development process (out of fear it may get sued, the same excuse certain video hosting service uses to give no credit to the open-source projects it uses). He was the one who blocked removing features, no matter how obsolete they were, and by extension replacing modules with freshly designed and implemented ones. And it was required as his code tended to turn out into unmaintainable monolithic mess. I’d say his main advantages were his willingness to throw a lot of time on the problem, his willingness to spend it on optimising even minute details and his deep knowledge of H.263. To me it seems that without him it would take more time to support various flavours of it but, with XviD and x264 people around, the project would probably have less features in general but it would still have all popular features and probably with cleaner and nicer design.

Finally, how would it fare without all the corporate support it got throughout the years? I think it would change nothing except some things would have been done slower and some more controversial things would not be there. The only cases of corporate support I’m aware about are Summer of Code program, one time some companies sponsored a new server and one time when some French company paid for libswscale relicensing to LGPL. I used SoC mostly as an excuse to allocate more time to work as I would work on the task even without a bounty (but I’d have to earn them in another way), thus I expect some other students would do the same. The server did not cost too much so we all could chip in and buy it regardless. And the proper thing to do with libswscale would be to replace it with something else entirely but you should know by now who was against it. Some companies sponsored developing certain features they wanted but it was between them and individual developers (and it was already at the stage when the project was popular already and the stuff would be contributed regardless).

See, unlike many people I don’t regard FFmpeg as something unique (in the sense that it’s a project only Fabrice Bellard could create). It was nice to have around and it helped immeasurably but without it something else would fill the niche. There were other people working on similar tasks after all (does anybody remember transcode? or gmerlin?). Hopefully you got an idea on how many talented unsung heroes had been working on FFmpeg and libav over the years.

In either case the project gave me enough joy (and a share of other strong emotions) along with new friends (and, I suppose, enemies). It was a part of my life as well as the part of the open-source software landscape that I’m not going to forget any time soon.

P.S. I was asked to also write about x264 developers. I’m not qualified to do that because I don’t know much neither about them nor the overall history of the project beside some anecdotes (that concern the personalities of those developers and thus are better not made public anyway). In case of FFmpeg/libav I was involved in the project not merely by contributing patches, I was on IRC channels where developer discussions were held, attended conferences and took part in discussions happening there. In other words, I have collected memories that were only partly recorded (on mailing lists, but good luck finding SourceForge era mails; CEmpeg also had its IRC channel publicly logged at Michael’s insistence for a while) or not recorded at all. And without knowing those details you can’t have a full picture of what was going on and why. Obtaining them for an outsider is not an easy task either: consider how many influential people left the project more than a decade ago, how many developers would rather not remember it at all and how many of those who are willing to share their memories will be biased (I’m no exception, hopefully my biases are obvious so you can discount them). I’m not arrogant enough to write something serious about a project I have not participated in.

P.P.S. At least this should not be a problem for NihAV as it’ll never be popular enough for others to care about its history (and it is documented in this blog already).

This entry was posted on Friday, January 20th, 2023 at 3:47 am and is filed under FFhistory. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.