Kostya's Boring Codec World

This is my look on what happened but I’ll try to remain objective.

A bit about me (in the very unlikely case you don’t know already and care). I’ve learned about FFmpeg in 2004 or so, just saw it along with other packages in Mandrake release. For several months I downloaded source snapshots at cybercafe (even dial-up was impossible then). I had long interest in general data compression methods and some interest in codecs sparkled by XAnim and desire to play M$ ADPCM files on Linux and FFmpeg got new decoders every week or so (mostly for packed YUV formats but nevertheless quite useful).

One day I tried to reverse-engineer some codec (just for fun), looked at sample produced by TechSmith Camtasia and realized that it’s packed with zlib and after some time guessed correctly they use M$ RLE. In order to test it I wrote a decoder and hacked it into FFmpeg. Eventually it worked and I send my decoder to Mike Melanson. On the 14th of August 2004 it was committed to FFmpeg codebase and it made me proud for my work for a week (those were the times!). After another decoder or two I’ve learned and started to read ffmpeg-devel mailing list (it was on SourceForget then). I think I started submitting my patches there with Indeo2 decoder or so.

After a while I was offered CVS commit access which I refused because of technical limitations. Finally in March 2006 I got display for my MacMini and I was ready for more active development. Google Summers of Code gave me opportunity to dedicate a bit more time for FFmpeg since I could say “hey, I’m payed for it!”. I still try to contribute even if I’m no longer student, have job and too little free time.

And now to the business.

As you may know, most active group of developers had disagreements on how FFmpeg was managed. First that resulted into an attempt to move old-style development elsewhere and reinstating new development under old name. Since Fabrice was in favour of old group and controls ffmpeg.org DNS entry, new model development group was forced out and now is known under a name of Libav.

But what is the root of disagreements? The “legendary” leader, Michael Niedermayer. Legendary in the sence that it’s a legend and not a reality.

At least since 2004 (when I joined the project) FFmpeg was rather a self-organizing community of developers, each with his own goals. Somebody wanted to play movie trailers encoded with QuickTime (hi there, Mike!), somebody wanted to play obscure game formats, somebody just wanted to support anything that he could reverse engineer (that’s me and probably Mike and other people as well). Diego Biurrun tried to bring project in shape by introducing formatting conventions (in early days nobody cared about style much), he and later Måns Rullgård made FFmpeg build system almost perfect, also Måns and Baptiste Coudurier (and many other people) worked on improving or introducing support for common formats.

Later when FFmpeg started participating in GSoCs, at first it was handled by Mike and now by Ronald Bultje. Our test system — FATE started as Mike’s experiment for automated testing regressions for many parts. Later it was completely redesigned and rewritten by Måns who also used a lot of his own hardware to provide test results so FFmpeg was tested on variety of platforms and compilers (most non-x86 things at our FATE are because of his work).

Bug tracking system was set up by Luca and he also found a hosting for it. A lot of services for FFmpeg were run on hardware of Attila Kinali (and even bandwidth and hosting for main server was his achievement). And recent Subversion -> Git transition with merging history from SwScaler is mostly done by Janne Grunau.

So, what’s the role of leader in FFmpeg? None! Almost every significant action was done by somebody else. Were they following some roadmap devised by him? There is no such thing either. Maybe it’s his social skills that kept community together? Wrong again, he caused some people to leave project (and not only the last year, Baptiste would serve good example) and different service maintainers too — by forcing his idiosyncrasies on project (like long-standing DTS guessing issue) or ordering service maintainers around.

And his role as lead developer has been diminishing probably since 2004. I can’t deny he did outstanding work on optimising H.263-based encoders and decoders and writing H.264 decoder, writing and developing some other stuff and providing reviews for patches. But what does he do in recent time? I can’t name anything significant. And from technical point he can’t serve as example: he never cared much about architectures beside x86 nor about his code being easily understandable.

Thus, some developers had had enough and forked. It’s still self-organized community with people contributing to what they seem important and nobody to order around (and not that much stalling on patch reviewing like in times of designated maintainers either).

This fork seems to moved murky waters and some trolls (mostly from MPlayer project that have no relation to FFmpeg at all) reappeared after long time; I cannot directly blame Michael on it but it seems suspicious for me. And the messages I’ve read on ffmpeg-devel between forking and creating Libav made me mostly disgusted, so I’ve unsubscribed from FFmpeg mailing lists and don’t participate in FFmpeg anymore. What goes there is not my concern anymore and I’m happy with Libav.

P.S. Also since most of new things in FFmpeg were introduced despite of him (like Git transition and releases), I can’t forget one historical analogy. In German “the leader” is “der Führer”, but that word is rarely used nowadays because there was another Austrian who completely spoiled its meaning.

This is written mainly as a response to some flamewars.

All codecs may be divided into two categories — mature codecs and developing codecs. In the first case we have frozen bitstream format and not so many enhancements to codebase supporting that codec. In the second case we have codec that may change bitstream format and (what is quite important) encoder features.

FFmpeg itself went that way — from highly experimental H.26x encoder and decoder to rather stable set of almost all decoders available around and several encoders. Since there are some coding rules and conventions and existing framework in it, it makes it very convenient place to implement decoders — you can reuse a lot of code optimised for many platforms (so you don’t have to care about DCT speed, for example) and users don’t have to worry about adding new decoding interface and new lines in configure script, it’s all handled inside libavcodec. And “NIH syndrome” also gives a benefit here — you don’t have to worry about additional libraries (and original codec devs will have their codec specs tested as well).
You know the other advantages of this approach too.

In the same time those features make FFmpeg a bad place for having still evolving encoders for they are not likely to fit into existing framework so easy. The best this tension could be viewed in our interaction with certain encoder. They constantly modify this encoder, so existing FFmpeg options and presets are not good for them and it’s hard to tell how well it will work. Now let’s see what happens if x264 code will be merged into FFmpeg. It will put a rather harsh constraint on x264 developers because it’s hard to tell what change breaks other codecs (changes behaviour, whatever) or vice versa. The same applies to codec-specific features (like muxer using some encoder information, think H.264+MPEG-TS).

On the other hand, it is much easier to incorporate into FFmpeg an encoder not changing so much — some compromises should be made on common interface, some parts replaced with standard FFmpeg routines and voila!

I think that’s the reason we have a lot of decoders and not so many lossy encoders (especially not so many lossy encoders with good quality) in last N years. And it’s the reason why encoder should be originated as standalone projects and merged when they are stable. I’d also like to note that FFmpeg has standing issues with providing better framework for non-H.261 based codecs and descendants (where is codec-independent rate control and motion estimation?), maybe this affected Snow development as well. Anyway, let’s live and see how all these things will be resolved.

Here’s the thing either Compn, known for his passion to document codecs, or Mike, known for his passion to diagrams, charts and codecs, should have done loooooong time ago.

While the same information may be obtained from Multimedia Wiki, a graphical layout should be more handy for claims like “… include reverse-engineering of all Real video formats” here. I am also aware of list of supported codecs in MPlayer documentation but it’s also boring and not very useful as a reference.

Here’s how I like it — green status for supported codecs, red for unsupported. But from a glance on it you can see what’s missing and what should be added to my beloved video conversion tool.

Note: I know that we have to enhance FFmpeg in other areas than different formats support (filter system, for example). Patches welcome.

Ok, let’s see what I’ve done this year:

• simple IMA ADPCM encoder – Apple variant
• worked a bit on different codecs – BMP, Fraps, Monkey’s Audio, TIFF, VC-1
• got RV40 and RV30 finally working more or less as supposed (some garbage still occurs on some B-frames, but mostly both decoders produce watchable video)

So, what are the plans for the next year:

• Find more time for FFmpeg development
• Take part in GSoC (it gives a good reason to work on FFmpeg and also makes a good source of T-shirts)
• Go abroad
• And, of course, make FFmpeg closer to the world domination

For the last step I need:

• Add more formats support to FFmpeg (WavPack lossy, Lucas Arts games formats, Bink, etc.)
• Convince Mike finish his Xan4 decoder
• Convince Robert finish his AMR-NB decoder (unless somebody beats him to it) and AAC-HE decoding support (those messages about SBR not supported are really nagging me)
• Convince Kostya finish AAC encoder (hey, that’s me!)

So, let’s see what we get in the upcoming year.