Kostya's Boring Codec World

And now is finally the time when we should discuss the man who made FFmpeg into what it is today: large, successful and toxic mess. The man who was the project leader for a long time and definitely-not-project-leader since. Of course I’m talking about Michael Niedermayer.

From what I can gather, he started as yet another MPlayer developer adding crucial bits to it, namely the library for video output postprocessing (deblocking, deinterlacing and such) and the infamous libswscale for image format conversion and rescaling that grew out of it. There he got interested in FFmpeg, started to contribute various fixed and optimisations to it and eventually Fabrice left the project to him.

Unlike the rest of people, he dedicated all of himself to the work. Few people met him in person as he never goes to any conferences, at least back in the day he lived at the minimum income in order not to waste time on other work. How do I know that bit? I was involved in libswscale relicensing to LGPL and splitting the reward for doing that afterwards with several other people; boy, was it a mess for Diego (who handled the official financial part).

So what has he done to the project in all this years:

• improving the speed of decoding and encoding by introducing countless new x86-specific optimisations and overall tweaking of C code;
• extending the amount of H.263 (or MPEG-4 part 2) based codecs supported, both for decoding and encoding;
• implementing support for other formats (especially H.264);
• inventing new formats himself (like FFV1 or Snow);
• adding crucial components (like replacing rather rudimentary imgconvert with libswscale and allowing it to be re-licensed under LGPL later);
• a lot of testing, bugfixing and mitigating security issues;
• and overall leadership of course.

What’s there not to like? Let’s see:

• tweaking C code for optimal performance is fine only until the next version of compiler decides it’s an undefined behaviour now (I remember this being an issue with pointer aliasing but there were more);
• having a lot of codecs if definitely good but trying to fit almost all video codecs into the same framework designed only for the H.263 likes is not. Mentioning MpegEncContext still makes some people shudder;
• pushing your own toys into a widely-used library is a dubious move. FFV1 proved out to be an excellent lossless video codec but Snow is a failed experiment that was competitive in DivX times but went irrelevant when x264 appeared;
• coding style (more about it below);
• and overall project decisions that led to the project split among other things (more about it below as well).

A small rant about coding style. I can’t claim that Niedermayer wrote outright bad code but I can’t say he wrote code that was easy to maintain. As in MpegEncContext mentioned above—initially it was the context for both decoding and encoding all those “8×8 DCT in 16×16 macroblock” formats but Michael kept using it for the codecs where it made less sense (like H.264) and all different codec flavours were handled inside the single file using extremely complex conditions and obscure context variables. And of course all of those variables were stored in one extremely large context (no points for guessing the name). It took years of refactoring to make it somewhat saner and understandable. If you argue that the code was perfect as is, I should point out that it had to be modified later in order to handle various standard extensions like YUV422 or 10-bit video and it was not a trivial thing to do.

And while we’re speaking about his technical decisions, beside creating impenetrable code monoliths he is also responsible for introducing various “fixes” for the very specific cases nobody has encountered (I understand the desire to support all more or less valid streams but you should draw a line somewhere instead of adding a hack for each sample somebody asked you to support in private). And the thing directly related to this: automagic guesstimation. The best example of it the infamous compute_pkt_fields() which tried to fill timestamps and duration properties of a packet. For the sane streams it was not needed at all and for the broken ones it functioned on GIGO principle, so not surprisingly there were complaints about it all the time.

And now finally a rant about leadership. Sometimes I wonder if it’s because he comes from a country with a hammer and a sickle on its coat of arms but his management way reminds me of a typical socialist country: selective application of rules, extreme conservatism and voting used to evade responsibility while ensuring the desired outcome.

What do I mean by selective application of rules? Just doing what would make other people get their commit rights suspended (I think Måns actually did that once after his post-commit reviews had no effect on Michael). I suspect that “Michael Niedermayer (and Carl Eugen Hoyos to some extent) can do whatever he wants” is still an unwritten but accepted rule in jbmpeg let alone previous iterations of the project.

What do I mean by extreme conservatism? Mostly feature hoarding: you may add a new feature, extend an already existing one but never to remove any. The extreme example of it would be libpostproc: this library had never been used by FFmpeg itself but when finally Derek tried to move it into stand-alone repository Michael added a special filter just to keep it inside. There’s Sonic—a lossless audio codec based on Bonk that has never been working good (or used by anybody). libswscale replacement never happening because the alternatives do not have the most important feature—runtime-generated MMX scaler. The few things that were removed after all (like ffserver) were done as the result of fierce battles against Michael. Or as a lighter example, he was always against moving to Git so it happened only after the third attempt and right before the split (fun fact: IIRC during the second attempt some guy called Linus Torvalds sent a mail in support of the move).

And what did he usually do when some concrete decision was needed? He usually resorted to “let everybody with SVN commit access to mplayerhq.hu vote” which meant some people who were not contributing to FFmpeg had a say while people who sent patches but were not granted commit rights were excluded. Of course there were people disagreeing with such behaviour and they tried to reform the project—resulting in the infamous split. And what do you know, during the initial talks between two groups of the developers Michael invited MPlayer-only developers again.

During the CEmpeg war against libav his behaviour was far from perfect. I can find badmouthing and changing project name in the files “because it existed before libav” a bit childish yet inoffensive but not the other things. For instance, poaching. I conducted an experiment once: took Derek’s name, translated it into German and committed some Indeo-related patches using this pseudonym to libav mailing list. Even before they were reviewed there I got a message from Michael that he committed them to FFmpeg and I should send them there instead. And there was a case with sound conversion library. Apparently he and Justin Ruggles worked jointly on the design (so it could be used by the both projects) but then for some reason Michael released his version under libswresample name instead (I heard claims that a certain corporation asked him to do that). Unrelated sad thing: the same happened to ProRes decoder but it was Maxim Poliakovski and Baptiste Coudurier involved there.

---

Overall, I believe Michael Niedermayer was an extremely useful man for the project until probably 2010 when his flaws and shifting norms in project development made him rather a detriment. The harsh split in 2011 made him rethink some of his former decisions and embrace things he previously denied (like releases). Yet it made him shift from a developer who works on great stuff to somebody who mostly does janitorial work (namely fixing various issues reported by the fuzzer). I’d say his last greatest work was H.264 decoder and his last substantial piece of work was libswresample and I don’t expect anything neither from him nor from jbmpeg in general.

P.S. Now with the two prominent figures described it’s time to talk about all the lesser-known people who actually made FFmpeg a great project, from Alex Beregszaszi to Vladimir Voroshilov, from Luca Abeni to Zdenek Kabelac, from Diego Biurrun to Benjamin Zores. Not all of them will be described in great details but I hope to give at least a passing mention.

This entry was posted on Tuesday, January 3rd, 2023 at 10:36 am and is filed under FFhistory. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.