FFhistory: Summer of Code students

Since 2006 FFmpeg (and libav when it existed and was active) has been participating in the Summer of Code program. Essentially it’s students working on tasks for different project with one corporation paying for the successful completion of the task during summer (back in the day it was $4.5k, no idea what happens now). While the sum is not remarkable by American standards, it was high by Central European and Asian standards. And of course there were students who were after money and disappeared after they got them (in one case with unbelievable claims about why the health problems prevented him from the completion). In one case there was a student who essentially plagiarised another project, in another case a student dumped a lot of code with unknown functionality so it was hard to understand or review. But here I want to talk about the students who actually stuck around after the program was over and even did something else.

The first such student would be Robert Swain. He had been around before and worked mostly on the x264 wrapper. He tried to implement AMR-NB decoder but failed to complete it (considering how convoluted those speech codecs are it’s hard to blame him) but he later redeemed himself by working on AAC decoder (which was a task another SoC student failed). Eventually he got hired by Collabora.

Reynaldo H. Verdejo Pinochet worked on QCELP decoder (and it took a long time to complete). IIRC he went to work for Samsung and tried to support FFmpeg from there.

Alexander Strange was probably the best student. He made some H.264 decoder optimisations before and was chosen to introduce multi-threaded decoding in libavcodec. IIRC he forked it into ffmpeg-mt, created the framework and implemented it for some codecs. So now you know whom to thank for multithreaded decoding.

Jai Menon is responsible for ALAC encoder. Afterwards he submitted random patches, often for libopenjpeg wrapper.

Zhentan Feng wrote MXF muxer and worked on MMS later.

Thilo Borgmann did MPEG-4 ALS decoder. Also he implemented CorePNG as his qualification task that required some API changes (the codec relies on keyframe status set by the container and that information was not passed to the decoder before). As I mentioned before, he’s still active in the project.

Marcelo Galvão Póvoa wrote AMR-WB decoder.

Alberto Delmás reverse engineered MSS2 which was not an easy task as it combines a non-trivial arithmetic coding for lossless parts with regions coded as WMV3 frames. He also REd Windows Media Image/Presentation codec which was WMV3-coded frames with various transforms applied to them.

And of course I can’t forget Mashiat Sarker Shakkhar. This student from Bangladesh worked on WMA Lossless decoder and improving VC-1 decoder. When he searched for a job he asked if I can give recommendations to his potential employer—which I did to my best and he got employed. When we met at VDD15 he was still happy to see me.

There was Guillaume Martres who single-handedly wrote most of the working H.265 decoder. It spawned three forks with different optimisations: beside the expected two there was also OpenHEVC (which was later merged).

From newer students I should mention Rostislav Pehlivanov who is responsible for creating AAC encoder for libavcodec (previously it was just bitstream writer as the quality was abysmal, as expected from yet another failed SoC project) and later he also wrote Opus encoder as well as Dirac encoder (this one on behalf of Kieran’s company).

Here I mentioned just newcomers, there were some long-standing developers participating in the program with varying degrees of success. For example, Diego Biurrun was selected to write Flash Screen Video 2 decoder using decompiled source as the reference (so it was mostly the matter of refactoring it) but he got passionate about other things and failed to do the tasks. Jan Ekström wrote Ut Video encoder. My projects were hit-and-miss (RTMP was okay, VC-1 decoder worked but not in full, RV4 decoder worked eventually but it took years to finish, AAC encoder failed).

Overall, in many cases even if the student failed to complete the task in time and left, the results could be salvaged and completed later. That’s how FFmpeg got some of its decoders and even libavfilter after all.

8 Responses to “FFhistory: Summer of Code students”

  1. And now for the origin story of how the project got involved in GSoC! 🙂

    In 2006, I thought it would be nice to have a native VC-1 decoder in FFmpeg, but I didn’t have time to do it myself. But I felt the task was well-defined enough that I could tutor someone else who wanted to do it.

    Then a week later, I learned about GSoC and realized it would be a perfect fit. I applied and even though I had missed the cutoff by a day, they still let us in.

  2. Kostya says:

    It took many summers and one random (Dutch?) guy to make the decoder complete though. And should I have known the story earlier I’d mentioned it in a section dedicated to you.

  3. Derek Prestegard says:

    You must do a posting on Dark_Shikari aka Fiona Glasser. Her contributions to x264 are legendary!

  4. Kostya says:

    I’m not going to write a history of x264 (I’ve never cared about it much), if somebody follows my suit and writes about other open-source projects it would be nice though.

  5. JEEB says:

    > Ut Video encoder

    Ah, this project. It was a very teaching experience. I never got too far with the SIMD, unfortunately (I think I had first-line left prediction working after BBB helped me with the basics, but then I moved back to finishing up the encoder).

    I had not touched this component for a very long time, but it was fun to see finally in 2020 as I made ffmpeg.c configure the encoder color space/range information based on the first decoded & filtered video frame that the automated codec_tag handling for BT.709/non-BT.709 finally worked automatically. I recall being somewhat bummed back when I wrote it that the tooling wouldn’t pass that information through, but apparently I only got annoyed enough in another context to improve the situation years later 🙂 .

  6. Kostya says:

    koda should be envious of you.

  7. Mashiat says:

    Quite an honor to make the list. Thanks Kostya!

    > worked on WMA Lossless decoder and improving VC-1 decoder

    Kind of you to gloss over the fact that I left both codecs in a buggy and incomplete state. The guilt still haunts me …

  8. Kostya says:

    I’d rather blame the codec design rather than you. Just look at the list of people in the copyright section of either decoder—you’re in a good company.