How Projects Wither and Die

March 22nd, 2014

For the last few years I feel some disappointment with my work building up and now I try to explain why.

What kept me working on FFmpeg and later Libav?

Money? Well, I admit that it brought me ~$20000 during all those years of work and it was very helpful in my student years but it’s not that much really and wouldn’t be enough for living even in Ukraine.

Of course the main driving reason was fun of writing code and joy of being useful. I still remember being proud for a week for this commit. I still remember how it was fun (sometimes) to reverse engineer a codec and warm feeling when it’s done. I remember users thanking for the work done and asking for features.

Where did all go?

The project matured and now the situation got different. Previously you mostly had millions of clueless users asking how to transcode something to FLV that were tiring but easy to deal with, now you have more enterprise users that use our code often without acknowledging or contributing back (in the old times Picsearch gave us a database of audio and video files in Internet that used codecs we didn’t support — that’s one of the most valuable contributions ever). But that’s not what kills the fun, “security holes” do.

With an advance of automatic fuzz tools it’s easy to generate millions of damaged files that crash your decoder and yet there are no tools for generating correct patches. Fixing those crashes is tedious, requires a lot of thinking (should I disable it? will it affect decoding correct files? etc.) and in other words not fun at all. You have to balance between having decent code, ability to handle corrupted files and being robust — and in order to account for all possible corner cases in the code from the very beginning you should be more paranoid than FFmpeg leader. And somehow you cannot avoid it, you’re expected to fix it or else. This is like you’re on maintenance contract but without any form of compensation, you just get a mountain of corrupt samples and “have you fixed it yet?” every week. Or you get some “security vulnerability” reports with the same effect. I repeat, this is not fun — so why should I do it for free?

There is only one exception around called VideoLAN. Those guys really show (and not only show) some care and they give back to us. Just in my case I gave them all they wanted and I could give them.

As for the rest, world domination is not my goal, I don’t see fun in maintenance and noone is paying me to do it. Why should I continue?

So I’ll try to finish whatever projects I still have around and end it all. I’ve been around for 9.5 years after all, that’s long enough.

P.S. Maybe I should move to Oljonsbyn.

A Glance at Mobiclip HD

January 27th, 2014

For no particular reason I’ve decided to look at it and the codec is quite interesting despite being of somewhat WTFy design.

First, it uses 6 buffers so motion compensation can reference any of them while decoding (except that buffer 0 is the current frame and it’s a skip block).

Second, it seems to operate on 16×16 macroblocks that can be split further into smaller subblocks or encoded as the whole. Yes, it’s essentially a quadtree. And motion coefficients seem to be read in the beginning for all macroblocks and then deltas from overall macroblock value are coded for all subblocks when they occur.

Third, it seems to have halfpel motion compensation and some very simple transform that depends on block size and number of coefficients coded. While it’s no DCT I’ve seen that it decodes (last, skip, level) triplets, unquantises them and calls transform function depending on the transform size and position of last nonzero coefficient. And I’m not completely sure but looks like some kind of spatial prediction like H.264 is invoked too.

Truly there are interesting codecs no-one cares about (including me).

Some Notes on Some Intermediate Codec Family

January 27th, 2014

A friend of mine Mario asked me to look at DNxHD 444. It turned out to be quite easy to support in libavcodec decoder (at least for CID 1256 for which I has sample) after I looked at the binary decoder. And I was curious what formats were there.

Here is the list of internal IDs supported by Avid decoder with a family they belong to, image parameters (width x height @ bitdepth) and other properties.

  • 1233: Avid_HD (1920×1080@10) interlaced (marked as debug format)
  • 1234: Avid_HD (1920×1080@10) interlaced (marked as debug format)
  • 1235: Avid_HD (1920×1080@10) progressive
  • 1236: Avid_HD (1920×1080@10) progressive (marked as debug format)
  • 1237: Avid_HD (1920×1080@8) progressive
  • 1238: Avid_HD (1920×1080@8) progressive
  • 1239: Avid_HD (1920×1080@8) interlaced (marked as debug format)
  • 1240: Avid_HD (1920×1080@8) interlaced (marked as debug format)
  • 1241: Avid_HD (1920×1080@10) interlaced
  • 1242: Avid_HD (1920×1080@8) interlaced
  • 1243: Avid_HD (1920×1080@8) interlaced
  • 1244: Avid_HD (1440×1080@8) interlaced
  • 1250: Avid_HD (1280×720@10) progressive
  • 1251: Avid_HD (1280×720@8) progressive
  • 1252: Avid_HD (1280×720@8) progressive
  • 1253: Avid_HD (1920×1080@8) progressive
  • 1254: Avid_HD (1920×1080@8) interlaced
  • 1256: DNx444 (1920×1080@10) progressive
  • 1257: DNx444 (1920×1080@10) interlaced
  • 1258: DNx100 (960×720@8) progressive
  • 1259: DNx100 (1440×1080@8) progressive
  • 1260: DNx100 (1440×1080@8) interlaced
  • 32768: AHD-DBG-1 Avid_HD (64×32@8) interlaced
  • 32769: AHD-DBG-2 Avid_HD (128×128@8) interlaced
  • 32770: AHD-DBG-3 Avid_HD (480×320@8) interlaced
  • 32771: AHD-DBG-4 Avid_HD (64×32@10) interlaced
  • 32772: AHD-DBG-5 Avid_HD (128×128@10) interlaced
  • 32773: AHD-DBG-6 Avid_HD (480×320@10) interlaced
  • 36864: AHD-DBG-3 Avid_HD (720×512@8) interlaced

If you look at this table you can see more formats than supported by libavcodec currently. Unsupported formats being debug ones, interlaced ones and not belonging to Avid_HD family.

While I fully approve not having interlaced formats support, the rest can be supported (especially if samples are provided).

Sigh, too much intermediate codecs I had looked at.

Bink2: Inter Block Residue

January 18th, 2014

Inter block residue decoding is not different from intra block decoding except that DCs are expected to be in -1023…1023 range instead of 0…2047 and quantisation matrix for luma is different.

Posts about reconstruction process might follow.

Bink2: Intra Block — Chroma

January 18th, 2014

Chroma block coding is similar to luma but with some changes since there are only 4 blocks coded here.

Thus, CBP is coded as two nibbles (real CBP and VLC switch) and it does not try to reuse nibbles from last CBP in code.

There are only 4 DCs here but they are decoded the same way.

AC block decoding is completely the same.

Bink2: Intra Block — Luma

January 18th, 2014

Intra luma block in Bink2 contains the following elements: CBP, quantiser, DCs and ACs.

CBP is coded as 32-bit bitmask depending on the previous CBP value. Internally top half is coded depending on bottom one and the whole bitmask is coded in nibbles starting from LSB.
Lower half decoding depends on the control bits:

  • 11 — simply return last CBP
  • 10 — use low 16 bit from last CBP
  • 0 — decode 4 low nibbles of CBP. Initial nibble value is set to (last_cbp >> 4) & 0xF, if the next bit is 1 then don’t change it, otherwise read new value from the bitstream (4 bits of course).

Now we can use these low 16 bits of CBP to restore high 16 bits. This is also done by nibbles and decoding depends on them (why nibbles? Because blocks are coded in groups of four).


pat4 = (last_cbp >> 20) & 0xF;
ref = cbp;
for (i = 0; i < 4; i++) {   if (!ones_count[ref & 0xF]) {    pat4 = 0;   } else if (ones_count[ref & 0xF] || getbit()) {    pat4 = 0;    for (bit = 1; bit < 0x10; bit <<= 1)     if ((ref & bit) && getbit())      pat4 |= bit;   } else {    pat4 &= ref & 0xF;   }   cbp |= pat4 << 16 + i * 4;   ref >>= 4
}

Essentially it decides what set bits to copy from the first part. And top 16 bits are not really a coded block pattern, it just tells decoder to use an alternative set of VLC codes in AC decoding.

Quantiser is coded with static VLC (plus sign bit for nonzero value) as a difference to the previous quantiser.

Quantisation table for DC: 4 4 4 4 4 6 7 8 10 12 16 24 32 48 64 128

16 DCs (coded with the same scheme as motion vector described in the previous post)

16 blocks of AC coefficients coded in groups of four. Each AC block is coded as (value, skip) pair where value is coded with static VLC that gives small levels (0-3) or number of bits for raw value to read. Skips have one peculiarity too: value coded with static VLC defines either skip (for values 0-10), escape value (when you got you need to read 6 bits with real skip value), end of block value and that the following 8 AC coefficients won’t have skip values coded after them.

Scan order is strange, here are first 8 indices from it: 0, 2, 1, 8, 9, 17, 10, 16, 24, 3, 18, 25, 32, 11, 33

Bink2: Frame structure

January 16th, 2014

Frames consist of ordinary macroblocks in 420 format with optional alpha. Bitstream is packed into 32-bit little-endian words.

Every macroblock is prefixed by 2-bit code determining its type.

Type 0 (intra block). Decode intra block.

Type 1 (skip block). Simply skip block.

Type 2 (motion block). Decode motion vectors and copy block with ½-pel precision for luma and alpha and ¼-pel precision for chroma. There is no residue coding.

Type 3 (inter block). Decode motion vectors and copy block with ½-pel precision for luma and alpha and ¼-pel precision for chroma. Now decode and add residue.

Motion vector differences are coded this way: motion vector element size in bits (3 bits, if read value is 7 then read additional 2 bits, so total is up to 7+3=10 bits per MV element); four motion vector elements; sign bits for non-zero motion vector elements.

Side note: looks like this codec employs this scheme (bit size in 3+2 scheme, fixed-size values, signs for nonzero values) for other elements too, e.g. DCs.

Luma hpel filter looks something like (A - 4*B + 19*C - 4*D + E + 1) >> 5
Chroma qpel: ¼ — (6*A + 2*B + 1) >> 3, ½ — (A + B + 1) >> 1

Next: intra macroblock decoding — luma.

Bink2: RE Notes

January 16th, 2014

Since I obviously have nothing better to do with my time I looked at Bink2 again. In the (hopefully) following blog posts I’ll try to document bitstream format; general codec design was presented much earlier.

At least container format remains the same (except that it uses e.g. KB2f or KB2g instead of BIKi).

My Stomach’s Guide to Sweden

January 15th, 2014

Sweden is awesome country and its food is plain but decent and heart-warming. And everything gets even better in late December.

There’s no Christmas in Sweden, they have Jul — it’s less Christian in nature but significantly better. As an Ukrainian I totally approve Swedish version because the national Ukrainian animal and products from it are well celebrated.

IMG_3397Yes, that one

Sweden has really good dairy products (I’d especially recommend cheeses and filmjölk), surprisingly good selection of meat (including horse, deer, reindeer and elk), outstanding herring (really, nothing beats fried fresh strömming) and other fish products. And oh so many variations of candies everywhere. And drinks.

IMG_2395Berries

In jul it all gets even better since special food appears — traditional julskinka, sylta (and especially julpressylta), marzipan pigs, special versions of prästost and other cheese… And of course julmust.

One can try traditional game of mine — try different variations of everything. For example, drinks (Trocadero, julmust/påskmust or filmjölk), cheese or even köttbullar (I’ve finally found köttbullar made from deer meat for example). Or at least look at designs of marzipan pigs, every konditori makes their own.

Fun fact — in Gävle they like unofficial symbol of the city so much that they have marzipan goats along with pigs.

And another fun fact — there is a drink from Norrland called Portello and it has unlicensed clones. The real Portello is produced by The Norrland Brewery (aka Vasa Bryggeri) but some other breweries have drinks with similar taste but different name. Guttsta Källa produces Ortello and Mora Bryggeri produces Candelo, though it’s a trollish brewery that makes Rio Cola with familiar Cuba Cola design, too lazy to produce Julmust and Påskmust (see picture) and less popular drinks like Guldus or Haiwa are all renamed too.

IMG_3414Mora Bryggeri — no julmust or påskmust

IMG_4016The Reference Drinks

I’ll continue exploring the wonderful world of Swedish food at every possible occasion.

And some fun things to try:

  • julmust with semlor (they are always available in Norrland, it seems);
  • fried gravad lax;
  • filmjölk with berries (I mix lingon, blueberries and raspberries — they are available frozen all year round);
  • julskinka and Wastgöta Kloster cheese on tunnbröd;
  • Swedish apples;
  • chewing candies;
  • and of course Trocadero from any brewery in Norrland!

Internetless

December 6th, 2013

I still don’t have Internet connection at home (for two months and counting) and not sure if I get any this year.

So here’s the list of rants I’d have written, had I had access:

  • SD cards as modern floppies — similarly ubiquitous and slow;
  • Swiss cheeses — some are OK, most are slightly sticky, coated in something very smelly and named “chas”;
  • WMV9 pre-RTM P-frames decoding: how it should be decoded and why our decoder is still wrong sometimes;
  • Bink2 decoder progress report (since I’ve not started working on it yet).

Do not stay tuned.