Kostya's Boring Codec World

Two days ago I’ve heard about the release of new screen codec from TechSmith. Since TSCC was the first codec I’ve REd (even without looking at decoder) I could not resist and looked at it.

This codec is completely different from its predecessor. While TSCC was simple deflate+RLE, this one operates on blocks. It splits frame into slices 16×8, which are split into 4×4 blocks in YUV 4:4:4 format and 16-240 range. And those blocks are coded with DCT-like transform and quantised with one of two possible quantisers. Pretty easy, isn’t it? Oh, and internally codec is called “Dora”.

And looks like DCT is what some more advanced screen codecs use (and they are usually not lossless anymore).

For example, Micro$oft has about five screen codecs (maybe more):

1. M$ Camcorder Video (CGDI) — somebody had rather stupid idea “ooh, let’s simply record GDI events and store them into frame”. Including commands for drawing text (and damned be you if your system fonts differ) and such. That’s the reason why we never get a decoder for it.
2. M$ Screen 1 (WMV 7 Screen, MSS1) — palettised codec that employed classical arithmetic coding and modelling and coded frame by recursive sub-partitioning and pixel prediction for image areas.
3. M$ Screen 2 (WMV 9 Screen, MSS2) — hybrid codec with enhanced coding method from Screen Codec 1, 16-bit RLE and coding some image areas with WMV 9. (And it was not REd by me BTW).
4. M$ ATC Screen (MSA1) — internally it’s still known as MSS3. This codec operates in YUV 4:2:0 and uses range coder and modelling for coding macroblocks as solid fill, vector-quantised image, Haar wavelet or DCT.
5. M$ Expression.Encoder Screen (aka Titanium Screen, MTS2) — this codec uses variable-length codes and only two coding methods left: DCT (exactly as in MSA1) and vector quantising which looks like a mix of MSA1 and MSS1.

I really should end it and move to something else…

Some of you may have heard of such thing as Mi***soft screen capture codecs (aka WM Screen decoder). Here I’ll try to present some known information about it.

First of all, this codec employs old-school arithmetic coder — traking high and low values instead of low and range, normalisation by one bit, and it’s used for decoding everything (1-bit numbers, 8-bit numbers and symbols with variable probabilities) without any additional context-dependency stuff.

This codec employs predicting pixel value by its 4 neighbours (up left, up, up right and left). Intraframes and interframes seem to be coded in same way, but intraframes may have first line(s) data coded explicitly as 8-bit values (or was that palette?).

Each frame is recursively partitioned, final partitions may be filled with some value, restored from predicted values or copied from somewhere else.

Partitioning is done in this way: decode mode, if its value is 2, do something on decoded rectangle, otherwise decode partitioning value and use it to split rectangle horizontally (when mode is zero) or vertically (mode=1) and perform the same operation on halves.

Do not expect decoder to be ready any time soon though. At least not if someone else makes it.

I’ve just committed fully working (I hope) support for newer Fraps videos. While RE’ing that stuff I made two big mistakes: misunderstood bit reader to read only 31 bits from 32-bit word (which is not correct) and that Fraps version 4 is not identical to Fraps version 2. All of these is corrected now.

And now is a new category that Mike forgot to add to MultimediaWiki (hint, hint).

Supported screen capture codecs:

• TechSmith codec (aka Ensharpen)
• VMWare VMnc
• DosBox ZMBV
• Fraps
• CamStudio

Unsupported screen capture codecs:

• M$ Screen codecs MSS1 and MSS2
• MSU Screen Capture Lossless Codec (SCLS)

Interesting – all those unsupported codecs use arithmetic coding (and as I suspect don’t work with palletized  images).

Here is what I currently know about Fraps v2 (v1 is supported by FFmpeg):

• all data is stored in 32-bit little-endian words
• first word indicates codec version and flag 0x8000000 – delta frame
• second word is zero
• next word is ‘FPSx’ magic
• next three words are offsets in frame (from position 8) to packed YUV planes
• each plane starts with 256 words – frequencies for each possible value
• the rest of data  is bits packed in words
• each bit looks like an input to state machine which outputs pair (value, skip) and changes its internal state
• the rest is still needs to be figured out