Archive for the ‘Useless Rants’ Category

OMG

Saturday, March 23rd, 2013

IMG_1466

I think this should be in every first aid kit.

A new record

Monday, March 4th, 2013

I thought that there’s no codec more horrible than Go2Despair and was proved wrong again. Here’s the even worse pair: Dxtory codec and its standalone companion PackBit.

DxtoryCodec.dll is 8.2MB which rivals Go2C++Hell because it’s standalone decoder without any additional code. PackBitCodec.dll is even more elegant. Of the total size of 672256 bytes more than 357000 bytes are occupied by the code of one function (that’s 3-6 times more than the ordinary zlib-based codec complete DLL file). Truly some people should not be allowed to program.

On Sweden and small details

Monday, February 4th, 2013

Probably Sweden was never a really influental in science besides chemistry but it has contributed a lot of small things that really made our world better: high-precision measuring tools, or official human rights representative, or Trocadero, even flatpacked furniture.

Here’s an example: in the swedishest corner of Sweden there was a guy who really changed our world with two simple things. One of them is centrifugal milk separator. Another one is the nozzle. Yes, a simple nozzle that is used in rocket and jet engines. Were those inventions that great and important? Maybe not. Were they useful and have they changed the world a bit to the better? Definitely yes.

And that’s how the place where he came from looks nowadays:

IMG_1107

IMG_1109

IMG_1119

(in case it was not obvious — I’ve been to Orsa and I’m proud about that).

A few words about G2M4 (that were not censored)

Sunday, November 4th, 2012

Okay, I looked into G2M4 closer, here’s the output:

As with the previous beast, there are two types of images combined in single tile — so-called synthetic layer and natural layer. What you see if the first layer decoded.

Here’s the general tile structure:

  • Compression subtype (top bits from the first byte).
  • Transparency colour (three bytes)
  • Number of palette entries minus one (one byte)
  • Palette entries (byte triplets)
  • Synthetic layer (16-bit BE chunk size plus deflated data, may be not present)
  • Natural layer (probably headerless JPEG data, too lazy to verify)

Synthetic layer image is (after decompression) contains packed bitmap that uses palette from above, each row is coded as 8-bit flag [packed row data]. If the flag is zero then row data is present (that’s my guess, it always seems to be zero). Row data is just palette indices stored as 1/2/4 or 8 bits per index depending on palette size. Sample output you can see above.

Feel free to complete RE.

FnAQ about G2M2/G2M3

Saturday, November 3rd, 2012

Just to clarify status: I’m not working on this anymore so anyone can pick it up and finish.

And here are some possible questions that might be asked but more probably won’t.

Q: who cares about this codec anyway?

A: Not me. VLC does.

Q: so why don’t you do it?

A: there are several reasons. First, now I have idea how it works and it’s not that interesting anymore. Second, it would require some debugging and I cannot run that decoder under MPlayer2 (and I don’t use Windows at all).

Q: but wait, there’s G2M4!

A: right, and it uses completely different coding. I might look at it but no promises either.

Q: so, how does it work?

A: the idea is simple. Every frame is divided into tiles and some tiles can be updated from the previous frame or not; some additional information (i.e. mouse cursor shape and position) is also stored in the frame.
G2M2/G2M3 use the technology licensed from Accusoft that combines JPEG and ELS-coded image.

Q: how do they do it?

A: the approach (I call it JPEG-Binary Koder or J-BK for short) is quite simple. Every tile has ELS-coded picture with possible transparency. Transparent areas should be replaced with headerless JPEG data (i.e. only scan data without any markers but with escapes).

Q: sounds easy, where’s the catch?

A: I’m too lazy to catch bugs in my quick JPEG decoder reimplementation and ELS-coded image requires some debugging which I can’t do.

Q: okay, I want to do it, where shall I start?

A: is it the first of April? No? Hmm… Okay, here’s what I would do: grab a copy of g2m.dll (there are enough of them around, in various sizes too), disassemble it.
Find the ELS thresholds table (referenced values are 0x10000, 0x12A00, 0x15C00, 0x19600, 0x1DA00, 0x22800, 0x28500, ...) — the function referencing it is the one used to update ELS coder state, go up from there. Feel free to look at the wiki entry about G2M. Bonne chance!

Teasing

Saturday, October 6th, 2012

In the recent month I was not very productive, so I’d like to talk about codecs that I’m not likely to finish soon (not that I’m going to finish any codec soon anyway).

GoToMeeting 2/3

G2M2 decoder output (the best I could get)

Here’s the best output I could get from G2M2 or G2M3 data by decoding JPEG part of the tiles. ELS part still needs some work since it’s boring — 10-neighbour prediction, differential pixel decoding and other wonders of binary coder.

Certain Intermediate Codec

I managed to reverse engineer some parts of it. First you have so-called fixed header, then you have strip sizes and then strip data with some header as well. The way it’s coded is also more or less clear. But some connecting details — like how those strips are divided (now it looks like 96×1 macroblocks or equally ridiculous).

Since it’s QuickTime it’s hard to say where are the entry points to the codec and what functions are invoked.
Also the only usable binary (with debug symbols) is PowerPC only. It’s nice platform but I still need to learn some of its peculiarities.

VoxWare MetaSound

It turns out that it is slightly simplified variant of TwinVQ. It does not have variable-length codes, all values are read as fixed bits (depending on sampling rate and bitrate of course). The only catch is that it’s hard to find where such description is retrieved or generated. And existing codebooks are somewhat different.

On buying tickets

Sunday, August 12th, 2012

I like to travel around, usually by railway — it’s the most comfortable means of transportation (unless you’re talking about Ukrainian trains or French TGV). And the fastest one for short to long distances (planes are technically faster but consider time getting to the airport, from the airport, security checks…).

So to travel around you need to buy tickets (I’m not made of money to afford some magic “travel anywhere, pay monthly through nose” card) and that’s what I’m complaining about. My requirements are rather easy: you should not need to interact with people and you should be able to see what are the possibilities for the travel.

France. Le facepaume. Ticket vending machines there reflect national spirit frighteningly perfect.

Ticket vending machines.

That picture was taken in one French town near the border. There are two vending machines. To the right is French one. Here’s a short comparison with its neighbour:

  • Languages — half a dozen for one, French for another.
  • Controls — touch screen for one, weird knob with a button in centre (and you cannot change audio tracks with it).
  • Destinations — countrywide and beyond in one case, one region in the other case.

To be fair there are SNCF ticket vending machines that should offer countrywide destinations and I heard you can even get e-ticket from them and they even support other languages than French (which is the hardest to believe). The only problem that you should handle them politely (i.e. point and stick you finger as hard as possible) and I rather value my fingers.

Germany. Three years ago it was a bit quirky but they’ve upgraded vending machines software and now it’s almost perfect. Half a dozen of possible languages, rather intuitive interface, some additional features. And you can buy a ticket to the destinations in neighboring civilised countries (Switzerland and Netherlands) not served by Deutsche Bahn directly. The main WTF is that sometimes you see the trains but you cannot buy a ticket for them there (probably some special trains?).

Netherlands. I’ve used it only once but I remember it being pretty decent.

Sweden. Pretty decent ticket vending machines, I like the additional features like printing bought e-ticket (and you can get it in many other places too). Also I like the fact they have both touch screen and real keyboard and trackball. The only downsides are that it’s a bit slow and that it does not deal with cash and accepts only cards. Sweden is an advanced country after all, you can pay with card almost everywhere but it sucks to be foreigner from a mostly cash-based country (i.e. me few years ago).

Switzerland. Simply WTFiest ticket vending machines. They might be the reason most people buy rail pass instead. You can buy a ticket but it will ask you which route you prefer and it’s not optional. But it’s compensated by the fact it does not offer you any information on trains. You bought your ticket from A to B via C, now go and find what train goes that way in some other place.

Ukraine. In my opinion it should just give up that automated system for ticket sales (made in Soviet times) and cashier should write tickets by hand. Then it will be perfect stone age. I heard there are some advancements in that area: you can now buy e-ticket (but you still need to go to the ticket office where they print it out) and there are talks about removing the requirement to show your ID during ticket purchase. And if you don’t speak Russian you’d better not try buying tickets at all.

On Monster Codec

Sunday, July 15th, 2012

There is one codec unrivaled in its monstrosity. I’m talking about GoToMeetingAndNeverReturn.

First of all, look at its size. The oldest version I could find (it decodes only G2M2 and G2M3) occupies 6196600 bytes. Newer one with G2M4 support (and that’s an additional compressor too) is 8247160 bytes, Current version (with seemingly the same functionality) is 15665528 bytes.

What does it contain? Everything. I kid you not, All (maybe except one) GoToUnpleasantPlace utilities refer this small file and actually export their real main() function out from it. And their size is about forty kilobytes each.

What does it contain from code point of view? Some libraries for networking, cryptography, speech codecs library, libjpeg, zlib, some internal code and tons of C++-induced crap. When you see wrapper calling wrapper calling wrapper calling something trivial (and the wrappers just pass arguments as is and do nothing else), or when you see that 90% of any function is used for exception handling, then we’re talking about enterprise-grade C++ indeed. And of course absolute bloatedness (including making indirect calls where unneccessary) and inability to run on Linux (not with Wine or MPlayer2 loader) greatly help debugging.

Now to the codec details: every frame consists of chunks which hold different kind of data — frame information, mouse cursor shape, mouse cursor position, image data, etc. Frame is divided into tiles (usually 8×8 tiles in frame) and each tile is coded separately.

There are two compression methods, known by their numbers stored in frame header. For G2M2 and G2M3 it’s compression method 2, for G2M4 it’s compression method 3. Both are horrible.

Method 2 seems to have some completely unrelated submethods.

Here’s the call graph from method 2 decompression function.

Looks like there are several possibilities when coding with this method.

There is a possibility to use JPEG compression but I don’t know under what circumstances it’s used.
Also I’ve discovered that MSA1(MSS3) and MTS2(MSS4) actually use standard recommended quantisation tables from ITU T.81 Appendix K.1 (they are not used by libavcodec JPEG decoder though) and quality to quantisation mapping from libjpeg6.

Another coding method employs ELS codes I’ve described in the previous post and uses it to code some bit plane and RGB triplet in dependent form (i.e. with prediction from two previous pixels and using R-G, G, B-G components).

For method 3 so far I know only a few facts. First byte contains image type and depending on it decoding may vary a bit. For example, for image type 0 and 3 three following bytes contain some RGB value, other image types don’t have it. And internally it’s called MPCCoder or SMPCCoder.

In conclusion I want to say that this codec is too large and horrible to use only static reverse-engineering (because it’s very hard even to determine what is the next function called indirectly) and debugging it requires Windows, so don’t expect me to RE it. But if you do you’ll get many thanks and some money from these guys. Good luck!

Some Notes on Un-RE’d Codecs

Saturday, June 23rd, 2012

If I haven’t REd a codec that doesn’t mean I haven’t looked at them at all.
So today I want to talk a bit about some un-REd codecs and what peculiarities they have.

Looks like that all interesting codecs can be divided into three groups: screen codecs, intermediate codecs and speech codecs.
Since I don’t understand the latter group I shan’t give details on it.

Screen codecs

We have lots of them and they can be divided into two categories: simple and monsters.
Simple codecs usually employ some standard data compression library (zlib, FastLZ, LZO or LZF) or Huffman coding with standard median prediction and interframe difference.
I.e. boring, let’s talk about monsters.

  • Windows Media Video 9 Screen (aka MSS2) — combines palettised regions coded like in its predecessor with WMV9-coded regions.
  • M$ Expressions Encoder Screen (aka Titanium Screen codec) — it uses variable-length codes and codes frames with one of two methods. One of them is DCT exactly the same as in M$ ATC Screen codec.
  • MSU Screen Lossless Codec — this one seems simply code R,G,B values with some arithmetic coder and lots of context modeling and prediction.
  • Go2Meeting codecs — a good demonstration of the fact that the best strategy against REing is employing shitty coding monkey.
    Version 4 of decoder was monolithic 8 MB .dll file, version 4 is 15 MB already, all in “fine” C++.
    There are two compression methods known.
    Version 2 employs some weird arithmetic coder substitution (suspiciously like ELS-coder by Wm.D. Withers).
    Version 3 employs libjpeg and zlib for coding image blocks somehow, frame data doesn’t look like it at all.

Intermediate codecs

Cineform — looks like they use Huffman coding and wavelets and it codes 10-bit video.

Fruit Intermediate Codec — looks a lot like its successor (ProRes) but with different bitstream format and fixed coding scheme instead of adaptive ones.

BitJazz SheerVideo — the main problem with it is that most of the codec code performs conversion between any of couple dozens of formats (8- and 10-bit YUV and RGB packed in any possible way). Actual decompression code gets lost somewhere.

On reverse-engineering codecs

Saturday, June 9th, 2012

Sometimes I’m asked what codec I’ll do next. So here’s the answer: I don’t really know.

Since I’m no longer student I don’t have much time to dedicate to reverse-engineering.
Thus I work on different codecs time from time and select such codecs more or less in random.
When I see that it’s nearly completed I spend more time on it till I write a decoder.
That’s how I got RALF decoder, for example.
Or that’s how I have not got Discworld III BMV decoder: long time ago I’ve figured out container format and audio compression, a month ago I’ve advanced in video decoder; but it’s far from completion and I don’t know when I’ll work on it again.

Of course some factors affect my selection of codecs: if I have some interest in it, if it serves some theoretical purpose (e.g. I did Indeo Audio and RALF not because I needed them but to have respective families of codecs fully supported) or if somebody convinces me to do it (two words – GSoC 2007).

So if you ask for some decoder you might get it after a while (but no obligations unless you send me a box of Trocadero).
Don’t forget that samples should be present too (and decoder for the more complex formats).