Kostya's Boring Codec World

Just in time I though I’ve fully understood RV4 loop filter. It uses both coded block pattern and some other pattern. I thought it was CBP from the previous frame, but it turned out to be some special deblocking pattern calculated for each block in interframes after decoding that block. That calculation is easy – it just selects a set of subblocks to check, compare some values and if the difference is less than 3 then set a bit in deblock pattern. Now the only thing left is to find out is where those values come from.

I’ve mostly understood how RV40 loop filter works.

Just not to forget main principles I document it here (this blog was created for such things after all).

• CBPs from left, top and bottom neighbours are used in filter, and if frame type is interframe then CBPs for those blocks in that reference frame are used as well.
• There are two actual filter types – weak and strong, both are described in H.264 drafts.
• Edges in subblock are filtered in the next order: bottom, left, top. Top edge is filtered only for the subblocks in the first row.
• There are many filter parameters passed: dither argument (for strong filter, depends on subblock position), two thresholds taken from ClipTable, threshold taken from alpha_tab, threshold taken from beta_tab and the same value multiplied by 3 or 4 (four is for Y plane filters in not extremely big pictures).
• The problem was to determine what ClipTable parameters should be used, as it has an additional dimension, more on it below.

There are seven values taken from ClipTable total:

1. ClipTable[0][current block quantiser]
2. ClipTable[2][current block quantiser]
3. ClipTable[2][global quantiser set in header]
4. ClipTable[x][current block quantiser]
5. ClipTable[x][top neighbour quantiser]
6. ClipTable[x][left neighbour quantiser]
7. ClipTable[x][bottom neighbour quantiser]

That x value is 2 for the intra block types and P-frame interblock with DCs coded separately, 1 otherwise.
As I understand, ClipTable[x][current block quantiser] is used by default and other valuer are used for corner cases (subblock on the side of the edge is uncoded, belongs to another macroblock or does not exist at all).

I should look at H.264 loop filter description (thanks to all who sent me the pointers to the book by Iain Richardson), it seems suspiciously similar.

While work on AAC encoder is slowly progressing (now it’s mostly psychoacoustics left to do and maybe HE-AAC if somebody will convince me), I’m looking at side tasks to make my life a bit more colourful.
For now those tasks are writing SSE2 optimization for Monkey’s Audio decoder (and that is the first piece of SIMD assembly I’ve ever written) and working on RV40 loop filter.
To give people false hope, it’s more understandable by now. Only one function argument is not obvious. And Dark Shikari, you were wrong – RV40 is 99,5% alike with H264 draft (not 99% you said), as loop filter is suspiciously similar to H264 one.