I’ve mostly understood how RV40 loop filter works.
Just not to forget main principles I document it here (this blog was created for such things after all).
- CBPs from left, top and bottom neighbours are used in filter, and if frame type is interframe then CBPs for those blocks in that reference frame are used as well.
- There are two actual filter types – weak and strong, both are described in H.264 drafts.
- Edges in subblock are filtered in the next order: bottom, left, top. Top edge is filtered only for the subblocks in the first row.
- There are many filter parameters passed: dither argument (for strong filter, depends on subblock position), two thresholds taken from
ClipTable, threshold taken fromalpha_tab, threshold taken frombeta_taband the same value multiplied by 3 or 4 (four is for Y plane filters in not extremely big pictures). - The problem was to determine what
ClipTableparameters should be used, as it has an additional dimension, more on it below.
There are seven values taken from ClipTable total:
ClipTable[0][current block quantiser]ClipTable[2][current block quantiser]ClipTable[2][global quantiser set in header]ClipTable[x][current block quantiser]ClipTable[x][top neighbour quantiser]ClipTable[x][left neighbour quantiser]ClipTable[x][bottom neighbour quantiser]
That x value is 2 for the intra block types and P-frame interblock with DCs coded separately, 1 otherwise.
As I understand, ClipTable[x][current block quantiser] is used by default and other valuer are used for corner cases (subblock on the side of the edge is uncoded, belongs to another macroblock or does not exist at all).
I should look at H.264 loop filter description (thanks to all who sent me the pointers to the book by Iain Richardson), it seems suspiciously similar.