[Live-devel] Frame granularity

[Matthew Hodgson]

Hi Ross,

Huge thanks for the speedy reply :)

On Sun, 21 May 2006, Ross Finlayson wrote:

>> I'm trying to write a custom MultiFramedRTPSource and Sink (and Framer?)
>> for a video payload format
>
> Out of curiosity - which codec is this for?  Is this payload format
> already described by a RFC (or IETF Internet Draft)?

This is 1996 vintage baseline H.263 - as described in RFC 2190.

>>  - but i'm slightly confused about what
>> granularity of frames I should be aiming for.  Should the Source yield
>> frames which refer to a discrete picture of video, or instead whatever
>> sub-picture fragment the individual received packets represent (e.g.
>> picture, GOB, slice or macroblocks)?
>
> The latter.  The reason for this is that these 'sub-pictures'
> ("slices" for MPEG video; "NAL units" for H.264) are useful to a
> decoder by themselves.  E.g., a MPEG video decoder can display the
> 'slices' that it has received, even if some of the other slices in
> the frame were note received.

Right.  I was slightly confused by the H263plusVideoSource, which 
reassembles inbound packets until it sees a marker bit, and then yields 
the result as a 'frame'.  I guess I'll try yielding a single frame of data 
from the sink for every received packet (which may well be a sub-picture 
element).

How is one meant to keep track of whether such a sub-picture frame 
represents the end of a picture - or any other information that might be 
useful for the destination Sink?  (e.g. the matching RTPSink would need to 
know this so as to correctly set the Marker bit - and would also need to 
know the picture type (I/P/B frame), image size etc. in order to correctly 
set special headers).

> The 'special headers' are RTP-specific.  Therefore, they should be
> added by the "RTPSink" (before the packet is sent), and should be
> stripped by the "RTPSource" (before the incoming packet data is
> delivered to the reader).  The various virtual functions in
> "MultiFramedRTPSink/Source" can do this for you.

Okay - but should one try to use the data stripped by the Source in the 
Sink, if fSource->fSpecialHeaderBytes[] is available?  Would a Filter 
which yielded sub-picture frames of data need to maintain an equivalent 
structure for the Sink to use?  Or should the Sink always recalculate the 
special headers from scratch by parsing the raw bytestream?  If the Sink 
did try to reuse the special headers, is there a way to get it to split 
the frame into the packets that the Source built it up out of?

>>  How would one tell
>> the extended MultiFramedRTPSink to only fragment the frame on the
>> boundaries implied by that frame's specialHeaders, rather than MTU-based
>> fragmentation?
>
> Without knowing more about the payload format (and the special
> headers) that you are using, it's hard for me to answer this.

Fair enough.  I was asking this because I can see no way to tell a Sink to 
fragment a frame based on any heuristic other than exceeding the 
maxPacketSize set for the Sink's OutPacketBuffer - and so wondered if a 
Framer should be used for fragmenting larger frames into smaller ones 
valid for outputting as individual packets by the Sink.

>> Alternatively, is the intention for a Framer to perform this operation
>
> 'Framers' are usually used only when reading byte-stream data from a
> file (or stdin).

Right - that makes sense; I will ignore them for now.

I hope this clarifies my confusion slightly - any further insights would 
be, of course, very gratefully received :)

thanks,

Matthew.