[Live-devel] H264VideoFramer truncating frames

Thu Mar 19 02:13:55 PDT 2015

Having tested this further I realised that the H264VideoStreamFramer 
assumes a constant 25fps. On a heavily loaded system which was dropping 
frames this caused the timestamps to lose sync with wall clock time so 
I'm still looking for a solution.

Hopefully I can subclass the H264VideoStreamFramer to handle discrete 
frame input in Annex B format somehow.

On 03/11/2015 01:54 PM, Robert Smith wrote:
> I've got this working now, I think my problem was conceptually 
> treating the byte stream as a series of frames instead of as a 
> continuous stream.
>
> Having looked at the ByteStreamMemoryBufferSource class it makes a lot 
> more sense now.
>
> On 03/11/2015 10:10 AM, Robert Smith wrote:
>> Firstly apologies for the multiple posts.
>>
>> The platform is a TI DaVinci DM8148 using TI's HDVICP2 video encoder 
>> hardware. As far as I can tell the encoder is capable of outputting 
>> discrete NALU's but we are only able to use the encoder via an 
>> OpenMAX API which doesn't expose this behaviour.. it's frustrating!.
>>
>> The encoder provides discrete 'frames' which consist of a buffer of 
>> concatenated NALU's prepended with the start codes.
>>
>> Now that I think about it, If the H264VideoFramer is expecting to 
>> read X bytes from a continuous stream, I should be able to give it 
>> only part of the 'frame' that I receive from the encoder? I'll try 
>> this and see how it works.
>>
>> Btw, On our other systems we use the Intel IPP H264 encoder and the 
>> Intel Media SDK encoder, I'm not as familiar with them but I 
>> understand that they also don't output discrete NALU's so I thought 
>> this was common amongst encoders.
>>
>>
>> Regarding slices, I can configure the encoder to encode a frame as 
>> multiple slices based either on a maximum slice size or number of 
>> MB's per slice but we're using GStreamer as one of our clients and I 
>> get a lot of image corruption with multiple slices enabled.
>>
>> I haven't had time to look into the problem deeper but using a single 
>> slice was a quick and easy solution.
>>
>> Thanks,
>>
>> Robert Smith.
>>
>> On 03/10/2015 01:49 AM, Ross Finlayson wrote:
>>>> The encoder unfortunately only supplies frames in Annex B byte 
>>>> stream format requiring the frames to be parsed.
>>>
>>> Are you sure about this?  (Often, hardware encoders have firmware 
>>> upgrades available.)
>>>
>>>
>>>> Previously I was using my own class to identify the NAL unit's in 
>>>> conjunction with the H264VideoDiscreteFramer which worked fine but 
>>>> it's heavy on the CPU. So I've been trying to use the 
>>>> H264VideoFramer and just pass the full frames in which works ok and 
>>>> is faster than my solution except that I'm seeing a lot of 
>>>> truncated frames.
>>>>
>>>> Having looked into the code it appears to be caused by the 
>>>> behaviour of the StreamParser class; specifically the 
>>>> ensureValidBytes1() method which calls getNextFrame() on my source 
>>>> with maxSize = BANK_SIZE - fTotNumValidBytes. The method switches 
>>>> banks to ensure that the larger of numBytesNeeded or the input 
>>>> sources maxFrameSize() will fit.
>>>>
>>>> I can 'fix' the problem by increasing BANK_SIZE and implementing 
>>>> maxFrameSize() on my source but I'm not totally happy with this 
>>>> solution because I would prefer not to modify the library source 
>>>> and I'm just guessing for the maxFrameSize() value.
>>>>
>>>> I was wondering whether it's possible to return a partial frame 
>>>> from my video source?
>>>
>>> Yes, but not in the way that you might think :-)  A H.264 encoder 
>>> actually delivers "NAL units".  "NAL units" are what actually get 
>>> parsed by our code, and packed into RTP packets.
>>>
>>> Often, a "NAL unit" is a complete frame.  It is possible, however, 
>>> for a 'key frame' to be split up - by your encoder - into multiple 
>>> 'slice' NAL units.  For datagram streaming (e.g., over RTP), it is 
>>> *much* better to have your key frames broken up into multiple 
>>> 'slice' NAL units, than to have the key frame be a single, large NAL 
>>> unit - which is what you have now.  This is especially true if your 
>>> key frames are exceptionally large: ~150000 bytes or larger, which 
>>> appears to be the case for you, because you are hitting the 
>>> BANK_SIZE limit (which was deliberately set to be larger than 
>>> realistically needed).
>>>
>>> Note that a 150000 byte key frame NAL unit will get transmitted as 
>>> more than 1000 RTP packets (datagrams).  (Our code automatically 
>>> handles the required fragmentation.)  If *any* of these 1000 packets 
>>> gets lost in transit, then the entire key frame will be undeliverable.
>>>
>>> If, instead, your encoder delivers each key frame as multiple 
>>> 'slice' NAL units, then your streaming will be much more resilient 
>>> to network packet loss.
>>>
>>> So, your first task should be to check whether your encoder:
>>> 1/ can be reconfigured to deliver discrete frames, rather than a 
>>> stream with each NAL unit prepended by a 0x00 0x00 0x00 0x01 'start 
>>> code', and
>>> 2/ can be reconfigured to deliver key frames as multiple 'slice' NAL 
>>> units, rather than as a single (ridiculously large) NAL unit.
>>>
>>>
>>> Ross Finlayson
>>> Live Networks, Inc.
>>> http://www.live555.com/
>>>
>>>
>>>
>>> _______________________________________________
>>> live-devel mailing list
>>> live-devel at lists.live555.com
>>> http://lists.live555.com/mailman/listinfo/live-devel
>>
>>
>>
>> _______________________________________________
>> live-devel mailing list
>> live-devel at lists.live555.com
>> http://lists.live555.com/mailman/listinfo/live-devel
>
>
>
> _______________________________________________
> live-devel mailing list
> live-devel at lists.live555.com
> http://lists.live555.com/mailman/listinfo/live-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.live555.com/pipermail/live-devel/attachments/20150319/8b238b4e/attachment.html>