[Live-devel] H.264 Byte-stream (with audio), framer to use and syncing audio

[Ross Finlayson]

> http://lists.live555.com/pipermail/live-devel/2014-September/018686.html

Yes.  Everyone please read this again.


> So in the event you have a Byte-stream source, is there anyway to sync up the audio when using the H264VideoStreamFramer?

No.  If you want to properly synchronize video with audio, then (of course) you will need to have accurate presentation times for each (video and audio) frames.  Which means that you *cannot* use a “H264VideoStreamFramer” class (because that class tries to calculate presentation times itself, as it parses the byte stream).  (Having an unstructured video byte stream (fed to a “H264VideoStreamFramer”) is fine if you’re streaming video only, but it doesn’t work well if you are also streaming (synchronized) audio; see below.)

In other words, if you want your video to be properly synchronized with audio, then you must (i) configure your encoder to deliver one NAL unit at a time, *not* an unstructured byte stream; and (ii) give each of these NAL units a proper presentation time (which must be aligned with ‘wall-clock’ time - i.e., the time that you’d get by calling “gettimeofday())”); and (iii) feed each of these NAL units, one at a time, *without* a preceding ‘start code’, to a “H264VideoStreamDiscreteFramer” (*not* to a “H264VideoStreamFramer”).

(The reason why NAL units fed to a “H264VideoStreamDiscreteFramer” must not begin with a 'start code' is because we don’t want this ‘start code’ to appear in the resulting RTP packet - because that would make the RTP packet contents non-standards-compliant (i.e., not compliant with RFC 6184), meaning that compliant RTP receivers might not handle the data properly.  This is why our code outputs an error message if it sees a ‘start code’ at the beginning of a NAL unit fed to a “H264VideoStreamDiscreteFramer”, and why you should not modify the code to remove the outputting of that error message!)

I don’t know now much clearer I can be about all of this; I have explained it many times now (including in the link noted above).


Ross Finlayson
Live Networks, Inc.
http://www.live555.com/