[Live-devel] Mpeg2 ES and audio PCM stream Synchronization with Live555

[Guy Bonneau]

I would like to provide some feedback of my synchronization issues 
with mpeg video and audio PCM. I think it might be useful to other 
users of the library.

I have slightly modified 2 testing applications included with Live555: 
testMPEG1or2VideoStreamer and testMPEG1or2VideoReceiver. I have included 
these 2 applications with this email.

I have added logging in the testing application testMPEG1or2VideoReceiver
to a log file. Logging to the console was causing "stress" to the 
environment thread and starved the library from CPU processing.

I first start the testMPEG1or2VideoStreamer application. That application
streams an Mpeg2 Video Elementary Stream and a PCM audio wave. The audio
file must be 48KHz 2 Channels 16 bits. Otherwise modification
to the testMPEG1or2VideoReceiver app will be needed. I have introduced 
a delay of 3 seconds between the creation of the source objects of the 
library and the actual starting of the sinking that begin the streaming. 
This closely simulates our use of the library by our own application. 
Our application starts the sinking when the user press the "play" button 
but create the objects when the application is instantiated.

Once testMPEG1or2VideoStreamer begins streaming I start the second
application testMPEG1or2VideoReceiver. I have modified the application 
to closely match the design of the main receiving loop of the 
VLC Media application. The application receives both the video and
audio streams. I don't store the data in this testing case.

Here is an excerpt of the log on the receiving client before the 
library send the RTCP SR:

Video received 1311 bytes at time 0-00:20:13-702.255
Video received 1339 bytes at time 0-00:20:13-702.255
Video received 1348 bytes at time 0-00:20:13-702.255
Video received 1395 bytes at time 0-00:20:13-702.255
Video received 1068 bytes at time 0-00:20:13-702.255
Video received  506 bytes at time 0-00:20:13-702.255
Audio received 1400 bytes at time 0-00:20:13-720.633
Audio received 1400 bytes at time 0-00:20:13-727.924
Audio received 1400 bytes at time 0-00:20:13-735.215

(...)

At this point both the Video and Audio are closely in sync.
I guess using the client wall clock until RTCP SR. 

Here is an excerpt of the log after the library sent the
RTCP SR:

Video received  506 bytes at time 0-00:20:13-702.255
Audio received 1400 bytes at time 0-00:20:13-720.633
Audio received 1400 bytes at time 0-00:20:13-727.924
Audio received 1400 bytes at time 0-00:20:13-735.215

Audio hasBeenSynchronizedUsingRTCP()

Audio received 1400 bytes at time 0-00:33:24-123.145

Video hasBeenSynchronizedUsingRTCP()

Video received 1047 bytes at time 0-00:33:21-122.121
Video received  987 bytes at time 0-00:33:21-122.121
Video received 1205 bytes at time 0-00:33:21-122.121
Video received 1152 bytes at time 0-00:33:21-122.121
Audio received 1400 bytes at time 0-00:33:24-130.436 (3 Sec GAP)
Video received 1160 bytes at time 0-00:33:21-122.121
Video received  994 bytes at time 0-00:33:21-122.121
Video received 1007 bytes at time 0-00:33:21-122.121
Video received  559 bytes at time 0-00:33:21-122.121
Audio received 1400 bytes at time 0-00:33:24-137.727
Audio received 1400 bytes at time 0-00:33:24-145.018
Audio received 1400 bytes at time 0-00:33:24-152.309
Video received 1053 bytes at time 0-00:33:21-155.487
Video received  991 bytes at time 0-00:33:21-155.487

As soon as the RTCP SR report is received at the client side the
library resync the presentation time to the server wall clock but 
a gap is created between the audio and video. The presentation 
time gap matches the delay between the creation of the source 
objects and the actual starting of the sinking.

I noted that the object MPEGVideoStreamFramer creates the base 
presentation time when method MPEGVideoStreamFramer::reset() is called.
This method is called at creation time of MPEGVideoStreamFramer or when
the method flushInput() is called. The comment associated to this method
is: //called if there is a discontinuity (seeking) in the input.

In our case the Mpeg2 stream was streaming from a live source and the 
flushInput() method was never called. Thus the base presentation time
used when the Mpeg2 streaming began was from the object creation time. 
This created a huge gap between the audio and mpeg2 stream. And VLC
media player was baffled by the presentation time once RTCP sync was
received. VLC Media Player got lost trying to regain synchronization.

This gap can be solved if the library source objects are instantiated when
the sink are started. But this was unacceptable in my case. I can also
make a call to the method flushInput() just before starting the sinking.
This is the easiest solution.

A small modification could also be done to the library. Since the
computing of the presentation time by MPEGVideoStreamFramer is internal
to the library and cannot be reassigned I fell the initialization of
the base presentation time should be done when method 
computePresentationTime is called by object MPEGVideoStreamFramer. This
could be done like this:

void MPEGVideoStreamFramer::reset() {
  fPictureCount = 0;
  fPictureEndMarker = False;
  fPicturesAdjustment = 0;
  fPictureTimeBase = 0.0;
  fTcSecsBase = 0;
  fHaveSeenFirstTimeCode = False;
  fPresentationTimeBase.tv_sec = 0;  //added
  fPresentationTimeBase.tv_usec = 0; //added
  fFlushedInput = True;  // Added this private member to object
  //Removed call to gettimeofday
}

void MPEGVideoStreamFramer
::computePresentationTime(unsigned numAdditionalPictures) { 
  // Use the current wallclock time as the base 'presentation time':
  if( fFlushedInput == True )
  {
    fFlushedInput = False;
    gettimeofday(&fPresentationTimeBase, NULL);
  }
(...)
}

Or

I suggest the adding of the calling to the method flushInput() in
the testing application testMPEG1or2VideoStreamer of the library
before starting the sink.





-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: testMpeg1or2VideoAndLPCMStreamer.cpp
Url: http://lists.live555.com/pipermail/live-devel/attachments/20080212/8ba7dba1/attachment.ksh 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: testMPEG1or2VideoReceiver.cpp
Url: http://lists.live555.com/pipermail/live-devel/attachments/20080212/8ba7dba1/attachment-0001.ksh