[Live-devel] How to synchronize audio and video[already read FAQ]

[Shixin Zeng]

On 7/31/05, Ross Finlayson <finlayson at live.com> wrote:
> 
> At 08:43 PM 7/30/2005, you wrote:
> >Hello, all
> >I've read the FAQ about synchronization. It says that the parameter
> >presentationTime passed to afterGetFrame() can be used to synchronize,
> 
> Yes, but note that this FAQ applies to *receiving* RTP streams (i.e.,
> over a network). I.e., it applies to subclasses of "RTPSource".
> 
> > but I can't figure out how to use the parameter. I even can't
> > understand the exact mean of the parameter. Is it the same as the
> > MediaSource::fPresentationTime exactly? In other words, if in my
> > MultiFramedMediaSource::doGetNextFrame() implementation, I set
> > fPresentationTime={1000,1000} for one particular frame, the
> > receiver should get a presentationTime={1000,1000} for this frame
> > when call afterGettingFrame, shouldn't it?
> 
> It's not clear from your description exactly what you are using your
> "MultiFramedMediaSource" for. But, if you are using it as a source
> for an outgoing RTP stream (i.e., feeding it into a "RTPSink"), then
> your presentation times should be aligned with real, 'wallclock'
> time. I.e., the first time you generate "fPresentationTime" in your
> source object, you should do so using the value obtained by calling
> "gettimeofday()". Plus, of course, you must have a "RTCPInstance" -
> at both the sender and receiver - for synchronization to work.


Yes, I use FramedSource as the source for SimpleRTPSink
my implementation is:

class BufferedFramedSource: public FramedSource{
public:
BufferedFramedSource(UsageEnvironment & env,
const std::string & type);
virtual ~BufferedFramedSource();
private:
void doGetNextFrame();

std::string category;
};


BufferedFramedSource::BufferedFramedSource(UsageEnvironment & env, 
const std::string & type)
:FramedSource(env), fInBuffer(ib), category(type)
{
gettimeofday(&fPresentationTime, NULL);
};
BufferedFramedSource::~BufferedFramedSource()
{
}

void BufferedFramedSource::doGetNextFrame()
{
unsigned frameSize = 1024;
if(fMaxSize < frameSize){
fNumTruncatedBytes = frameSize - fMaxSize;
fFrameSize = fMaxSize;
}else{
fNumTruncatedBytes = 0;
fFrameSize = frameSize;
}
fPresentationTime.tv_sec += 10;

//fill the buffer with the discription of fPresentationTime
std::stringstream ss(std::stringstream::out);
ss << category 
<<" presentation time : "<< fPresentationTime.tv_sec <<"s, "
<< fPresentationTime.tv_usec << "microseconds";
memcpy(fTo, ss.str().c_str(), fFrameSize);

fDurationInMicroseconds = 1000;

nextTask() = envir().taskScheduler().scheduleDelayedTask(0,
(TaskFunc*)afterGetting, this);
}

On the client peer, I use SimpleRTPSource to receive the stream, and try to 
print the presentation asocciated with the frame.

class BufferedSink: public MediaSink{
public:
BufferedSink(UsageEnvironment &env, 
ofstream & file,
unsigned maxSize=1024);
~BufferedSink();
protected:
static void afterGettingFrame(void *clientData, 
unsigned frameSize, 
unsigned numTruncatedBytes, 
struct timeval presentationTime, 
unsigned durationInMicroseconds);

virtual Boolean continuePlaying();
private:
virtual void afterGettingFrame1 (unsigned frameSize, 
struct timeval presentationTime,
unsigned durationInMicroseconds);
unsigned char *fBuffer;
ofstream & ofile;
};



BufferedSink::BufferedSink(UsageEnvironment &env, 
ofstream & file,
unsigned maxSize):
MediaSink(env), fBufferSize(maxSize),
ofile(file)
{
fBuffer = new unsigned char[maxSize];
}

BufferedSink::~BufferedSink()
{
delete []fBuffer;
}

void 
BufferedSink::afterGettingFrame(void* clientData, 
unsigned frameSize,
unsigned numTruncatedBytes,
struct timeval presentationTime,
unsigned durationInMicroseconds) 
{
BufferedSink* sink = (BufferedSink*)clientData;
sink->afterGettingFrame1(frameSize, 
presentationTime, 
durationInMicroseconds);
}


void BufferedSink::afterGettingFrame1(unsigned frameSize,
struct timeval presentationTime, 
unsigned durationInMicroseconds) {
ofile << "presentationTime: " << presentationTime.tv_sec << "s,"
<< presentationTime.tv_usec << "micros, "
<<"durationInMicroseconds: "<< durationInMicroseconds 
<<", size: " <<frameSize << " bytes" << endl;
ofile << "Content of the buffer is:\n" << fBuffer << endl << endl;
if(presentationTime.tv_sec == 0 
&& presentationTime.tv_usec == 0){
// The output file has closed. Handle this the same way as if the
// input source had closed:
std::cout<< "End of the stream...";
onSourceClosure(this);

stopPlaying();
return;
}
// Then try getting the next frame:
continuePlaying();
}

Boolean
BufferedSink::continuePlaying()
{
if (fSource == NULL) return False;

fSource->getNextFrame(fBuffer, fBufferSize,
afterGettingFrame, this,
onSourceClosure, this);

return True;
}

I also created a RTPCPInstance for the stream. What I got from the output of 
the client is:

presentationTime: 1122790619s,599583micros, durationInMicroseconds: 0, size: 
1024 bytes
Content of the buffer is:
video presentation time : 1123141214s, 536290microseconds

presentationTime: 1122790629s,599583micros, durationInMicroseconds: 0, size: 
1024 bytes
Content of the buffer is:
video presentation time : 1123141224s, 536290microseconds

Apparently, the received presentationTime is not the same as the sending 
presentationTime. What's wrong with my codes?


>Another attribute of MediaSource I don't understand is
> >fDurationInMicroseconds. In the above example, I set it to 10000 in
> >MultiFramedMediaSource::doGetNextFrame, but I always get a 0 for
> >durantiontimeInMicroseconds in afterGetttingFrame(), why?
> 
> "duration in microseconds" - unlike "presentation time" - is not a
> parameter that gets passed within RTP (i.e., from sender to
> receiver). Instead, it is a parameter that is used only internally
> within a chain of "Media" objects. In particular, for data sources
> that feed into a "MultiFramedRTPSink" (subclass), "duration in
> microseconds" tells the "MultiFramedRTPSink" how long to delay
> between sending each outgoing RTP packet.


That is to say the durationTime from RTPSource is useless? 

To understand how the "LIVE.COM <http://LIVE.COM> Streaming Media" code 
> works, I
> suggest that you start by examing the code for the existing demo
> applications ("testProgs"), before trying to modify this to develop
> your own code.


I have read testMP3Streamer and testMP3Receiver. but synchronization is not 
involved in them.
Thanks!

_______________________________________________
> live-devel mailing list
> live-devel at lists.live.com
> http://lists.live.com/mailman/listinfo/live-devel
> 



-- 
Best regards

Shixin Zeng
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.live.com/pipermail/live-devel/attachments/20050731/cd655dec/attachment-0001.html