[Live-devel] Synchronizing audio and video streams

[Severin Schoepke]

Hello again,

I'm referring to the following mail:


I'm working on an application that streams live generated content (audio 
and video) using the Darwin Streamnig Server. I decided to use ffmpeg 
for the encoding part and live555 for the streaming part. The basic 
architecture is as follows: In a first thread I generate the content, 
and in a second one I stream the content to the DSS. Data is passed 
between the threads using two buffers.
I implemented two FramedSource subclasses for the streaming part, one 
for audio and one for video data. Both work as follows: They read raw 
data from their input buffer and encode it using ffmpeg (using MPEG4 and 
MP2 codecs) and write to their output buffers (fTo). My sources are then 
connected to an MPEG4VideoStreamFramer and an MPEG1or2AudioStreamFramer 
respectively. These are connected to a DarwinInjector (based on the code 
of testMPEG4VideoToDarwin).
The problem I have now is that the streams are not synchronized when 
viewing them (using QuickTime or VLC). Based on debug printf's I found 
out that the audio source's doGetNextFrame() is called much more that 
the video source's. Therefore, the audio stream is played ahead and the 
video stream is lagging behind.
I set correct presentation times for both streams, so I thought that 
live555 does 'the rest' synchronizing the streams, but it seems not to 
work. Therefore I'd like to ask you if I should implement a 
synchronization of the streams by myself, or if I'm doing something 
wrong...
I followed your suggestion and tried it using your server...


I got an answer:

Ross Finlayson schrieb:
> We're no longer providing support for the "DarwinInjector" code. 
> Instead, we recommend that you use our own RTSP server 
> implementation, rather than a separate "Darwin Streaming Server".
>   


So I tried to use your RTSP server but it didn't change the situation. 
My streams are still diverting. I investigated the problem further and 
fond out, that live555 queries my sound source much more often than it 
queries the video source. I suspect it has something to do with data 
size: my audio source provides only about 300 bytes per invocation, 
whereas the video source provides between 3000 and 8000bytes per 
invocation (I'm referring to a call of doGetNextFrame() as an 
invocation). I suspect that live555 waits with streaming out video until 
it has aboout the same size of audio data. This way, my audio source 
gets polled about 10 times as much as the video source, even though it 
can only provide about twice as much frames than the video source...
I know, the above sounds a bit strange and is not very clear, but I hope 
someone understands what I mean and I hope that this person tells me 
wether this behaviour is to be expected and how I could change it.

cheers, Severin