[Live-devel] RTSPClient w/significant frame loss

[Ross Finlayson]

This is getting silly.  You're doing a lot of mental 'flailing around'; yet the answer is right in front of you:

> So I eliminated all memory allocation and memcpy of the 23K of data into the new frame buffer from the receive buffer, and I was back to getting almost zero frame loss. But as soon as I added in anything else, virtually immediately returned to about 2/3 (66%) frame loss again. But now I have a good specific example to discuss -- ffmpeg decodding. Now actually, I did not add everything back in that is required in processing, just a single call to avcodec_decode_video2 -- no scaling, color conversion, or rendering was done.The processing time for this one call averaged around ~19 milliseconds

There's the problem.  It's taking your CPU a whopping 19ms to decode each frame.  That means that - even if your CPU did absolutely nothing else - you couldn't handle any more than 50 frames-per-second.  In practice, your limit will be a lot less than this.  It's clear that your CPU is woefully underpowered.

It's really quite simple: *Every* LIVE555-based application that receives and processes a RTP stream is structured as follows:
	while (1) {
		Step A: Get a frame of data from a socket;
		Step B: Do something with this frame of data;
	}
That's it.  "Step A" is what the LIVE555 library does.  "Step B" is the new code that you add.  Every receiving application is structured like this - including VLC.  (The reason the VLC code looks a bit different is because it uses its own event loop, not ours.  But its structure is essentially the same.)

Note that your computer needs to be powerful enough so that it can execute each iteration of this loop (i.e., Step A + Step B) at least as fast as frames arrive.  If your computer can't handle frames faster than they arrive, then you will lose data.  End of story.  If this happens, then increasing the OS's socket buffer size can't save you.

Note also that - for each frame - "Step A" needs to be done before "Step B".  Thus the loop.  There's no way you can restructure this to make it more efficient.

(But wait, I hear some people say.  What if you used threads, so that one thread did
	while (1) {
		Step A: Get a frame of data from a socket;;
	}
and another thread did
	while (1) {
		Step B: Do something with this frame of data;
	}
?  That wouldn't help you.  Because each "Step B" relies upon a frame that was generated by "Step A", you'd need to write your own (thread-safe) queueing mechanism to buffer the frame data that gets passed between the two threads.  But this would just be duplicating - less efficiently - the socket buffering that's already inside the operating system.  So it would actually end up being less efficient than the single-threaded loop.  Note also that "Step A" consumes relatively little CPU, so you'd be unlikely to get any benefit even if you have multiple cores.)

If - as appears to be the case for you - your CPU isn't powerful enough to handle the incoming stream of frames, then you have three possible solutions:
	1/ Reduce the incoming frame rate, or the size of complexity of each incoming frame.  I.e., you'll need to change your encoder.
	2/ Improve your implementation of "Step B" - i.e., your code.  (If (and only if) you have a multi-core CPU, then this is where you *might* be able to use threads - just within your "Step B" code - to improve performance.)
	3/ Use a faster CPU.

The basic problem here is more fundamental.  Most people's exposure to the Internet these days is solely through HTTP, which - being a TCP-based protocol - means that data never arrives faster than the receiver can process it.  But datagram-based applications are different, and unfortunately many people these days - who think that the World-Wide Web is the entire Internet - don't appreciate this.  They try using the LIVE555 code and then, when they see network data loss for the first time in their life, think that the LIVE555 code has to be to blame.  Quite frankly, I'm getting tired of this.

This will be my last posting on this particular topic.


Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.live555.com/pipermail/live-devel/attachments/20120322/912cc7c1/attachment.html>