[Live-devel] RTSPClient w/significant frame loss

Wed Mar 21 18:38:53 PDT 2012

Ross, 

Thanks for the reply. I appreciate the pointer to the additional information. I did read it and followed its advice, but in a nutshell, I'm still experiencing around 60% packet loss though doing very little processing in the afterGettingFrame() method callback in my MediaSink subclass. The FAQ makes the statement:

"It's important to understand that because a LIVE555 Streaming Media application runs as a single thread - never writing to, or reading from, sockets concurrently - if packet loss occurs, then it must be happening either (i) on the network, or (ii) in the operating system of the sender or receiver. There's nothing in our code that can be 'losing' packets."

I do not dispute that, but I need to know more about the implications of Live555 behavior, because even though it might not explicitly lose packets, the implications of its approach may result in the same for any app actually doing any processing. I hope there's something I'm missing, which can improve performance. I'll explain. 

Earlier in the thread, I mentioned that the big picture of my use case is receiving an H.264 video stream via RTSP with Live555, then decoding the received data with ffmpeg, and displaying the resulting video frame to the screen. I originally thought the data loss was occurring in the ffmpeg decoding and rendering to the screen, until I removed decoding and rendering entirely, and discovered the frame loss was experienced before even leaving the afterGettingFrame method. Here's the specifics of what I did: 

1. I followed the FAQ's recommendations, which increased the receiveBuffer to (VLC's recommended) 2000000. (Note, I also tried 400000, same results as 2000000, also note there are no truncated bytes).

2. I removed all ffmpeg decoding, rendering to the screen, and all processing whatsoever from the afterGettingFrame() callback on the MediaSink subclass, except for logging counts of frames received. Combined with the receiveBuffer increase in 1, this resulted in receipt of 28-29fps, which tracks with the 30fps being sent by the server. At this point, I was fairly certain that I had the Live555 side of the processing pipeline figured out, and there was nothing left to do. 

3. <Abridged here, it has taken me days to discover this> I added back in the minimal processing I was doing in the afterGettingFrame() callback of the MediaSink subclass *but didn't add any decoding/displaying* -- all I am doing is copying a 4-byte start code and and the data in the client buffer to another buffer to send it downstream. Specifically these two chunks of code: 

    // add the start code
    len = 4;            
    memcpy((newFrame + offset), start_code, len);
    offset += len;

    // add the buffer data
    memcpy((newFrame + offset), fReceiveBuffer, frameSize);

when the code above was commented out, I get no frame loss, around 28-29fps. When the code is included, I only get 10-14fps, i.e. around 60% frame loss. Note, nothing is being done in afterGettingFrame() or anywhere else with this buffer...after this code above, continuePlaying() is called. The reason I put this code in was for the sole purpose of gauging what effect if any this had on frame loss. Clearly, it has a huge impact. This now probably warrants some explanation of our stream being processed. 

Our H.264 stream is very simple. gop size is 1, each one with 3 sequential NAL units being sent repeatedly: 

NAL 7 | NAL 8 | NAL 5

NAL units 7 and 8 are 4-9 bytes each whose general purpose is just to communicate frame size. NAL unit 5 is consistently around 23K and is the frame image data. So in reference to the code snippets above, when afterGettingFrame() is called, I copy both a start code and the actual data into a new buffer. (Now in actual processing I'll copy prop records info as well, and then pass this on to ffmpeg, but this was all commented out for this test). That memcpy is basically copying 15 bytes of data for the first two afterGettingFrame calls and 23K on the third afterGettingFrame call for what essentially is one video frame. 

This brings us to the heart of the matter: the only processing being done here is just simply the creation of a new buffer. Why would this result in a 60% frame loss? No decoding or displaying of any kind is even being attempted. 

My assumption is that I'm missing or misunderstanding something fundamental about Live555, and there's a knob or switch to be turned or flipped somewhere that will solve the problem. Because as it is, I'm not seeing how any processing at all of this stream is realistically feasible (even handing off the processing) without losing over half the stream data. 

So I return to the statement in the FAQ -- if the culprit in frame loss is essentially afterGettingFrame() processing, is there any metric or quantification on whether any real processing (or even handing off of data) can actually be done without prohibitive data loss? There's got to be a way to receive a video stream and actually do something minimal with it without losing so much data. 

Ross, yours (and anyone else's) expertise and ideas here are greatly appreciated. 

Thanks, 

Brad

Brad O'Hearne
Founder / Lead Developer
Big Hill Software LLC
http://www.bighillsoftware.com

On Mar 10, 2012, at 11:08 PM, Ross Finlayson wrote:

>> 1. What factors / culprits could be causing this massive frame loss? 
>> 
>> 2. Is there anything in the base RTSPClient or MediaSink subclasses (or elsewhere) that can be configured to reduce frame loss? 
> 
> See <http://www.live555.com/liveMedia/faq.html#packet-loss>
> 
> You should also, of course, make sure that your receiver ("MediaSink" subclass)'s buffer is big enough for the incoming NAL units.  (You can do this by checking whether "numTruncatedBytes" is ever >0.)
> 
> 
> Ross Finlayson
> Live Networks, Inc.
> http://www.live555.com/
> 
> _______________________________________________
> live-devel mailing list
> live-devel at lists.live555.com
> http://lists.live555.com/mailman/listinfo/live-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.live555.com/pipermail/live-devel/attachments/20120321/ae76d6b8/attachment-0001.html>