[Live-devel] Server socket not closed after TCP request from client

[Bruno Abreu]

Hello Ross,

while trying to replicate a problem that occasionally arises with a small set
of streaming servers, we detected something that might be an issue with
LIVE555.

Those servers are behind a firewall which blocks UDP packets (not under our
control). So the clients request data via TCP, not UDP. On occasion the
servers block or crash.

We still have to delve deeper into this and so, trying to replicate this
situation in our own environment, we ran testOnDemandRTSPServer and used
openRTSP as the client with the following parameters:

$ ./openRTSP -d 10 -t rtsp://<server>:8554/h264ESVideoTest

What we've noticed is that after the TEARDOWN from each client we can see a
socket left in CLOSE_WAIT state.

Since we're using a script (attached) that calls openRTSP repeatedly, the
number of CLOSE_WAIT sockets increases as the clients continue to make
requests.

The trick here seems to be the conjunction of the -d and -t flags.

$ lsof -i
testOnDem 32741 smartcodec    3u  IPv4 8466411      0t0  TCP *:8554 (LISTEN)
testOnDem 32741 smartcodec    4u  IPv4 8466414      0t0  TCP *:8000 (LISTEN)
testOnDem 32741 smartcodec    5u  IPv4 8466417      0t0  TCP
smartcodec-ddt2:8554->smartcodec-ddt2:47728 (CLOSE_WAIT)
testOnDem 32741 smartcodec    6r  IPv4 8466664      0t0  TCP
smartcodec-ddt2:8554->smartcodec-ddt2:47731 (CLOSE_WAIT)
testOnDem 32741 smartcodec    7u  IPv4 8466674      0t0  TCP
smartcodec-ddt2:8554->smartcodec-ddt2:47732 (CLOSE_WAIT)
testOnDem 32741 smartcodec    8u  IPv4 8466700      0t0  TCP
smartcodec-ddt2:8554->smartcodec-ddt2:47735 (CLOSE_WAIT)
testOnDem 32741 smartcodec    9u  IPv4 8466425      0t0  TCP
smartcodec-ddt2:8554->smartcodec-ddt2:47729 (CLOSE_WAIT)
testOnDem 32741 smartcodec   10r  IPv4 8466708      0t0  TCP
smartcodec-ddt2:8554->smartcodec-ddt2:47736 (CLOSE_WAIT)
testOnDem 32741 smartcodec   11u  IPv4 8466744      0t0  TCP
smartcodec-ddt2:8554->smartcodec-ddt2:47739 (CLOSE_WAIT)
testOnDem 32741 smartcodec   12u  IPv4 8466681      0t0  TCP
smartcodec-ddt2:8554->smartcodec-ddt2:47733 (CLOSE_WAIT)
...
testOnDem 32741 smartcodec   80u  IPv4 8467707      0t0  TCP
smartcodec-ddt2:8554->smartcodec-ddt2:47808 (ESTABLISHED)
testOnDem 32741 smartcodec   83u  IPv4 8467712      0t0  UDP *:6972
testOnDem 32741 smartcodec   84u  IPv4 8467713      0t0  UDP *:6973
testOnDem 32741 smartcodec   85u  IPv4 8467715      0t0  TCP
smartcodec-ddt2:8554->smartcodec-ddt2:47809 (ESTABLISHED)
testOnDem 32741 smartcodec   87u  IPv4 8467721      0t0  UDP *:6974
testOnDem 32741 smartcodec   88u  IPv4 8467722      0t0  UDP *:6975

Eventually, the limit of open sockets is reached and, when that happens, the
server usually blocks using a lot of CPU. Streaming stops and no more client
connections are accepted. Although, once, I've seen the server crash. Sorry I
didn't collect a core dump so I could print a back-trace.

I've also attached the logs from the server (class RTSPServer compiled with
DEBUG flag) after a run with the limit of open file descriptors set to 32.

Also, please notice the following change made to ByteStreamFileSource.cpp so
that our server could run continuously with a small file:

@@ -91,8 +91,9 @@

 void ByteStreamFileSource::doGetNextFrame() {
   if (feof(fFid) || ferror(fFid) || (fLimitNumBytesToStream &&
fNumBytesToStream == 0)) {
-    handleClosure(this);
-    return;
+    // handleClosure(this);
+    // return;
+    rewind(fFid);
   }

This doesn't seem to be the cause of our own server blocks/crashes, but we
thought we should report.

Thank you,
Bruno Abreu


-- 
Living Data - Sistemas de Informação e Apoio à Decisão, Lda.

LxFactory - Rua Rodrigues de Faria,   Mobile: +351 963428802
103, edifício I - 4º piso             Phone:  +351 213622163
1300-501 LISBOA                       Fax:    +351 213622165
Portugal                              URL: www.livingdata.pt
-------------- next part --------------
A non-text attachment was scrubbed...
Name: openRTSP-testOnDemand.sh
Type: application/x-shellscript
Size: 1054 bytes
Desc: not available
URL: <http://lists.live555.com/pipermail/live-devel/attachments/20131125/8f9b737f/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: testOnDemandRTSPServer.log.gz
Type: application/x-gzip
Size: 12570 bytes
Desc: not available
URL: <http://lists.live555.com/pipermail/live-devel/attachments/20131125/8f9b737f/attachment-0001.bin>