[Live-devel] An implicit bug in RTSPClient::getResponse()1 on Windows

[Brain Lai]

Dear Sir:

In an application where the RTSP server has a limit on the number of
connections, the server may send something like "RTSP/1.0 406 Not
Acceptable\r\n\r\n" and/or close the incoming socket immediately because the
max number of connection is reached.

If the sample agent openRTSP(VLC, too) connects to the server in the
meanwhile, it will be stuck in RTSPClient::getResponse()1 on Windows while
read 0 due to "connectin reset by peer" on Linux.

So, I step inside RTSPClient::getResponse1() and find the following code
causes the problem:

  unsigned bytesReadNow = readSocket(...);
  where readSocket(...) has return type int and may return -1 in the above
scenario.
  Such that the condition never holds:
  if (bytesReadNow == 0) {
      envir().setResultMsg("RTSP response was truncated");
      break;
  }

  This is because (unsigned)-1 is greater than zero. At this time, the
client agent enters an infinite read loop and causes the CPU fulll load.
  Thererfore, I suggests to modify the code as the following:

 int bytesReadNow = readSocket(...);
  if (bytesReadNow <= 0) {
      envir().setResultMsg("RTSP response was truncated");
      break;
  }

  This should ends the probelm.

BR.
Brain Lai
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.live555.com/pipermail/live-devel/attachments/20071115/cf76a56b/attachment.html