[Live-devel] Live555 Proxy Server sometimes does not proxy source after connection loss

[Erik Montnemery]

Regarding issue 2)
>> the state for the stream has timed out, and the proxy server then attempts to re-open a RTSP connection to the ‘back-end’ server. However, this appears to be hanging - again, apparently due to your ‘back-end’ server being broken.
>The issue is triggered when there is network outage, and can be
>reproduced by forcing the 'back-end' devices to repeatedly reconnect
>to the WIFI.
>After reproducing a few more times, I suspect the reestablishing of
>network connection (sometimes) happens because there is a race
>condition/timing issue:
I don't think I made much sense, let me rephrase:
Issue 2) is triggered when there is network outage, and can be
reproduced by forcing the 'back-end' devices to repeatedly reconnect
to the Wi-Fi network.

After reproducing a few more times, it is clear there is an issue if
ProxyRTSPClient::reset is called while a connection to 'back-end' is
in progress.
In the logs, this is triggered by ProxyRTSPClient::continueAfterLivenessCommand.
In this case, the connection attempt will be reset by
RTSPClient::reset, but on the next call to RTSPClient::sendRequest,
fRequestsAwaitingConnection.isEmpty() will return false because the
queue is not empty.

Should the fRequestsAwaitingResponse queue be cleared by
RTSPClient::reset to prevent this?

Or, should the fRequestsAwaitingConnection queue be cleared in
RTSPClient::reset?

Or would it be correct to modify RTSPClient::sendRequest from:
if (!fRequestsAwaitingConnection.isEmpty()) {
To:
if (!fRequestsAwaitingConnection.isEmpty() && fInputSocketNum >= 0) {

/Erik

2016-09-05 21:28 GMT+02:00 Erik Montnemery <erik at montnemery.com>:
> Thanks Ross,
>
> Regarding issue 1)
> You might be right, I'll try to reproduce while recording network traffic.
>
> Regarding issue 2)
>> the state for the stream has timed out, and the proxy server then attempts to re-open a RTSP connection to the ‘back-end’ server. However, this appears to be hanging - again, apparently due to your ‘back-end’ server being broken.
> The issue is triggered when there is network outage, and can be
> reproduced by forcing the 'back-end' devices to repeatedly reconnect
> to the WIFI.
> After reproducing a few more times, I suspect the reestablishing of
> network connection (sometimes) happens because there is a race
> condition/timing issue:
>
> Example 1)
> Sep 05 18:24:18 ProxyServerMediaSubsession[(NULL),H264]::closeStreamSource()
> Sep 05 18:24:18
> ProxyServerMediaSubsession[(NULL),H264]::~ProxyServerMediaSubsession()
> Sep 05 18:24:18
> ProxyServerMediaSubsession[(NULL),MPEG4-GENERIC]::~ProxyServerMediaSubsession()
> Sep 05 18:24:18 Opening connection to 192.168.0.126, port 554...
> Sep 05 18:24:18 ProxyRTSPClient[rtsp://192.168.0.126/ch0_0.h264]: lost
> connection to server ('errno': 115).  Resetting...
> <----- "Reset" triggered in
> ProxyRTSPClient::continueAfterLivenessCommand while reconnect to the
> 'back-end' is in progress
>
> Example 2)
> (Note, I added some additional debug prints)
> Sep 05 19:53:16 StreamState::reclaim()
> Sep 05 19:53:16 ProxyServerMediaSubsession[(NULL),H264]::closeStreamSource()
> Sep 05 19:53:16
> ProxyServerMediaSubsession[(NULL),H264]::~ProxyServerMediaSubsession()
> Sep 05 19:53:16
> ProxyServerMediaSubsession[(NULL),MPEG4-GENERIC]::~ProxyServerMediaSubsession()
> Sep 05 19:53:16 Opening connection to 192.168.0.126, port 554...
> Sep 05 19:53:16 Setting up BackhroundHandling for socket 7
> Sep 05 19:53:16 ProxyRTSPClient[rtsp://192.168.0.126/ch0_0.h264]: lost
> connection to server ('errno': 115).  Resetting...
> <----- Again, "Reset" triggered in
> ProxyRTSPClient::continueAfterLivenessCommand while reconnect to the
> 'back-end' is in progress
>
> /Erik
>
> 2016-09-05 10:12 GMT+02:00 Ross Finlayson <finlayson at live555.com>:
>>> Most of the time, the proxy server reconnects to the source after a
>>> loss of connection to a source, but every now and then I observe one
>>> of the following issues:
>>
>> In each case, the problem (from what I can tell) seems to be the fault of the ‘back-end’ server - i.e., it seems to be broken.
>>
>>
>>> 1) The proxyserver reconnects to the source, but no data frames are
>>> forwarded to the clients.
>>
>> If no data are forwarded to the (‘front-end’) clients, then that must mean that the ‘back-end’ server was not sending any data to the proxy server.  If the ‘back-end’ server appears responsive in spite of this (i.e., the RTSP connection remains open, and the server continues responding to “OPTIONS” requests from the proxy server), then it must be broken in some way…
>>
>>
>>> 2) The proxy server does not reconnect to the source, and clients to
>>> the proxy server are being told:
>>> RTSP/1.0 404 File Not Found, Or In Incorrect Format
>>> An example log of scenario 2) is pasted below
>> […]
>>> ProxyServerMediaSubsession[(NULL),H264]::~ProxyServerMediaSubsession()
>>> Jul 26 20:25:44
>>> ProxyServerMediaSubsession[(NULL),MPEG4-GENERIC]::~ProxyServerMediaSubsession()
>>> Jul 26 20:25:44 Opening connection to 192.168.0.126, port 554…
>>
>> In this case the state for the stream has timed out, and the proxy server then attempts to re-open a RTSP connection to the ‘back-end’ server.  However, this appears to be hanging - again, apparently due to your ‘back-end’ server being broken.
>>
>> I suggest just replacing your ‘back-end’ servers (i.e., web cams or whatever).
>>
>>
>> Ross Finlayson
>> Live Networks, Inc.
>> http://www.live555.com/
>>
>>
>> _______________________________________________
>> live-devel mailing list
>> live-devel at lists.live555.com
>> http://lists.live555.com/mailman/listinfo/live-devel