[Live-devel] StreamReplicator with FileSink problem

[Bruno Abreu]

Hello Ross,

We have been running for some time a RTSPServer that acquires video from a live
source (with our own DeviceSource) from which we create 2 or 3 replicas: one for
multicast streaming and another to feed a FileSink for file storage. The third
replica is created on demand for unicast streaming.

Recently we detected a problem on some servers when the file partition on which
the files are being stored becomes full. When that happens, all pipelines
(FileSink, multicast and unicast) stop running.

The FileSink halt was expected and we are currently implementing a strategy to
deal with disk full issues. But, at least, the multicast streaming should
continue and we were somewhat surprised with these symptoms, so we tried to
understand what was going on.

This is what we concluded: when the partition becomes full, an error must be
detected in FileSink::afterGettingFrame(...) when fflush(fOutFid) is called.
After that, no next frame is requested from our upstream source, even though
there are several replicas in the replicator.

We think this is because in MediaSink::onSourceClosure(...) fSource is set to
NULL and, when MediaSink::stopPlaying() is called, fSource->stopGettingFrames()
is NOT CALLED since it was previously set to NULL.

And, since fSource->stopGettingFrames() is not being called, neither is
StreamReplica::doStopGettingFrames(). Now, we believe that our file sink replica
was the master replica in StreamReplicator and, since it's not being
deactivated, no other replica requests any more frames.

The following patch to MediaSink solved our current problem:

--- liveMedia/MediaSink.cpp (revision 16076)
+++ liveMedia/MediaSink.cpp (working copy)
@@ -92,7 +92,7 @@
 
 void MediaSink::onSourceClosure(void* clientData) {
   MediaSink* sink = (MediaSink*)clientData;
- sink->fSource = NULL; // indicates that we can be played again
+ // sink->fSource = NULL; // indicates that we can be played again
   if (sink->fAfterFunc != NULL) {
     (*(sink->fAfterFunc))(sink->fAfterClientData);
   }

We're just not sure that this is a correct solution or if we should override
MediaSink::stopPlaying() to set the MediaSink::fSource before 'if (fSource !=
NULL) fSource->stopGettingFrames()' is called, or something else entirely.

I've attached a small modification to testProgs/testReplicator.cpp that creates
2 replicas from a ByteStreamFileSource that reads the video
"bipbop-gear1-all.ts" (available from the LIVE555 site).

We run this app with the following script:

#--------
#!/bin/sh

./testReplicator &

DISK_AVAIL=`df -h . | sed '1d' | awk '{print $4}' | cut -d'%' -f1`

echo "type <Return> to continue and allocate ${DISK_AVAIL} on current partition"
echo "ATTENTION: This will allocate all free space"

read cont

fallocate -l ${DISK_AVAIL} dummy.1
#--------

It allows us to check that, after fallocate completely fills up the available
space, both the file and the udp sink replicas stop working.

Any help will be appreciated.

Thank You,
Bruno Abreu

-- 
Living Data - Sistemas de Informação e Apoio à Decisão, Lda.

LxFactory - Rua Rodrigues de Faria, 103,
edifício I - 4º piso                  Phone:  +351 213622163
1300-501 LISBOA                       Fax:    +351 213622165
Portugal                              URL: www.livingdata.pt

-------------- next part --------------
A non-text attachment was scrubbed...
Name: testReplicator.cpp
Type: text/x-c++src
Size: 4663 bytes
Desc: not available
URL: <http://lists.live555.com/pipermail/live-devel/attachments/20130110/ceab4984/attachment.bin>