[Live-devel] Crash in MPEG4/H264/H265 VideoFileServerMediaSubSession

[Robert Suijker]

Hi Ross,

I ran into a crash when adding the RTSP server to our storage engine. I have
come up with a work around but I'm not sure if it is the right place to fix.

The work around I've implemented is adding a check to the checkForAuxSDPLine1()
method:

void H264VideoFileServerMediaSubsession::checkForAuxSDPLine1() {
  char const* dasl;

+  if (fDoneFlag) return;
+
  if (fAuxSDPLine != NULL) {
	// Signal the event loop that we're done:


But I'm not sure if this is the right place to do the check since it seems more
related to the order of how the events are handled. This same fix (if it is a
fix) should also be applied to H265VideoFileServerMediaSubsession and the
MPEG4VideoFileServerMediaSubsession class.

Explanation of what I figured out so far:

It started with my own source filter (based on DeviceSource) that extracts
recorded data (raw MPEG-4 or H.264). But sometimes there isn't data recorded
in which case the Source filter calls handleClosure() to shutdown. What then
happens is I get a crash in fDummyRTPSink->auxSDPLine() called from
checkForAuxSDPLine1() which tries to access a freed source filter.
Since fDummyRTPSink is just a pointer to the created source filter but no longer
pointing to a valid source filter.

Then I checked if it was a specific problem with my source filter by running
the DynamicRTSPServer and request a RTSP session with VLC for an empty file
(zero byte) 'media.h264' file. I got the same crash in the server with the
H264VideoFileSource as I had with my source filter.

What I see happening is when a DESCRIBE comes in a RTP sink is created to
extract auxiliary SDP lines by calling into getAuxSDPLine(). This starts the
source filter in the startPlaying() call.

The source filter's doGetNextFrame() is called which calls handleClosure()
which triggers afterPlayingDummy1() which will set the done flag and unschedules
any tasks (which aren't yet started).

Next checkForAuxSDPLine1 will be called which schedules a task to checkForAuxSDPLine()
since there is no data available. Then in getAuxSDPLine() the event loop is
started that eventually runs the scheduled task and checkForAuxSDPLine1() will
be entered again, but this time the source filter is freed and the pointer
fDummyRTPSink points to a chunk of memory no longer being a source filter
the result is a crash.

When I prevent the source from calling handleClosure() I can't shutdown my
application since it runs the even loop from the H264VideoFileServerMediaSubsession
which won't check my flag. It will also crash the server eventually if enough
RTSP describe requests have been received with a stack overflow since it
basically recurse the getAuxSDPLine() method.

This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.