[Live-devel] A question about (how to debug your crash)

[Ross Finlayson]

I've changed the Subject: line of this email, because it's not clear what, if anything, your crash has to do with threads.

> I pull and record from hundreds of cameras. Because of some OS limitaitions I can only wait on a maximum of 64 events.
>  
> So I have threads called processors for each group of 60 sources. Each of these has their own UsageEnvironment and TaskScheduler and event loop.
>  
> Is this a reliable architecture?

Yes, provided that *all* of the following are true:
1/ No thread ever accesses a "LIVE555" object that was created by another thread.  (The sole exception to this: A thread may call "triggerEvent()" on another thread's "TaskScheduler" object.)
2/ Your application code (that uses the LIVE555 libraries) is 'thread safe'.
3/ The system runtime library (that's used by both LIVE555 and your application code) is 'thread safe'.

That's why if you want to have multiple threads of control, it's better, if possible, to give each thread of control its own process, rather than having multiple threads of control running in the same process.


> today’s problem is distinctly different.
> It only occurs with one brand of camera.

What kind of camera is this?  I.e., what codec(s) does it stream?  Is there anything 'distinct' about it?


>     Am I correct in assuming no camera, no matter how bad the stream, should be able to crash live555?

Yes, of course.  Remember, though, that your application consists not just of the LIVE555 code, but also your application code, and the runtime libraries.  A bug in any one of these might cause a crash that could (e.g., because it steps on a wrong piece of memory) give the appearance of being in the LIVE555 code.


>  The access violation always occurs in the same location. 
>  On the delete packet; line of
>  ReorderingPacketBuffer::freePacket(BufferedPacket* packet){
>     if (packet != fSavedPacket) {
>       delete packet;
>     } else {
>       fSavedPacketFree = True;
>     }
>   }
>  
> In all inspections of memory the packet, whose pointer is not null, appears to already be destructed. The fBuf and vPtr are a special value 0xdddddddd which is Microsofts way of saying this memory was explicitly deleted.

Your first task should be to make sure that you're running the latest version of the LIVE555 code.  (I recently fixed a bug in the code that was causing symptoms similar (though not identical) to yours.)

Your second task, IMHO, should be to figure out if "packet" *really* has already been deleted.  One way to do this is to instrument both the constructor and destructor of "ReorderingPacketBuffer".  E.g., add something like
	fprintf(stderr, "created ReorderingPacketBuffer:%p\n", this);
to the "ReorderingPacketBuffer" constructor, and
	fprintf(stderr, "deleting ReorderingPacketBuffer:%p\n", this);
to the "ReorderingPacketBuffer" destructor.  Then, looking at your log after the crash, you'll be able to tell whether or not "packet" points to an actual "ReorderingPacketBuffer" object that had already been deleted.

It's more likely, though, that "packet" will be a 'garbage' pointer - i.e., not pointing to a (currently or recently allocated) "ReorderingPacketBuffer".  If this is the case, then the bug is that something must have (erroneously) written on top of the pointer.


> The call stack always shows we are a few frames down from MultiFramedRTPSource::networkReadHandler1() and I was trying to figure it out.
> I came across a question. Given the implementation of bytesAvailable() should the line “if bPacket->bytesavailable() == 0)…” be if bPacket->bytesavailable() <= 0)…”

No, because the "bytesAvailable()" function returns an unsigned value, i.e., always >= 0.  This is almost certainly a 'red herring'.


>  I am looking for suggestions on where to look for the source of this problem. Simple or scaled down examples do not have a problem.

Nonetheless, you should try to find the simplest possible environment that causes your crash.  First, you should make sure that the (unmodified) "testRTSPClient" demo application does not crash when it streams from this network camera.  Then, you should try running a single-threaded version of your application - first with just this single network camera as a source, then with more sources (up to your maximum of 60).  If the single-threaded version of your application crashes - even once - then you will know that multi-threading in your application is not to blame.

In any case, if the crash occurs only with your application (and not with (e.g.) "testRTSPClient"), then it's unlikely that I'll be able to debug it for free on this mailing list, unfortunately...

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.live555.com/pipermail/live-devel/attachments/20130407/f9f0d96e/attachment.html>