[Live-devel] Possible bugs in ByteStreamFileSource and MPEG2TransportStreamFramer

[Wiser, Tyson]

>>  >Hmm, reads from the input file (in this case, a pipe) are supposed to
>>>be non-blocking, returning only as many bytes as are currently
>>>available (which will always be >0, because the read is happening
>>>only in response to a return from "select()" on the input file's
>>>socket).  Perhaps there's some way you can configure your pipe to
>>>ensure that reads from it don't block?
>>
>>I agree that, since a select call was made, we are guaranteed that there
>>is at least one byte available for reading.  I think the problem lies in
>>the fread() function.  From the Linux man page and other online documentation
>>I get the impression that fread() will block until the requested amount of
>>data has been read, there is an error, or EOF is reached.  On the other
>>hand, the read() function is not blocking and returns the requested amount
>>of data or less if the requested amount is not currently available.
>
>I'm not convinced that any such distinction between the behavior of 
>"fread()" and "read()" is portable across OSs.  (For example, on 
>FreeBSD, the man page for "fread()" doesn't mention blocking at all.)
>
>A better solution is to add the following code to the 
>"ByteStreamFileSource" constructor (which is currently empty):
>
>#ifndef READ_FROM_FILES_SYNCHRONOUSLY
>	makeSocketNonBlocking(fileno(fFid));
>#endif
>
>and continue to implement "doReadFromFile()" using "fread()" rather 
>than "read()".
>
>Please let us know if that works OK for you.  If so, I'll make this 
>change in the next release of the software.

I have been unable to reproduce the event loop hang to test if this suggestion
fixes it.  It must have been a combination of the two issues mentioned in my
original post that caused this.

However, I have been able to verify that your suggested addition to the
"ByteStreamFileSource" constructor still does not behave as I would expect it
to when using "fread()".  My video source can be as slow as 2 FPS.  In this case
I would expect that after I write a complete frame to the pipe (again this is
H.264 video wrapped in an MPEG2-TS that is getting written to the pipe) that the
entire frame would be read and sent out over the network before the next frame
arrives with the last Ethernet packet of the frame potentially containing less
than 1316 bytes of data.

I put a debug statement in "ByteStreamFileSource::doReadFromFile()" that prints
the value of "fMaxSize" and "fFrameSize" just after the "fread()" call.  With or
without your suggested addition to the constructor both values are always equal
to 1316 (meaning it never reads less than that).  This results in some data not
being read and sent out on the network until the next frame arrives.  I have
seen as much as two of the 1316 byte reads get held back each frame.  This means
that the client does not receive the complete frame until the next frame comes
in, which can be up to 500 milliseconds later.

If I change the "fread()" call to a "read()" call, however, the behavior is what
I expect.  The majority of the debug statements show that the values of "fMaxSize"
and "fFrameSize" are still equal to 1316, but when we reach the end of the frame
the last read (and thus the value of "fFrameSize") is less than 1316.  This
results in the entire frame being read and sent out on the network as soon as it
is written to my pipe (I've confirmed this with WireShark).

I checked the FreeBSD man page for "fread()" and it appears to be essentially the
same as the Linux man page.  Neither one specifically mention blocking.  You have
to read between the lines. Quoting from the Linux man page:

"The function fread() reads nmemb elements of data, each of size bytes long..."
"fread() and fwrite() return the number of items successfully read or written
(i.e., not the number of characters). **If** an error occurs, or the end-of-file
is reached, the return value is a short item count (or zero)."

I emphasized "If" because that suggests to me that the error or EOF condition is
the only way that the returned count would not be equal to the requested count,
thus implying blocking.  http://www.cplusplus.com/reference/clibrary/cstdio/fread/
seems to also imply that "fread()" is blocking (emphasis is mine again):

"The total number of elements successfully read is returned as a size_t object,
which is an integral data type.  **If this number differs from the count parameter,
either an error occurred or the End Of File was reached**."

Thus, for my application, using "fread()" gives incorrect behavior while using
"read()" gives the correct behavior.  You are obviously much more cognizant of the
ramifications of this change than I am, so you may decide that the change should
not be included in the live555 library.  I ask that you at least consider it again.
Meanwhile I will continue to try to reproduce the event loop hang.

Thanks,
Tyson