[Live-devel] MP4 video trak samples: were do 'non-frame' NAL units belong?

[Joel Keller]

Greetings experts,

Live555 is playing a key role in our industrial machine vision application
- thank you for the library!  I am experiencing a problem saving MP4 format
video files using openRTSP.  The RTSP/RTP stream is coming from a camera,
and I can successfully save the 'raw/elementary' H264 NAL stream with
openRTSP.  However, when I save the video as MP4, with something like this:

openRTSP -d 30 -b 1000000 -B 1000000 -4 -y -w 1536 -h 1536 -f 15 rtsp://
10.0.3.11/live.sdp > myvideo.mp4


I find that the generated MP4 file contains NAL units in the 'mdat' box
which do not correspond to video frames, specifically NAL units of type
SPS, PPS, and SEI.  These three NAL units are periodically interspersed
with the frame data.  This interspersion itself is of course not a problem
as the media data area can contain non-frame data, and this meta data is of
course needed to interpret the frame.  However these NAL units are treated
as 'MP4 samples' in the MP4 'Sample Table Boxes'.  Specifically, in the
stts (Decoding time to sample box), these samples have a "delta" of zero,
implying that their presentation time is the same as the samples
which precede them (this makes sense, because they are not frames of the
video, but it violates ISO spec, which states that "The sample entries are
ordered by decoding time stamps; therefore the deltas are all
non-negative.").   I find that this causes some (most?) video players to
"stutter" when playing back the MP4 file (avidemux2 for example),  because
they determine the video's frame rate and then go sample-by-sample through
the video.  When the player gets to these non-frame samples, it appears to
delay for the frame-duration, but it cannot decode the frame (because the
sample is not a frame).  This leads to a video with ~15 smooth frames, then
3 pause frames, then ~15 smooth frames.

My questions are:

1)  What is the correct thing to do with these NAL units in the MP4 file?

2)  How can I make openRTSP/live555 create an MP4 file without this
stuttering?



Possible answers I have to #1 are:

a.  "non-frame" NAL units are appropriate samples in MP4 video traks.  But
then what to do with the stts deltas?

b.  "non-frame" NAL units should not show up as samples (see avcC box), and
should be 'skipped over' based on MP4 'chunks'.  But how do I get live555
to do this?

c.  "non-frame' NAL units should be "combined" their following "frame" NAL
unit to be a single 'MP4 sample'.  (ie a sample is not necessarily a single
NAL unit). Again how would I get live555 to do this?



Thank you for reading this long post.  I am most grateful for any help.


Thanks,

Joel Keller
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.live555.com/pipermail/live-devel/attachments/20130319/6467ce8d/attachment.html>