[Live-devel] Urgent help needed!

[Morgan Tørvolt]

> OK, I guess I was assuming that the OS's file system implementation
> is smart enough to detect sequential reads within a file (even  if
> they're interleaved with sequential reads within other files), and
> optimize this by doing read-ahead - for each file - into its file
> system cache.  However, I'm not sure in practice how smart OSs are
> about this, but if the OS kernel *could* do this automatically, that
> would certainly be better than having the application implementing
> its own read-ahead in user space.  In any case, file systems
> typically read from the disk in large block sizes (certainly much
> larger than 188 bytes), so you get some 'read-ahead' for free there.
> (OTOH, we had a major VOD customer that had no choice but to use
> Windows, and they ended up deciding to do their own
> read-ahead/buffering of Transport Stream file data in user space
> (replacing "ByteStreamFileSource" and "MPEG2TransportStreamFramer").
> But perhaps other OSs aren't as brain-damaged as Windows about
> this...)

We run Linux, and we have seen some significant improvements by
reading larger chuncks, but if they get too large, we got PCR jitter
problems due to the "slow read" of the harddrives. It seemed like when
three or more clients needed read-time after one-another (happened
randomly from time to time), the time it took to finish those three
reads made the following packets delayed. This was with very large
buffers though. With other clients (like VLC) one could probably
increase the buffer size even more than we do.
A disk with 60MB/s throughput can give you 600kB in (avg-seek-time +
600k/60M) = 8ms + 10ms = 18ms. 60kB would be achieved in 9ms. Half the
time, one tenth of the data.

The filesystem reads in larger chunks than a TS packet, that is true,
but I believe never larger than one block, since the next block could
very well be on a different place on the disk. If you have 4k block
size, that would be about 21 TS packets. If it has some spare time it
could do some read ahead, but that only helps until the amount of
requests starts piling up, making read-ahead on one process/request a
time-burden on the next process/request.

> Are you using our index file format, or a different one that you created?

We use our own that we made a about year/year and a half ago. Would
like to use yours though, as we would not need to maintain our code
anymore, but I have not had time to look at it yet.

> That's true, but in an Internet environment (as opposed to, say, a
> satellite network, where jitter can presumably be tightly controlled)
> you can't avoid nontrivial network jitter.  The *only* way to
> overcome this is to add buffering at the client, between the network
> interface and the decoder.  If a client is too 'cheap' to have enough
> buffering to overcome typical Internet network jitter, then it's not
> going to work (well) on the Internet, no matter how precisely the
> Transport Packets are transmitted from the source.

Yes, you are quite right. Satellite networks are easier in that
aspect.  SDH and and even to some extent ATM networks are also able to
do this. Ethernet is unfortunately  broken by design on this aspect,
and one do need buffers. One have to weigh the responsiveness against
the network jitter demand. Usually a semi closed network, like most
IPTV "cable TV" providers and hospitality use have quite good control
of its network delay, so having more than a few hundred ms buffer
would be end-user-experience degrading. Having that much jitter added
by the server would probably not be the best thing that happened then.

> Ultimately, I can envision there being three Transport Stream 'framer' classes:
> 1/ "MPEG2TransportStreamDiscreteFramer", as described above, for live
> streams that deliver discrete Transport Stream packets at precise
> times.
> 2/ "MPEG2TransportStreamFramerFromIndexFile", for use with
> prerecorded Transport Stream files that have a corresponding index
> file.
> 3/ The current "MPEG2TransportStreamFramer", as a fallback for everything else.

That could be quite nice actually. Since MPEG4 transmitted in
terrestrial and satellite networks also use the same transportstream
layer, it would work with those as well. That sounds like a good plan.
If you indeed do implement this, remember to do time adjusting also in
the index-file version, as the double precision is not good enough.
Gave us problems every half hour or so.

If I could interest you in another very nice feature, I have a
suggestion for the trick play functionality. I see that you use I
frames now. What we have done is to use GOPs. We play a full GOP, then
jump over "scale" amounts of GOPs to play the next one. We found this
to be a much sweeter way for fast forwarding as you don't just get
random pictures but half a second of movie between each jump. This
makes it much easier to see exactly where in the stream you are. For
rewind i think the I-frame solution works best. You will need to edit
the MPEG stream to enable this GOP streaming, though, setting the
broken_GOP flag in the PES header while fast forwarding, as well as
index on sequence header packets as well as the actual I frame
(usually the same or very close I guess).


Best regards
-Morgan-