[Live-devel] Read ahead on Linux

[Morgan Tørvolt]

> > OK, I guess I was assuming that the OS's file system implementation
> > is smart enough to detect sequential reads within a file (even  if
> > they're interleaved with sequential reads within other files), and
> > optimize this by doing read-ahead - for each file - into its file
> > system cache.  However, I'm not sure in practice how smart OSs are
> > about this, but if the OS kernel *could* do this automatically, that
> > would certainly be better than having the application implementing
> > its own read-ahead in user space.  In any case, file systems
> > typically read from the disk in large block sizes (certainly much
> > larger than 188 bytes), so you get some 'read-ahead' for free there.
> > (OTOH, we had a major VOD customer that had no choice but to use
> > Windows, and they ended up deciding to do their own
> > read-ahead/buffering of Transport Stream file data in user space
> > (replacing "ByteStreamFileSource" and "MPEG2TransportStreamFramer").
> > But perhaps other OSs aren't as brain-damaged as Windows about
> > this...)
>
> We run Linux, and we have seen some significant improvements by
> reading larger chuncks, but if they get too large, we got PCR jitter
> problems due to the "slow read" of the harddrives. It seemed like when
> three or more clients needed read-time after one-another (happened
> randomly from time to time), the time it took to finish those three
> reads made the following packets delayed. This was with very large
> buffers though. With other clients (like VLC) one could probably
> increase the buffer size even more than we do.
> A disk with 60MB/s throughput can give you 600kB in (avg-seek-time +
> 600k/60M) = 8ms + 10ms = 18ms. 60kB would be achieved in 9ms. Half the
> time, one tenth of the data.

Just following up on this issue. Rumors say that the next kernel
release will incorporate adaptive read-ahead. Here is some old reading
material I dug, although I am not sure that it is exactly this that is
being included in the upcoming kernel release:
http://kerneltrap.org/node/6642

It seems like our fixes that does read-ahead in the application will
now become obsolete. For releases like redhat and others, it will
probably take some time before the update arrives.

When this is up and running, it will make all our lives much easier =)

-Morgan-