[Live-devel] Live555 performance configuration

Sat Feb 7 14:00:07 PST 2009

>> 1. best speed of disks 7200 rpm.
>
> Better 15k SAS/scsi, to serve a lot of stream the seek time is really an
> important feature.

Not necessarily.

> Load it with RAM too, which is cheap nowadays and serve
> as "seek buffer" very well (on linux systems).

This is of course true. As much as the motherboard can handle.

>> 2. how many disk and type of confirue raid 1
>
> I would use raid 5 for a vod if you have a lot of content, it is cheaper at
> $/GB.

This is not a very good idea though. What you achieve do by doing
this, is that you make sure that all disks move their heads for every
read of the disk. Given 100 different open files, and an average disk
access time of 10ms, each process will be able to read exactly once
per second if in a RAID5/6. Even though you get a very high read
speed, it is only for a very short period, and not very often. If you
only read a couple of TS packets each time, you are in a shitload of
trouble right there. More explanation further down.

>> I am looking to service around 200 amino set top box request with a
>> possible 100 vod movies downloaded at any one time.
>
> 100 movies at which rate?
>
> I use live555 on Debian/Linux and I assure it can serve that load with above
> mentioned configuration with 4.5 Mbps mpeg2 streams without problems.

100 movies at 4,5Mbit is more that 50MB/s. On real random access,
there is no harddrive in the world that will be able to give you that
throughput, no matter how big a RAID5 system you have. If you have
lots of cache hits, the maybe, but on many large files that is very
unlikely. Raid does not help you here at all.

The problem is simple. Random access will cause the disks to spend
most of their time moving the read head around. Have a look at
storagereview.com and their random access tests. 100MB/s sustained on
sequential data, but <2MB/s on random access. Say your RAID has 1GB/s
transfer rate, it will still not be able to overcome the issue of the
>8 ms it takes moving the read disk read heads around.

> I cannot spell "windows" and "server" in the same sentence, I'm sorry :-)

Your main problem with this is loss of control actually, but putting
that aside for now.

What you need to do to be able to "survive" a huge amount of
simultaneous clients is to do the RAID thing manually. Do not set up a
RAID, because that causes the disks to read and move the disk heads at
the same time. If you have two disk with the same content on, but not
in RAID, then each disk could be used better. Making the VOD server
read ahead a lot (like 1 MB), would make a 4Mbit/s stream read every 2
seconds, but for longer periods. The disk would actually spend only
20ms every 2 seconds for this (given 10ms seek time and 100MB/s
sustained data rate). If reads don't crash, this would be able to give
you 100 streams off one single disk. This gives a higher read duty
cycle, giving you a lot higher throughput as you see. Also, doing this
on two separate instead of one raid1 (which would give you 15ms read
time per 2 seconds per process, and a maximum of 133 stream totally)
you get a >50% higher throughput using two separate disks.

In essence, forget RAID, raid is for high throughput sequential read.
Make a manual mirror, and utilize the disks well by reading ahead a
lot. Avoid a lot of threads reading a little data each time from the
same disk. If you are really serious about getting high throughput,
abstract the disk access away to a separate service who's only job is
to make sure that all reads and writes from the disk are serialized,
and make all data go trough this. Now the reads will never crash, and
the theoretical 100 clients per hard-drive is not sot theoretical any
more.

The calculations are easy. A disk with 100MB/s read speed and 10ms
access time will give you 1MB per 10ms read time. If you read 1MB at a
time on random spots on the disk, you will use 10ms per read, and 10ms
on average for moving to the next spot on the disk. Effectively giving
you a maximum 50MB/s because half of your read time goes to moving the
disk read heads. If you only read 10KB per read, the read will take
only 0.1ms, but the head will still use 10ms to move around. Theory
will then give you 0.1ms read time every 10.1ms, giving you somewhere
around 1MB/s throughput. Cache and block sizes might improve this
somewhat, but you get the idea. Read large chucks.

Now for a _very_ important lesson if you need disk access speed.
7200rpm disks can have faster access time than 10k disks. When moving
the head, the disk also needs to rotate to the correct position for a
read to start. On average 0.5 rotations, which is 4.2ms on a 7200rpm
disk and 3ms on a 10k disk. A regular 7200rpm disk has an average
access time of 13ms or so, while a 10k disk has 8ms. This makes the
7200rpm disk use 9ms moving the head, and the 10k 5ms. Now, given an
expensive 147GB 10k disk, and a 1TB 7200rpm disk, if you make a
partition on the 1TB disk of 147GB, and only use that part of the
disk, the head will only need to move 10% of what it would using the
entire disk ( there is more data per rotation on the outer parts of
the disk platters ). This could reduce the 9ms to about 1ms. 4.2ms +
1ms = 5.2ms, which is _faster_ than the 10k disk, and really more in
the 15k realm, and you have 800GB to spare. Test it if you don't
believe it. This can also give you 4 times as many disks for the same
price possibly, which will make absolutely sure that the total
throughput from these 4 cheap disks will surpass the 15k disks by a
mile.

So, go for huge SATA disks before expensive 10-15k disks, have lots of
them ( 1/4 of the price hints to 4 times as many :-p which alone will
heavily increase the throuhtput ), use as small a main partition as
you can, and make sure you do not use RAID but spread movies on
several pairs of manually mirrored disks to increase the read duty
cycle. Also make sure to read a lot at a time and do it less
frequently.

I might of course be totally wrong on all this, but the tests I have
done seems to agree. Hopefully I got it all right in my explanation
here. Obviously, doing this will demand more from the developers, but
it might very well be worth it.

Best regards

-Morgan-

> --
> ESC:wq
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
> _______________________________________________
> live-devel mailing list
> live-devel at lists.live555.com
> http://lists.live555.com/mailman/listinfo/live-devel
>