[Live-devel] Zero Latency Live Streaming

[Warren Young]

On Dec 28, 2018, at 10:31 AM, Ross Finlayson <finlayson at live555.com> wrote:
> 
> 1/ In your H.264 encoder - i.e., between the creation of a raw video frame, and the time that it is encoded into (one or more) H.264 NAL units.

Ross didn’t go into any detail on this point, Kevin, so I thought I’d weigh in on it. 

I think this is the largest contributor to your stream’s latency.  The simple fact is that H.264 is not designed as a low-latency codec.  It can be *tuned* to be so, but only by throwing away most of its advantages over, say, MJPEG.

Any time you introduce inter-frame compression — meaning that you need more than one frame in the decoder’s buffer RAM to decode at least some frames — you introduce a delay proportional to N * M where N is the inverse of frames per second in milliseconds and M is the maximum number of of frames needed.

For 30 fps video, N is 33 ms.

For H.264, M can vary considerably: https://planetcalc.com/3321/

If your encoder is configured to have a maximum reference frame span of 5 frames, and you’ve configured Live555 to send each frame out in true *streaming* fashion, then you can expect to have roughly 160 ms of latency minimum due to this factor alone.  That’s human-scale latency!

If you need to use H.264 and must achieve low latency, you’ll have to modify your encoder to use a very conservative GOP structure.  Best to use no B frames at all.

You can make an argument for no P frames, either.  That is, I-frame-only video, like MJPEG, but with about 20 years of advancement on still picture coding, making it about 2x more efficient.

Bottom line, TANSTAAFL.  The high compression rate of an H.264 encoder tuned for low-bandwidth web streaming comes at a price, one facet of which is high latency.

If you want an extreme counterexample for comparison, consider HD-SDI, which requires 1.485 Gbit/sec to send 1080p60 video, but it does so with as close to zero latency as is practical:

    https://en.wikipedia.org/wiki/Serial_digital_interface#Bit_rates

You don’t get down into the single digit Mbit/sec range for the same frame size and frame rate without paying certain costs.