[Live-devel] How to send extra data linked to frames of a RTSP stream?

[Ross Finlayson]

> On May 12, 2020, at 9:32 AM, Matthew Czarnek <mczarnek at edge360.com> wrote:
> 
> I would like to send some small pieces of data that is associated with a RTSP packet alongside the packet within the stream.
> 
> I see that RTSP supports subtitles, is it possible to use subtitles to send information associated with a frame to the client? I'm seeing a few 15 year old posts about live555 not supporting subtitles but not seeing any that are up to date.. which I assume means still not supported.
> 
> Is there some other way to do it within the RTSP stream itself? Maybe if I send a custom packets? Or does it have to be sent separately via something like sockets, and linked to the stream via timestamps or something like that?

Remember that you’re not just sending data.  Someone else is also receiving it.  And, in particular, you want a standards-compliant RTSP/RTP client to have a chance to understand this data.  Therefore, you should use an IETF standard RTP payload format, rather than some format that you’ve conjured up out of thin air.

If you really want to send this data in a way that a standard RTSP client has a chance to handle, then I suggest adding it as a real-time text stream (that would be sent as its own stream, alongside the existing video stream (and audio stream, if any).  Specifically:
	- In your server, add a separate, new “ServerMediaSubsession” object.
		- The “ServerMediaSubsession” object’s “createNewStreamSource()” virtual function would create an instance of a new “FramedSource” subclass (that you would write)
			- Your new “FramedSource” subclass would implement “doGetNextFrame()” by converting the “small piece of data” into ASCII (or at least UTF-8), and delivering the data as a string.  The “fPresentationTime” in each case should be the same as that of the corresponding video frame.
		- The “ServerMediaSubsession” object’s “createNewRTPSink()” virtual function would create an instance of “T140TextRTPSink”.


Ross Finlayson
Live Networks, Inc.
http://www.live555.com/