[Live-devel] H264 video conflict with AAC audio using same ServerMediaSession

Thu May 7 08:30:35 PDT 2015

Hi Ross,

After checking all my code carefully, it turned out to be my fault, I
create rtcpGroupAudiosock using the rtp port for audio instead of rtcp
port, which fix the problem, and both audio and video can stream together,
and sync well, but one problem still there when using VLC, two time stamp
still try to display alternately at the time control bar( it seem like
audio and video try to dominate the time stamp ), and after playing for a
while, around 5 mins, both audio and video have 50/50 percentage missing,
leaving only audio or video, I know that maybe the problem of VLC, it there
any possible that something is wrong when the Server was created. Thank you.

The SDP file when using openRTSP.
--------------------------
v=0
o=- 1259782324707606 1 IN IP4 10.0.4.1
s=Streaming
i=H264/AAC
t=0 0
a=tool:LIVE555 Streaming Media v2015.03.19
a=type:broadcast
a=control:*
a=source-filter: incl IN IP4 * 10.0.4.1
a=rtcp-unicast: reflection
a=range:npt=0-
a=x-qt-text-nam:PointMaker Streaming
a=x-qt-text-inf:H264/AAC
m=video 18888 RTP/AVP 96
c=IN IP4 232.182.197.175/255
b=AS:764
a=rtpmap:96 H264/90000
a=fmtp:96
packetization-mode=1;profile-level-id=64002A;sprop-parameter-sets=J2QAKqwbKgKA9oCg,KP4Briw=
a=control:track1
m=audio 18890 RTP/AVP 97
c=IN IP4 232.182.197.175/255
b=AS:100
a=rtpmap:97 MPEG4-GENERIC/48000/2
a=fmtp:97
streamtype=5;profile-level-id=1;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3;config=1190
a=control:track2
---------------------------

main code of the RTSPServer

TaskScheduler* scheduler = BasicTaskScheduler::createNew();

env = BasicUsageEnvironment::createNew(*scheduler);

struct in_addr destinationAddress;

destinationAddress.s_addr = chooseRandomIPv4SSMAddress(*env);

unsigned short rtpPortNum = 18888;

unsigned short rtcpPortNum = rtpPortNum+1;

unsigned char ttl = 255;

  /*For AAC audio RTP seesion */

unsigned short rtpPortAudioNum = 18890;

unsigned short rtcpPortAudioNum = rtpPortAudioNum + 1;

Port rtpPort(rtpPortNum);

Port rtcpPort(rtcpPortNum);

Port rtpAudioPort(rtpPortAudioNum);

Port rtcpAudioPort(rtcpPortAudioNum);

Groupsock rtpGroupsock(*env, destinationAddress, rtpPort, ttl);

Groupsock rtcpGroupsock(*env, destinationAddress, rtcpPort, ttl);

Groupsock rtpGroupAudiosock(*env, destinationAddress, rtpAudioPort, ttl);

Groupsock rtcpGroupAudiosock(*env, destinationAddress, rtpAudioPort, ttl);

rtpGroupsock.multicastSendOnly(); // we're a SSM source

rtcpGroupsock.multicastSendOnly(); // we're a SSM source

rtpGroupAudiosock.multicastSendOnly(); // we're a SSM source

rtcpGroupAudiosock.multicastSendOnly(); // we're a SSM source

RTPSink* videoSink;

RTPSink* audioSink;

RTSPServer* rtspServer;

videoSink = H264VideoRTPSink::createNew(*env, &rtpGroupsock, 96);

audioSink = MPEG4GenericRTPSink::createNew( *env, &rtpGroupAudiosock,97,
48000, "audio", "AAC-hbr", "1190", 2);

const unsigned estimatedSessionBandwidth = 500; //in kbps; for RTCP b/w
share

const unsigned estimatedSessionBandwidthAudio = 64;

rtcp = RTCPInstance::createNew(*env, &rtcpGroupsock, estimatedSessionBandwidth,
CNAME, videoSink, NULL /* we're a server */, isSSM /* we're a SSM source */
);

rtcpAudio = RTCPInstance::createNew(*env, &rtcpGroupAudiosock,
estimatedSessionBandwidthAudio,
CNAME, audioSink, NULL, isSSM);

rtspServer = RTSPServer::createNew(*env, 8554, NULL);

ServerMediaSession* sms = ServerMediaSession::createNew(*env, "testStream",
"H264/AAC", "Test Streaming",  isSSM/*SSM*/);

sms->addSubsession(PassiveServerMediaSubsession::createNew(*videoSink,
rtcp)); //rtcp video multicast

sms->addSubsession(PassiveServerMediaSubsession::createNew(*audioSink,
rtcpAudio));//rtcpAudio audio multicast

rtspServer->addServerMediaSession(sms);

devSource = DeviceSource::createNew( *env );

aacSource = AACFramedSource::createNew( *env );

videoES = devSource;

audioES = aacSource;

videoSource = H264VideoStreamFramer::createNew( *env, videoES, True );

videoSink->startPlaying(*videoSource, NULL, videoSink);

audioSink->startPlaying(*audioES, NULL, audioSink);

env->taskScheduler().doEventLoop( &stopEventLoop );

On Wed, May 6, 2015 at 1:08 PM, Ross Finlayson <finlayson at live555.com>
wrote:

> Right now, I have problem make them stream out together, it seems audio
> will hang the video immediately when connect the RTSP server using VLC
>
>
> I suggest that you first use our “openRTSP” RTSP client application <
> http://www.live555.com/openRTSP/> to test receiving both audio and video
> from your server.  Using “openRTSP”, you should see (nonempty) files for
> both the audio and video stream.  (“openRTSP” is a much simpler client
> application than VLC, because it doesn’t render the incoming media, and
> doesn’t care whether their presentation times are correct.)
>
>
> I doubt the problem that the time stamp for video and audio are not
> correct, since during playing using VLC, I notice that two different time
> are shown alternately, for the video, I set the "fPresentationTime" as
> below: since the frame rate of my H264 video is 30fps, the time for the
> first video frame is based on the "gettimeofday", then increased based on
> frame rate
>
> ----------
>
>   if( encoder_video_ts.tv_sec == 0 && encoder_video_ts.tv_usec == 0 )
>
> Are you certain that this branch is actually getting executed - i.e., that
> your “encoder_video_ts” variable (both the “tv_sec” and “tv_usec” fields)
> is actually initialized to 0?
>
>     gettimeofday( &encoder_video_ts, NULL ); //Set the first stream time
> stamp from wall clock
>
>   else {
>
>     unsigned uSeconds = encoder_video_ts.tv_usec + 1000000/30; //Framerate
> is 30fps
>
> This is correct *only if* each NAL unit that you deliver is a complete
> ‘access unit’ - i.e., picture.  If, instead, your stream includes PPS and
> SPS NAL units, and/or ‘slice’ NAL units, then this calculation won’t be
> correct.
>
> In any case, I suggest simplifying your code by just setting
> “fPresentationTime” for *each* of your video NAL units by calling
> “gettimeofday()” - as you already do for the audio frames.
>
> since audio and video are feeding using two separated thread, is this the
> correct way to sync the A/V ?
>
>
> Yes, if the “fPresentationTime” values are accurate, *and* aligned with
> ‘wall clock’ time (i.e., times that would be generated by
> “gettimeofday()”), then audio and video will get properly synchronized by
> standards-conforming RTP/RTCP clients (like VLC).
>
> Ross Finlayson
> Live Networks, Inc.
> http://www.live555.com/
>
>
>
> _______________________________________________
> live-devel mailing list
> live-devel at lists.live555.com
> http://lists.live555.com/mailman/listinfo/live-devel
>
>

-- 
Xingjun Chen
M.S. Degree *Earned*
Electrical and Computer Engineering
University of Arizona, Tucson AZ 85721
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.live555.com/pipermail/live-devel/attachments/20150507/02a58346/attachment.html>