<div dir="ltr">Hi Ross,<div><br></div><div>After checking all my code carefully, it turned out to be my fault, I create rtcpGroupAudiosock using the rtp port for audio instead of rtcp port, which fix the problem, and both audio and video can stream together, and sync well, but one problem still there when using VLC, two time stamp still try to display alternately at the time control bar( it seem like audio and video try to dominate the time stamp ), and after playing for a while, around 5 mins, both audio and video have 50/50 percentage missing, leaving only audio or video, I know that maybe the problem of VLC, it there any possible that something is wrong when the Server was created. Thank you.</div><div><br></div><div>The SDP file when using openRTSP.</div>
<div>--------------------------</div><div><div>v=0</div><div>o=- 1259782324707606 1 IN IP4 10.0.4.1</div><div>s=Streaming</div><div>i=H264/AAC</div><div>t=0 0</div><div>a=tool:LIVE555 Streaming Media v2015.03.19</div><div>a=type:broadcast</div><div>a=control:*</div><div>a=source-filter: incl IN IP4 * 10.0.4.1</div><div>a=rtcp-unicast: reflection</div><div>a=range:npt=0-</div><div>a=x-qt-text-nam:PointMaker Streaming</div><div>a=x-qt-text-inf:H264/AAC</div><div>m=video 18888 RTP/AVP 96</div><div>c=IN IP4 <a href="http://232.182.197.175/255" target="_blank">232.182.197.175/255</a></div><div>b=AS:764</div><div>a=rtpmap:96 H264/90000</div><div>a=fmtp:96 packetization-mode=1;profile-level-id=64002A;sprop-parameter-sets=J2QAKqwbKgKA9oCg,KP4Briw=</div><div>a=control:track1</div><div>m=audio 18890 RTP/AVP 97</div><div>c=IN IP4 <a href="http://232.182.197.175/255" target="_blank">232.182.197.175/255</a></div><div>b=AS:100</div><div>a=rtpmap:97 MPEG4-GENERIC/48000/2</div><div>a=fmtp:97 streamtype=5;profile-level-id=1;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3;config=1190</div><div>a=control:track2</div></div><div>---------------------------</div><div><br></div><div><br></div><div>main code of the RTSPServer</div><div><br></div><div><br></div><div>
<p><span>TaskScheduler</span><span>* </span><span>scheduler</span><span> = </span><span>BasicTaskScheduler</span><span>::createNew();</span></p><p><span>
</span></p><p><span>env = </span><span>BasicUsageEnvironment</span><span>::createNew(*scheduler);</span></p><p><span>
</span></p><p><span>struct</span><span> </span><span>in_addr</span><span> </span><span>destinationAddress</span><span>;</span></p></div><div>
<p><span>destinationAddress.s_addr = chooseRandomIPv4SSMAddress(*env);</span></p></div><div>
<p><span>unsigned</span><span> </span><span>short</span><span> </span><span>rtpPortNum</span><span> = 18888;</span><span> </span></p>
<p><span>unsigned</span><span> </span><span>short</span><span> </span><span>rtcpPortNum</span><span> = rtpPortNum+1;</span></p>
<p><span>unsigned</span><span> </span><span>char</span><span> </span><span>ttl</span><span> = 255;</span></p>
<p><span> </span><span>/*For AAC audio RTP seesion */</span></p>
<p><span>unsigned</span><span> </span><span>short</span><span> </span><span>rtpPortAudioNum</span><span> = 18890;</span></p>
<p><span>unsigned</span><span> </span><span>short</span><span> </span><span>rtcpPortAudioNum</span><span> = rtpPortAudioNum + 1;</span></p><p><span>Port</span><span> </span><span>rtpPort</span><span>(rtpPortNum);</span></p><p><span>
</span></p><p><span>Port</span><span> </span><span>rtcpPort</span><span>(rtcpPortNum);</span></p></div><div>
<p><span>Port</span><span> </span><span>rtpAudioPort</span><span>(rtpPortAudioNum);</span></p>
<p><span>Port</span><span> </span><span>rtcpAudioPort</span><span>(rtcpPortAudioNum);</span></p><p><span>Groupsock</span><span> </span><span>rtpGroupsock</span><span>(*env, destinationAddress, rtpPort, ttl);</span></p>
<p><span>Groupsock</span><span> </span><span>rtcpGroupsock</span><span>(*env, destinationAddress, rtcpPort, ttl);</span></p>
<p><span>Groupsock</span><span> </span><span>rtpGroupAudiosock</span><span>(*env, destinationAddress, rtpAudioPort, ttl);</span></p>
<p><span>Groupsock</span><span> </span><span>rtcpGroupAudiosock</span><span>(*env, destinationAddress, rtpAudioPort, ttl);</span></p><p><span>rtpGroupsock.multicastSendOnly(); </span><span>// we're a SSM source </span></p><p><span>rtcpGroupsock.multicastSendOnly(); </span><span>// we're a SSM source </span></p><p><span>rtpGroupAudiosock.multicastSendOnly(); </span><span>// we're a SSM source </span></p><p><span>rtcpGroupAudiosock.multicastSendOnly(); </span><span>// we're a SSM source</span></p><p><span>RTPSink* videoSink;</span></p><p>RTPSink* audioSink;<span><br></span></p><p>RTSPServer* rtspServer;</p><p><span>videoSink = </span><span>H264VideoRTPSink</span><span>::createNew(*env, &rtpGroupsock, 96);</span><br></p><p><span>audioSink = </span><span>MPEG4GenericRTPSink</span><span>::createNew( *env, &rtpGroupAudiosock</span>,97<span>, 48000, </span><span>"audio"</span><span>, </span><span>"AAC-hbr"</span><span>, </span><span>"1190"</span><span>, 2);</span></p><p><span>const</span><span> </span><span>unsigned</span><span> </span><span>estimatedSessionBandwidth</span><span> = 500; </span><span>//in kbps; for RTCP b/w share </span><br></p><p>
</p><p><span>const</span><span> </span><span>unsigned</span><span> </span><span>estimatedSessionBandwidthAudio</span><span> = 64;</span></p><p><span>rtcp = </span><span>RTCPInstance</span><span>::createNew(*env, &rtcpGroupsock, </span>estimatedSessionBandwidth, CNAME, <span>videoSink, </span><span>NULL</span><span> </span><span>/* we're a server */</span><span>, </span><span>isSSM </span><span>/* we're a SSM source */</span><span>);</span></p><p><span>rtcpAudio = </span><span>RTCPInstance</span><span>::createNew(*env, &rtcpGroupAudiosock, </span>estimatedSessionBandwidthAudio, CNAME, <span>audioSink, </span><span>NULL</span><span>, </span>isSSM);</p><p>
</p><p><span>rtspServer = </span><span>RTSPServer</span><span>::createNew(*env, 8554, NULL);</span></p><p><span>ServerMediaSession</span><span>* </span><span>sms </span><span>= </span><span>ServerMediaSession</span><span>::createNew(*env, </span><span>"testStream"</span><span>, </span><span>"H264/AAC"</span><span>, </span><span>"Test Streaming"</span><span>, </span><span> isSSM</span><span>/*SSM*/</span><span>);</span></p><p><span>sms->addSubsession(</span><span>PassiveServerMediaSubsession</span><span>::createNew(*videoSink, rtcp)); </span><span>//rtcp video multicast</span></p><p><span>
</span></p><p><span>sms->addSubsession(</span><span>PassiveServerMediaSubsession</span><span>::createNew(*audioSink, rtcpAudio));</span><span>//rtcpAudio audio multicast</span></p><p><span>
</span></p><p><span>rtspServer->addServerMediaSession(sms);</span></p><p><span>
</span></p><p><span>devSource = </span><span>DeviceSource</span><span>::createNew( *env );</span></p><p><span>
</span></p><p><span>aacSource = </span><span>AACFramedSource</span><span>::createNew( *env );</span></p><p><span>videoES = devSource;</span></p><p><span>
</span></p><p><span>audioES = aacSource;</span></p><p><span>
</span></p><p><span>videoSource = </span><span>H264VideoStreamFramer</span><span>::createNew( *env, videoES, True );</span></p><p><span>videoSink->startPlaying(*videoSource, </span><span>NULL</span><span>, videoSink);</span></p><p><span>
</span></p><p><span>audioSink->startPlaying(*audioES, </span><span>NULL</span><span>, audioSink);</span></p><p><span>
</span></p><p><span>env->taskScheduler().doEventLoop( &stopEventLoop );</span></p></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 6, 2015 at 1:08 PM, Ross Finlayson <span dir="ltr"><<a href="mailto:finlayson@live555.com" target="_blank">finlayson@live555.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><span class=""><blockquote type="cite"><div><div dir="ltr"><div><div>Right now, I have problem make them stream out together, it seems audio will hang the video immediately when connect the RTSP server using VLC</div></div></div></div></blockquote><div><br></div></span>I suggest that you first use our “openRTSP” RTSP client application <<a href="http://www.live555.com/openRTSP/" target="_blank">http://www.live555.com/openRTSP/</a>> to test receiving both audio and video from your server. Using “openRTSP”, you should see (nonempty) files for both the audio and video stream. (“openRTSP” is a much simpler client application than VLC, because it doesn’t render the incoming media, and doesn’t care whether their presentation times are correct.)</div><div><br></div><div><span class=""><br><blockquote type="cite"><div dir="ltr"><div><div>I doubt the problem that the time stamp for video and audio are not correct, since during playing using VLC, I notice that two different time are shown alternately, for the video, I set the "fPresentationTime" as below: since the frame rate of my H264 video is 30fps, the time for the first video frame is based on the "gettimeofday", then increased based on frame rate </div><div><br></div><div>----------</div><div><p><span> </span><span>if</span><span>( encoder_video_ts.tv_sec == 0 && encoder_video_ts.tv_usec == 0 )</span></p></div></div></div></blockquote></span><div>Are you certain that this branch is actually getting executed - i.e., that your “encoder_video_ts” variable (both the “tv_sec” and “tv_usec” fields) is actually initialized to 0?</div><span class=""><br><blockquote type="cite"><div dir="ltr"><div><div><p><span> gettimeofday( &encoder_video_ts, </span><span>NULL</span><span> ); </span><span>//Set the first stream time stamp from wall clock </span></p><p><span> </span><span>else</span><span> {</span></p><p><span> </span><span>unsigned</span><span> </span><span>uSeconds</span><span> = encoder_video_ts.tv_usec + 1000000/30; </span><span>//Framerate is 30fps</span></p></div></div></div></blockquote></span>This is correct *only if* each NAL unit that you deliver is a complete ‘access unit’ - i.e., picture. If, instead, your stream includes PPS and SPS NAL units, and/or ‘slice’ NAL units, then this calculation won’t be correct.</div><div><br></div><div>In any case, I suggest simplifying your code by just setting “fPresentationTime” for *each* of your video NAL units by calling “gettimeofday()” - as you already do for the audio frames.</div><div><span class=""><br><blockquote type="cite"><div dir="ltr"><div><div><p></p></div><div>since audio and video are feeding using two separated thread, is this the correct way to sync the A/V ?</div></div></div></blockquote><div><br></div></span>Yes, if the “fPresentationTime” values are accurate, *and* aligned with ‘wall clock’ time (i.e., times that would be generated by “gettimeofday()”), then audio and video will get properly synchronized by standards-conforming RTP/RTCP clients (like VLC).</div><span class="HOEnZb"><font color="#888888"><br><div>
<span style="border-collapse:separate;color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="border-collapse:separate;color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Ross Finlayson<br>Live Networks, Inc.<br><a href="http://www.live555.com/" target="_blank">http://www.live555.com/</a></span></span>
</div><div><br></div>
<br></font></span></div><br>_______________________________________________<br>
live-devel mailing list<br>
<a href="mailto:live-devel@lists.live555.com">live-devel@lists.live555.com</a><br>
<a href="http://lists.live555.com/mailman/listinfo/live-devel" target="_blank">http://lists.live555.com/mailman/listinfo/live-devel</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div>Xingjun Chen<br>M.S. Degree <span><em>Earned</em></span></div>
<div>Electrical and Computer Engineering<br>University of Arizona, Tucson AZ 85721</div></div>
</div>