[Live-devel] H264 video conflict with AAC audio using same ServerMediaSession

Thu May 7 10:03:40 PDT 2015

Hi, Ross,

And here are the TimeStamp I printed out to screen, and it seems that video
is always ahead for around 20 frames before audio begin, so is there any
problem on this time stamp? Thank you

-----------------------------------------------------
Timestamp(video): 1259780427.812096
Timestamp(video): 1259780427.825078
Timestamp(video): 1259780427.849170
Timestamp(video): 1259780427.890521
Timestamp(video): 1259780427.928453
Timestamp(video): 1259780427.968674
Timestamp(video): 1259780428.8373
Timestamp(video): 1259780428.48704
Timestamp(video): 1259780428.98299
Timestamp(video): 1259780428.135787
Timestamp(video): 1259780428.168545
Timestamp(video): 1259780428.208479
Timestamp(video): 1259780428.248502
Timestamp(video): 1259780428.331133
Timestamp(video): 1259780428.344304
Timestamp(video): 1259780428.385989
Timestamp(video): 1259780428.410665
Timestamp(video): 1259780428.448498
Timestamp(video): 1259780428.488449
Timestamp(audio): 1259780428.495363
Timestamp(audio): 1259780428.521098
Timestamp(video): 1259780428.531169
Timestamp(video): 1259780428.569797
Timestamp(audio): 1259780428.575766
Timestamp(audio): 1259780428.582072
Timestamp(audio): 1259780428.610186
Timestamp(video): 1259780428.626195
Timestamp(audio): 1259780428.637294
Timestamp(audio): 1259780428.650332
Timestamp(video): 1259780428.656169
Timestamp(audio): 1259780428.672040
Timestamp(video): 1259780428.688736
Timestamp(audio): 1259780428.691176
Timestamp(audio): 1259780428.700389
Timestamp(audio): 1259780428.717058
Timestamp(video): 1259780428.728756
Timestamp(audio): 1259780428.756663
Timestamp(audio): 1259780428.762358
Timestamp(video): 1259780428.768766
Timestamp(audio): 1259780428.789704
Timestamp(audio): 1259780428.798148
Timestamp(video): 1259780428.808646
Timestamp(audio): 1259780428.816236
Timestamp(audio): 1259780428.821639
Timestamp(audio): 1259780428.838771
Timestamp(video): 1259780428.848818
Timestamp(audio): 1259780428.867066
Timestamp(audio): 1259780428.876592
Timestamp(audio): 1259780428.890574
Timestamp(video): 1259780428.892247
Timestamp(audio): 1259780428.912069
Timestamp(audio): 1259780428.921644
Timestamp(audio): 1259780428.932578
Timestamp(video): 1259780428.933324
Timestamp(audio): 1259780428.952768
Timestamp(audio): 1259780428.962698
Timestamp(audio): 1259780428.973102
Timestamp(video): 1259780428.973788
Timestamp(audio): 1259780428.989784
Timestamp(audio): 1259780429.5626
Timestamp(video): 1259780429.8911
Timestamp(audio): 1259780429.26718
Timestamp(audio): 1259780429.35422
Timestamp(video): 1259780429.48761
Timestamp(audio): 1259780429.60430
Timestamp(audio): 1259780429.70219
Timestamp(audio): 1259780429.75564
Timestamp(video): 1259780429.98452
Timestamp(audio): 1259780429.104823
Timestamp(video): 1259780429.128612
Timestamp(audio): 1259780429.131253
Timestamp(audio): 1259780429.152763
Timestamp(video): 1259780429.168746
Timestamp(audio): 1259780429.181418
Timestamp(audio): 1259780429.198946
Timestamp(video): 1259780429.208612
Timestamp(audio): 1259780429.209002
Timestamp(audio): 1259780429.227990
Timestamp(audio): 1259780429.232854
Timestamp(video): 1259780429.248753
Timestamp(audio): 1259780429.256110
Timestamp(audio): 1259780429.261718
Timestamp(audio): 1259780429.282645
Timestamp(video): 1259780429.288910
Timestamp(audio): 1259780429.309122
Timestamp(video): 1259780429.328835
Timestamp(audio): 1259780429.332635
Timestamp(audio): 1259780429.361411
Timestamp(audio): 1259780429.370254
Timestamp(video): 1259780429.397703
Timestamp(audio): 1259780429.402252
Timestamp(video): 1259780429.408766
Timestamp(video): 1259780429.448676
Timestamp(audio): 1259780429.468867
Timestamp(video): 1259780429.496035
Timestamp(audio): 1259780429.499250
Timestamp(audio): 1259780429.509263
Timestamp(audio): 1259780429.519708
Timestamp(video): 1259780429.528744
Timestamp(audio): 1259780429.537210
Timestamp(audio): 1259780429.542289
Timestamp(video): 1259780429.568949
Timestamp(audio): 1259780429.577568
Timestamp(audio): 1259780429.596234
Timestamp(audio): 1259780429.602707
Timestamp(video): 1259780429.608950
Timestamp(audio): 1259780429.621322
Timestamp(audio): 1259780429.641266
Timestamp(video): 1259780429.648665
Timestamp(audio): 1259780429.659844
Timestamp(audio): 1259780429.669236
Timestamp(video): 1259780429.688742
Timestamp(audio): 1259780429.706423
Timestamp(audio): 1259780429.715664
Timestamp(audio): 1259780429.737233
Timestamp(video): 1259780429.738925
Timestamp(audio): 1259780429.762702
Timestamp(video): 1259780429.768581
Timestamp(audio): 1259780429.782619
Timestamp(video): 1259780429.808556
Timestamp(audio): 1259780429.810332
Timestamp(audio): 1259780429.829258
Timestamp(video): 1259780429.848669
Timestamp(audio): 1259780429.850344
Timestamp(audio): 1259780429.871560
Timestamp(video): 1259780429.888661
Timestamp(audio): 1259780429.899719
Timestamp(audio): 1259780429.908551
Timestamp(audio): 1259780429.918580
Timestamp(video): 1259780429.928617
Timestamp(video): 1259780429.980931
Timestamp(audio): 1259780429.981592
Timestamp(audio): 1259780429.990447
Timestamp(video): 1259780430.9109
Timestamp(audio): 1259780430.10411
Timestamp(audio): 1259780430.22045
Timestamp(video): 1259780430.48567
Timestamp(audio): 1259780430.52625
Timestamp(audio): 1259780430.62529
Timestamp(audio): 1259780430.75703
Timestamp(video): 1259780430.88803
Timestamp(audio): 1259780430.100140
Timestamp(audio): 1259780430.121773
Timestamp(video): 1259780430.148481
Timestamp(audio): 1259780430.150373
Timestamp(video): 1259780430.168749
Timestamp(audio): 1259780430.181554
Timestamp(audio): 1259780430.192204
Timestamp(video): 1259780430.208985
Timestamp(audio): 1259780430.212909
Timestamp(audio): 1259780430.229808
Timestamp(video): 1259780430.248740
Timestamp(audio): 1259780430.252553
Timestamp(audio): 1259780430.272066
Timestamp(video): 1259780430.288604
Timestamp(audio): 1259780430.299078
Timestamp(audio): 1259780430.319278
Timestamp(video): 1259780430.328566
Timestamp(audio): 1259780430.340239
Timestamp(video): 1259780430.368764
Timestamp(audio): 1259780430.376474
Timestamp(audio): 1259780430.391928
Timestamp(audio): 1259780430.397143
Timestamp(video): 1259780430.408545
Timestamp(audio): 1259780430.420702
Timestamp(video): 1259780430.448765
Timestamp(audio): 1259780430.469398
Timestamp(video): 1259780430.496224
Timestamp(audio): 1259780430.501760
Timestamp(audio): 1259780430.520669
Timestamp(audio): 1259780430.528676
Timestamp(video): 1259780430.529006
Timestamp(audio): 1259780430.560397
Timestamp(audio): 1259780430.570918
Timestamp(video): 1259780430.572975
Timestamp(audio): 1259780430.592193
Timestamp(audio): 1259780430.598936
Timestamp(video): 1259780430.609019
Timestamp(audio): 1259780430.619331
Timestamp(audio): 1259780430.631390
Timestamp(video): 1259780430.648716
Timestamp(audio): 1259780430.652739
Timestamp(audio): 1259780430.678291
Timestamp(video): 1259780430.688827
Timestamp(audio): 1259780430.698216
Timestamp(audio): 1259780430.718915
Timestamp(video): 1259780430.728615
Timestamp(audio): 1259780430.748697
Timestamp(video): 1259780430.769219
Timestamp(audio): 1259780430.776459
Timestamp(audio): 1259780430.782756
Timestamp(video): 1259780430.808762
Timestamp(audio): 1259780430.809276
Timestamp(audio): 1259780430.828512
Timestamp(video): 1259780430.848757
Timestamp(audio): 1259780430.851372
Timestamp(audio): 1259780430.871244
Timestamp(video): 1259780430.888754
Timestamp(audio): 1259780430.889594
Timestamp(audio): 1259780430.911814
Timestamp(video): 1259780430.930554
Timestamp(audio): 1259780430.947126
Timestamp(audio): 1259780430.952891
Timestamp(video): 1259780430.969168
Timestamp(audio): 1259780430.979588
Timestamp(audio): 1259780430.989776
Timestamp(video): 1259780431.9692
Timestamp(audio): 1259780431.26371
Timestamp(audio): 1259780431.32381
Timestamp(video): 1259780431.48556
Timestamp(audio): 1259780431.68791
Timestamp(video): 1259780431.88579
Timestamp(audio): 1259780431.90912
Timestamp(audio): 1259780431.109843
Timestamp(video): 1259780431.128633
Timestamp(audio): 1259780431.129292
Timestamp(audio): 1259780431.150196
                            .
                            .
                            .
                            .
                            .
                            .
                            .
------------------------------------------------------

On Thu, May 7, 2015 at 8:30 AM, Kevin Chen <xjchen at email.arizona.edu> wrote:

> Hi Ross,
>
> After checking all my code carefully, it turned out to be my fault, I
> create rtcpGroupAudiosock using the rtp port for audio instead of rtcp
> port, which fix the problem, and both audio and video can stream together,
> and sync well, but one problem still there when using VLC, two time stamp
> still try to display alternately at the time control bar( it seem like
> audio and video try to dominate the time stamp ), and after playing for a
> while, around 5 mins, both audio and video have 50/50 percentage missing,
> leaving only audio or video, I know that maybe the problem of VLC, it there
> any possible that something is wrong when the Server was created. Thank you.
>
> The SDP file when using openRTSP.
> --------------------------
> v=0
> o=- 1259782324707606 1 IN IP4 10.0.4.1
> s=Streaming
> i=H264/AAC
> t=0 0
> a=tool:LIVE555 Streaming Media v2015.03.19
> a=type:broadcast
> a=control:*
> a=source-filter: incl IN IP4 * 10.0.4.1
> a=rtcp-unicast: reflection
> a=range:npt=0-
> a=x-qt-text-nam:PointMaker Streaming
> a=x-qt-text-inf:H264/AAC
> m=video 18888 RTP/AVP 96
> c=IN IP4 232.182.197.175/255
> b=AS:764
> a=rtpmap:96 H264/90000
> a=fmtp:96
> packetization-mode=1;profile-level-id=64002A;sprop-parameter-sets=J2QAKqwbKgKA9oCg,KP4Briw=
> a=control:track1
> m=audio 18890 RTP/AVP 97
> c=IN IP4 232.182.197.175/255
> b=AS:100
> a=rtpmap:97 MPEG4-GENERIC/48000/2
> a=fmtp:97
> streamtype=5;profile-level-id=1;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3;config=1190
> a=control:track2
> ---------------------------
>
>
> main code of the RTSPServer
>
>
> TaskScheduler* scheduler = BasicTaskScheduler::createNew();
>
> env = BasicUsageEnvironment::createNew(*scheduler);
>
> struct in_addr destinationAddress;
>
> destinationAddress.s_addr = chooseRandomIPv4SSMAddress(*env);
>
> unsigned short rtpPortNum = 18888;
>
> unsigned short rtcpPortNum = rtpPortNum+1;
>
> unsigned char ttl = 255;
>
>   /*For AAC audio RTP seesion */
>
> unsigned short rtpPortAudioNum = 18890;
>
> unsigned short rtcpPortAudioNum = rtpPortAudioNum + 1;
>
> Port rtpPort(rtpPortNum);
>
> Port rtcpPort(rtcpPortNum);
>
> Port rtpAudioPort(rtpPortAudioNum);
>
> Port rtcpAudioPort(rtcpPortAudioNum);
>
> Groupsock rtpGroupsock(*env, destinationAddress, rtpPort, ttl);
>
> Groupsock rtcpGroupsock(*env, destinationAddress, rtcpPort, ttl);
>
> Groupsock rtpGroupAudiosock(*env, destinationAddress, rtpAudioPort, ttl);
>
> Groupsock rtcpGroupAudiosock(*env, destinationAddress, rtpAudioPort, ttl);
>
> rtpGroupsock.multicastSendOnly(); // we're a SSM source
>
>
> rtcpGroupsock.multicastSendOnly(); // we're a SSM source
>
>
> rtpGroupAudiosock.multicastSendOnly(); // we're a SSM source
>
>
> rtcpGroupAudiosock.multicastSendOnly(); // we're a SSM source
>
> RTPSink* videoSink;
>
> RTPSink* audioSink;
>
> RTSPServer* rtspServer;
>
> videoSink = H264VideoRTPSink::createNew(*env, &rtpGroupsock, 96);
>
> audioSink = MPEG4GenericRTPSink::createNew( *env, &rtpGroupAudiosock,97,
> 48000, "audio", "AAC-hbr", "1190", 2);
>
> const unsigned estimatedSessionBandwidth = 500; //in kbps; for RTCP b/w
> share
>
> const unsigned estimatedSessionBandwidthAudio = 64;
>
> rtcp = RTCPInstance::createNew(*env, &rtcpGroupsock, estimatedSessionBandwidth,
> CNAME, videoSink, NULL /* we're a server */, isSSM /* we're a SSM source
> */);
>
> rtcpAudio = RTCPInstance::createNew(*env, &rtcpGroupAudiosock, estimatedSessionBandwidthAudio,
> CNAME, audioSink, NULL, isSSM);
>
> rtspServer = RTSPServer::createNew(*env, 8554, NULL);
>
> ServerMediaSession* sms = ServerMediaSession::createNew(*env, "testStream",
> "H264/AAC", "Test Streaming",  isSSM/*SSM*/);
>
> sms->addSubsession(PassiveServerMediaSubsession::createNew(*videoSink,
> rtcp)); //rtcp video multicast
>
> sms->addSubsession(PassiveServerMediaSubsession::createNew(*audioSink,
> rtcpAudio));//rtcpAudio audio multicast
>
> rtspServer->addServerMediaSession(sms);
>
> devSource = DeviceSource::createNew( *env );
>
> aacSource = AACFramedSource::createNew( *env );
>
> videoES = devSource;
>
> audioES = aacSource;
>
> videoSource = H264VideoStreamFramer::createNew( *env, videoES, True );
>
> videoSink->startPlaying(*videoSource, NULL, videoSink);
>
> audioSink->startPlaying(*audioES, NULL, audioSink);
>
> env->taskScheduler().doEventLoop( &stopEventLoop );
>
> On Wed, May 6, 2015 at 1:08 PM, Ross Finlayson <finlayson at live555.com>
> wrote:
>
>> Right now, I have problem make them stream out together, it seems audio
>> will hang the video immediately when connect the RTSP server using VLC
>>
>>
>> I suggest that you first use our “openRTSP” RTSP client application <
>> http://www.live555.com/openRTSP/> to test receiving both audio and video
>> from your server.  Using “openRTSP”, you should see (nonempty) files for
>> both the audio and video stream.  (“openRTSP” is a much simpler client
>> application than VLC, because it doesn’t render the incoming media, and
>> doesn’t care whether their presentation times are correct.)
>>
>>
>> I doubt the problem that the time stamp for video and audio are not
>> correct, since during playing using VLC, I notice that two different time
>> are shown alternately, for the video, I set the "fPresentationTime" as
>> below: since the frame rate of my H264 video is 30fps, the time for the
>> first video frame is based on the "gettimeofday", then increased based on
>> frame rate
>>
>> ----------
>>
>>   if( encoder_video_ts.tv_sec == 0 && encoder_video_ts.tv_usec == 0 )
>>
>> Are you certain that this branch is actually getting executed - i.e.,
>> that your “encoder_video_ts” variable (both the “tv_sec” and “tv_usec”
>> fields) is actually initialized to 0?
>>
>>     gettimeofday( &encoder_video_ts, NULL ); //Set the first stream time
>> stamp from wall clock
>>
>>   else {
>>
>>     unsigned uSeconds = encoder_video_ts.tv_usec + 1000000/30; //Framerate
>> is 30fps
>>
>> This is correct *only if* each NAL unit that you deliver is a complete
>> ‘access unit’ - i.e., picture.  If, instead, your stream includes PPS and
>> SPS NAL units, and/or ‘slice’ NAL units, then this calculation won’t be
>> correct.
>>
>> In any case, I suggest simplifying your code by just setting
>> “fPresentationTime” for *each* of your video NAL units by calling
>> “gettimeofday()” - as you already do for the audio frames.
>>
>> since audio and video are feeding using two separated thread, is this the
>> correct way to sync the A/V ?
>>
>>
>> Yes, if the “fPresentationTime” values are accurate, *and* aligned with
>> ‘wall clock’ time (i.e., times that would be generated by
>> “gettimeofday()”), then audio and video will get properly synchronized by
>> standards-conforming RTP/RTCP clients (like VLC).
>>
>> Ross Finlayson
>> Live Networks, Inc.
>> http://www.live555.com/
>>
>>
>>
>> _______________________________________________
>> live-devel mailing list
>> live-devel at lists.live555.com
>> http://lists.live555.com/mailman/listinfo/live-devel
>>
>>
>
>
> --
> Xingjun Chen
> M.S. Degree *Earned*
> Electrical and Computer Engineering
> University of Arizona, Tucson AZ 85721
>

-- 
Xingjun Chen
M.S. Degree *Earned*
Electrical and Computer Engineering
University of Arizona, Tucson AZ 85721
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.live555.com/pipermail/live-devel/attachments/20150507/50aaaf9e/attachment.html>