[Live-devel] UDP packets appear to be empty, forcing switch to TCP in VLC

[Dave McMordie]

On Fri, May 8, 2015 at 1:23 AM, Ross Finlayson <finlayson at live555.com>
wrote:

> Having ruled out firewall and UDP transmission issues, what else could
> cause RTP-over-UDP to fail when RTP-over-TCP works fine?
>
>
> Nothing comes to mind, unfortunately.  Assuming that you haven’t modified
> the supplied code, I can’t think of any reason (other than firewalls) why
> RTP-over-TCP would work, but RTP-over-UDP would not.
>

Just to be sure, I wiped all the live555 code yesterday, untarballed it and
rebuilt from scratch-- same result.  We subclass
OnDemandServerMediaSubsession, FramedSource (for our DeviceSource class)
and create an RTSP server instance based on your example code.


>
> Let’s see your “OnDemandServerMediaSubsession” subclass.  In particular,
> how have you implemented the “createNewStreamSource()” and
> “createNewRTPSink()” virtual functions?
>

#include "H264SMS.h"
#include "Log.h"
#include "ImageRingBuffer.h"
#include <queue>

#include "leakcheck.h"  // always include this last in any file


H264SMS::H264SMS( UsageEnvironment& env, bool reuseFirstSource,
                  GetMoreEncodedDataCallback* moreDataCallback, void*
userCtx,
                  std::vector<uint8_t> useSPS,
                  std::vector<uint8_t> usePPS
                  )
    : OnDemandServerMediaSubsession( env, reuseFirstSource ),
      sink(NULL),
      source(NULL),
      myPPS(usePPS),
      mySPS(useSPS),
      moreDataCb(moreDataCallback),
      userContext(userCtx)
{

}

H264SMS *
H264SMS::createNew( UsageEnvironment& env, bool reuseFirstSource,
                    GetMoreEncodedDataCallback moreDataCallback, void*
userCtx,
                    std::vector<uint8_t> useSPS,
                    std::vector<uint8_t> usePPS)
{
    return new H264SMS( env, reuseFirstSource, moreDataCallback, userCtx,
                        useSPS, usePPS
                        );
}

H264SMS::~H264SMS()
{
    //Medium::close(sink);
    //Medium::close(source);
}

FramedSource* H264SMS::createNewStreamSource(unsigned,  unsigned&
estBitrate )
{

    estBitrate = 20000;
    unsigned timePerFrame = 1000000/30; // microseconds

    source =  H264DeviceSource::createNew( envir(), timePerFrame);
    source->registerGetMoreEncodedDataCallback(moreDataCb, userContext);

    H264VideoStreamDiscreteFramer* framer =
H264VideoStreamDiscreteFramer::createNew(envir(), source);

    return framer;
}

RTPSink* H264SMS::createNewRTPSink(Groupsock* rtpGroupsock,
                                   unsigned char rtpPayloadTypeIfDynamic,
                                   FramedSource* inputSource)
{
    (void)inputSource;

    OutPacketBuffer::maxSize = 2000000;

    sink = H264VideoRTPSink::createNew( envir(), rtpGroupsock,
rtpPayloadTypeIfDynamic,
                                        &mySPS[0], mySPS.size(), &myPPS[0],
myPPS.size() );
    return sink;
}


> Here is how I am sending the encoded frames which I get from libavcodec to
> the DeviceSource (in case this rings any alarm bells):
>
>
> No, this looks OK.  But what does your implementation of
> “doGetNextFrame()” do?
>

I spent some time going through older posts on this mailing list and
noticed the requirement that we not send multiple NALs to the
H264VideoStreamDiscreteFramer.  I put in a test looking for multiple NALs
by matching the 0x00000001 start code and found that most encoded frame
buffers have two NALs-- the first one appears to be SPS/PPS information.
So I added code to split these NALs and deliver them one-at-a-time without
the start sequence.  The result was no video over TCP or UDP even though
the NALs looked correct.

For reasons I don't fully understand (which may have to do with the
behaviour of our HW encoder) we are manually extracting the SPS and PPS and
feeding it to the constructor of the ServerMediaSubsession subclass (as you
see above).  Since this was suspect, I checked both that saveCopyOfSPS
and saveCopyOfPPS we being called (they are) and that the SPS and PPS
sequences match what we have extracted (they do).  Just to be extra sure, I
hacked in a pile of code from Fraunhofer H.264 reference codebase to
validate the SPS and PPS we were sending you.  They appear to parse okay
and give the following result:

2015-May-08 09:28:14  SPS: profile_idc
2015-May-08 09:28:14  SPS: constrained_set0_flag
2015-May-08 09:28:14  SPS: constrained_set1_flag
2015-May-08 09:28:14  SPS: constrained_set2_flag
2015-May-08 09:28:14  SPS: constrained_set3_flag
2015-May-08 09:28:14  SPS: reserved_zero_4bits
2015-May-08 09:28:14  SPS: level_idc
2015-May-08 09:28:14  SPS: seq_parameter_set_id
2015-May-08 09:28:14  SPS: log2_max_frame_num_minus4
2015-May-08 09:28:14  SPS: pic_order_cnt_type
2015-May-08 09:28:14  SPS: num_ref_frames
2015-May-08 09:28:14  SPS: gaps_in_frame_num_value_allowed_flag
2015-May-08 09:28:14  SPS: pic_width_in_mbs_minus1
2015-May-08 09:28:14  SPS: pic_height_in_map_units_minus1
2015-May-08 09:28:14  SPS: frame_mbs_only_flag
2015-May-08 09:28:14  SPS: direct_8x8_inference_flag
2015-May-08 09:28:14  SPS: frame_cropping_flag
2015-May-08 09:28:14  SPS: vui_parameters_present_flag

Very glad you have you cast your eyes over the implementation below and
above.  I suspect our code is doing something wrong, but I am still not
seeing where or how.  Is there any extra tracing or validation you can
recommend to make sure the stream and SDP are 100% correct going out the
door on the server side?

void H264DeviceSource::doGetNextFrame() {

   if (fNeedAFrame) {
        Log::w("doGetNextFrame called while deliverFrame active");
        return;
    }

    fNeedAFrame = True;
    deliverFrameToClient();

}

void H264DeviceSource::deliverFrameToClient() {

    fNeedAFrame = False;

    pEncodedFrameAndMetadata frameAndMetadata;



// Dequeue the next encoded frame object
while (running){

if (moreDataCallback != NULL)
frameAndMetadata = moreDataCallback(userCtx);
else
return;

if (frameAndMetadata.get() != NULL &&
frameAndMetadata->encodedFrame.get() != NULL &&
!frameAndMetadata->wasStreamed)
{
if (!frameAndMetadata->encodedFrame->empty()) {
break;
} else {
Log::i("Empty frame!!!");
}
}

boost::this_thread::sleep(boost::posix_time::milliseconds(20));
}

frameAndMetadata->wasStreamed = true;

//Log::i("RTSP WE HAVE H264 FRAME.");

lastImageAcquisitionTime = VTime::currentTimeMillis();

// Set the 'presentation time': the time that this frame was sent

gettimeofday(&fPresentationTime, NULL);

fDurationInMicroseconds = 27027;

startCapture(); // prepare the next frame


if (!running){
return;
}

// strip start sequence and copy to the fTo buffer:
populateEncodedImage(frameAndMetadata);



    if (fFrameSize >= fMaxSize) {
        Log::e("H264DeviceSource::deliverFrameToClient(): read maximum
buffer size.  Frame may be truncated");
    }

    frameDeliveryRate.countEvent();


}

Thanks again for your help.

Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.live555.com/pipermail/live-devel/attachments/20150508/4ce20d27/attachment.html>