Video Decoding Tutorial

1. Overview

AUTEL Mobile SDK offers the functionality of obtaining video streams from a camera. It allows you to enable video streams, add observers to obtain video streams, and further implement data decoding and video playback. Alternatively, you can use VideoStreamPlayManager and VideoOpenGLView provided by Autel_UX_SDK demo for video decoding and playback. In this tutorial, you will learn how to use the above two methods to decode videos.

2. Video Decoding Process

The following call flow diagram shows the interactions between the UI layer, video playback business, and Mobile SDK.

3. Video Decoding APIs

The video decoding functionality provided by the Mobile SDK is mainly implemented by AUTELVideoFeed and AUTELVideoDataPacket classes.

AUTELVideoFeed is mainly used for enabling or disabling video streams or adding or removing an observer. The Mobile SDK allocates the stream data of every frame to the added observer.

3.1 Enable Video Stream Monitoring

+ (void)openVideoFeed;

3.2 Disable Video Stream Monitoring

+ (void)closeVideoFeed

3.3 Add an Observer for Video Streams

When the Mobile SDK receives video streams and prepares the stream data packets, it will allocate the packets to the queue in which the observer that you add is located.

+ (void)addVideoFeedListener:(id<AUTELVideoFeedListener>)listener withQueue:(dispatch_queue_t)queue;

3.4 Remove a Video Stream Observer

After you remove a video stream observer, the SDK will not allocate any video streams to that observer.

+ (void)removeVideoFeedListener:(id<AUTELVideoFeedListener>)listener;

3.5 Stream Data Callback

After you add a video stream observer in Section 2.2.3, you must implement callbacks in the class where the observer is located. The following describes the callback APIs:

Callback to be executed when the stream data packet changes:

- (void)videoFeed:(AUTELVideoFeed *)feed didUpdateVideoData:(AUTELVideoDataPacket *)packet;

Callback to be executed when the connection status of the camera changes:

As all video streams come from the camera, if the camera is disconnected, no video streams will be produced. Therefore, we need to monitor the connection status of the camera.

- (void)videoFeed:(AUTELVideoFeed *)feed didUpdateConnecteState:(BOOL)state;

You can obtain video stream data (AUTELVideoDataPacket) and connection status of the camera by using the stream data packet callback. You can decode and play the data after you obtain AUTELVideoDataPacket.

3.6 Video Stream Data

AUTELVideoDataPacket is a stream data class used for implementing the callback from the upper layer after an observer is added. The following describes the properties of AUTELVideoDataPacket.

@property (nonatomic, assign, readonly)uint8_t *data;// Obtains the stream data of the current frame

@property (nonatomic, assign, readonly)size_t size; // The size of the code stream data

@property (nonatomic, assign, readonly)int64_t pts; // Obtains the frame rate of the current frame

@property (nonatomic, assign, readonly)int32_t width; // Obtains the width of the video image of the current frame

@property (nonatomic, assign, readonly)int32_t height; // Obtains the height of the video image of the current frame

@property (nonatomic, assign, readonly)int32_t keyFrame; // Indicates whether it is a key frame

@property (nonatomic, assign, readonly)AutelVideoEncodeFormat videoFormat; //H.264 encoding format, H.265 encoding format

- (instancetype)initWithData:(uint8_t *)data withSize:(size_t)size withPts:(int64_t)pts;
// Added for supporting H.265 encoding format. For the definition of codeType, see AutelVideoEncodeFormat
- (instancetype)initWithData:(uint8_t *)data withSize:(size_t)size withPts:(int64_t)pts encodeType: (int)codeType;
// Added for supporting H.265 encoding format. For the definition of codeType, see AutelVideoEncodeFormat
- (instancetype)initWithData: (uint8_t *)data
                    withwidth: (int32_t)width
                    withheight: (int32_t)height
                  withKeyFrame: (int32_t)keyFrame
                    withSize: (size_t)size
                     withPts: (int64_t)pts
                  encodeType: (int)codeType;

- (instancetype)initWithData:(uint8_t *)data;
// Added for supporting H.265 encoding format. For the definition of codeType, see AutelVideoEncodeFormat
- (instancetype)initWithData:(uint8_t *)data encodeType: (int)codeType;

By now, you have learned how to obtain the video data. If you want to implement video decoding and playback by yourself, you have achieved your purpose now.

4. Implement Video Decoding

The following uses Autel_UX_SDK demo as an example to explain how to develop video decoding and playback functionalities.

In this example, to facilitate functionality development, we encapsulated the underlying video streams of the Mobile SDK again. The involved classes include VideoStreamPlayManager and VideoOpenGLView.

• VideoStreamPlayManager: The manager class for enabling streams, decoding streams, and playing video frames. You can implement video playback by simply calling this class.
• VideoOpenGLView: video playback view rendered by openGL. You can just create it normally and set up the layout.

The above classes are all encapsulated in the AUTELWidget.framework dynamic library.

4.1 Create a Video Playback View

lazy var videoPlayView : VideoOpenGLView = {
    let view = VideoOpenGLView.init()
    return view
}()

view.addSubview(videoPlayView)
videoPlayView.snp.makeConstraints { (make) in
    make.edges.equalToSuperview()
}

4.2 Start Video Playback

func startPlay() {
    guard !self.isPlaySetup else {
        return
    }
    self.isPlaySetup = true
    print("VideoStreamPlayManager startPlay called")
    VideoStreamPlayManager.sharedInstance().setPlayView(to: videoPlayView)
}

4.3 Stop Video Playback

func stopPlay() {
    print("VideoStreamPlayManager stopPlay called")
    VideoStreamPlayManager.sharedInstance().stopPlayAndRemoveView()
    self.isPlaySetup = false
}

4.4 Other Functionality

• Enable the feature of showing overexposed parts in a video.

  Mark overexposed parts in a video and show them in the video screen in real time.

  VideoStreamPlayManager.sharedInstance().showOverExpose = true

• Set the callback to be executed when the video aspect ration changes.

  VideoStreamPlayManager.sharedInstance().ratioHandle = { [weak self] ratio in
      GCDUtils.main {
          self?.updatePlayerView(ratio)
      }
  }

• Obtain the width and height of the video.

  VideoStreamPlayManager.sharedInstance().renderAspectRatio