Skip to main content

Camera Stream

Overview​

To assist developers in more flexible aircraft flight control, developers can use the APIs provided by PSDK to access the camera stream data of Autel Robotics aircraft. By combining this with image recognition algorithms, developers can create applications tailored to specific use-case requirements.

Basic Concepts​

Camera Stream​

To meet the need for obtaining camera stream functionality in applications developed using PSDK, PSDK provides the ability to access the camera stream, supporting the retrieval of the H.264 camera stream from Gimbal 1.

Note:

  • Since the callback function for obtaining the camera stream from Gimbal 1 operates in a separate thread, and OpenCV's imshow module only supports running in a single thread, only the H.264 camera stream from Gimbal 1 is supported for developers.
  • After retrieving the camera stream, please use a decoder such as FFmpeg to decode it.
  • For more details on the H.264 standard stream, please refer to H.264 Stream Standard .

Resolution and Frame Rate​

  • PSDK supports developers in obtaining the camera stream from Gimbal 1 on the aircraft. Developers or users can mount different models of cameras based on actual needs and configure the frame rate according to the camera model and working mode.
  • Photo mode：
    • Camera mode: Supports retrieving images with a resolution of 960 × 720 (720p).
    • Video mode: Supports retrieving videos with a resolution of 1280 × 720 (720p).

Note：

  • The frame rate for obtaining the main camera stream is 30 fps.

Camera H.264 Stream​

The process for obtaining the H.264 stream from the aircraft camera is as follows:

  1. Before using the function to get the H.264 camera stream, developers must first implement the liveViewSampleCb function to retrieve and process the H.264 camera stream.
  2. Call the startH264Stream() interface, specify the camera from which the stream is to be obtained, provide the callback function to receive the camera stream, and pass in the user information.
  3. Start the aircraft and user payload device, running the application developed using PSDK. The aircraft will then push the H.264 stream to the user payload device.
  4. After the user's payload device receives the H.264 stream data, it will trigger the callback function set by the developer in the application developed using PSDK.
  5. The liveViewSampleCb function designed by the developer will process the obtained H.264 stream by performing actions such as storage, decoding, and forwarding.

Using the Function to Retrieve Camera H.264 Stream​

1. Start Receiving the Camera H.264 Stream​

Control the application to receive the H.264 stream from the specified camera.

/**
 * @brief Start the camera H264 stream from the specified position.
 * @param position: Camera position for the H264 stream output.
 * @param source: sub-camera source for the H264 stream output.
 * @param callback: Callback function in a callback thread when a new h264 frame is received
 */
T_UAVReturnCode UAV_LiveView_StartH264Stream(int32_t payload_id, E_UAVLiveViewCameraSource source, UAV_LiveView_H264Callback callback);

2. Stop Receiving H.264 Stream​

Control the application to stop receiving the H.264 camera stream.

/**
 * @brief Stop the Camera H264 Stream from the specified position.
 * @param position: Camera position for the H264 stream output.
 * @param source: sub-camera source for the H264 stream output.
 */
T_UAVReturnCode UAV_LiveView_StopH264Stream(int32_t payload_id, E_UAVLiveViewCameraSource source);

3. Save or Process H.264 Stream​

After the PSDK-based application retrieves the H.264 stream, developers can perform the necessary actions on the obtained stream.

/**
 * @brief Liveview camera h264 stream callback.
 */
typedef void (*UAVLiveView_H264Callback)(int32_t payload_id, const uint8_t *buf, uint32_t len);

static void UAV_LiveView_H264StreamCallback(int32_t payload_id, const uint8_t *buf, uint32_t len)
{
    LOG_INFO("camera id: {}, len: {}", payload_id, len);

    if(nullptr != s_fp)
        fwrite((const void *)buf, 1, len, s_fp);
    fsync(fileno(s_fp));
}

Note

  • After retrieving the H.264 stream from a specified camera, developers can use the command ffplay -flags2 showall xxx.h264 to play the H.264 file.
  • Using the sample provided by PSDK, developers can retrieve H.264 stream data and record the received stream as an H.264 file locally, with the filename userData.
  • By using Sample camera-stream-callback-sample and camera-stream-poll-sample, developers can decode the H.264 stream using FFmpeg. The sample can help implement the required functionality.

Common Issues​

Decoder Errors During Decoding​

Due to platform computational limitations, applications developed using PSDK may encounter the following issues during encoding and decoding:

  • Slow decoding speed: The decoder may require some time to decode the first frame.
  • Frame loss: Insufficient computational power of the platform.
  • Errors when decoding with FFmpeg: Try decoding on Ubuntu 16.04, and ensure that the application running the decoder has the correct RNDIS, USB, and network drivers installed, allowing the application to recognize the EVO Max series multirotor aircraft correctly.

Occasional I-Frames in H.264 Stream with GDR Encoding Format​

When switching camera views in the Autel Enterprise system, the H.264 stream from the camera may insert I-Frames to help build the image for the Autel Enterprise live feed. Therefore, I-Frames may appear in the H.264 stream, but this does not affect the decoding process of the GDR format.