Saved searches

Use saved searches to filter your results more quickly

Cancel Create saved search Sign up Reseting focus

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

Python examples on how to use GStreamer within OpenCV. Now with GPU support! 🔥🔥🔥

Notifications You must be signed in to change notification settings

mad4ms/python-opencv-gstreamer-examples

This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Go to file

Folders and files

Last commit message Last commit date

Latest commit

History

View all files

Repository files navigation

Python3 examples for the usage of GStreamer in OpenCV

I give you the light of Eärendil GStreamer, our most beloved star lib. May it be a light for you in dark places, when all other lights go out.

Short intro

Grabbing of standard OpenCV videocapture device

import cv2 # Cam properties fps = 30. frame_width = 1920 frame_height = 1080 # Create capture cap = cv2.VideoCapture(0) # Set camera properties cap.set(cv2.CAP_PROP_FRAME_WIDTH, frame_width) cap.set(cv2.CAP_PROP_FRAME_HEIGHT, frame_height) cap.set(cv2.CAP_PROP_FPS, fps)

Grabbing of v4l2src videocapture device via GStreamer

# The following string usually works on most webcams webcam2appsink_YUY2_640_480 = "v4l2src device=/dev/video0 ! video/x-raw, format=YUY2, width=640, height=480, pixel-aspect-ratio=1/1, framerate=30/1 ! videoconvert ! appsink"

Writing of OpenCV frames to shared memory

gst_str = "appsrc ! videoconvert ! shmsink socket-path=/tmp/foo sync=true wait-for-connection=false shm-size=10000000"

Grabbing of shared memory sources

cap = cv2.VideoCapture("shmsrc socket-path=/tmp/foo ! video/x-raw, format=BGR, width=640, height=480, pixel-aspect-ratio=1/1, framerate=30/1 ! decodebin ! videoconvert ! appsink")

Writing of OpenCV frames to shared memory, file and RTP

gst_str_rtp = "appsrc ! videoconvert ! x264enc noise-reduction=10000 tune=zerolatency byte-stream=true threads=4 " \ " ! h264parse ! mpegtsmux ! rtpmp2tpay ! udpsink host=127.0.0.1 port=5000"

Usage of hardware acceleration features for encoding and decoding

# mfxh264enc does all the HW encoding on the INTEL HD GPU appsink2file = "appsrc ! videoconvert ! mfxh264enc ! \ video/x-h264, profile=baseline ! \ matroskamux ! filesink location=the_gstreamer_enjoyer.mkv"

Portable to CLI usage

Since you are here, you probably know why you want to use GStreamer and OpenCV and I'm not gonna list all the advantages that GStreamer brings to the table. However, if you find this repo helpful or even remotely funny, consider leaving a star. Or not. Your choice.

News

2021-03-30 Updated README; further install instructions; Raspi HW enc pipeline examples incoming
2021-03-23 Updated README; preparation for nvidia examples
2021-03-01 Updated for usage with Intel HD GPUs. NVIDIA examples as well as more complex stuff like splitting coming soon

Prerequisites

Be a unix enthusiast. The GPU encoding pipelines in GStreamer are extremly powerful, but are hard to install.
Ubuntu 20.04
Correct drivers for GPU (CUDA + CUDNN if necessary)
Install GStreamer

sudo apt-get install gstreamer1.0* libgstreamer-plugins-bad1.0-0 libgstreamer-plugins-base1.0-0 libgstreamer-plugins-base1.0-dev libgstreamer1.0 libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgtk-3-dev

Prerequisites for HW accelerated encoding/decoding

Intel-CPUs
- Supported platforms: Intel Haswell / Broadwell / Skylake with Intel HD / Iris Pro graphics / Apollo Lake) (>= Gen4)
- Noticable features:
  - HW H264/H265 encoding and decoding
  - Bitrate control (CBR, VBR, CQP).
  - Selectable profiles up to High Profile.
  - LibVA
  - GmmLib
  - Intel(R) Media Driver for VAAPI
  - Intel Media SDK
  - Gstreamer-Media-SDK
  - x264enc (Software encoding): 25 ms
  - mfxh264enc (Hardware encoding): 1 ms
  - Supported platforms: CUDA enabled NVIDIA GPU (List of CUDA GPUs) (tested on GTX1070, GTX1080Ti and RTX 2070)
  - Noticable features:
    - HW H264/H265 encoding and decoding
    - Bitrate control (CBR, VBR, CQP)
    - get a NVIDIA GPU (or feel free to contribute method)
    - not sponsored by NVIDIA, but by them
    Building OpenCV
    - Build & install OpenCV 4.x (4.2 works good for me; ROS works with it)
      - Mind here that we need to change a lot of CMake flags, so I highly recommend cmake-gui ( sudo apt-get install cmake-qt-gui ); search and click the features you want to have enabled (even after your exec'd a usual cmake -D flag)
      - Guide for building with CUDA support on Ubuntu 20.04 (18.04 here)
      - Mind that CUDA requires opencv-contrib modules (do not forget to check out the correct version here as well)
      - enable deprecated OPENCV_GENERATE_PKGCONFIG files. Idk why OpenCV thinks they aren't needed anymore.
      - or enable gstreamer 1.0 support with -D WITH_GSTREAMER=ON
      Tests
      - Test if GStreamer installation was successful (You should see your webcam's image):
```
$ gst-launch-1.0 v4l2src ! xvimagesink
```
      - Test if Intel MFX installation was successful:
```
$ gst-inspect-1.0 | grep mfx mfx: mfxh264dec: MFX H264 decoder mfx: mfxhevcdec: MFX HEVC decoder mfx: mfxh264enc: MFX H.264 encoder mfx: mfxhevcenc: MFX H.265 encoder
```
      - Test if NVIDIA NVENC installation was successful:
```
$ gst-inspect-1.0 | grep nvenc nvenc: nvh264enc: NVENC H.264 Video Encoder
```
      Usage
      - gst_device_to_shm grabs the VideoCapture(0) and puts the raw image in a shared memory. Mind here to define your own webcam properties.
      - gst_shm_to_app grabs the shared memory frame from gst_device_to_shm and pipes it to a VideoCapture.
      - gst_device_to_rtp grabs the VideoCapture(0) ,encodes the frame and streams it to rtp://localhost:5000
      - gst_shm_to_rtp grabs the shared memory frame from gst_device_to_shm ,encodes the frame and streams it to rtp://localhost:5000 .
      - gst_intel_device_to_app_to_file grabs the v4l2src /dev/video0 (usually webcam) to OpenCV format and writes it as an h264 encoded file.
      - gst_intel_device_to_app_to_rtp grabs the v4l2src /dev/video0 (usually webcam) to OpenCV format and writes it as an h264 encoded rtp stream. Use the provided *.sdp files for VLC viewer.
      - gst_shm_to_rtp grabs the shared memory frame from gst_device_to_shm ,encodes the frame and streams it to rtp://localhost:5000 .
      - [gst_nvidia_device_to_app_to_file] (coming_soon™)
      - [gst_nvidia_device_to_app_to_rtp] (coming_soon™)
      - [gst_raspberrypi_device_to_app_to_file] (coming_soon™)