Implemented optional EGL based GL/CL interop by kallaballa · Pull Request #22704 · opencv/opencv

kallaballa · 2022-10-26T23:46:08Z

The pertaining issue: #22703

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

asmorkalov · 2022-10-27T10:42:37Z

OpenCV may use GLX for several purposes:

Rendering itself in highgui
OpenGL interop in core with cv::Mat
OpenGL interop for CUDA and OpenCL
Indirect interop in context of video codecs
EGL integration is useful, but should be optional on CMake level.

modules/core/CMakeLists.txt

kallaballa · 2022-10-27T10:48:36Z

OpenCV may use GLX for several purposes:

Rendering itself in highgui

OpenGL interop in core with cv::Mat

OpenGL interop for CUDA and OpenCL

Indirect interop in context of video codecs
EGL integration is useful, but should be optional on CMake level.

But why would you use GLX if you already have EGL (e.g. for android CL/GL interop). Could you please explain a use case where having EGL & GLX provides a benefit? Because all of those things (and more... e.g. offscreen rendering) can be done with EGL.

asmorkalov · 2022-10-27T11:15:04Z

Let's say, historical reason. Some code was contributed long time ago and not developed actively. I'm not against of EGL, but prefer safe way - not destroy existing features.

kallaballa · 2022-10-27T11:29:47Z

I see now what you mean. There is another place where GLX is used:

opencv/modules/core/src/gl_core_3_1.cpp

Line 131 in 778fadd

#include <GL/glx.h>

Anyway, I understand your point but on the other hand I wouldn't mind also replacing the CV_GL_GET_PROC_ADDRESS implementation and be done with GLX. :)

kallaballa · 2022-11-01T11:24:13Z

Anyway... even if you don't want to replace GLX completely, this pull request still makes sense. Replacing GLX with EGL for OpenGL interop brings many advantages, like:

Compatibility to Wayland (Without the limits of XWayland)
Offscreen rendering to GBM (drm), EGLDevice and pbuffers.
Support for DMABUF
Direct support for VAAPI

kallaballa · 2022-11-05T14:40:06Z

EGL integration is useful, but should be optional on CMake level.

I understand now. and will implement it like that.

kallaballa · 2022-11-05T15:08:23Z

Done de6bb16

…L_INTEROP)

kallaballa · 2022-11-09T19:17:46Z

I improved the implementation by defining a proper OCV_OPTION OPENCV_ENABLE_EGL_INTEROP that is OFF by default and depends on WITH_OPENGL. if OPENCV_ENABLE_EGL_INTEROP is ON then the compile flag -DHAVE_EGL_INTEROP for the file modules/core/src/opengl.cpp is set which in turn enables EGL-based OpenGL-interop.

kallaballa · 2022-11-15T08:13:07Z

I thought about it and maybe it would be a good idea to infer which window-system API (GLX or EGL) to use by probing eglGetCurrentContext() and glXGetCurrentContext() but only if HAVE_EGL_INTEROP. That way both systems can be supported.
What do you think?

asmorkalov · 2022-11-15T12:12:53Z

I tried both branches on my Ubuntu 18.04 host. Both implementations work, but final executable has significantly different dependencies. I do not think that dependency from both libEGL and libGLX is good idea.

asmorkalov · 2022-11-15T12:18:16Z

Also not all Linux systems provide EGL support. Please check OpenGL_EGL_FOUND beforehand: https://cmake.org/cmake/help/latest/module/FindOpenGL.html

kallaballa · 2022-11-15T16:59:56Z

Also not all Linux systems provide EGL support. Please check OpenGL_EGL_FOUND beforehand: https://cmake.org/cmake/help/latest/module/FindOpenGL.html

Alright!

I tried both branches on my Ubuntu 18.04 host. Both implementations work, but final executable has significantly different dependencies. I do not think that dependency from both libEGL and libGLX is good idea.

I understand your intuition in this (I had the same) but since i really want this feature i went to find evidence. ;)

First i tried to find examples of libraries on my system linking to both

find /usr/lib64/ -maxdepth 1 -name "*.so" | while read lib; do
    deps="$(sudo ldd "$lib" 2>/dev/null | tr -d '\n')"; 
    echo "$deps" | grep -q -E "libGLX.so.*libEGL.so|libEGL.so.*libGLX.so"; 
    if [ $? -eq 0 ]; then 
        echo "$lib"; 
    fi; 
done

And i found the following files:

/usr/lib64/libshotwell-authenticator.so
/usr/lib64/libshotwell-plugin-common.so
/usr/lib64/libshotwell-plugin-dev-1.0.so
/usr/lib64/libpangocairo-1.0.so
/usr/lib64/liblibreofficekitgtk.so
/usr/lib64/libgailutil.so
/usr/lib64/libgdk-x11-2.0.so
/usr/lib64/libgtk-x11-2.0.so
/usr/lib64/librsvg-2.so
/usr/lib64/libgailutil-3.so
/usr/lib64/libcairomm-1.0.so
/usr/lib64/libpangomm-1.4.so
/usr/lib64/libgdkmm-2.4.so
/usr/lib64/libgtkmm-2.4.so
/usr/lib64/libgdkmm-3.0.so
/usr/lib64/libgtkmm-3.0.so
/usr/lib64/libcairomm-1.16.so
/usr/lib64/libpangomm-2.48.so
/usr/lib64/libgtk-4.so
/usr/lib64/libgtkmm-4.0.so
/usr/lib64/libglade-2.0.so
/usr/lib64/libgladeui-2.so
/usr/lib64/libadwaita-1.so
/usr/lib64/libgnome-bluetooth-ui-3.0.so
/usr/lib64/libgdk-3.so
/usr/lib64/libgtk-3.so
/usr/lib64/libgimp-2.0.so
/usr/lib64/libgimpcolor-2.0.so
/usr/lib64/libgimpconfig-2.0.so
/usr/lib64/libgimpmodule-2.0.so
/usr/lib64/libgimpui-2.0.so
/usr/lib64/libgimpwidgets-2.0.so
/usr/lib64/libgstgl-1.0.so
/usr/lib64/libcairo-gobject.so
/usr/lib64/libcairo-script-interpreter.so
/usr/lib64/libcairo.so
/usr/lib64/libQt5EglFSDeviceIntegration.so
/usr/lib64/libQt5EglFsKmsSupport.so
/usr/lib64/libgdkglext-x11-1.0.so
/usr/lib64/libgtkglext-x11-1.0.so
/usr/lib64/libavahi-ui-gtk3.so
/usr/lib64/libperf-gtk.so

So at least somebody is actively linking EGL and GLX at the same time. But what struck my eye quickly is that it's mostly GTK stack.... So i went to search in their repo and found this: https://gitlab.gnome.org/GNOME/gtk/-/merge_requests/3540

They are basically using GLX as a fallback to EGL.

kallaballa · 2022-11-15T17:34:42Z

Half a year later firefox did the same: https://mozillagfx.wordpress.com/2021/10/30/switching-the-linux-graphics-stack-from-glx-to-egl/

Btw. I just realized I left -maxdepth 1 in my find command. There are more examples.

asmorkalov · 2022-11-17T06:55:48Z

Thanks a lot for the research! Could you check if the GTK stack does lazy linkage with EGL and GLX?

kallaballa · 2022-11-17T08:31:30Z

I am on it :)

kallaballa · 2022-11-20T01:39:07Z

I have following statement from a GTK developer: "EGL has issues with properly reporting the RGBA VIsual, which is why GTK defaults to GLX. You can often just force the RGBA Visual and it'll work, but GTK isn't doing that yet (Firefox and Chrome switched recently I think, maybe it got fixed or there's a an official way, didn't investigate yet)
in any case: GTK does pick one or the other at startup and then disallows the other."

About the exact circumstances of linking i still have to investigate and I am also going to branch out to the Firefox community and see if they have advice.

kallaballa · 2022-11-20T02:04:07Z

GTK doesn't do any special linking but has routines for probing the subsystems and only uses one at the time:
https://gitlab.gnome.org/GNOME/gtk/-/blob/main/gdk/x11/gdkdisplay-x11.c#L2947
https://gitlab.gnome.org/GNOME/gtk/-/blob/main/gdk/gdkdisplay.c#L1638
https://gitlab.gnome.org/GNOME/gtk/-/blob/main/gdk/x11/gdkglcontext-glx.c#L883
https://gitlab.gnome.org/GNOME/gtk/-/blob/main/gdk/gdkglcontextprivate.h#L52

kallaballa · 2022-11-20T05:35:04Z

Statement from a Firefox dev: "Hi! So unfortunately Mesas X11 EGL implementation still has some open issues (e.g. https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9989 ....) which is why GTK4 still falls back to GLX. Generally GTK4 and OBS-Studio might be simpler examples than FF."

kallaballa · 2022-11-20T11:35:49Z

Also not all Linux systems provide EGL support. Please check OpenGL_EGL_FOUND beforehand: https://cmake.org/cmake/help/latest/module/FindOpenGL.html

There is no check for GLX in place either (as far as i can see). Can't hurt to add that too?

kallaballa · 2022-11-20T12:45:54Z

Also not all Linux systems provide EGL support. Please check OpenGL_EGL_FOUND beforehand: https://cmake.org/cmake/help/latest/module/FindOpenGL.html

Done 735a0f2

There is no check for GLX in place either (as far as i can see). Can't hurt to add that too?

Done 49f64dd

opencv-alalek · 2023-09-24T23:54:02Z

modules/highgui/cmake/init.cmake

  endif()
 endmacro()

-add_backend("gtk" WITH_GTK)


Need to recover GTK plugins support.

Yes. I need to push my latest changes. They include working GTK.

Sorry. needed quite some to come back to this, but now i pushed the latest changes :)

opencv-alalek · 2023-09-24T23:57:22Z

modules/core/CMakeLists.txt

  ocv_append_source_file_compile_definitions("${CMAKE_CURRENT_LIST_DIR}/src/va_intel.cpp" "OPENCV_LIBVA_LINK=1")
 endif()
+if(OPENCV_ENABLE_EGL_INTEROP)
+  ocv_append_source_file_compile_definitions("${CMAKE_CURRENT_LIST_DIR}/src/opengl.cpp" "HAVE_EGL_INTEROP")


Perhaps HAVE_EGL and HAVE_OPENGL should be checked too.

It does in the main CMakeLists.txt

Not really.

User could specify WITH_OPENGL=ON but there could be no OpenCL dev packages on build system (or for build target platform in case of cross-compiling).

WITH_ / OPENCV_ENABLE_ is a user intention (lets enable this if available).
Next, find_package() should be executed.
Preferable try_compile() should be run to validate dependency configuration.
After checks above HAVE_ variable is set to ON / 1.
Feature usage should be guarded by HAVE_ checks.

All dependencies should be properly detected on configuration (cmake) stage.
Build stage (make) should not fail due to dependency misusing.

I see now. Will probably create a PR today.

opencv-alalek · 2023-09-24T23:59:05Z

modules/core/CMakeLists.txt

    "${ZLIB_LIBRARIES}" "${OPENCL_LIBRARIES}" "${VA_LIBRARIES}"
-    "${OPENGL_LIBRARIES}"
-    "${GLX_LIBRARIES}"
+    "${OPENGL_LIBRARIES}" "${OPENGL_egl_LIBRARY}"


@asmorkalov Does OpenGL subsystem in OpenCV uses dynamic loading? or direct linkage?

In a way it does both and with libGLVND the situation is a bit more complex. But anyway, it is directly linked but the function pointers are loaded a runtime.

No, linkage is static for now:

opencv-build/lib$ readelf -d ./libopencv_core.so Dynamic section at offset 0xe7d750 contains 31 entries: Tag Type Name/Value 0x0000000000000001 (NEEDED) Shared library: [libz.so.1] 0x0000000000000001 (NEEDED) Shared library: [libGL.so.1] 0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6] 0x0000000000000001 (NEEDED) Shared library: [libm.so.6] 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] 0x000000000000000e (SONAME) Library soname: [libopencv_core.so.412] 0x000000000000001d (RUNPATH) Library runpath: [::::::::::::::] 0x000000000000000c (INIT) 0x7c000 0x000000000000000d (FINI) 0xbd84cc 0x0000000000000019 (INIT_ARRAY) 0xe65860 0x000000000000001b (INIT_ARRAYSZ) 120 (bytes) 0x000000000000001a (FINI_ARRAY) 0xe658d8 0x000000000000001c (FINI_ARRAYSZ) 8 (bytes) 0x000000006ffffef5 (GNU_HASH) 0x2f0 0x0000000000000005 (STRTAB) 0x14608 0x0000000000000006 (SYMTAB) 0x4fd8 0x000000000000000a (STRSZ) 101596 (bytes) 0x000000000000000b (SYMENT) 24 (bytes) 0x0000000000000003 (PLTGOT) 0xe7efe8 0x0000000000000002 (PLTRELSZ) 26304 (bytes) 0x0000000000000014 (PLTREL) RELA 0x0000000000000017 (JMPREL) 0x75800 0x0000000000000007 (RELA) 0x2e9b8 0x0000000000000008 (RELASZ) 290376 (bytes) 0x0000000000000009 (RELAENT) 24 (bytes) 0x000000006ffffffe (VERNEED) 0x2e768 0x000000006fffffff (VERNEEDNUM) 4 0x000000006ffffff0 (VERSYM) 0x2d2e4 0x000000006ffffff9 (RELACOUNT) 11124 0x0000000000000000 (NULL) 0x0

kallaballa · 2024-10-09T06:19:47Z

I have this in use on a couple of Linux systems (OpenSuSe, Debian, Ubuntu) mostly Intel iGPU systems but also one with Nvidia GPU. Intel requires EGL while Nvidia requires GLX. On the NVidia system the Intel iGPU is available as well and works - so i can switch between them with the same binary due to the GLX fallback mechanism.
What is missing?

kallaballa · 2025-01-02T06:21:20Z

Tested using an Intel A770 and Mesa 24.3.1 as well

kallaballa · 2025-01-02T07:40:25Z

Resolved merge conflict. testing.

kallaballa · 2025-01-03T23:38:23Z

Tested. Still works as expected.

opencv-alalek · 2025-05-13T03:21:10Z

cmake/OpenCVFindLibsGUI.cmake

  if(WITH_WIN32UI OR (HAVE_QT AND QT_QTOPENGL_FOUND) OR HAVE_GTK3 OR (HAVE_GTK AND NOT HAVE_GTK3 AND HAVE_GTKGLEXT))
    find_package (OpenGL QUIET)
+    if(OpenGL_EGL_FOUND)
+      set(HAVE_EGL TRUE AND OPENCV_ENABLE_EGL)


CMake's set doesn't support complex boolean values.

opencv-alalek · 2025-05-13T03:23:34Z

modules/core/src/opengl.cpp

+#    if defined(OPENCV_ENABLE_EGL)
+#      include <EGL/egl.h>
+#    endif
+#    if defined(OPENCV_ENABLE_GLX)


Where are these vars defined?

opencv-alalek · 2025-05-13T03:23:37Z

modules/core/CMakeLists.txt

 if(OPENCV_LIBVA_LINK)
  ocv_append_source_file_compile_definitions("${CMAKE_CURRENT_LIST_DIR}/src/va_intel.cpp" "OPENCV_LIBVA_LINK=1")
 endif()
+if(OPENCV_ENABLE_EGL_INTEROP)


Where do we define this flag?

kallaballa · 2025-08-19T07:50:27Z

i fixed the issues

replaced glx with egl

6eb76a1

asmorkalov requested a review from alalek October 27, 2022 06:31

asmorkalov added category: core category: build/install labels Oct 27, 2022

asmorkalov reviewed Oct 27, 2022

View reviewed changes

modules/core/CMakeLists.txt Show resolved Hide resolved

made egl based opengl/opencl interop optional

de6bb16

kallaballa changed the title ~~replaced glx with egl~~ Implemented optional EGL based GL/CL interop Nov 5, 2022

kallaballa added 2 commits November 9, 2022 19:53

Merge branch 'opencv:4.x' into replace_glx_with_egl

2a824fc

create a proper option for egl-based opengl-interop (OPENCV_ENABLE_EG…

5fe0596

…L_INTEROP)

properly detect and report EGL

735a0f2

opencv-alalek reviewed Sep 24, 2023

View reviewed changes

kallaballa and others added 3 commits October 26, 2023 07:16

ported latest EGL/OpenGL changes proper branch

13bbd26

Merge branch 'opencv:4.x' into replace_glx_with_egl

de7d0ff

recover gtk plugins

27f612e

asmorkalov added this to the 4.11.0 milestone Jul 5, 2024

Merge branch '4.x' into replace_glx_with_egl

0ec678b

asmorkalov modified the milestones: 4.11.0, 4.12.0 Dec 20, 2024

Merge branch '4.x' into replace_glx_with_egl

0607e13

kallaballa and others added 2 commits January 2, 2025 08:43

Removed obsolete code

78d50d0

opengl.cpp: fixed macro flow

66f9b6f

Simplified cmake checks to presume existing behaviour.

3b3f415

asmorkalov force-pushed the replace_glx_with_egl branch from d243a30 to 3b3f415 Compare May 12, 2025 16:01

asmorkalov approved these changes May 12, 2025

View reviewed changes

opencv-alalek reviewed May 13, 2025

View reviewed changes

opencv-alalek added the incomplete label May 13, 2025

asmorkalov modified the milestones: 4.12.0, 4.13.0 Jun 11, 2025

kallaballa added 3 commits August 18, 2025 23:13

fixed interop macros

f887d38

added missing defines

21ffbd0

remove EMSCRIPTEN clause

c779e46

kallaballa added 2 commits September 7, 2025 11:03

Update OpenCVFindLibsGUI.cmake: Dont limit EGL and GLX by interop

7e6b1ec

Update opengl.cpp: fixsed variable naming

d486634

asmorkalov modified the milestones: 4.13.0, 4.14.0 Dec 8, 2025

Uh oh!

Conversation

kallaballa commented Oct 26, 2022

Pull Request Readiness Checklist

Uh oh!

asmorkalov commented Oct 27, 2022

Uh oh!

Uh oh!

kallaballa commented Oct 27, 2022

Uh oh!

asmorkalov commented Oct 27, 2022

Uh oh!

kallaballa commented Oct 27, 2022

Uh oh!

kallaballa commented Nov 1, 2022

Uh oh!

kallaballa commented Nov 5, 2022

Uh oh!

kallaballa commented Nov 5, 2022

Uh oh!

kallaballa commented Nov 9, 2022

Uh oh!

kallaballa commented Nov 15, 2022

Uh oh!

asmorkalov commented Nov 15, 2022

Uh oh!

asmorkalov commented Nov 15, 2022

Uh oh!

kallaballa commented Nov 15, 2022

Uh oh!

kallaballa commented Nov 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asmorkalov commented Nov 17, 2022

Uh oh!

kallaballa commented Nov 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kallaballa commented Nov 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kallaballa commented Nov 20, 2022

Uh oh!

kallaballa commented Nov 20, 2022

Uh oh!

kallaballa commented Nov 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kallaballa commented Nov 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

opencv-alalek Sep 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kallaballa commented Oct 9, 2024

Uh oh!

kallaballa commented Jan 2, 2025

Uh oh!

kallaballa commented Jan 2, 2025

Uh oh!

kallaballa commented Jan 3, 2025

kallaballa commented Nov 15, 2022 •

edited

Loading

kallaballa commented Nov 17, 2022 •

edited

Loading

kallaballa commented Nov 20, 2022 •

edited

Loading

kallaballa commented Nov 20, 2022 •

edited

Loading

kallaballa commented Nov 20, 2022 •

edited

Loading

opencv-alalek Sep 27, 2023 •

edited

Loading