OpenCV for ZGE

Use of external libraries (DLLs) from ZGE.

Moderator: Moderators

User avatar
Rado1
Posts: 775
Joined: Wed May 05, 2010 12:16 pm

OpenCV for ZGE

Post by Rado1 »

Based on discussion about new possible ZGE extensions, I created an experimental version of the ZgeCV library, a wrapper for OpenCV computer vision library.

The current demo version works only on Windows and just captures video from connected cameras and returns the data to be renderable by OpenGL. That's the first important step and next will come:

- allow for multiple camera capturing
- managing multiple frames/images to be processed independently
- import/export of images and videos from files
- various image/video filters
- object detection
- face recognition, etc.

OpenCV has a lot of possibilities, but at the other hand, an unpleasant aspect is its size. With the current video capturing features it is about 6.3 MB, when added some image processing features, it is about 12 MB. Just compare it to the compiled and uncompressed demo created in ZGE which is about 240 KB.

The library can be downloaded from here. The demo ZGE project is attached.

Your comments are welcome.
Attachments
screenshot from camera capturing demo
screenshot from camera capturing demo
scr.jpg (29.14 KiB) Viewed 27230 times
demo1.zgeproj
demo ZGE project
(2.52 KiB) Downloaded 696 times
User avatar
VilleK
Site Admin
Posts: 2274
Joined: Mon Jan 15, 2007 4:50 pm
Location: Stockholm, Sweden
Contact:

Post by VilleK »

Thumbs up Radovan! :-)
StevenM
Posts: 149
Joined: Thu Jan 20, 2011 10:03 am

Awsome

Post by StevenM »

Thanks for doing this! I know opencv is time consuming to set-up. I'm glad you were able to get the project started. For the average computer user, OpenCV s not easy to get up and running - ZGE with OpenCV can make computer vision more accessible, and that is great.


Your camera demo is working perfect.


Computer Vision has so many applications, but be weary of face recognition, it's a very difficult problem. Face /object/shape detection (is there a face?) is simple with opencv, but recognition (who's face is it?) is complicated and not very accurate. If you start there, you'll get frustrated for sure.
User avatar
Rado1
Posts: 775
Joined: Wed May 05, 2010 12:16 pm

Re: Awsome

Post by Rado1 »

StevenM wrote:Thanks for doing this! I know opencv is time consuming to set-up.
This was not so difficult. The new version 3.0.0 beta comes with nice pre-built .dll and .lib files, so compilation and linking is quite an easy task. Also the API is relatively straightforward.

To avoid frustration from face recognition... (even if I do not expect bigger troubles than recognizing my face in a mirror by myself soon in the morning) I rather started with simpler things. The current update contains:

- support for several cameras simultaneously
- support for arbitrary number of images; this is useful for image operations
- an initial set of image operations/filters: color transformation, flipping, Gaussian blur, and Canny edge detection

Enjoy the new demo project and please download the latest dll to make it working. I wondered myself how OpenCV is fast and all the effects can be applied in runtime on captured video.

Next steps:
1. There is a lot of image operations in OpenCV, and I do not think I'll add all of them to ZgeCV library... if you have some preferences, just tell me.
2. I could try, for instance, object detection or motion analysis. StevenM do you have some experience with these topics? Could you please give me some good tutorials to be able to understand it fast and correctly?
Attachments
screenshot form new demo
screenshot form new demo
scr1.jpg (64.79 KiB) Viewed 27190 times
demo2.zgeproj
demo project
(6.78 KiB) Downloaded 632 times
User avatar
Kjell
Posts: 1876
Joined: Sat Feb 23, 2008 11:15 pm

Re: Awsome

Post by Kjell »

Hi Rado1,
Rado1 wrote:an initial set of image operations/filters: color transformation, flipping, Gaussian blur, and Canny edge detection
Are you doing these using OpenCV? Since ZGE has built-in components ( Blur / Convolution / Expression ) for such effects*, or you could use a shader of course.

*Not sure whether BitmapLoad works properly when you've adjusted the size of a Bitmap using glTexImage2D though.

K
User avatar
Rado1
Posts: 775
Joined: Wed May 05, 2010 12:16 pm

Re: Awsome

Post by Rado1 »

Kjell wrote:Are you doing these using OpenCV? Since ZGE has built-in components ( Blur / Convolution / Expression ) for such effects*, or you could use a shader of course.
I'm using OpenCV filters, because they are applied to the captured images in runtime... and are relatively fast. How could be ZGE bitmap components used for captured bitmap which is set by glTexSubImage2D?
Kjell wrote:*Not sure whether BitmapLoad works properly when you've adjusted the size of a Bitmap using glTexImage2D though.
Yes, that's the question. Original bitmaps are 16x16, then scaled to 640x480 with my camera...
User avatar
Kjell
Posts: 1876
Joined: Sat Feb 23, 2008 11:15 pm

Re: Awsome

Post by Kjell »

Hi Rado1,

Just tried it .. looks like this using BitmapBlur :cry:

Image

Obviously this wouldn't be a problem if the Bitmap component had integer values for width & height ( instead of the drop-down menus ). And using shaders it works just fine of course.

K
User avatar
Rado1
Posts: 775
Joined: Wed May 05, 2010 12:16 pm

Re: Awsome

Post by Rado1 »

Kjell wrote:And using shaders it works just fine of course.
Sure, shaders would probably be the best solution for displaying images. However, the OpenCV image filters can also be used to prepare input images/videos for more complex image processing, such as edge detection, object recognition, motion recognition, etc. So they are usable...
StevenM
Posts: 149
Joined: Thu Jan 20, 2011 10:03 am

some tips for face detection

Post by StevenM »

2. I could try, for instance, object detection or motion analysis. StevenM do you have some experience with these topics? Could you please give me some good tutorials to be able to understand it fast and correctly?
I've used OpenCV face detection/tracking. It is easy to do. Most examples of face tracking, that i have seen, is just face detection on every frame.

Just note :

The cascade. Cascades are .xml files that contain data needed by the detector to detect a specific object and the files can be found in the ?:\opencv\sources\data folder.
These files would need to be copied to a specific folder with the zge DLL in order to implement face detection. There are also some other cascades on the web.

Detecting geometric shapes, blobs, lines does not require a cascade:
https://opencv-code.com/tutorials/detec ... -an-image/

2.prep image for face detection. You can use the image as is and that is what I normally do - but in order to improve speed and accuracy - scale down hi-res image. applying filters can improve detection and prevent false positives.

3.face_cascade.detectMultiScale( frame_gray, faces, 1.1, 2, 0|CV_HAAR_SCALE_IMAGE, Size(30, 30) );


parameters can be found on this page:

http://docs.opencv.org/modules/objdetec ... ation.html

faces might be tricky - it is a the variable that holds the face detection results array vec4(rectangular upper and lower corners) . To overlay a rectangle on the display, the rectangle need to be scaled back up if the image was scaled down for face detection.

the value 2(minimum neighbors) here has caused me to get a lot of false positives. raising this value can reduce false positives, but too high will result in non-detection and a performance hit.

CV_HAAR_SCALE_IMAGE - This constant in indicates: scale the image instead of the detector, arguably better than the default method.

Size = smallest (Width,Height) in pixels of a detected face.

I use python, but c++ code for face detection is here:

minimal example:
http://answers.opencv.org/question/2006 ... cognition/

opencv doc example:
http://docs.opencv.org/doc/tutorials/ob ... ifier.html
User avatar
VilleK
Site Admin
Posts: 2274
Joined: Mon Jan 15, 2007 4:50 pm
Location: Stockholm, Sweden
Contact:

Post by VilleK »

Cool :). I wouldn't recommend using ZGE's built-in filters though, they are not really designed for realtime use. If fast filters are already built-in OpenCV I would use that.
User avatar
Kjell
Posts: 1876
Joined: Sat Feb 23, 2008 11:15 pm

Re: Awsome

Post by Kjell »

Hi guys,
Rado1 wrote:However, the OpenCV image filters can also be used to prepare input images/videos for more complex image processing.
Ah, i see .. makes sense.
VilleK wrote:I wouldn't recommend using ZGE's built-in filters though, they are not really designed for realtime use.
Alright, never mind then :)

K
User avatar
Rado1
Posts: 775
Joined: Wed May 05, 2010 12:16 pm

Post by Rado1 »

ZgeCV contains now also a simple object detection based on Haar Feature-based Cascade Classifier. After updating the dll you can run the attached demo project to detect frontal face and eyes. The quality of detection is not 100% and highly depends on the provided cascade classifier files. The attached project uses examples provided by OpenCV distribution.

Next plans:
1. Experiment with hand detection. I saw some interesting results on web able to detect hand and finger positions. I'll try to use it as a controller for ZGE games.
2. Porting the library to Android.
Attachments
screenshot from face detection demo
screenshot from face detection demo
scr2.jpg (35.58 KiB) Viewed 27121 times
demo3.zip
face detection project file
(146.69 KiB) Downloaded 578 times
StevenM
Posts: 149
Joined: Thu Jan 20, 2011 10:03 am

Thank you

Post by StevenM »

Yes! This is fantastic - downloading now.
Experiment with hand detection. I saw some interesting results on web able to detect hand and finger positions.
I've seen this too. I think this is done using a combination of detection and tracking, but not sure.
User avatar
VilleK
Site Admin
Posts: 2274
Joined: Mon Jan 15, 2007 4:50 pm
Location: Stockholm, Sweden
Contact:

Post by VilleK »

This looks great. I can see it being used for interesting Visualizer effect experiments among other things. Hope I get a chance to use it.

Also clever that you include family members in your testing, perhaps they'll have more understanding why you spend much time in front of the computer ;)

edit: just tried it and it works fine here.
User avatar
Rado1
Posts: 775
Joined: Wed May 05, 2010 12:16 pm

Post by Rado1 »

I'm glad you like it. BTW I tested OpenCV 2.4.10 and it produces 4 times smaller dll than 3.0.0 beta; I achieved 3.17MB with the same functionality.

However, I'm still not satisfied. The problem is that both versions of OpenCV have memory leaks in very basic functions such as cvtColor(). Running of such a program results in many page faults and memory consumption is increasing. StevenM do you have the same experience?
Post Reply