FPS issue for video recording using OpenCV 2 [CvbPython]

Hey,

In advance I will let you know that I not an expert in Computer Vision :frowning: , that’s why there is a chance that my questions might look silly for you and for that I apologize :upside_down_face:.

So, I am trying to make a video with the following snippet code (testing now only).

import os
import cvb
import cv2
FPS = 25
CAMERA_WIDTH = 1920
CAMERA_HEIGHT = 1080
CAMERA_VERTICAL_OFFSET = 32
CAMERA_HORIZONTAL_OFFSET = 32
frameSize = (1920, 1080)



def deviceConfiguration(dev_node_map):
    dev_node_map['Std::Width'].value = CAMERA_WIDTH
    dev_node_map['Std::Height'].value = CAMERA_HEIGHT
    # set the Frame rate in Hz
    dev_node_map['Std::AcquisitionFrameRate'].value = FPS
    dev_node_map['Std::OffsetX'].value = CAMERA_HORIZONTAL_OFFSET
    dev_node_map['Std::OffsetY'].value = CAMERA_VERTICAL_OFFSET
    dev_node_map['Cust::autoBrightnessMode'].value = "Active"

if __name__ == '__main__':
    rate_counter = None
    out = cv2.VideoWriter('recording.avi', cv2.VideoWriter_fourcc(*'DIVX'), 25, frameSize)
    with cvb.DeviceFactory.open(cvb.install_path() + "/drivers/GenICam.vin", port=0) as device:
        deviceConfiguration(device.node_maps["Device"])
        device.node_maps["Device"]['Cust::timestampControlLatch'].execute()
        stream = device.stream
        rate_counter = cvb.RateCounter()

        stream.start()
        for i in range(10):
            rate_counter.step()
            image, status = stream.wait()
            if status == cvb.WaitStatus.Ok:
                np_image = cvb.as_array(image, copy=False)
                # save the frames
                out.write(np_image)
                imageRawTimeStamp = "{:.0f}".format(float(image.raw_timestamp))

                print("Acquired image: " + str(i) + " | Timestamp: " + str(imageRawTimeStamp))
                print("Acquired with: " + str(rate_counter.rate) + " fps")

        stream.abort()

I checked other examples for python to resolve this issue. But no sucess, even if I specify the FPS to be 25, I am getting Low FPS or even getting no fps at all. You can check the output “nan” or some frames I am getting 0 value with the rawtimestamp you can see the result below:

Acquired image: 0 | Timestamp: 1612365745947438848
Acquired with: nan fps
Acquired image: 1 | Timestamp: 1612365745987438848
Acquired with: nan fps
Acquired image: 2 | Timestamp: 1612365746027438848
Acquired with: 12.048192771084338 fps
Acquired image: 3 | Timestamp: 1612365746067439104
Acquired with: 13.333333333333334 fps
Acquired image: 4 | Timestamp: 1612365746107439104
Acquired with: 14.285714285714286 fps
Acquired image: 5 | Timestamp: 1612365746147439104
Acquired with: 14.925373134328359 fps
Acquired image: 6 | Timestamp: 1612365746187439104
Acquired with: 15.384615384615385 fps
Acquired image: 7 | Timestamp: 1612365746227439360
Acquired with: 15.873015873015873 fps
Acquired image: 8 | Timestamp: 1612365746267439360
Acquired with: 16.39344262295082 fps
Acquired image: 9 | Timestamp: 1612365746307439360
Acquired with: 16.666666666666668 fps

I know the issue is with writing video using openCV, if I comment out the code for openCV, than I getting 25 fps, exactly. Additionally, I change my code to save the Images directly instead of video with the following code:

                         if status == cvb.WaitStatus.Ok:
                             
                            np_image = cvb.as_array(image, copy=False)
                            rgb_imag = cv2.cvtColor(np_image, cv2.COLOR_BGR2RGB)
                            cv2.imwrite(imageRawTimeStamp+".bmp",np_image)

For .png format fps is also getting dropped, the best is for .bmp . I am getting 25 FPS but the problem is that it has storage issue. I have 1 TB SSD even I ran out of the DISK space very quickly. I got this information regarding the performance for different formats from this post different-performance-when-saving-different-formats and also tested it locally with different samples programs provided by stemmer imaging https://help.commonvisionblox.com/NextGen/14.0/cvbpy/examples.html

Now, my questions are:

  • How can I skip maybe 10 or 15 frames in the beginning before the actual acquisition?
  • What is the best way to save a multiple videos for 1 hour like for example 6 videos without getting FPS dropped? Also is there any kind of a timer where I can make use to make 10 minutes video (I have checked cvb.StopWatch()) but no success, I am lost. I know that there movie2 for video recording but It does not support Linux.
  • When movie2 will be supported in Linux?
  • Can I get information like how many frames are dropped during the acquisition? I already enabled the option in device setting that not to resend packets but I am curious If I can log this kind of information.

Thanks in advance, any kind of help would be appreciated.
Aftab

1 Like

Hi @aftab

There is nothing to apologize for! We are all here to learn :nerd_face:

You will need to implement that logic for skipping frames in your software. Like:

image, status = stream.wait()
images_acquired += 1
if images_acquired > 15:
    # save image

For longterm recording, the biggest bottleneck is the harddrive performance. If your video doesn’t fit in RAM, you will need to use a high performance SSD that can handle your desired bandwidth.
Another option is saving the images to a video container using some kind of compression. This will decrease the harddisk requirements dramatically but you will need a strong CPU for the encoding (or GPU if your codec supports GPU accelerated encoding)

You know your camera’s FPS - why don’t you simply count your frames and stop after n images?

To use the words of James Hetfield:

Waiting for the one
The day that never comes

(Oh boy I hope Metallica won’t sue me for copyright infringements now… :zipper_mouth_face: )

But maybe @illusive has a more professional answer for us?

Indeed, you can! The stream object has a statistics property:

buffers_lost = device.stream.statistics[cvb.StreamInfo.NumBuffersLost]

Have a look at the full list of available stream statistics:

https://help.commonvisionblox.com/NextGen/14.0/cvbpy/d4/d34/classcvb_1_1_stream_info.html

I hope this helps!

Cheers
Thomas

2 Likes

Thanks, @rtp_derahs and hi again, @aftab

I fear there is not much insight I can add to that. As far as “Movie2 for Linux” is concerned I am very sorry, but that won’t happen anytime soon. Movie2 is built on top of the DirectShow infrastructure of Windows which simply isn’t available on Linux. So it would basically have to be a rewrite based e. g. on something like Video4Linux and I fear that there isn’t a strong business case for that (not much demand as far as I can tell).

As for your performance questions: Much of it has been answered, I guess. However I am not fully on board with the hard disc limitation diagnosis. If you do the math, you are effectively trying to record something like 200 MB/s, which I agree is hard to stomach for a magnetic disc. But you wrote that when writing the data to a sequence of uncompressed *.bmp files then it works - but your disc fills up far too quick, which to me sounds like your harddisc speed is not your problem.

Judging by your FourCC code you are doing DivX encoding. DivX is an excellent codec, but for encoding it is also a very power-hungry one. Some 7 or 8 years back I did a project where we had a dual-HD (3840x1080) RGB stream to encode, 400 MB/s and a big fat Xeon CPU was barely keeping up with that - with all cores close to 100% simultaneously. Not sure how much parallelization support the codec used with OpenCV offers, but to me it sounds like this is your limiting factor.

If that is the case then your best bet would probably be to use something else for encoding. If you can live with consecutively saved images with at least some degree of compression, then try TIFF (it did perform suprisingly well on the tests I did for https://forum.commonvisionblox.com/t/different-performance-when-saving-different-formats/252/4 - but of course its compression i nowhere near what DivX could do for you…

3 Likes

Thank you so much for the detailed and informative feedback @rtp_derahs @illusive.

I am trying to play around with different formats and for sure I will get back as I am able to resolve the issue with FPS. But, yes Mr @illusive, you have pointed rightly that codec used with OpenCV is causing the problem. I came to this conclusion because I have posted a question on the OpenCv forum here and was hinted about the problem.

Given the fact, I cannot tolerate to lose any kind of information due to compression, while on the other side, the storage issue is also problematic as well as the OpenCV. I Know there is no “One size fits all” solution, but I have to take the middle path.

I will give a try to play with Tiff format and OpenCV on how to overcome this issue trying to check other codec and I will get back to you guys.

Thank you and have a nice week. Stay safe and blessed.

Regards,
Aftab

2 Likes