Mismatch in number of images delivered VS FPS of line scan camera

Dear Team,
I have a SW4000Q line scan camera which delivers RGB and NIR streams. I noticed strange behavior when operating the camera at maximum resolution. The maximum resolution the camera can handle is 4096x4096. So at this resolution, I can set the line rate to a maximum value of 72 kHz for an exposure time of 10 us. With this settings the FPS of camera comes to around 16 to 17 FPS. Now, using CVB++, I am trying to acquire the images from both the streams for a total of 10 secs. Ideally I would expect the camera to deliver 160 images per stream. But, the camera only delivers around 80 to 90 images per stream. This behavior is not seen at lower resolutions like 2560 x 2048. If I reduce the resolution, for the maximum line rate the FPS goes upto 33. I also see that 330 images per stream getting delivered from the camera which is as expected.

Any help regarding this is appreciated

Hi @keerthitheja

I am not entirely sure what the cause is, but one thing that strikes me as odd is the transfer rates involved here.

  • Your expectation is to transfer 4096 x 4096 @17 fps. Now 4096 x 4096 x 4 = 64 MB per frame. At 16 frames per second we end up at 1 GB per second. This is roughly the theoretical limit for a 10 GigE connection.
  • What you did receive at this resolution is about 9 fps, so roundabout 576 MB per second.
  • Reducing to 2560 x 2048 pixels (20 MB per frame) gets the frame rate up to 33 fps, which is roundabout 660 MB per second.

So the total transfer rate at reduced resolution is practically more consistent with the 9 fps at full resolution than with the theoretical maximum data rate of 1 GB per second.

Generally speaking, our experience with 10 GigE is the following:

  • Under benign circumstances (decent 10 GigE adapter, powerful CPU, good wiring) you can expect to see around 800 to 900 MB per second.
  • Not all adapter brands perform equally well (:arrow_forward: What adapter are you using?)
  • Depending on the adapter brand there are a few settings that can be tried out to improve performance (like reducing the reserved bandwidth for video streaming)
  • The overall overhead of UDP + GEV can be a bit of a burden for the CPU to handle. With the average i3 or i5 CPU we’ve seen situations where the CPU limits the throughput at something between 50% and 65% of the theoretical limit (:arrow_forward: what CPU are you using?)

Can you provide a bit more detail on your setup?

Hello @keerthitheja,

as already told there are several factors which might limit you maximum framerate and bandwidth.

1. Hardware Limitations

First you should be aware about the topology of your mainboard and CPU you are using.

  • PCIe version?
  • amount of used lanes?
  • who is providing the lanes, chipset or CPU?
  • system memory type, amount and connections? (DDR4, dual channel, etc.)
  • NIC performance settings configured?
  • does the amount of buffers suit the current framerate?
  • Jumbo Frames configured in driver on the host system?

The following image is showing a very simplified representation of a PC system using Intel Core CPUs.

Only the PCIe bus provided by the CPU can deliver appropriate bandwidths for image acquisition. The chipset of some mainboards are providing PCIe lanes as well. But those are most likely very limited in the bandwidth. Some lanes could even share the bandwidth with other lanes also resulting in a reduced bandwidth.
Because of the used Ethernet protocol CPU utilization could be very high when using 10GigE resulting in packet loss. We are recommending the following settings for the NIC in order to keep CPU load low.

1.1 NIC Configuration

We want to maximize the size of each Ethernet packet carrying image data. The supported packet sizes of the NIC can be found in the settings of the NIC. In order to keep CPU utilization low we have to maximize the size of each Ethernet packet carrying image data (setting: Jumbo Frame).

In addition we have to minimize the amount of interrupts the NIC could perform and therefore disrupt the complete system, even on multicore systems (setting: Interrupt Moderation Rate).

The last very important setting is the amount of receive buffers. We recommend to maximize the amount of receive buffers in the NIC settings.

1.2 Driver / Camera Configuration

Up to this point we have just enabled the usage of so called Jumbo Frames. We have to tell the driver on the host system to actually use the large packet size. The image of the CVB GenICamBrowser is showing how to configure the packet size used by the camera.

The packet size must be divisible by four with no remainder. When the NIC is supporting a packet size of 9014 Bytes I would recommend to set the packet size to 8192 Bytes in CVB.

Please be aware of that the packet size can’t be stored in the user settings of the camera. The settings must be stored in the driver settings, e.g.: GenICam.ini file.

2. Software Limitations

Beside hardware limitations Windows could also be resonsible to limit you bandwidth. Windows could throttle link bandwidth in order to lower CPU load and prioritize multimedia applications. Please note that this setting is not related to QoS (Quality of Service) mechanism of Windows.

The throttling of the bandwidth can be disabled by using the registry key NetworkThrottlingIndex. This key is stored under the following path.

Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Multimedia\SystemProfile

For HEX value 0xffffffff thorttling is diabled. To enable throttling again the HEX value 0xa has to be used.

3. CVB Statistics
CVB could output various statistics helping to determine the bottleneck of the system. The following image is showing the statistics when system performance isn’t sufficient to sustain the frame rate. The driver is receiving corrupt or incomplete images.

e.g.:
grafik

When the NIC is receiving corrupt images possible reason is a bad Ethernet cable. In this case the packets are already corrupt when received.

Another possible reason for packet loss on the NIC is that all internal buffers have been already used. In this case new received packets have to be discarded.

When using CVB we could get another statistics directly from the acquisition driver. CVB is providing a command line tool in the subfolder “%CVB%\3Hardware\StemmerImaging\Utilities”. Name of the tool is “GevFDDiag.exe”.

e.g.:
Device Index: 4
Device Handle: 0000000000000110
Virtual Name: \DEVICE{70E47909-8896-4A83-95B3-1C9B342BAF81}
Device Name: {70E47909-8896-4A83-95B3-1C9B342BAF81}-{3798D586-26C4-43C8-922E-17DB536F3A1B}-0000
Vendor Description: " Aquantia AQtion Network Adapter"
Medium: 802.3
MAC Address: 78-D0-04-2C-65-AD
Link Speed: 0

Device Index: 4
SendNetBufferListsExternal: 19170
SendNetBufferListsInternal: 3
SendNetBuffersInternal: 3
ReceiveNetBufferLists: 0
ReceiveNetBuffers: 16077827
ReceiveBytesTotal: 24327660988
ReceiveIP4PacketsTotal: 16076855
ReceiveUDPPacketsTotal: 15914627
ReceiveUDPBytesTotal: 24085072702
ReceiveUDPPacketsDropped: 15914627
ReceiveUDPBytesDropped: 24085072702

Device Index: 4
Vendor Description: " Aquantia AQtion Network Adapter"
MAC Address: 78-D0-04-2C-65-AD
OID_GEN_SUPPORTED_LIST = not supported
OID_GEN_HARDWARE_STATUS = 0
OID_GEN_MEDIA_SUPPORTED = 0
OID_GEN_MEDIA_IN_USE = 0
OID_GEN_MAXIMUM_LOOKAHEAD = 512
OID_GEN_MAXIMUM_FRAME_SIZE = not supported
OID_GEN_LINK_SPEED = not supported
OID_GEN_TRANSMIT_BUFFER_SPACE = 16740352
OID_GEN_RECEIVE_BUFFER_SPACE = 66945060
OID_GEN_TRANSMIT_BLOCK_SIZE = 16348
OID_GEN_RECEIVE_BLOCK_SIZE = 16348
OID_GEN_VENDOR_ID = 738513016
OID_GEN_VENDOR_DESCRIPTION = not supported
OID_GEN_CURRENT_PACKET_FILTER = 11

OID_GEN_DRIVER_VERSION = 1586
OID_GEN_MAXIMUM_TOTAL_SIZE = 16348

OID_GEN_MAC_OPTIONS = 717
OID_GEN_MEDIA_CONNECT_STATUS = not supported
OID_GEN_MAXIMUM_SEND_PACKETS = 4096

OID_GEN_XMIT_OK = 1234954
OID_GEN_RCV_OK = 25270724966
OID_GEN_XMIT_ERROR = 0
OID_GEN_RCV_ERROR = 0
OID_GEN_RCV_NO_BUFFER = 21474804480

OID_GEN_DIRECTED_BYTES_XMIT = 157086
OID_GEN_DIRECTED_FRAMES_XMIT = 2454
OID_GEN_MULTICAST_BYTES_XMIT = 9047
OID_GEN_MULTICAST_FRAMES_XMIT = 98
OID_GEN_BROADCAST_BYTES_XMIT = 1069013
OID_GEN_BROADCAST_FRAMES_XMIT = 16628
OID_GEN_DIRECTED_BYTES_RCV = 25270725570
OID_GEN_DIRECTED_FRAMES_RCV = 16656756
OID_GEN_MULTICAST_BYTES_RCV = 0
OID_GEN_MULTICAST_FRAMES_RCV = 0
OID_GEN_BROADCAST_BYTES_RCV = 64
OID_GEN_BROADCAST_FRAMES_RCV = 1
OID_GEN_RCV_CRC_ERROR = 0
OID_GEN_TRANSMIT_QUEUE_LENGTH = 0
OID_GEN_GET_TIME_CAPS = not supported
OID_GEN_GET_NETCARD_TIME = not supported
OID_GEN_NETCARD_LOAD = not supported
OID_GEN_DEVICE_PROFILE = not supported

OID_GEN_RECEIVE_SCALE_PARAMETERS = not supported
OID_GEN_RECEIVE_SCALE_CAPABILITIES = not supported

We are especially interested in the following counters.

OID_GEN_XMIT_OK = 1234954
OID_GEN_RCV_OK = 25270724966
OID_GEN_XMIT_ERROR = 0
OID_GEN_RCV_ERROR = 0
OID_GEN_RCV_NO_BUFFER = 21474804480

When the counter OID_GEN_RCV_ERROR is incremented then the packets are already corrupted on the Ethernet cable. When counter OID_GEN_RCV_NO_BUFFER is incremented then all buffers of the NIC are had been in usage.

CPU load

The second most likely reason for packet loss is the CPU itself. Because of the used protocol each packet is causing little CPU load. Please check the CPU usage using Windows Resource Monitor.

e.g.:
grafik

You can also use some third party tools like HWiNFO64 to gather more information’s about the system. A bunch of values concerning RAM and PCIe connection are shown. In addition the speed of the former front side bus which is called Ring Bus on some Intel Core CPUs.

e.g.:

The tool is outputting some sensor values as well. The following values indicate if the internal BUS of the CPU or graphics chip is limiting.

e.g.:
grafik

4. Auto negotiation

NIC and camera are auto negotiating the link speed. Please check the negotiated link speed.

e.g.:
grafik

3 Likes

Hi @illusive

We are currently using Intel(R) Ethernet Controller X550 10 gigE adapter.

The CPU we are using is Intel i7-9700TE-CPU @ 1.8 GHz
We have a total RAM of 32 GB.

Hi @KVo and @illusive
I followed the instructions that you mentioned. I am seeing a strange behavior with the camera when set to maximum resolution of 4096x4096. So, after a power cycle, I can open and close the camera with the default setting in it. When i set the resolution to 4096x4096 and maximum line rate possible, and do the acquisition, the camera behaves normally. But, when I try to open the camera again with the settings inside the camera, the genicam browser tries to open the camera but it fails and the camera stops responding. You can see the camera stays open and there is a cross mark on the camera icon in the configured devices tab.

I see this issue occurring only with the GeniCam Browser and CVB (even with CVB++). When I try the same thing with Jai control Tool, the camera opens and closes normally as expected. I have been facing this issue for the past 2 weeks.

@illusive Could this be happening because of the issue I raised here ?

https://forum.commonvisionblox.com/t/multi-stream-gige-vision-camera-through-put-is-strange/1732?u=keerthitheja

Hi @keerthitheja

this with the LAG setup right? Or without?

hi @c.hartmann
This happens with and without LAG setup.

Hi, It is possible to dedicate a specific CPUs to Jai Program, which they usually recommend that. Do you use this setting?
How many physical CPUs do you have? Is Hyper-threading activated?

Hi @MandanaS
I have 8 cores in my processor. Currently, i dont think we are dedicating any specific cores to the JAI program. I can try this setting. Let me also check if hyper threading is enabled

@keerthitheja I’m realy sorry for my dellayed reply. Two things are corious in the screenshot showing GenICamBrowser.

  1. Camera is already occupied by another process
  2. CVB filter driver isn’t present any more. Other screenshots have shown that this driver is installed in general.

First thing to check is if there is a leftover from the previous section in Windows Task Manager. In case ther is no leftover from the previous session you could try deleting the subfolders under C:\ProgramData\boost_interprocess

e.g.:
PS C:\ProgramData\boost_interprocess> ls

  • Verzeichnis: C:\ProgramData\boost_interprocess*

Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 29.06.2022 11:17 1656331749
d----- 29.06.2022 15:04 1656490355
d----- 29.06.2022 16:29 1656511441
d----- 30.06.2022 09:27 1656518115
d----- 30.06.2022 12:44 1656585354
d----- 30.06.2022 13:15 1656586183
d----- 30.06.2022 13:40 1656587811
d----- 05.07.2022 09:59 1656945815

When the problem still persists using CVB LogGUI would help greatly. I will cover the LogGUI in a later post if you like.