Firstly, a little background before I start sketching our problem:
We have an application which contains 9 Jai-Go 5000M-PGE cameras. These camera’s are divided in two camera groups (G1:4 cams & G2:5 cams) and both camera groups are triggered separately. The time between two triggers is around 4-5 seconds (so a really low frame rate). The camera’s are connected with GigE to managed Cisco switch and then to a quadport which is installed on the backside of our Windows PC (Intel i5 processor). The camera’s are evenly divided over the quadport (2-2-2-3).
We configured the settings of the camera’s in Common Vision Blox Manager as described below:
Resolution: 1280 x 1024 pixels
Buffer Size: 3
Monochrome camera (1pixel=1byte)
Packet Size: 8192 bytes/frame
Timestamp Thick Frequency Unit: 62 500 000
Packet delay varies from camera to camera:
C1: 15000 timestamp thick units
C2: 17000
C3: 19000
…
C9: 31000
We played already a lot with the packet delay.
The adapter settings:
Jumbo packets: 9014 bytes
Receive buffers: 2048 bytes
Interrupt moderation Rate: Extreme
Problem:
Our current image shows strokes of pixels of three images ago in our buffer as our buffer size equals 3. So, the buffer is not cleared before writing the new image in it. Otherwise we should see strokes of black pixels. We played already a lot with the packet delay but this doesn’t seems to solve our problem. We also checked the bandwidth of the switch which seems enough.
Hello @SimDest welcome to our forum, and thank you for the detailed description.
Normally this is a question for our technical support, because this issue is not really CVB related.
But as a lot of people are facing such problems, I will try to give you some advice here as well.
Lost packets and Bandwidth Issue
At first, it really looks like lost packets. At least if you rotated the image because the stripes should be horizontal not vertical as in your image.
These lost packets using several GigE cameras are mostly related to a bandwidth issue.
Based on your description, it seems that you already did everything to prevent this.
For everyone else who reads this. A low frame rate does not mean that you can summarize the overall bandwidth related to the low frame rate. By default, the cameras send new images over the network as fast as possible. Even if this happens only once per minute. If every camera connected over the same network does this at the same time, we could reach the bandwidth limit of GigE and then get lost packets.
Interrupt Moderation
Back to your issue. You already played with the packet delay to lower the transfer bandwidth.
One thing which is worth testing is to deactivate the Interrupt Moderation. Normally, dedicated network cards do not have such an issue from our experience. We saw an issue related to the Interrupt Moderation only on compact PCs with integrated network chips.
But it is easy to test and worth a try.
PC Bus Bandwidth
Even if you lowered the transferred bandwidth to a value that every camera connected to one NIC could transfer all its data, you could reach another bottleneck. The bus on your PC where the network card is connected to. Please check what’s your available bus bandwidth on your system.
When you know that, you can raise the packet delay to values that all cameras together do not exceed this bandwidth limit.
Approach to find a suitable packet delay.
Know your Bandwidth limit (depending on Network card on one NIC around 110MB/s)
change the transferred bandwidth in the camera divided by the number of connected cameras to one NIC. A lot of cameras already have a feature to do that directly and the packet delay is calculated and set automatically (e.g. StreamBytesPerSecond)
On this JAI camera, it is possible that you do not have such a value. Then you need to find a suitable value by letting the camera acquire images in free running mode and raise the packet delay until you reach a frame rate which is lower than your target bandwidth.
**Do not forget to reduce the Frame rate setting in the camera to a lower value in the end if you have a free running setup to avoid other issues **
If you have a PC Bus Bandwidth issue the same approach from point 1 and 2 need to be made by including all cameras connected to the network card not only connected to one NIC.
just a little addition to @Sebastian s answer here:
I had customers dealing with the same problem as you currently are and I was told, that having one VLAN per Camera can also help avoiding collisions.
Maybe this is something for you to consider as well.
First of all, thanks for the advice you gave me, as we tried some topics you proposed. Here some feedback on our problem:
After much troubleshooting, we discovered that the NIC (Intel(R) Ethernet Converged Network Adapter X710-T) was the component which caused these problems. We subjected this NIC to different tests and under different settings, but the strokes of pixels of 3 images ago in our buffer kept on returning.
We were able to simulate this phenomenon more frequently by applying a tool which caused a higher CPU&disk load.
We did the same test for these two NICs:
Intel ET2 Quad Port Server Adapter (a NIC we normally always use, and never had any issues with)
Intel I350 Gigabit Network Connection
Both NICs seems to perform better and we weren’t able to simulate the ‘sticking’ pixels even under a high CPU&Disk load.
