Camera disconnects on app shutdown after specific ordering of property access

I’m seeing some unusual behaviour after exiting my application. I am running with Ubuntu 1804 on ARM64 with two JAI USB cameras.

After my app has exited, if I run “lsual” I get an error from one of the cameras which reads “error: in CVUAL message: failed to read basic U3V info.”.

If I run “lsual” again it appears as expected but this time the address is 23 not 21 (as in the image above - I can only put one image in a post as a new user) - this number keeps increasing each time we perform this test cycle.

If I don’t run ‘lsual’ twice then Discover fails to find at least one of the cameras (but that doesn’t stop seem to property access? At least not consistently.)

I’ve reduced this down to an example that consistently fails and one that consistently works… I don’t understand why there should be such a difference in behaviour…

  1. (works) Open camera 1, Access camera 1 properties, Open camera 2, Access camera 2 properties
  2. (fails) Open camera 1, Open camera 2, Access camera 1 properties, Access camera 2 properties

I’ve included a compilable code example below…

#include <iostream>

#include <cvb/device_factory.hpp>
#include <cvb/device.hpp>
#include <cvb/driver/stream.hpp>
#include <cvb/driver/software_trigger.hpp>
#include <cvb/driver/stream_image.hpp>
#include <iCVCUtilities.h>

const std::string DriverPath =
    Cvb::InstallPath() +
    CVB_LIT("drivers/GenICam.vin");

const int Camera1Port = 0;
const int Camera2Port = 1;
const int BoardNumber = 0;

using namespace Cvb;

void PrintProperties(DevicePtr camera)
{
    auto properties = camera->NodeMap(CVB_LIT("Device"));
    auto idNode = properties->Node<StringNode>("DeviceUserID");
    std::cout << idNode->Value() << std::endl;
}

// Doesn't fail
void Test_A()
{
    auto discoveredDevices = DeviceFactory::Discover();
    std::cout << "Discovered " << std::to_string(discoveredDevices.size()) << " device(s)" << std::endl;

    auto camera1 =
        DeviceFactory::Open(
            DriverPath,
            Camera1Port,
            BoardNumber
        );

    PrintProperties(camera1);

    auto camera2 =
        DeviceFactory::Open(
            DriverPath,
            Camera2Port,
            BoardNumber
        );

    PrintProperties(camera2);

    PrintProperties(camera1);
    PrintProperties(camera2);
}

// Fails
void Test_B()
{
    auto discoveredDevices = DeviceFactory::Discover();
    std::cout << "Discovered " << std::to_string(discoveredDevices.size()) << " device(s)" << std::endl;

    auto camera1 =
        DeviceFactory::Open(
            DriverPath,
            Camera1Port,
            BoardNumber
        );

    auto camera2 =
        DeviceFactory::Open(
            DriverPath,
            Camera2Port,
            BoardNumber
        );

    PrintProperties(camera1);
    PrintProperties(camera2);
}

// Running lsual after test_fails() sees one of the cameras reset...
// Running lsual after test_no_failure() works fine...
int main(int argc, char** argv)
{
    if (argc == 0) goto fail;
    else
    {
        std::string option(argv[1]);
        if (option == "A")
            Test_A();
        else if (option == "B")
            Test_B();
        else goto fail;

        std::cout << "Now run 'lsual'" << std::endl;
        return 0;
    }

fail:
    std::cout 
        << "Invalid arguments provided. "
        << "Use 'A' = Test A (no failure) or 'B' = Test B (fails)";
    return -1;
}

Hi matt,

it is possible that the camera is still opened/not closed fast enough after your property query. Thus when calling lsual, the device is still open and the call fails.

When you run your second test (which fails) the program internally checks all cameras for a license, then opens the first camera, checks again all devices for a license before opening the second camera, reads properties from cameras etc. This license checking process can slow down the closing of the device, which in most cases is not a big deal. If you cannot use the working first test case code in the order that worked, you can try to do a Cvb::Utilities::SystemInfo::WaitForLicense(std::chrono::duration) for the license to be checked beforehand, so no further checks will be applied to the device opening if the license was already successfully found.

Your statements here raise all sorts of questions for me about how to use this API in a way that ensures consistent behaviour but I’ll respond first to your suggestion of adding SystemInfo::WaitForLicense…

I have tried this first at the beginning of the main() method and second after the Discover() and neither of these have changed the outcome. Have I understood the correct place to put the call?

I’m still seeing the error I reported. If I run lsual several times the address of (at least one) camera on the USB bus changes and both cameras appear.

Hi Matt,

This seems to suggest that a disconnect happens at some point.
(Don’t know if that is happening, so this is not a diagnosis yet)

Like @usernv said: Only one process may access any USB camera and there is a delay in closing devices (CVB no longer has access to the cam, but the Linux system may still have some open/blocked handles).
How much time passes in your test cycle? I.e. app exited then called lsual?

Please check via “dmesg -w” if something weird/bad happens on the kernel side. The -w parameter is important because of the timing of the error.

Yes, the call should be at the beginning of the main function. I.e. before discover.
But the call is successful? And no timeout?

As a side note:
Are you using Camerasuite licenses? I.e. /opt/cvb/camsuite.lic
If yes, how many license entries are there? (too many may delay opening camera, which may cause timeout issues on slow usb controllers)

@matt maybe you discovered a bug which might be beyond the scope of the forum. Could you please get in touch with our support-team with your issue?

de.support@stemmer-imaging.com

You can refer to this thread and please also write us your CVB version, used hardware (cameras and board) and (if possible without too much trouble) attach the exported log file from the LogGUI (set Configuration: module --> default to debug and export the log as JSON).

We will then further investigate and get back to you.

Many thanks in advance!

Thanks. I will send something across today.

I’ve sent a couple of updates via the support email but haven’t had any confirmation that it has been received - would it be possible to get an update? (even if that is simply “we’ve received it”). Thanks. Kind regards.

Hi @matt,
Sorry for the delayed reply. We received your request and are working on it. Please notice that due to the current circumstances (Corona) and the complexity of the request (multiple cameras on an embedded system with a possible licensing issue on Ubuntu) it might take a while for us to find a solution since we will recreate your issue on the same hard- and software. Hopefully you can until then use the first working code case so you will not be blocked in your work by the issue.

Best,
Nico

Thanks for the reply. I understand that this will take some time to investigate - the combinations of hardware and software are tricky - I needed to spend some time already to cut the example down to as far as I have. Unfortunately the code above represents a cut-down version of our software and the “working” form will be difficult to implement. If you need any more information please let me know.

For anyone else who comes across this problem: I have been in touch with the support team who have been able to reproduce the problem (potentially with some extra information not available on this forum). As of this moment they have not been able to find a solution.