Drawing arbitrary content as an overlay

Programming Questions
1

Hi,

I intend to perform arbitrary mouse painting on a CVB forms display. From my understanding, that should be possible with a User Overlay.

What’s the most simple path to get there in a way that

  • is non-destructive to the image (ie. enables painting in a dedicated plane) and
  • considers non-marked pixels as transparent (so the underlying planes remain visible)?

Another use case I’d like to cover is combining a binary mask with an image in a Forms display.
How can I add a binary mask as an image plane and display it in a way (again Forms display) where only 255 is shown in a given color while 0 is handled as transparent?

2

Hi @MarSch,

yes this should be possible with the CVB overlay functionality. You should look at the following tutorial:
%CVB%Tutorial\Image Manager\VB.Net\VBOverlayObject.NET

Or the following tutorial. There for the ellipse drawing single pixels are drawn.
%CVB%Tutorial\Foundation\CSharp\CSEllipseFit

But note that these tutorials are not written with the new wrappers.

To use a binary mask you can simply multiply the image with the mask image using the CVB Foundation MultiplyImages method.

3

Hi Arno,

thank you for the references. I already used overlays in the past in partly creative ways, but there seem to be some hard limits. E.g. the code at the bottom below works kind of, but isn’t quite the optimum.

What I can’t find is

  1. an Integrate() function in the Overlay class which allows the integration of an overlay into an existing one.

  2. a way to create/pull an image from one or several overlays, e.g. OverlayToImage(List overlays, Color transparentValue).

  3. a straightforward way to convert a binary image into an overlay with a value that’s handled as transparent.

    The options I can see are

    • first creating a point list by linear access and writing out single coordinates into a point array, which is called with e.g. UserOverlay (in theory, but I can see no actual overlay graphics in my test)
    • using the BitmapOverlay (while the conversion to the system type is somewhat quirky)
    • using Cvb.Forms.Overlays.ImageOverlay().

    However both BitmapOverlay and ImageOverlay seem to lack a transparency feature.

    For BitmapOverlay, it doesn’t seem to honor the transparency key of the Bitmap. The areas with the transparency value are displayed in a medium gray when such a bitmap is used as an overlay.

    For the ImageOverlay, neither 255 or 0 are displayed transparent, and I can’t find a parameter to set such a value either.

I’d be very happy if you could show me a way to get those things done in CVB. I had quite a close look at the documentation, however it still might be that I overlooked something or didn’t understand it properly.

private void mainDisplay_MouseDown(object sender, MouseEventArgs e)
{
  isDrawing = true;
}

private void mainDisplay_MouseUp(object sender, MouseEventArgs e)
{
  isDrawing = false;
}

private void mainDisplay_MouseMove(object sender, MouseEventArgs e)
{
  double[] valueAtPos = new double[1];
  imagePos = mainDisplay.ClientToImage(new System.Drawing.Point(e.X, e.Y));
  valueAtPos = mainDisplay.Image.GetPixel(imagePos.X, imagePos.Y);

  toolStripLabelPos.Text = 
    "  |  X, Y, Value {" + imagePos.X.ToString() + " , " + imagePos.Y.ToString()  + " , " + valueAtPos[0].ToString() + " }";

  if (isDrawing)
  {
    var drawingPoints = new System.Drawing.Point[] { imagePos };
    mainDisplay.Overlays.Add(new Cvb.Forms.Overlays.CircleOverlay("Circle", false, System.Drawing.Color.Red, false, true, drawingPoints[0], 5));
  }
}
4

Hi @MarSch!

What you want to achieve is indeed somewhat tricky. The overlay objects by and large are basically wrappers around the overlay plugins formerly used by the CVDisplay.ocx - wrapped as conveniently as possible for use in a C# application. However, there were subtle limits to that. For example as soon as you create a System.Drawing.Graphics object that wraps the DC of the :cvb: GDI display, transparency and XOR painting will stop working properly (whoever tried it did soon notice that once a System.Drawing.Graphiccs object has been created, overlays that are dragged around on the screen will start leaving a trail behind them). We’re not entirely sure what causes this, but XOR painting is probably a bit too '90s for .Net :wink:.

But let me address you points bit by bit…

The current architecture of the overlay plugins does not really allow for that. They are independent COM objects that only communicate with the Display object, but not among themselves (and as this is not present in the unmanaged layer it cannot be simply modeled in the managed wrappers). The closest thing one could create would be some kind of container plugin that’s capable of accumulating other plugins. But given the points further down this approach doesn’t seem reasonable.

This is hampered for the same reason, plus if we want the overlays to be paintable to an image, we’ll need to implement a means for them to paint themselves to a memory bitmap instead.

Same as above. As soon as the .Net runtime has been involved our unmanaged display doesn’t apply transparency properly any more (for example if with the CVDisplay.ocx just specify a file name and the transparency key in the members lpstrFileName and dwTransparent it’ll work, but with a bitmap handle set from a System.Drawing.Bitmap it won’t.

As for the options you suggest

This is tedious, but strictly speaking something like this is possible with the UserOverlay class in Stemmer.Cvb.Forms.dll. This is basically a wrapper for the former UserObject functionality on the CVDisplay.ocx, so it’s basically an object that will handle the points for you and call a delegate whenever the actual painting is required. To circumvent the issues with the ROP-mode mentioned earlier we have added the objects in the Stemmer.Cvb.Aux.Drawing namespace (located in the Stemmer.Cvb.Aux.dll). These simply map some of the more frequently used DC drawing functions of GDI to .Net without passing through System.Drawing.Graphics. Extending the set here would not generally be a problem. The bar for having these paint to a System.Drawing.Graphics object should not be too high either. For a look at the UserObject have a loot at the Cvb.Net Display tutorial (MainForm.cs, line 140 would be a good entry point).

This will not solve the transparency issue, so I don’t think this is a viable approach.

Also not a desirable approach because the concept of transparency would have to be introduced for this object first (currently it just uses the ImageToDC function which doesn’t have that). Plus: The set of painting functions on Image objects is actually more limited than what is available on Stemmer.Cvb.Aux.Drawing.UnmanagedGraphics.

To summarize, from my point of view that leaves only two options, one of them being only semi attractive and the other hitherto unmentioned:

  1. Make use of the classes in Stemmer.Cvb.Aux.Drawing namespace to achieve the desired effect. Only an option of the limited set of features is not too far away from what’s needed and the ctor would currently have to be called through reflection if this should be used on a System.Drawing.Graphics object that refers to a System.Drawing.Image.
  2. Instead of the Forms display, use the WPF display inside an ElementHost object. The WPF display integrates more closely with the .Net runtime than overlay objects of the forms display do (the latter always go through the underlying COM and CVCDisplay.dll infrastructure which limits their possibilities somewhat). With the WPF display you can select freely from the range of available features and use them for overlays. You can also easily use Bitmap objects (which can be painted arbitrarily) with the display including proper transparency. For a hint on how this may be used I recommend having a look at the “ClassifierTests” example for Polimago (specifically the CreateColorOverlay and the binding of the resulting bitmap).

I hope that helps a little further with your use case. If things are unclear or you want to discuss further you know how to reach us :slight_smile:

5 Likes