# Drawing arbitrary content as an overlay

**URL:** <https://forum.commonvisionblox.com/t/drawing-arbitrary-content-as-an-overlay/1282>\
**Category:** Programming Questions\
**Created:** [December 3, 2020, 9:29am UTC](https://forum.commonvisionblox.com/t/drawing-arbitrary-content-as-an-overlay/1282 "2020-12-03T09:29:54Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![MarSch](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.commonvisionblox.com/marsch/32/23_2.png) [@MarSch](https://forum.commonvisionblox.com/u/MarSch)\
**Post date:** [December 3, 2020, 9:29am UTC](https://forum.commonvisionblox.com/t/drawing-arbitrary-content-as-an-overlay/1282/1 "2020-12-03T09:29:54Z")

</div>

Hi,

I intend to perform arbitrary mouse painting on a CVB forms display. From my understanding, that should be possible with a User Overlay.

What’s the most simple path to get there in a way that

- is non-destructive to the image (ie. enables painting in a dedicated plane) and
- considers non-marked pixels as transparent (so the underlying planes remain visible)?

Another use case I’d like to cover is combining a binary mask with an image in a Forms display.  
How can I add a binary mask as an image plane and display it in a way (again Forms display) where only 255 is shown in a given color while 0 is handled as transparent?

---

<div class="post-metadata">

**Author:** ![Arno](https://avatars.discourse-cdn.com/v4/letter/a/a5b964/32.png) [@Arno](https://forum.commonvisionblox.com/u/Arno)\
**Post date:** [December 3, 2020, 10:05am UTC](https://forum.commonvisionblox.com/t/drawing-arbitrary-content-as-an-overlay/1282/2 "2020-12-03T10:05:54Z")

</div>

Hi @MarSch,

yes this should be possible with the CVB overlay functionality. You should look at the following tutorial:  
`%CVB%Tutorial\Image Manager\VB.Net\VBOverlayObject.NET`

Or the following tutorial. There for the ellipse drawing single pixels are drawn.  
`%CVB%Tutorial\Foundation\CSharp\CSEllipseFit`

But note that these tutorials are not written with the new wrappers.

To use a binary mask you can simply multiply the image with the mask image using the CVB Foundation MultiplyImages method.

---

<div class="post-metadata">

**Author:** ![MarSch](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.commonvisionblox.com/marsch/32/23_2.png) [@MarSch](https://forum.commonvisionblox.com/u/MarSch)\
**Post date:** [December 3, 2020, 12:17pm UTC](https://forum.commonvisionblox.com/t/drawing-arbitrary-content-as-an-overlay/1282/3 "2020-12-03T12:17:37Z")

</div>

Hi Arno,

thank you for the references. I already used overlays in the past in partly creative ways, but there seem to be some hard limits. E.g. the code at the bottom below works kind of, but isn’t quite the optimum.

What I can’t find is

1. an Integrate() function in the Overlay class which allows the integration of an overlay into an existing one.

2. a way to create/pull an image from one or several overlays, e.g. OverlayToImage(List overlays, Color transparentValue).

3. a straightforward way to convert a binary image into an overlay with a value that’s handled as transparent.

I’d be very happy if you could show me a way to get those things done in CVB. I had quite a close look at the documentation, however it still might be that I overlooked something or didn’t understand it properly.

```
private void mainDisplay_MouseDown(object sender, MouseEventArgs e)
{
  isDrawing = true;
}

private void mainDisplay_MouseUp(object sender, MouseEventArgs e)
{
  isDrawing = false;
}

private void mainDisplay_MouseMove(object sender, MouseEventArgs e)
{
  double[] valueAtPos = new double[1];
  imagePos = mainDisplay.ClientToImage(new System.Drawing.Point(e.X, e.Y));
  valueAtPos = mainDisplay.Image.GetPixel(imagePos.X, imagePos.Y);

  toolStripLabelPos.Text = 
    " | X, Y, Value {" + imagePos.X.ToString() + " , " + imagePos.Y.ToString() + " , " + valueAtPos[0].ToString() + " }";

  if (isDrawing)
  {
    var drawingPoints = new System.Drawing.Point[] { imagePos };
    mainDisplay.Overlays.Add(new Cvb.Forms.Overlays.CircleOverlay("Circle", false, System.Drawing.Color.Red, false, true, drawingPoints[0], 5));
  }
}
```

---

<div class="post-metadata">

**Author:** ![illusive](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.commonvisionblox.com/illusive/32/61_2.png) [@illusive](https://forum.commonvisionblox.com/u/illusive)\
**Post date:** [December 4, 2020, 8:41am UTC](https://forum.commonvisionblox.com/t/drawing-arbitrary-content-as-an-overlay/1282/4 "2020-12-04T08:41:40Z")

</div>

Hi @MarSch!

What you want to achieve is indeed somewhat tricky. The overlay objects by and large are basically wrappers around the overlay plugins formerly used by the CVDisplay.ocx - wrapped as conveniently as possible for use in a C# application. However, there were subtle limits to that. For example as soon as you create a `System.Drawing.Graphics` object that wraps the `DC` of the :cvb: GDI display, transparency and XOR painting will stop working properly (whoever tried it did soon notice that once a `System.Drawing.Graphiccs` object has been created, overlays that are dragged around on the screen will start leaving a trail behind them). We’re not entirely sure what causes this, but XOR painting is probably a bit too '90s for .Net 😉.

But let me address you points bit by bit…

> [@MarSch](#):
>
> What I can’t find is
> 
> 1. an Integrate() function in the Overlay class which allows the integration of an overlay into an existing one.

The current architecture of the overlay plugins does not really allow for that. They are independent COM objects that only communicate with the Display object, but not among themselves (and as this is not present in the unmanaged layer it cannot be simply modeled in the managed wrappers). The closest thing one could create would be some kind of container plugin that’s capable of accumulating other plugins. But given the points further down this approach doesn’t seem reasonable.

> [@MarSch](#):
>
> 1. a way to create/pull an image from one or several overlays, e.g. OverlayToImage(List overlays, Color transparentValue).

This is hampered for the same reason, plus if we want the overlays to be paintable to an image, we’ll need to implement a means for them to paint themselves to a memory bitmap instead.

> [@MarSch](#):
>
> 1. a straightforward way to convert a binary image into an overlay with a value that’s handled as transparent.

Same as above. As soon as the .Net runtime has been involved our unmanaged display doesn’t apply transparency properly any more (for example if with the CVDisplay.ocx just specify a file name and the transparency key in the members `lpstrFileName` and `dwTransparent` it’ll work, but with a bitmap handle set from a `System.Drawing.Bitmap` it won’t.

As for the options you suggest

> [@MarSch](#):
>
> - first creating a point list by linear access and writing out single coordinates into a point array, which is called with e.g. UserOverlay (in theory, but I can see no actual overlay graphics in my test)

This is tedious, but strictly speaking something like this is possible with the `UserOverlay` class in Stemmer.Cvb.Forms.dll. This is basically a wrapper for the former `UserObject` functionality on the CVDisplay.ocx, so it’s basically an object that will handle the points for you and call a delegate whenever the actual painting is required. To circumvent the issues with the ROP-mode mentioned earlier we have added the objects in the `Stemmer.Cvb.Aux.Drawing` namespace (located in the Stemmer.Cvb.Aux.dll). These simply map some of the more frequently used DC drawing functions of GDI to .Net _without_ passing through `System.Drawing.Graphics`. Extending the set here would not generally be a problem. The bar for having these paint to a `System.Drawing.Graphics` object should not be too high either. For a look at the `UserObject` have a loot at the [Cvb.Net](http://Cvb.Net) Display tutorial (MainForm.cs, line 140 would be a good entry point).

> [@MarSch](#):
>
> - using the BitmapOverlay (while the conversion to the system type is somewhat quirky)

This will not solve the transparency issue, so I don’t think this is a viable approach.

> [@MarSch](#):
>
> - using Cvb.Forms.Overlays.ImageOverlay().

Also not a desirable approach because the concept of transparency would have to be introduced for this object first (currently it just uses the `ImageToDC` function which doesn’t have that). Plus: The set of painting functions on Image objects is actually more limited than what is available on `Stemmer.Cvb.Aux.Drawing.UnmanagedGraphics`.

To summarize, from my point of view that leaves only two options, one of them being only semi attractive and the other hitherto unmentioned:

1. Make use of the classes in `Stemmer.Cvb.Aux.Drawing` namespace to achieve the desired effect. Only an option of the limited set of features is not too far away from what’s needed and the ctor would currently have to be called through reflection if this should be used on a `System.Drawing.Graphics` object that refers to a `System.Drawing.Image`.
2. Instead of the Forms display, use the WPF display inside an `ElementHost` object. The WPF display integrates more closely with the .Net runtime than overlay objects of the forms display do (the latter always go through the underlying COM and CVCDisplay.dll infrastructure which limits their possibilities somewhat). With the WPF display you can select freely from the range of available features and use them for overlays. You can also easily use Bitmap objects (which can be painted arbitrarily) with the display including proper transparency. For a hint on how this may be used I recommend having a look at the “ClassifierTests” example for Polimago (specifically the `CreateColorOverlay` and the binding of the resulting bitmap).

I hope that helps a little further with your use case. If things are unclear or you want to discuss further you know how to reach us 🙂