If I am correct then the stencil buffer is used when you want to avoid drawing something, otherwise someone please correct me

The stencil vector in that demo is done using a software render, and this is why its easy to do stencil vectors. What they have done is creating a poly texture fill rutine and a buffer the size of the screen (or alternatively having the texture pointers capped to the size of the texture) and then they feed the poly texture fill with different values depending on where on the screen its drawn. Adding shading values to darken/lighten the texture can enhance the effect. I have done this effect a couple of times using BlitzPlus and a software render.
If you are looking to do a similar effect, and do not want to write a complete software render in Blitzmax, then you might be able to do something similar, if you create a texture that is applied to all polygons, and then manipulate the texture UVs. The data is there in the image type somewhere. Someone at the blitzmax forum did their own textured polygon function and there is code showing how its manipulated.
I am not at my regular computer right now, but I could take a look for something later on though.