omino pixel blog

This post will show a handful of simple tricks for “drawing” using Pixel Bender. That’s not what it’s for, especially in Flash. But it’s still interesting to see how it’s done.

But first, the demo:

And I’m sure the handy tabs across the top for the source code won’t escape your attention, either!

Pixel Bender Metadata

In the demo above, you can notice a couple of things. Some of the parameters are drawn as sliders, but also there’s a checkbox, some color pickers, and, in the drawing area, some draggable spots. This is done by a collusion between the Pixel Bender kernel, and the Flash viewer code. Here’s the parameters from the Pixel Bender kernel:

    parameter float3 backgroundColor<kind:"color";defaultValue:float3(1,1,1);>;

    parameter float spotRadius<defaultValue:50.0;minValue:1.0;maxValue:800.0;>;
    parameter float3 spotColor<kind:"color";defaultValue:float3(0,0,0);>;
    parameter float spotSquare<kind:"checkbox";defaultValue:0.0;>;

    // This parameter just gets poked with the rendering dimensions, always
    // Doesn't get shown to user. In my custom viewer, that is.
    parameter float2 dims<kind:"dstsize";defaultValue:float2(300,300);>;

    // "point" means it lives somewhere in the image, as a coordinate.
    parameter float2 lineStart<kind:"point";minValue:float2(0,0);maxValue:float2(400,400);defaultValue:float2(10,10);>;
    parameter float2 lineEnd<kind:"point";minValue:float2(0,0);maxValue:float2(400,400);defaultValue:float2(30,100);>;
    parameter float lineWidth<minValue:0.5;maxValue:30.0;defaultValue:2.0;>;

I’ve added a kind property to some of the parameters. Pixel Bender ignores it, but in ActionScript we can access it as part of the ShaderData. For example, it is true that shader.data.spotSquare.kind == "checkbox".

These aren’t special or magic! My Pixel Bender kernel says it, and my Flash viewer reads it. It’s a handy trick, if you happen to be in control of both parts.

How to Draw a Circle

It’s so easy! Given a center and a radius, you check each pixel to see if it’s within that distance of the center. Like so:

        float d = distance(co,center);
        if(d < spotRadius)
            dst = spotColor;

How to Draw a Square

This is ever so slightly trickier. Again, with a center and a radius, we want to see that both the x- and y-distances from the center are within the radius. Like so:

        float d = max(abs(co.x - center.x),abs(co.y - center.y));
        if(d < spotRadius)
            dst = spotColor;

How to Draw a Line

Now this is getting mathy. In an earlier post about Gradient fills, I derive an equation for “position between two points”, like so:

t = dot(uv,xy) / dot(xy,xy)

where a line segment is given by the endpoints (0,0) and (x,y), and some screen spot is given by (u,v). If t is within the range 0 to 1, then (u,v) is somewhere along that line segment if it’s drawn infinitely wide. That’s a short stubby line no matter how you slice it.

To constrain the width of the line, we need to know how far from the center (u,v) is, which is given by:

d = abs(length(xy) * dot(uv,yx) / dot(xy,xy))

(If you followed the Gradient post, it’s the right half — the y-component — of the final 2×2 matrix, times the length of the line, for scaling. But it’s not important.)

So, the combined code to draw a line between two endpoints, with a particular line width, and square-capped ends, is:

       // and now, a bit of fun math, as we plot a line between two points.
        float2 uv = co - lineStart;
        float2 xy = lineEnd - lineStart;
        float xy2 = dot(xy,xy);
        float g = dot(uv,xy) / xy2;
        // if we are between the two points, then g is between 0 and 1.
        float dl = abs(dot(uv,float2(-xy.y,xy.x)) / xy2 * length(xy));
        // dl is now the distance, in pixels, from outCoord to the line

        // if we're between the endpoints and close enough, draw.
        if(g >= 0.0 && g <= 1.0 && dl < lineWidth / 2.0)
            dst = float3(1,0,0); // draw the line in RED.

Ridiculous?

To reiterate: This is a ridiculous use for Pixel Bender. But the ideas may come in handy. Doing something with a squarish area-of-influence isn’t so outlandash… and while a solid-colored line is a bit unsubtle, a shaded field or beam area begins to seem plausible. Pixel Bender is so powerful and flexible, I know we’re going to continue to see unexpected and novel results from it.

Back to the cubes. This Pixel Bender generator shows a volume subdivided into cubes, and constrained to a spherical limit.

And here’s the source code.

Can’t wait to start incorporating this stuff into After Effects CS4… but haven’t uncovered any details on exactly how Pixel Bender integrates to it. Presumably some magic parameters may tell you things like the source or dest extant, and maybe you can add “typing” information to parameters. Looking for links… since it may be a little while before I can afford to upgrade.

I was trying to do something else. But, as you well know, sometimes bugs are really cool. Here’s my bug of the night. Click Go, once, and Scramble a few times.

Geometrical interpretation: a perfectly nonreflective sphere is in each cubical cell, getting cross-sected just like the cubes.

And here’s the source code. It’s based on “cubes” from a few posts back.

Featuring mix(), clamp(), and dot().

Very often I need a simple linear-gradient in After Effects, and end up sing the four-color gradient because that’s all I can find. And so, here, for your perusal, is a Pixel Bender implementation of a Linear Gradient Generator.

After that, we’ll see the code, and after that, a quick review of the math.

Here is the .swf. (With some improvements on the Pixel Bender viewer — you can drag any float2 parameters as points, now.)

Isn’t that fun? Well, I’m easily amused.

The Code

Here’s the Pixel Bender code.

kernel Gradient
{
    output pixel4 dst;
    parameter float2 p1<minValue:float2(0,0);maxValue:float2(400,400);defaultValue:float2(10,10);>;
    parameter float2 p2<minValue:float2(0,0);maxValue:float2(400,400);defaultValue:float2(130,130);>;

    parameter float3 color1<defaultValue:float3(0,0,0);>;
    parameter float3 color2<defaultValue:float3(1,1,1);>;

    void evaluatePixel()
    {
        float2 co = outCoord();

        // shift everything relative to p1.
        float2 uv = co - p1;
        float2 xy = p2 - p1;

        // the math.
        float g = dot(uv,xy) / dot(xy,xy);
        g = clamp(g,0.0,1.0);

        // the color.
        dst.a = 1.0;
        dst.rgb = mix(color1,color2,g);
    }
}

The Math

I’m ok at math, but not always fluent in it. I can remember what a matrix is, but not necessarily exactly what a cross-product is. Maybe you’re like me. In which case this quick run through the math and logic behind the gradient fill will be quite useful in your own Pixel Bender bending!

And be assured, a few pages of high-school-algebra-style scribblings, and the occasional google (How do I invert a rotation again? What’s the formula for perspective?) really can lead to workable results. Good clean fun!

Let’s go.

Consider a trivial gradient, where our two control points are at (0,0) and (1,0). That would be easy! Just take the x-value of any pixel, and that’s your color. (That is, the position on the ramp between your two gradient colors.)

By the way, this picture is from Ron Avitzur’s “Graphing Calculator”. The story of its development is legendary; less well known is that it’s actually a very powerful and useful tool.

I found the easiest way to think about a general two-point gradient was to ask, “How do we rotate and scale our control points back to this trivial gradient?” And let’s assume that the first control point is always at (0,0).

Here is matrix for a basic rotation, which moves any input point by a rotation around (0,0):

[1]

A rotate-and-scale matrix looks like that, times a constant on each element.

We want to think of our general gradient as a transformation on the trivial gradient. Let’s say our second control point is (X,Y), and our first control point is (0,0) as mentioned above. The rotate-and-scale matrix being applied, conceptually, is:

[2]

To figure out the 0-to-1 trivial position of any of our input points, we need to invert it. A matrix inversion looks like:

[3]

So, we invert the second equation [2] by plugging it in to [3] and get:

[4]

Now, let’s call each pixel position of our output (U,V). For our gradient, we actually don’t care about the resulting Y position; just X gives us our gradient value. So we get:

[5]

Now, the definition of the dot product of (A1,A2,A3,…) and (B1,B2,B3,…) is (A1B1 + A2B2 + A3B3 + …). So, from the above, UV+XY is, conveniently, (U,V) dot (X,Y). Similarly, X² + Y² is (X,Y) dot (X,Y).

And now, when you read this part of the code a second time, it should be much clearer what’s going on:

    void evaluatePixel() {
        float2 co = outCoord();

        // shift everything relative to p1.
        float2 uv = co - p1;
        float2 xy = p2 - p1;

        // the math.
        float g = dot(uv,xy) / dot(xy,xy);
        g = clamp(g,0.0,1.0);

        // the color.
        dst.a = 1.0;
        dst.rgb = mix(color1,color2,g);
    }

Simple and useful!

Next up, soon, more cubes.

Featuring distance().

One of the important games we play while writing a script or software — or other kinds of black boxes, too — is, “How can we tell what’s going on inside?”

Modern debuggers in modern programming languages make this a nearly-solved problem. Sometimes. Maybe it’s a new language and the tools aren’t there yet, or maybe you’re too cheap to purchase them. Or maybe you just like debugging with print statements. That’s cool.

In the case of Pixel Bender, I’ve been lately using several float2 variables to represent locations on the input or output. It turns out to be quite handy to drop “print” markers right in the output, to better see what’s going on.

The trick is extremely simple. If the distance to float2 value is within a few pixels, draw a color. Here’s the code to do it. It should be done last, after other pixel computations. That way it can be easily commented out or deleted before shipping.

        float p1Dist = distance(p1,outCoord());
        if(p1Dist < 3.0)
            dst.rgb = float3(1,0,0); // draw red
        else if (p1Dist < 5.0)
            dst.rgb = float3(0,0,0); // black outline

Handy!

Haven't figured out how to stamp little text figures into the output yet. You'd need to decompose your desired output on a per-pixel decision tree... would be an interesting logic-reduction problem, but perhaps impractical. It would be possible and useful to stamp 3 floating point values as "special debug pixels" at (0,0) and other easy to find locations.

At any rate, here's the whole kernel, ready for copy-and-pasting.

<languageVersion : 1.0;>
kernel DebugMarkers<namespace:"ns";vendor:"omino.com";version:1;> {
    input image4 src;
    output pixel4 dst;
    parameter float2 p1<minValue:float2(0,0);maxValue:float2(400,400);>;
    parameter float2 p2<minValue:float2(0,0);maxValue:float2(400,400);>;

    void evaluatePixel() {
        float2 co = outCoord();
        dst = sampleNearest(src,co);
        float p1Dist = distance(p1,co);
        if(p1Dist < 3.0)
            dst.rgb = float3(1,0,0);
        else if (p1Dist < 5.0)
            dst.rgb = float3(0,0,0);
        float p2Dist = distance(p2,co);
        if(p2Dist < 3.0)
            dst.rgb = float3(0,1,0);
        else if(p2Dist < 5.0)
            dst.rgb = float3(1,1,1);
    }
}

In which I present instances of a family of closed-cell geometric objects which are computationally trivial to render.

The Concept

Imagine moving a plane through a three dimensional mesh of cubes.

You start by rotating the plane to your liking. That is, for each pixel on the plane (since that is our rendering reference point) we rotate and translate it as we like, from (x, y, 0) to (x’, y’, z’). To find out which “cell” we’re in, we just look at the integer parts of the prime-coordinates. In the animation above, the cells have been “checkered” by whether their integer coordinates’ sum is even or odd. (Actually, I just used LightWave’s 3d checker texture function. But you get the idea.)

Drawing the checkerboard is easy. Drawing some depth in the cubes is only a little bit trickier. We need a perpendicular vector from the plane, to cast a ray, so we also rotate the perpendicular vector (0,0,1) by the same amount. Then we project that ray from the fractional position within the cell until it hits one of the 6 walls of the cube. Lastly, we give a non-optical depth cue based on how far the intersected wall is.

The Implementation

Here’s the math part of the rotation. There’s probably an easier way to get the 3-axis rotation matrix… But this works.

// concatenate three simple rotations to spinM
float3x3 spinXM = float3x3(1,0,0,0,cos(spin.x),spin.x,0,-sin(spin.x),cos(spin.x));
float3x3 spinYM = float3x3(cos(spin.y),0,sin(spin.y),0,1,0,-sin(spin.y),0,cos(spin.y));
float3x3 spinZM = float3x3(cos(spin.z), sin(spin.z), 0,-sin(spin.z), cos(spin.z), 0, 0, 0, 1 );
float3x3 spinM = spinXM * spinYM * spinZM;

float2 center = float2(400,500) / 2.0; // YUCK! hardcoded for my blog entry.
float2 oc = (outCoord() - center) / zoom; // adjust "input coordinates"

// we rotate the requested coordinate by 3 axes, giving the
// R3 point on the plane of the screen (after the mesh is turnt)
float3 p = float3(oc.x,oc.y,0) * spinM; //oc.x * axis1 + oc.y * axis2;

// perp is our viewing line, straight down from the screen.
float3 perp = float3(0,0,1) * spinM;

Here is an interactive demo of the algorithm, in Flash and Pixel Bender. (I added a full screen button, whee!)

And here is the Pixel Bender kernel:

And for reference, here is the template for the Flash displayer, compilable by mxmlc.

Flex trivia: stage.addEventListener(FullScreenEvent.FULL_SCREEN,screenChange);
works from mx:Application applicationComplete because the UI and display list is all instanced, but not from mx:Application creationComplete which happens a little bit earlier.

Next up: Suppose we didn’t intersect with a simple plane… !

Since I love After Effects, I need to learn Pixel Bender. To talk about Pixel Bender in this blog, it sure is handy that the new Flash Player can show it, live. But first I had to learn Flash. Darn!

So, below is a program written for Flex, compiled with Adobe’s free mxmlc command line tool. Requires Flash 10 (currently in beta). It incorporates a simple Pixel Bender kernel to draw on a rectangular area. It doesn’t use an input image.

For reference, here’s the snippet of Flex code that simply fills in an area with the Bended Pixels:

        // "draw" is the name of a UIComponent instance,
        // and "shader" is a Pixel Bender kernel instance.
        var g:Graphics = draw.graphics;
        g.clear();
        g.beginShaderFill(shader);
        var m:Number = 20;
        g.drawRect(m,m,draw.width - 2*m,draw.height - 2*m);
        g.endFill();

And, below is the swf, live, in action. Click the “animate” checkbox for most fun.

As you can see, it displays each of the kernel’s numeric parameters as a slider. Since it knows the min- and max- of each parameter, we can randomize and animate them, which is much more fun that twiddling the slider by hand, yes? It also suggests some of the possibilities when animated in After Effects CS4.

Importantly, this Flex program discerns the parameter values and ranges at run time, so you can pop in another kernel, recompile, and play some more.

Here’s the kernel code. Just a bunch of Math fun… really it was just to test out the Flash displayer, but it came out kind of psychedelic.

And here’s the Flex program. To pop in our own kernel, just alter the Embed line.

Stay tuned for some Fun With Cubes.

The mod() Function

In GL Shading Language, and Pixel Bender, the mod function is defined like so: mod(x,y) => x - y * floor(x / y). The result is always positive. We can see quickly that y’s sign goes away. And the floor function pushes towards negative infinity so y * floor(x / y) is always “lower” (not “smaller”) than x.

The mod() Function, GPU vs CPU

Here is a short Pixel Bender kernel.

<languageVersion : 1.0;>
kernel modTest2 <namespace : "x";vendor : "omino.com";version : 1;description : "mod() bug";>
{
    // on PBT build 35, produces different display on GPU/CPU
    parameter float span;
    output pixel4 dst;

    void evaluatePixel()
    {
        float x = outCoord().x / span - 1.0;
        float y = outCoord().y / span - 1.0;
        float m = mod(x,y);
        float t = x - y * floor(x / y); // computed by definition
        dst = float4(m / y,abs(m) / y,t / y,1);
    }

    region generated(){return region(float4(0,0,2.*span,2.*span));}
}

On the GPU, it produces:

And on the CPU:

The blue component is correct on both CPU and GPU because it’s computed according to the formula. The other two color components reveal the nature of the bug. (Probably that x/y is being rounded towards zero.)

As an aside: Notice that some of the CPU rendering’s lines are quite a bit smoother than the GPU rendering’s. Subtly different arithmetics.

The Workaround

A workaround might not be needed, in practice. The mod() function seems to work correctly in Flash, even though it’s on the CPU. (The PBT and Flash implementations are, clearly, if surprisingly, different, since you can turn off the Flash errors & warnings in PBT. Also confirmed in this note by Adobe engineer Tinic Uro.)

On the other hand, After Effects CS4’s CPU-renderer might still have the bug. On the gripping hand, perhaps Adobe will fix it… I’ve had trouble signing up onto the JIRA system to report it, though.

But to get correct render results on the CPU within PBT build 35 on Macintosh, simply use the expanded formula.

float m = mod(x,y); // may have a bug
float m = x - y * floor(x / y); // always works.

Pixel Bender is Fun!

Ok. This post will show the source code for the tiniest, simplest Pixel Bender generator you can imagine. A generator is like a filter, with no inputs; it just creates an output. Afterwards, some thoughts about integrating into Flash.

To run it, just download the Pixel Bender Toolkit from Adobe, and paste this code into it.

<languageVersion : 1.0;>

kernel ShowAColor <namespace : "x";vendor : "x";version : 1;>
{
    parameter float3 the_color < defaultValue : float3(0.8,0.1,.5) ; >;
    output pixel3 dst;

    void evaluatePixel()
    {
        dst = the_color;
    }

    region generated() // (but not supported in Flash)
    {
        return region(float4(0,0,100,100));
    }
}

When you run it, you’ll get to input a color with sliders…

And as you choose the color, a 100×100 square of GPU-accelerated pixels will display it.

Nothing too high-concept here, but it does show the two main methods. evaluatePixel() does the actual rendering; it gets called over and over for each pixel of output. generated() just tells Pixel Bender how big an image to create.

Adding Another Parameter

Lets do one further little tweak, and then get on to a discussion about Flash and such. We’ll add a control to invert the color.

<languageVersion : 1.0;>

kernel ShowAColor <namespace : "x";vendor : "x";version : 1;>
{
    parameter float3 the_color < defaultValue : float3(0.8,0.1,.5) ; >;
    parameter int invert_color < minValue : 0 ; maxValue : 1 ; >;
    output pixel3 dst;

    void evaluatePixel()
    {
        if(invert_color > 0)
            dst = float3(1,1,1) - the_color;
        else
            dst = the_color;
    }

    region generated() // (but not supported in Flash)
    {
        return region(float4(0,0,100,100));
    }
}

Now we get one more control, which inverts the color.

My Comments, Observations, and (Constructive) Critique

I’ll assume here that the reader is using other references as well to learn about Pixel Bender… that you know about the float and float3 and float4 types, and that you can have an input image to process, and so forth.

This particular kernel won’t work in Flash 10, for two reasons:

• The output in Flash must be of type pixel4, not pixel3. That is, it must have an alpha channel.
• Flash doesn’t support the generated() function. In the Pixel Bender Toolkit, omission of the generated() function (for an inputless kernel) causes a blank display when you run it. In Flash, however, a kernel is alway run as a “Filter” on something, and so assumes its size.

In the second example, there are two declared parameters. the_color is first, and invert_color is second. But they’re displayed in opposite order. The Pixel Bender Toolkit always displays them alphabetically. That’s not bad, but it would be nice if you, the author, could control the display order.

Also, it would be nice if the_color was shown as a color picker instead of three sliders. For that matter, it would be nice if invert_color showed up as checkbox. I think it would be nice if you could type:

    parameter float3 the_color
        < defaultValue : float3(0.8,0.1,.5) ; type : "colorpicker" >;
    parameter int invert_color
        < minValue : 0 ; maxValue : 1 ; type : "checkbox" >;

It turns out that you could do this in a generic Flash framework. All of the metadata in the <this:that> is visible to ActionScript as named properties of kernel’s instance (an instance of class ShaderFilter). You’d have to control the ordering with another assignment, but, in principle the following code would give such a hypothetical Flash-based parameter-editor enough clues:

    parameter float3 the_color
        < defaultValue : float3(0.8,0.1,.5) ; _index : 1 ; _type : "colorpicker" >;
    parameter int invert_color
        < minValue : 0 ; maxValue : 1 ; _index : 2 ; _type : "checkbox" >;

I’m hoping that integration with After Effects CS4 uses something like that… Hi Adobe… listening?

One Last Gotcha / Complaint

Annoyingly, Pixel Bender doesn’t do type-promotion. That is, you get errors if you ever mix integers with floats. Each line of the following code produces errors!

	parameter float xParam <defaultValue:2;>; // Ha! must be 2.0
	float x = 0; // Ha! must be 0.0
	if(x < 0) x = -x; // Ha! still must be 0.0
	float3 xyz = float3(1,2,3); // well, this is allowed! hooray

Next up, adventures with finally, finally learning a bit of Flash.

Fun with Pixel Bender.

Code to be posted soon, as well as interactive Flash 10 swf (if I can ever figure that out).