Technically, this is a really stupid thing to do, and move over it isn't the cleverest thing to do, regardless though I've done it just for the hell of it.

The aim of this post is to prevent anybody else from making the same stupid mistakes! end product isn't too bad though.

To give a quick over view, Sound.extract() gives access to raw sound data, as a ByteArray of floats; typically you setup two sound objects, the first loads the sound data from source, the second requests sound data from the first at regular intervals, extracting 2048-8192 samples per request and then playing it - idea is that in the method that requests the samples you can quickly pre-process it before its played and make some nice effects.

So my illogical train of thought was to visualize the raw sound data in real time as it plays to create some nice effects.

The Numbers Game

Music in flash is sampled at 44100 Hz Stereo - 44100 samples per second,  per channel - we're getting it at intervals of either 2048, 4096 or 8192 samples - meaning our visualization can run at either 21.53fps,  10.76fps or 5.45fps - which lets be honest is hardly going to make a nice smooth effect is it.

So our only option is to somehow synchronize the "timeline" (the screen updating) with the raw sound being played.

The Approach

I decided the only real way to do this was to make a continual buffer of sound that could be read independently by both the SampleDataEvent and by my own code to do the visualizations - after some playing around with numbers I finally gathered that if one was to run at a framerate of 63fps and render the 700 samples per channel at a time then I'd be consuming (63*700 = 41000 Hz / samples per second) = the data in realtime.

To make this possible I needed some kind of fast buffer to store the data in, obvious choice was fast memory which is a ByteArray and can be read and written to directly.

More number came in to play when the size of the buffer needed calculated, had to be a multiple of both the render speed (700) and the data feed rate (2048); which is a buffer of 358400 samples, or sample requests. (358400/175 = 2048).

To get the data in to the buffer I was finding the next offset at which i should write the data to my buffer, sound.extract'ing the data in to the buffer and using a modulo to create wrap around so after 175 samples data was written back at the start.

This covers getting the sound in to a fast memory buffer without consuming too many resources, next up to get it out and render it!

The Rendering

Rendering uses the same idea of requesting samples from the buffer, however this time the magic number is (358400/700 =) 512.

I implemented a simple framecounter that increments each frame and resets to 0 every 512 frames (giving the loop on the buffer), then quickly read 700 pairs of samples from the buffer each frame.

The left channel float value is then scaled by 300 used as an X position, whilst the right channel creates the Y position - both x and y are then fed to setPixel() on BitmapData whilst a simple calculation of 0xFFFFFF*leftChannel / 2 sets the color value.

Each frame the 700 dual channel samples are rendered creating 700 realtime 2D particles (hopefully) in time with the sound, and a simple blur filter is used to clear the previous frame in a nicer way.

The Point

It's all stupid, pointless and just stick to computeSpectrum - you only get 256 samples rather than 700 but it's perfectly in time and allows you do something other than exactly 63 fps; further it'll always be in time with the music.

The Result

Well, it's here - I've got it tuned in to my other halfs radio station citybeat radio at the minute (which i wholeheartedly recommend for any electronic / trance music fans) and well.. here it is with source.

PixelBender is one of these technologies, as advertised "You can use the Pixel Bender kernel language to implement image processing algorithms (filters or effects) in a hardware-independent manner"; however this isn't the whole truth of the matter.

PixelBender is a highly under-rated and under-used technology imho. Simply put PixelBender is actually a very fast way of converting or manipulating large sets of numerical data, whether it's image based or not. In this post I'm only going to skim the surface and give some pointers.

The standard input types cover Vector, ByteArray and BitmapData, likewise with output data, additionally many types are supported for parameters, everything from ints up to 4x4 Matrix' of floats.

Some obvious and simple uses include:

• 3D Data Manipulation (Float3 input and output - rgb or xyz)
• Raw Sound Data Manipulation (from Sound.extract, you could even build an FFT implementation in PixelBender)
• Visual Effects Calculations
• Batch Processing and Data Transformation.

Starting with Batch Processing and Data Transformation, here is the simplest PixelBender usage and example you could ever get.

Problem

Turn a BitmapData in to a Vector of RGB float values where each value is float < 1 and each set of 3 values comprises a pixel.

Doing this with pure AS3 or haXe or anything else is quite intensive, then you add PixelBender in to the equation

Solution
PixelBender Code:

<languageVersion : 1.0;>
kernel Identity
<   namespace : "org.webr3.pb";
 vendor : "WebR3";
 version : 1;
 description : "Converts BitmapData to Vector Number (rgb)";
>
{
 input image3 src;
 output pixel3 dst;

 void evaluatePixel()
 {
 dst = sampleNearest( src, outCoord() );
 }
}

And to use this we can create a simple wrapper class in AS3:

package test
{
 import __AS3__.vec.Vector;

 import flash.display.BitmapData;
 import flash.display.Shader;
 import flash.display.ShaderJob;
 import flash.utils.ByteArray;

 public class ShaderVector
 {
 [Embed("/rgb.pbj", mimeType="application/octet-stream")]
 private var shaderClass : Class;

 private var shader : Shader;

 public function ShaderVector()
 {        
 shader = new Shader(new shaderClass() as ByteArray);
 }

 public function convert( img : BitmapData ) : Vector.<Number>
 {
 var output : Vector.<Number> = new Vector.<Number>( (img.width*img.height) * 3 , false );
 shader.data.src.input = img;
 var job : ShaderJob = new ShaderJob( shader , output , img.width , img.height );
 job.start( true );
 output.fixed = true;
 return output;
 }

 }
}

Now we've got a kind of worker / utility class which we can farm all the hard work on to; usage is simple:

var worker : ShaderVector = new ShaderVector();
var rgb : Vector.<Number> = worker.convert( someBitmapData );

that's it, in one line we've solved the problem.

PixelBender is a lot easier than you may think, the above code is a small model you can easily manipulate just embed different pixel bender files.

Here's another simple kernel file; this one will take a BitmapData of perlin noise and convert it in to Vector of 3D positions for us to use.

It takes the bitmap data, converts every pixels r,g,b values in to float -0.5 to +0.5 values, then scales them up and finally returns back a vector of what are essentially 3D Perlin Noise Particles (like the ones in my earlier experiments)

<languageVersion : 1.0;>
kernel Scaler
<   namespace : "org.webr3.pb";
 vendor : "WebR3";
 version : 1;
 description : "For Perlin Noise, Converts BitmapData to 3D x,y,z positions";
>
{
 input image3 src;
 output pixel3 dst;

 parameter float scaleX
 <
 defaultValue : float(1.0);
 minValue     : float(0.1);
 maxValue     : float(100000.0);
 >;

 parameter float scaleY
 <
 defaultValue : float(1.0);
 minValue     : float(0.1);
 maxValue     : float(100000.0);
 >;

 parameter float scaleZ
 <
 defaultValue : float(1.0);
 minValue     : float(0.000001);
 maxValue     : float(100000.0);
 >;

 void evaluatePixel()
 {
 dst = sampleNearest( src, outCoord() );
 dst -= 0.5;
 dst.r *= scaleX;
 dst.g *= scaleY;
 dst.b *= scaleZ;
 }
}

and the modified shader vector, only 3 new lines to set up the scale values:

package test
{
 import __AS3__.vec.Vector;

 import flash.display.BitmapData;
 import flash.display.Shader;
 import flash.display.ShaderJob;
 import flash.utils.ByteArray;

 public class ShaderVector
 {
 [Embed("/rgb.pbj", mimeType="application/octet-stream")]
 private var shaderClass : Class;

 private var shader : Shader;

 public function ShaderVector()
 {        
 shader = new Shader(new shaderClass() as ByteArray);
 shader.data.scaleX.value = [200];
 shader.data.scaleY.value = [200];
 shader.data.scaleZ.value = [200];
 }

 public function convert( img : BitmapData ) : Vector.<Number>
 {
 var output : Vector.<Number> = new Vector.<Number>( (img.width*img.height) * 3 , false );
 shader.data.src.input = img;
 var job : ShaderJob = new ShaderJob( shader , output , img.width , img.height );
 job.start( true );
 output.fixed = true;
 return output;
 }

 }
}

If you're thinking this is complex, it's honestly not - try coding this in pure AS3 and it'll be a lot of complex slow lines, been there and done that!

I'm going to leave the examples there for now, if your looking for a 3D example then check the PixelBender files in Ralphs 3D Particle example.

Conclusion.

AS3 together with PixelBender, haXe and Alchemy is fast becoming an immense tech for us developers, have a play around you may be suprised by the results you can achieve by utilizing these techs.

Coming very soon I'll be posting some examples which show what can be done when you use PixelBender, Fast Memory, Vectors and haXe / AS3 together, but in unconventional ways. Until then I hope this is enough to get some more dev's playing with PixelBender.

peace ::

image is a still from I am Legend : Awakening a very well made 5 minute short animation.

Other than the really obvious bit that its direct access to a block of ram/fast memory which works extremely quickly, there is one other small but vital detail that's haXe specific and makes a vast difference between alchemy and haxe.

The Big Difference

You can think of flash.Memory as a static ByteArray, thus it is accessible from both static class and instance methods.

Now, another feature which offers huge speed increases in haXe is inlining.

Spelling it out - You can only inline static private methods, flash.Memory is all static access. So stick all your data (or big chunks of it) in flash.Memory and then inline  most of your heavy number crunching class.

All inlined, all using fast memory = very very quick swfs!

Ideas and Tricks

First of all, get the thought of only being able to have one chunk or type of data in fast memory at a time, you can partition it up and load everything in there.

A common setup I use is to have the first 1024 bytes as fast storage for float, int, byte, uint variables, the next xxx-thousand bytes as raw bitmap data, then all the other data I may want - like a block of 500k x,y,z floats for 3d data, followed by a few lookup charts.

the first 1024 bytes as fast storage for float, int, byte, uint variables - because you can then access the most commonly used vars using small int offsets (0-255) - and since its a small int your using it avoids all stacks completely, the bytecode will only contain the opcode to get the value from fast memory followed by the offset actually in the bytecode - for anything 256 and over its going to hit the int stack to get the offset first (which is barely noticable, but you know every fraction counts)

the next xxx-thousand bytes as raw bitmap data - because you can then work on your bitmap data ultra fast and write it back to the BitmapData instance using setPixels; very fast indeed, faster than a fixed sized vector even.

then all the other data - this is partially obvious, what may not be is the use of lookup charts, but I'll cover that in a moment.

Accessing..

Remember flash.Memory differs from a byte array in that you have to add in the offset for each value you want to grab

• Byte = 1 Byte
• Int = 4 Bytes
• Float = 4 Bytes
• Double = 8 Bytes

To use flash.Memory with functions which require a ByteArray.. [ like BitmapData.getPixels() -  Sound.extract() ] .. you can access it directly from flash.system.ApplicationDomain.currentDomain.domainMemory for example:

// set position to start of the data we need
flash.system.ApplicationDomain.currentDomain.domainMemory.position = SOME_OFFSET;
// use the fast memory
bitmapData.setPixels( bitmapData.rect , flash.system.ApplicationDomain.currentDomain.domainMemory );

note: don't be fooled though, you can't simply call the functions on the bytearray in currentDomain, although it is the data that is in fast memory, it does not use the opcodes, which are what speed it up!

Lookup Charts

Often you'll find yourself calling the same code over and over and over; perhaps without realising just how much; consider we make an animation which uses perlin noise to generate 2d or 3d data - like this example - the example reads about 40k uint values from perlin noise, then converts each pixel to individual r,g,b values, then turns it in to a float -1 to 1 using a simple calculation. Thing is.. that means the same calculation runs 3x40000 times per frame at 50 frames a second.. 6 million times a second!

In this scenario, there are only 256 possible inputs to the calculation and 256 results - so rather than working it out 6 million times we can load it all in to a lookup chart, stick the lookup chart in fast memory and then lookup values rather than calculating - often you can really speed things up by doing this. Practical example with comments here.

Hope this is of use to somebody and clears things up a bit; I use the methods often so you'll see them in the source of my experiments.

Been doing some more playing with Perlin Particles, this time going for more natural, and indeed more useful!

Full source and comments included with this one: 3D Perlin Particle Light Cloud

Doesn't need much of an introduction, after my previous post "Amazing what happens when you modify a Sphere"; I realised that I'd missed a few combinations, and as such thought it'd be a good idea to finish off the obvious combinations.. you know, to see what they looked like.

One of the best, and the one that makes the most sense formula wise is M16.. here's the formula for it

x = Math.cos(a) * Math.cos(r);
y = Math.cos(a) * Math.sin(r);
z = Math.sin(a) * Math.sin(r);

makes sense to me! anyway, on with the show - here are the rest:

• M8
• M9
• M10
• M11
• M12
• M13
• M14
• M15
• M16

Taking the 3d particle primatives a bit further (and adding color) I've found that some very nice and rather interesting models can be made when you simply change a sphere a little bit.

Normally I'd do these in order, but in this case I'd recommend you checkout M4 first to get a taste (it's certainly my fav).

Before the list, a very brief description; here we have the code to generate a sphere:


for( i in 0...PARTICLES ) {
a = (PARTICLES/360)*i;
r = (i/PARTICLES) * Math.PI;
x = Math.cos(a) * Math.cos(r);
y = Math.cos(a) * Math.sin(r);
z = Math.sin(a);
}


now all we are going to be doing is adding in some steps, or changing the odd cos/sin with cos/sin/tan - nothing complex.

• M1 Variation
• M2 Variation
• M3 Variation
• M4 Variation (my fav)
• M5 Variation
• M6 Variation
• M7 Variation
  

Now.. M7 is a rather interesting one.. because it has two very interesting variations (imho) primarily the eye which produces a flat, yet 3d shaded circle..?

• M7 Strange Variation - The Eye
• M7 Strange Variation - The Feather

In addition there are a few more which are created by simply stepping colors or values:

• 3D Abstract Rainbow
• One, Three, TwentyFour
• Three, Six, Twelve
• Six, Six, Twelve

again, thanks to haXe for allowing this - remember it's all just particles rendered in realtime.

Following earlier unpublished work, and my version of 300k particles in flash, I found that I've created a nice little setup for rendering the results of mathematical functions in flash; specifically rendering the results as 300k/500k particles - I'm kind of fascinated with particles and plotting functions, as are many others - my preference is to work with strange attractors; however I thought I'd shoot right back to basics and see what can be made with primatives and modifications of.

pre-warning, if you're looking for eye candy, forget it - these are simple practical examples and a bit boring, but useful if you need to know how

Starting with the simplest we have the Particle Cube - 500k random particles plotted in 3d space within an invisible cube. Then I modified it a bit..

• Sine Modified Particle Cube (it's all skewed)
• Sine and Time Modified Particle Cube (double cube with a flow of particles between)
• Sine, Tan and Cos Modified Particle Cube (one function applied to each axis)
• Alternative Flattened Sine, Tan, Cos Modified Particle Cube (patchwork circular flattened cube!)

Next up the Particle Sphere - another 500k particles in 3d space, all random, and then some modified versions..

• Spherical Circles (evenly distributed particles, creates a single line wrapped around a sphere)
• Emphasized Particle Sphere (double spaced to create a messy particle sphere - but more like particles)
• Half a Sphere (go on, guess)
• White Particle Bubble (evenly distributed to create a kind of bubble effect)
• Split Sections of a Sphere (two opposing but differing chunks of sphere, again all particles, like a swing bin lid)

And finally a couple of simple experiments..

• Random Warped Particle Plane (random x, random y, atan2(x,y) for z)
• Beach Bubble (the white bubble with some colour, like a beach ball but translucent)

In summary, it's good to see how easily different effects and primatives can be build, and even better to see how easily they can be modified with basic mathematical functions. (and how good haXe is :p)