Magic Music Visuals

Has the ability to sample pixel values at a coordinate been suggested before?

Use case would be an array of objects that vary based on the values of the pixels beneath them, or at another specified point, allowing video content to be interpreted using objects, such as a grid of spheres, or a chunky heightfield made up of cubes scaled in Z.

Nope, I think that's a reasonable idea.

Though as I think about it more, I'm not sure how it would fit in to the current framework to get linked values from video. Might not be simple. I'll ponder.

The Simplicity Paradox is a thing. When something appears simple, the back end infrastructure is always conversely complex.

Correct! Making things simple for the user is often complex for the developer.

I wrote very simple shader that would invert an image based on how bright it was at a specified pixel. This is very useful for keeping bright frames under control. Could the same technique be adapted, to sample an image and then apply that colour to the other input. Colour, of course, is the logical choice and would be relatively easy. Other attributes such as scale and position might be easy programmatically, but artistically how would one translate the sampled pixel to another attribute - colour might be a bit erratic for most attributes other than colour. If the video/image is a monochrome graduation then scaling a grid of object based on pixel value might make for a nice ripple effect.

a very interesting topic and discussion !