MMV VS Stable Diffusion - "Real-time" Mixing Demo

[rdrt]

I've been working with MMV for some time now and wanted to show off some of the "behind-the-scenes" scenes I've made.

https://vimeo.com/rdrt/spectrumdemo

The main scene presents Standard Diffusion output based on a theme and attempts to display images with similar "tone" as the song.
To achieve this effect, many globals were created to provide information related to tone and pitch to be summarized into rules for mixing the jpegs.

In order to see how the rules responded, I ended up with dozens of sub-scenes that no-one ever gets to see. That seemed like a shame so I added them to a mix and here they are.

[TKS]

The "making-of" part of your video is pretty cool, you do some interesting stuff there. I'll be sure to watch it again later.

But, to be honest, the finished video in no way represents the effort you put in, the images change so quickly that a random display, perhaps pseudo-colored by volume, would give similar results.

I would display each image much longer so that the influence of the music on the images is actually visible.

[rdrt]

Thanks for the input!

I've yet to find an image progression method that I'm pleased with.

Currently image FPS is driven by: (BPM_ALL/64)*(2*(GblTone0+GblPitch0))
Which translates to: (An attempt to wrangle the number that the BPM capture global comes up with)*(2*what's activity like right now)

[rdrt]

For anyone interested - Here's a more lengthy video with a deeper description of the tool set in the comments (included)

https://vimeo.com/rdrt/hybridanalog

What is SPECTRUM?
- It's a Hybrid Analytical Engine implementation, focused on auditory changes, providing rigging output via MMVs many paths
- Spectrum listens to live input and creates a "real time" VJ mix, matching the tone of the song to the tone of the media displayed, utilizing rules to present media in a music video style structure
- Presented via a series of scenes, containing modules and global references in MMV
- Comprised of a number of globals, aka "analog" shift and hold registers and weighted if statements
- It is designed to be supported by several GPUs, allowing a continuously refresh-able matrix of images via Stable Diffusion (or whatever input media).

Main Scene Features
- History vs Input Mixing for Tone, Pitch and (8x) Octaves
- History vs Input Mixing for (4x) Octave Pairs, aka Actors

Functions
- Listens to input
- Creates Globals for capturing pitch, tone and (8x) octaves over 40+ bars History
- Creates Globals for mixing Pitch and Tone History vs input
- Attempts capture of BPM, leaves global rigging for improvement
- Attempts capture of Song Structure and Structure History over period (GBLVol0 > 0)
- Creates Globals for Song Structure comparison to input
- Attempts capture of (8x) Octave Structure, History, Significant Break vs input
- Creates Globals for (8x) Octave comparison to input
- Attempts to capture "Actor" movement by grouping (8x) octaves into (4x) actors
- Creates three mixing outputs based on actors, leaves rigging for improvement

Bonus Points
- Rigged for real-time refresh of media content
- Diagnostic Scenes for all Globals and Components
- Digit Counter Integration
- Snip Integration
- etc.

OOh/AAh
- Over 750 Globals (Analog Shift Registers ftw)
- NVidia 30x Gen or more for predictable performance

rdrtlabs

*Everything_is_always_WIP
hybrid analog is where it all goes down.

[rdrt]

Here is the first set of response tests with a final effects layer added driven by tone / pitch relationships over different lengths of bars.

https://vimeo.com/showcase/10183317

[rdrt]

...and here is a final scene post-FX:

https://vimeo.com/800137301/80e3bde512

The images mixes are controlled by globals and mix based on octave pair groupings.
So the tone of the image attempts to match the song. Effects try to capture major tone and pitch changes for each octave grouping.

[rdrt]

... and here is a final mix.

https://vimeo.com/rdrt/cognitionmix

[rdrt]

I've continued on with this concept further and really like the depth it can create in a single, real-time setup.

Current example of my work in MMV:

https://vimeo.com/820106355