Previous topicNext topic

programmatic mechanism for generating/modifying scenes

Suggestions for new features for Magic.
Post Reply
nosuchtim
Posts: 5
Joined: Fri Nov 24, 2017 10:27 pm

programmatic mechanism for generating/modifying scenes

Post by nosuchtim »

I'd like to explore program-generated projects or scenes. The design of the OSC input source mechanism is excellent, but I'd like to do other things like create/modify/rearrange a pipeline of effects under the control of another program. I can currently create a pipeline of effects and selectively enable/disable effects with OSC control, but I can't (for example) rearrange the order of the effects programmatically. If the .magic file format was readable or documented, I could generate .magic files dynamically - that would be one way to accomplish it. Are there techniques that could be used to dynamically construct or modify a scene?

...Tim...
Magic
Site Admin
Posts: 3440
Joined: Wed Apr 09, 2014 9:28 pm

Re: programmatic mechanism for generating/modifying scenes

Post by Magic »

I'd like to do other things like create/modify/rearrange a pipeline of effects under the control of another program
Ok, just so I understand, why exactly do you want to do that?
I can't (for example) rearrange the order of the effects programmatically.
No, you can't do that, but you could easily use the InputSelector module to switch between different chains of effects.
If the .magic file format was readable or documented
We don't currently have any plans to open it. Not because we're secretive, just because it's pretty complicated :)
nosuchtim
Posts: 5
Joined: Fri Nov 24, 2017 10:27 pm

Re: programmatic mechanism for generating/modifying scenes

Post by nosuchtim »

>> I'd like to do other things like create/modify/rearrange a pipeline of effects under the control of another program
> Ok, just so I understand, why exactly do you want to do that?
>> I can't (for example) rearrange the order of the effects programmatically.
> No, you can't do that, but you could easily use the InputSelector module to switch between different chains of effects.

I create installations that have custom user interfaces/programs which combine and control other programs. See http://spacepalette.com for an older example, and https://www.youtube.com/watch?v=e-jUvLaHZZ4 for a newer example. APIs make this possible - I've been a longtime fan and user of Resolume and Plogue Bidule because they can be controlled with OSC. I even have an FFGL plugin which runs inside Resolume and uses OSC back to Resolume to turn effects on and off (it's like an alien organism controlling its host, unknown to the host). One of the main reasons that I'd like to move to using Magic is because of its support for ISF. I also like the way you allow the OSC control to be customized.

This API-using approach lets me create a custom interface to the programs being controlled, as well as adding features that the underlying programs don't have. Parameter control via OSC can be used to accomplish a lot of these things, but sometimes I want to do things that the underlying programs don't have ready-made APIs for - e.g. rearranging a pipeline of effects. I've found (as a result of actually building a prototype of an ISF-based system) that being able to quickly rearrange the order of effects in a pipeline can have dramatic effects on the output, and I want to be able to quickly explore such rearrangements. Manually constructing all possible arrangements to be selected with InputSelector doesn't seem practical. If I have a set of only 12 effects, and use 4 of them at a time, there are 12*11*10*9 different pipelines (or 12*12*12*12 if the same effect can be used multiple times with different parameters.

Thoughts? I imagine that expanding API control in general would be useful to others besides me, but I also see that you're open to custom software modifications, perhaps I should be exploring that.

...Tim...
Sadler
Posts: 1139
Joined: Sat Aug 02, 2014 7:10 pm
Location: London, UK

Re: programmatic mechanism for generating/modifying scenes

Post by Sadler »

I would guess that in the space of 11,800 combinations, not many, at least you couldn't predict which, would be aesthetically pleasing. Certainly, when I am composing a scene I must have fine control of almost every single parameter. In some cases, if parameters aren't exact, there's nothing to see at all even with most things normalised to 1. Also, and this is just my own visual taste, combining more than a couple of effects often creates video soup which no-one can get their heads around.

Using Resolume with IR ClipSA is certainly great for a range of control over media and effects but Resolume is different from Magic in (at least!) two main ways: 1. Resolume is much more structured with comp-layer-clip paradigm and, as such provides; 2) absolute OSC addresses which are programmed into the control surface or program, whereas Magic comps are much more organic and the OSC is learned (like MIDI) from within Magic and so absolute addresses are not required to be published.

You've got to ask yourself, even if you did have an API into Magic, how would you specify programmatically how to disconnect one node and connect it to another. With Resolume you can say L1->C4->E2->P2 but node based programs aren't that structured.

One last point about the power of magic and the control over combinatorial selection of effects...with globals driving parameters (esp through the expression modifier) into hierarchies of scenes containing arrays of effects controlled by various selectors and input sources is just as powerful, if not more so, (and less painful!) than using application programming interfaces to do the same job. You've probably tested a wide range of technologies in the creation of your rig but if you haven't already you should check out Iannix and GlovePie.
nosuchtim
Posts: 5
Joined: Fri Nov 24, 2017 10:27 pm

Re: programmatic mechanism for generating/modifying scenes

Post by nosuchtim »

Hi Sadler, thanks much for your thoughts. I agree with a number of your points, but my experience in exploring this type of work shows that for me there's a lot of benefit to the approach in spite of things that need to be addressed or avoided. Besides the work for which I've already provided links, LoopyCam is another instrument I built that leverages a flexible pipeline of effects - in that case it was pipelines of FF 1.0 and FFGL effects in a single piece of software.

> You've got to ask yourself, even if you did have an API into Magic, how would you specify programmatically how to disconnect one node and connect it to another.

Indeed, that's why my first suggestion was to document the format of the .magic file. I understand why that may be awkward, embarrassing, or undesirable to document. On the other hand, I've also seen large benefits when using systems that expose their settings in textual (while still undocumented and unsupported) form.

> You've probably tested a wide range of technologies in the creation of your rig but if you haven't already you should check out Iannix and GlovePie.

Thanks for those suggestions. Yes, I've investigated (and used when appropriate) a wide range of technologies, including those. Fugio is the closest thing to Magic, in terms of the features I'm looking for. I often roll-my-own systems using lower-level standards like OSC, TUIO, FreeFrame (1.0 and FFGL), Spout, etc. but am always looking around to see if some system (like Magic or Fugio) can help me avoid reinventing-the-wheel too much. A primary current goal I have is to convert my FFGL-based work to ISF. I've prototyped my own ISF-based stuff, but I'm trying to elevate my efforts a bit without sacrificing too many of the features that my experience guides me to want.

...Tim...
Magic
Site Admin
Posts: 3440
Joined: Wed Apr 09, 2014 9:28 pm

Re: programmatic mechanism for generating/modifying scenes

Post by Magic »

Yeah I agree with pretty much everything Sadler said. It would be challenging (though not impossible) to provide an OSC-based command system that would let you manipulate module connections. For all the work it would require, I don't see it as being a worthwhile investment because only a tiny, tiny percentage of users would want to use it. Maybe only you ;).

Same thing for the .magic file. There is nothing awkward or embarrassing about it -- it's just a very complicated file format, because each individual module, module parameter, and module connection is described in detail. Even if we opened it, it would still be very difficult to use. Manipulating it would likely create all kinds of errors when you tried to load it, and for us to support/document it would again mean a huge time investment for only a small number of users.

Perhaps most importantly though: after watching the video example you provided, I don't see how rearranging effects programatically would get you any better results (from an aesthetic perspective) than just using the existing functionality in Magic to change scenes and/or enable/disable certain modules. Your end users probably won't know the difference, especially if your control pad (as shown in your video) only has 12 preset options? It seems like having 12 Magic scenes would be enough to satisfy your needs? I wouldn't worry so much about the inner workings because your users won't even know about them. Focus on the end-user experience :) .

If you really do want to have such a detailed control over things, my suggestion would be to investigate lower-level node-based environments such as TouchDesigner, Max/Jitter, Isadora, vvvv, etc. They are more difficult to use and don't provide as much out-of-the-box high-level functionality as Magic, but you could create totally customizable graphical interfaces (and haptic interfaces), for example.
nosuchtim
Posts: 5
Joined: Fri Nov 24, 2017 10:27 pm

Re: programmatic mechanism for generating/modifying scenes

Post by nosuchtim »

> It seems like having 12 Magic scenes would be enough to satisfy your needs?

It's the process of coming up with those 12 "best of" scenes (or 24 in the latest Space Palette Pro) that would benefit from the large expansion of pipelines that could be generated and evaluated more quickly. The reason there are only 12 or 24 choices in those systems is because it's so painful to explore the possibilities that exist - the inability to programmatically re-order the pipeline in Resolume is one of the things (besides ISF support) that is motivating me to explore other options. I'm not doing this without knowing that the benefit is real - in LoopyCam, I built and heavily used a mechanism to generate/rearrange new pipelines instantly, allowing the huge space of potential pipelines to be explored interactively and improvisationally, only saving the ones that turn out worth saving. As far back as my early work with KeyKit, the ability to generate guided (not completely random) algorithmic possibilities (musical or visual) in under a second, combined with the human ability to decide almost instantaneously "yes I like that" or "no I don't", has proven its value to me repeatedly. In visuals, the human evaluation is even quicker than in music.

I'm very familiar with all of the other systems that have been mentioned so far.

> There is nothing awkward or embarrassing about it -- it's just a very complicated file format, because each individual module, module parameter, and module connection is described in detail.

That's exciting to hear, actually! I've submitted a request to see how much it would cost to expose that. If nothing else, it will be good motivation for me to go back to rolling my own stuff which, using things like Spout, can still be combined with and benefit from systems like Magic or Resolume. I greatly appreciate the time you're taking to comment on my needs.

...Tim...
Magic
Site Admin
Posts: 3440
Joined: Wed Apr 09, 2014 9:28 pm

Re: programmatic mechanism for generating/modifying scenes

Post by Magic »

It's no problem :). But I hope you realize that in order to read/generate your own .magic files, you would have to develop your own internal representation of Magic's scene structure, which would take you a very long time to get right. Especially since you are doing it only for your own research, it would probably be faster just to use Magic's GUI to rearrange things. One of the basic purposes of Magic is to provide a convenient GUI that eliminates the need for the text-based programming you are looking to do.

The other point Sadler made is important. Tweaking the individual module parameters, rather than the more general order of modules, is really what's going to make the biggest difference for how your scenes look. And your programmatic methods won't improve the speed of the parameter tweaking or your ability to evaluate the differences.
nosuchtim
Posts: 5
Joined: Fri Nov 24, 2017 10:27 pm

Re: programmatic mechanism for generating/modifying scenes

Post by nosuchtim »

> Tweaking the individual module parameters, rather than the more general order of modules, is really what's going to make the biggest difference for how your scenes look.

My experience shows that the order accesses a different and often more diverse space of possibilities that is difficult to predict and exciting to explore, along with tweaking of parameters. With the much larger selection of ISF effects available, being able to quickly try them in different combinations is key to my goals. In any case, congratulations and thank you for your support of ISF!

...Tim...
Post Reply