Audio delay(s) to compensate for latency

[Terry Payman]

Accumulated latencies can significantly reduce the visual impact of audio-responsive elements of a scene.

(Audio DSP filtering window) + (Audio envelope smoothing) + (Shader/buffer latencies) + (GPU double-buffering latency) + (Projector/display latency) = signifcant lag

I suggest compensating for these by adding a facility for variable audio delays via a slider control to facilitate "tuning". One such delay would be very useful, but finer-grained control would be possible if additional delays could be used to feed audio inputs to scenes/shaders having less than the longest delay, such that all paths had the same total delay.

Useful for when Magic is playing an audio file, both realtime or when recording a video. Perhaps also usable realtime if external audio could be "passed through" Magic with the delay.

A workround is to use the existing Start Time facility to delay the start of the audible sound, whilst having a muted duplicate file (with different name) to drive the Magic inputs. This is obviously inefficient on resources, needing the entire length of an audio track to be buffered in memory rather than ~200ms.

[Magic]

Thanks, I do understand the real-time case. But I recommend that you don't incorporate delay into a video file -- it should be hardware-independent.

A workround is to use the existing Start Time facility to delay the start of the audible sound, whilst having a muted duplicate file (with different name) to drive the Magic inputs. This is obviously inefficient on resources, needing the entire length of an audio track to be buffered in memory rather than ~200ms.

If you don't load your audio files into memory, the resource usage is extremely minimal. Your workaround is actually a pretty good solution .

[Terry Payman]

.... I recommend that you don't incorporate delay into a video file -- it should be hardware-independent....

On reflection, I woulldn't want to incorporate compensation for projector lag or GPU double-buffering into a video file - I agree the file should be hardware-independent.

I would still expect some delays due to audio envelope smoothing, & shader buffers - surely I still need to compensate for these?

...Your workaround is actually a pretty good solution...

It would be very much easier to optimise the delay if it could be adjusted in the same way as the audio level, with a slider.

[Magic]

I would still expect some delays due to audio envelope smoothing, & shader buffers - surely I still need to compensate for these?

Do you mean the Smooth modifier? I'm not 100% sure what you mean by shader buffers.

[Terry Payman]

I would still expect some delays due to audio envelope smoothing, & shader buffers - surely I still need to compensate for these?

Do you mean the Smooth modifier? I'm not 100% sure what you mean by shader buffers.

Yes, the Smooth modifier. The buffers I have in mind are frame buffers used by Trails and similar modules, where the output is partly a function of a previous frame, and not just the current inputs. Perhaps shaders is the wrong term.

[Magic]

Ok I think I understand.

I would still expect some delays due to audio envelope smoothing, & shader buffers - surely I still need to compensate for these?

IMHO, "surely" is relative . I would say it's an aesthetic preference. And if you keep in mind that Magic was primarily designed for live audio input, you'll understand why there isn't any latency compensation functionality (yet). You'd need a time machine to know what audio is coming in the near future; there's no way to "read ahead" in real life. All you can deal with is what's happening right now, and if the visuals lag a little bit because of smoothing/buffering, then that's what you get .

I don't mind thinking about how to add some functionality for latency compensation for audio files (beyond just one global slider), but it would be very tricky. Modules/modifiers only know about the audio data that is happening right now (either coming into the sound card or playing out of it). It would require a significant redesign for every individual buffered module and every individual Smooth modifier to ask for its own separate read-ahead audio track. Lots of caveats to consider.

[Terry Payman]

Ok I think I understand.

I would still expect some delays due to audio envelope smoothing... surely I still need to compensate for these?

IMHO, "surely" is relative . I would say it's an aesthetic preference.

With great respect, no . The aesthetic decision to compensate for latency is implicit in the specific context. We were discussing which delays should be considered in the production of latency-compensated but hardware-independent video files. Either the Smooth modifier introduces such a delay or it doesn't. I say "surely" it does .

I don't mind thinking about how to add some functionality for latency compensation for audio files (beyond just one global slider)

... but it would be very tricky...It would require a significant redesign for every individual buffered module and every individual Smooth modifier to ask for its own separate read-ahead audio track. Lots of caveats to consider.

IMHO it can all be done within the Audio/MIDI window with little more than the present functionality. No need for any changes to modules or modifiers. The essence is to give a selection of differently timed Inputs from which a subjective choice of the optimum delay compensation can be made by the user as each modifier is being adjusted.

A possible means to facilitate the requested functionality, assuming just one audio file and avoiding the present need to make duplicates:

1) Remove the restriction on multiple "import" instances of the file. Each instance can then be given an independent start time as well as level. Ideally an existing "Imported" entry could be copied in the same manner as a Playlist entry.

2) The Audio/MIDI window's Input assignment can then be used to give multiple timing options to modules/modifiers. Each Input would have a different delay. Appropriate input naming would help subsequent choice (eg between inputs named "+t","++t","+++t", "++t -6dB")

3) When designing a scene, each modifier can then be fed from to whichever of the differently timed "Input"s gives the best subjective result. If no existing Input is satisfactory, another virtual audio file with a different delay and/or level can readily be created and assigned to an additional Input.

Arguably, and perhaps preferably, each Input could request a start-time offset (as well as the existing level offset) from whatever was feeding it (Audio or MIDI file), avoiding the multiple import instances of the same file needed in the above example.

And if you keep in mind that Magic was primarily designed for live audio input, you'll understand why there isn't any latency compensation functionality (yet). You'd need a time machine to know what audio is coming in the near future; there's no way to "read ahead" in real life. All you can deal with is what's happening right now, and if the visuals lag a little bit because of smoothing/buffering, then that's what you get

With great respect, I disagree. In the context of music timing, one can be almost certain when the next beat will occur. No time machine needed, just BPM
If one delayed live audio to a modifier/module/scene by (1 beat - latency) it would give that scene an effectively instant response, albeit timed from the previous beat.

I'd been content with the "little bit" of lag in the visuals until very recently, when I saw an AV presentation in which the response to the sound appeared instant. Despite very little change in the visuals, the instant response gave a huge visual impact. On enquiry I discovered that the sound feed had been delayed. Subsequent experiment confirmed that a significantly higher visual impact can be achieved by this technique. Ground zero

[Magic]

Either the Smooth modifier introduces such a delay or it doesn't. I say "surely" it does .

Of course it does -- I agree . That's the consequence of the way it works. But whether or not you need to compensate for it is entirely up to you. There is no implicit need. The Smooth modifier can be used in any number of ways in which the delay is actually desirable, or irrelevant.

IMHO it can all be done within the Audio/MIDI window with little more than the present functionality.
1) Remove the restriction on multiple "import" instances of the file. Each instance can then be given an independent start time as well as level. Ideally an existing "Imported" entry could be copied in the same manner as a Playlist entry.

You're right, it would be fairly easy for me to allow more than one copy of an audio file to be imported. Or perhaps with a bit more work, allow for a "virtual" copy which references the original track's data (so there doesn't have to be a duplicate in memory). Either of those would solve the problem, with the exception of a more convenient slider instead of the Start Time dialog. I'm not sure I understand what your other steps are for?

With great respect, I disagree. In the context of music timing, one can be almost certain when the next beat will occur. No time machine needed, just BPM
If one delayed live audio to a modifier/module/scene by (1 beat - latency) it would give that scene an effectively instant response, albeit timed from the previous beat.

That might work if all you care about is the overall beat, but remember that Magic isn't just listening for the beat. The Volume feature picks up all changes in volume not necessarily related to the beat, and even more importantly, the other features (Tone, Pitch, and arbitrary frequencies) have little or nothing to do with the beat. Magic was specifically designed to be instantly responsive to all aspects of the audio, and to go deeper than just BPM. If you only want BPM, you might want to try using an Oscillator modifier.

I'd been content with the "little bit" of lag in the visuals until very recently, when I saw an AV presentation in which the response to the sound appeared instant. Despite very little change in the visuals, the instant response gave a huge visual impact. On enquiry I discovered that the sound feed had been delayed. Subsequent experiment confirmed that a significantly higher visual impact can be achieved by this technique. Ground zero

Yes, of course the visual impact is much greater with proper synchronization! But, overall, the biggest source of synchronization issues is always hardware, not software. Manufacturers and technicians give a lot of thought to how nice their video displays look, but they often overlook the lag. That is why there are now several online resources such as http://www.displaylag.com/ and https://en.wikipedia.org/wiki/Display_lag. You may also want to have another look at https://magicmusicvisuals.com/downloads ... erformance, specifically the section called "Use the right display settings", to see if there are any further lag-reduction steps you can take.

[Terry Payman]

IMHO it can all be done within the Audio/MIDI window with little more than the present functionality.
1) Remove the restriction on multiple "import" instances of the file. Each instance can then be given an independent start time as well as level. Ideally an existing "Imported" entry could be copied in the same manner as a Playlist entry.

You're right, it would be fairly easy for me to allow more than one copy of an audio file to be imported. Or perhaps with a bit more work, allow for a "virtual" copy which references the original track's data (so there doesn't have to be a duplicate in memory). Either of those would solve the problem, with the exception of a more convenient slider instead of the Start Time dialog. I'm not sure I understand what your other steps are for?

The other steps detail the envisaged usage of my proposal to implement latency compensation via Inputs, and to illustrate that it need not be "very tricky" to provide multiple independent delay-compensated feeds into Magic. I further attempted to show that such usage naturally extended the present usage and versatility of your Input concept. Time as well as level can be controlled for each input , and the most appropriate combination can be selected by the user at design time for optimum subjective effect.

Slider(or MIDI) control of the latency compensation would hugely facilitate optimisation and dynamic modulation of this system, and indeed to compensate for whatever hardware constraints are imposed by a particular performance situation.

[Terry Payman]

... continued.

With great respect, I disagree. In the context of music timing, one can be almost certain when the next beat will occur. No time machine needed, just BPM
If one delayed live audio to a modifier/module/scene by (1 beat - latency) it would give that scene an effectively instant response, albeit timed from the previous beat.

That might work if all you care about is the overall beat, but remember that Magic isn't just listening for the beat. The Volume feature picks up all changes in volume not necessarily related to the beat, and even more importantly, the other features (Tone, Pitch, and arbitrary frequencies) have little or nothing to do with the beat. Magic was specifically designed to be instantly responsive to all aspects of the audio, and to go deeper than just BPM. If you only want BPM, you might want to try using an Oscillator modifier.

Did you mean Delay modifier rather than Oscillator modifier? I can see that Delay could provide the (1 beat - latency) adjustment I need to try this with live audio. Many thanks for the idea.

Yes indeed, in the time domain all I care about is the overall beat. Visceral impact.

I'd been content with the "little bit" of lag in the visuals until very recently, when I saw an AV presentation in which the response to the sound appeared instant. Despite very little change in the visuals, the instant response gave a huge visual impact. On enquiry I discovered that the sound feed had been delayed. Subsequent experiment confirmed that a significantly higher visual impact can be achieved by this technique. Ground zero

Yes, of course the visual impact is much greater with proper synchronization! But, overall, the biggest source of synchronization issues is always hardware, not software. Manufacturers and technicians give a lot of thought to how nice their video displays look, but they often overlook the lag. That is why there are now several online resources such as http://www.displaylag.com/ and https://en.wikipedia.org/wiki/Display_lag. You may also want to have another look at https://magicmusicvisuals.com/downloads ... erformance, specifically the section called "Use the right display settings", to see if there are any further lag-reduction steps you can take.

I'm well aware of hardware lag; it's a particular issue with my interactive video feedback installations. Thanks for the links - I hadn't see tabularised lag data for displays, and hope I can find something similar for projectors.

Belatedly, many thanks Eric for your detailed responses

[Magic]

Did you mean Delay modifier rather than Oscillator modifier? I can see that Delay could provide the (1 beat - latency) adjustment I need to try this with live audio. Many thanks for the idea.

That might work, but what I actually meant was using a Ramp (for example) to provide a BPM-like pulse. The Ramp's parameter is the number of cycles per second, so just divide your desired BPM by 60 and voila, you have a pulsing signal.

I suppose that if you wanted to adjust the exact timing of when the pulse is at its peak, you could use the Delay afterwards.

[Terry Payman]

Did you mean Delay modifier rather than Oscillator modifier? I can see that Delay could provide the (1 beat - latency) adjustment I need to try this with live audio. Many thanks for the idea.

... what I actually meant was using a Ramp (for example) to provide a BPM-like pulse.... if you wanted to adjust the exact timing of when the pulse is at its peak, you could use the Delay afterwards.

It's specifically the adjustment of the exact timing that I need, rather than a free-running source that has no fixed phase relationship to any audio beat (unless the ramp can be reset asynchronously by some means I have overlooked).
That said, when I tried the Delay approach it proved very tricky to determine the required delay empirically, and it therefore seems impractical. On the other hand, audio delay works surprisingly well, and there is no alternative if meaningful waveforms are to be displayed.

If Magic could provide a means of providing a slightly delayed audio output to the PA (from any source, say a DJ)
- DJ could monitor undelayed version on headphones.
- Video projection from camera trained on DJ could be compensated for latency so DJ's projected image would be in perfect time.
- Given a soundcard with two stereo outputs VJ too could easily monitor undelayed sound.

[Magic]

If Magic could provide a means of providing a slightly delayed audio output to the PA (from any source, say a DJ)
- DJ could monitor undelayed version on headphones.
- Video projection from camera trained on DJ could be compensated for latency so DJ's projected image would be in perfect time.
- Given a soundcard with two stereo outputs VJ too could easily monitor undelayed sound.

From my personal experience of playing live guitar for 20 years, and working with many live DJs and electronic musicians, I can assure you that no performer would ever want to have their sound delayed, especially for visuals which are almost always secondary in importance to the music (perhaps someday this will change ). But even if the performer has their own separate audio feed which is not delayed, they can still hear the ambient music coming out of the main speakers, and just a few tens of milliseconds' delay would be incredibly distracting -- not only for properly playing their instrument/equipment, but for the audience to respond/interact with them.

Delaying the audio for the sake of the visuals only makes sense in the context of playing imported audio files directly within Magic. But I can't imagine any DJ who would use Magic as their sole audio source when performing their set!

I do agree that there is a use for latency compensation in Magic, but only for three scenarios:

1) To adjust for your own display device while you are working on a project that will ultimately be exported to a video file. But as we discussed before, you wouldn't want the file itself to have the compensation baked in, because a file should be hardware independent. So really, you'd still have the problem when you played back the file, thus you might as well not worry about it in the first place, or find a different solution (such as tweaking your display device directly).

2) To adjust for someone else's display device when you are using Magic to "play back" your project. But Magic isn't generally meant to be used that way -- it's always best to render your project to a video, and play back the video. I'm not sure why you'd want to use Magic to present a project that isn't reacting to live input, but maybe you can explain .

3) To adjust for certain aspects of Magic that introduce a delay internally, such as the Smooth modifier. This is the most relevant case, and I think it is a valid one, but again, only for non-live scenarios.

[Terry Payman]

If Magic could provide a means of providing a slightly delayed audio output to the PA (from any source, say a DJ)
- DJ could monitor undelayed version on headphones.
- Video projection from camera trained on DJ could be compensated for latency so DJ's projected image would be in perfect time.

From my personal experience of playing live guitar for 20 years, and working with many live DJs
and electronic musicians, I can assure you that no performer would ever want to have their sound delayed,
especially for visuals which are almost always secondary in importance to the music (perhaps someday this
will change )...

...Delaying the audio for the sake of the visuals only makes sense in the context of playing imported audio files directly within Magic...

I didn't (and would never) suggest an audio delay for live music performance. I see great application with pre-recorded music from an external source.

Audio from a jukebox could usefully be delayed for the sake of the visuals, with no distraction to the person pre-selecting the track or playlist. Time will tell if any DJs choose the greater visual impact, but that was just an example of a possible application (in retrospect, the jukebox is a much better example and could have avoided misunderstanding).

Audio delay will be used by at least one performer, myself. I have been commissioned to build an experimental/evolutionary Magic-based system for a weekly club event that is itself experimental and evolutionary. Face-tracking avatars will form one of the live visual inputs (yet more delay, even with a 60Hz camera) and the output of my system will be fed into a digital vision mixer - so even more latency to be compensated. I want perfect lip-sync for the avatars, so anyone choosing to "appear" as an avatar will need to wear very good sound-cancelling headphones so that delayed room sound is not too distracting. I may have to buy some Bose QC35s .

As an external workaround is practicable I don't expect to find delaying a live audio feed as a Magic feature in the near future

For the agreed Magic non-live scenario (3) a means of slider/MIDI adjustment of start time(s) would be greatly appreciated .

[Magic]

That sounds very cool, but I do think you might have some unforeseen problems with the audio delays, especially as being confusing to the participants, so just keep everything I mentioned in mind as you move along.

I have the original Bose QC15s, and I really love them, but they don't cancel out voices very well.

[Terry Payman]

That sounds very cool, but I do think you might have some unforeseen problems with the audio delays, especially as being confusing to the participants, so just keep everything I mentioned in mind as you move along.

I have the original Bose QC15s, and I really love them, but they don't cancel out voices very well.

Many thanks for all your observations Eric. I'd better try rigging up my laptop to do the audio delay and take it to the Bose shop for a trial.