Use Audio Streams in Video as Inputs?

[hiniko]

Hi there,

First, this is an amazing piece of software. You will be getting money from me as soon as I get paid!

The only thing similar to this request is this post, however I felt it different enough to post.

Would you be able to have audio streams from the Video module optionally able to be used as an input source?

For a bit more context:

I have recorded a "Live set" of me playing around of my phone where the audio is being mixed, straight into the input of Quicktime which is recording via a webcam. As everything is externally mixed I don't really need to separate and mess with the audio streams. Except I can't use them as an input to Magic if I don't.

So for now I have, and I have noticed that after a while the video desyncs with the audio. This happens after about 5 minutes and gets worse as the resulting video continues (it's about 20 minutes long). I have tried a few ways of fixing this, internally and externally from Magic.

Internally, I have tried speeding the video up slightly. however without a way (that I have found) to reliably trigger the start of the video with playing from the Audio/MIDI window I can't know really know for sure until rendering has finished what the result is like.

Externally I have tried to stretch the video to match, but that is a *lot* of faffing about.
My current options are to run with the original recording, which kinda sucks because I had a nice pulse effect and a default branding + EQ visual that sit in a default magic project, spend that faff time, or to just live with it (I don't know if I can!)

Of course only you really know what is going on under the hood (My coder inquisitiveness makes me want to know as well =P) but I have a feeling from the current UI that it may not be trivial? My first thoughts are an option in the video module to expose streams as input or a "Audio Input Module" you can link to do the same.

Let me know what you think or if you have any other thoughts on how to achieve my goals.

And thanks again. Magic is amazing and so are you!

[Magic]

First, this is an amazing piece of software. You will be getting money from me as soon as I get paid!

Thanks!

Of course only you really know what is going on under the hood (My coder inquisitiveness makes me want to know as well =P) but I have a feeling from the current UI that it may not be trivial? My first thoughts are an option in the video module to expose streams as input or a "Audio Input Module" you can link to do the same.

Let me know what you think or if you have any other thoughts on how to achieve my goals.

For your situation it gets a little bit tricky. Magic wasn't designed as a video "editor" per se, but rather as a video "generator". So although it may seem strange and/or counterintuitive, it's actually not easy to sync up pre-existing audio and video. Even though Magic does allow you to import videos using the Video module, the intent is really to use the videos as creative elements in the scene which are manipulated in some way. This is especially true when the videos are looped, have their speed adjusted dynamically, or have other effects applied which don't follow a constant linear timeline. Know what I mean?

Internally, I have tried speeding the video up slightly. however without a way (that I have found) to reliably trigger the start of the video with playing from the Audio/MIDI window I can't know really know for sure until rendering has finished what the result is like.

Overall, you should be able to trigger the start of the video fairly precisely from the start of the audio playback. I described the process in this post: viewtopic.php?f=2&t=542. Have a look there and let me know if it helps any. It might solve the problem, although you may still need to speed up the video slightly.

[hiniko]

Hey Eric,

Thanks for getting back to me.

I do understand what you mean about intended use. It's a shame in a way because I feel the processing pipeline you have available to you in magic is so intuitive and easy to use to get great effects. There is nothing like abusing tools to do jobs they happen to do well =P

Thank you for that link to that post, I'm sorry I didn't find it before. I will keep that in mind next time I attempt this. Also I think the cause of the worsening desync might have been down to a recording corruption. There was a 3 second period where the frames in audio and video freeze and carry on. When you watch the original recording in a player this is fine, but split up it falls apart.

For now I took the video in to a video editor and finished it off, but in cause you're interested here is what I was going for and here is what I ended up with!

Thanks again!

[pinmode]

I'm seeing the same issue. Why does the video slowly desync with audio from the same file that is played externally? It's easy to manually start them at the same time but frustrating that this isn't a solution without more tweaking and time spent.

[pinmode]

It seems to line up with the speed set to 1.001x.

30 - 29.97 = 0.03

0.03 / 30 = 0.001

Maybe this is related to drop frame? The video file is 29.97.

[Magic]

Hard for me to say without seeing the exact video file. Is there any way you could share it via Dropbox or Google Drive and post the link here? Or you can contact us via https://magicmusicvisuals.com/contact if you want to send us the link privately.

[pinmode]

Sorry, unfortunately I won't be able to upload these. They're really big and my net connection isn't great at home. In lieu of that, here's some detail about the two files where this workaround was needed:

MPEG4 Video (H264) 1920x1080 30fps 71880kbps [V: Alias Data Handler [eng] (h264 main L5.0, yuv420p, 1920x1080, 71880 kb/s)]
MPEG4 Video (H264) 1920x1080 29.97fps 40271kbps [V: Alias Data Handler [eng] (h264 main L4.1, yuv420p, 1920x1080, 40271 kb/s)]

Setting it to 1.001 appears to sync it up every time up to the end of the video which is 1.5 hours. I'm playing the audio with Audacity.

[blackdot]

did i get that right that you want to sync an audio file with a video file and create a new video with sound? i'd just use a video editing tool like premiere, or blender if you're
limited to free stuff.

align your stuff and stretch one or the other accordingly if necessary.

[Magic]

In lieu of that, here's some detail about the two files where this workaround was needed:

MPEG4 Video (H264) 1920x1080 30fps 71880kbps [V: Alias Data Handler [eng] (h264 main L5.0, yuv420p, 1920x1080, 71880 kb/s)]
MPEG4 Video (H264) 1920x1080 29.97fps 40271kbps [V: Alias Data Handler [eng] (h264 main L4.1, yuv420p, 1920x1080, 40271 kb/s)]

Ok thanks, although those are pretty standard specs so it's still hard for me to say. Are you seeing the same problem for both files? Since the first one is 30fps and not 29.97, it seems unlikely that the problem is related to drop-frame.

I'm playing the audio with Audacity.

Out of curiosity, why are you playing the audio with Audacity rather than directly in Magic?

[pinmode]

@Eric - maybe a useful test would be to play it in Magic and VLC or MPC-HC at the same time, let it go 30 minutes and check the sync. I'll try this. I agree that drop frame might have nothing to do with it. In any case, I don't really consider it to be a problem considering your vision for Magic's architecture. As for not playing it in Magic's audio player, I just didn't consider it - I'm going to try this and see if the same behavior exists.

@blackdot - I've already synced the scene changes in Premiere and now I'm overlaying Magic visuals. It's a long piece and it would require some work to overlay visuals *and* trigger the scene changes in time, which is why I prepared it in Premiere like this.

[pinmode]

@Eric - just want to confirm that the Magic audio player didn't make a difference. The video played slightly too fast as before and using 1.001 speed fixes it. Thanks for your responses!

[dj0le]

One time, I was asked to overlay a bunch of audio reactive effects onto a video of a live performance. I think this is similar to what you initially were trying to do. I couldn't use the original video in Magic because I didn't want to re-render it and degrade the quality which wasn't super high to start with.

Anyway, the way I ended up doing it was: I used ffmpeg to rip out the audio track from the original movie. Then, in Magic, I used that audio to create the reactive effects on a transparent background. Once I had them created, I rendered out the effects as transparent PNGs. Finally, in ffmpeg, I used the filter_complex command to combine the original video and the PNG's together and it was nice and perfectly synced.

[o0_music]

I'm seeing the same issues. My process usually involves shooting live jam videos on one of more cameras ( GoPro ) with reference audio but record the real audio via USB straight to the computer. I then do a first pass sync and trim of the videos with the video and the clean audio. So far so good.

The problem is that I then use Magic to create some visual overlays that are triggered by the music, but that need to stay in sync with the original video layers. Everything starts in sync correctly, but gets progressively out of sync. I haven't run precise tests, but I think that the sync is based on the difference between 30fps and 29.97fps.

What happens when you use a video as both an audio input and also as one of the layers in the MMV scenes? And should I compensate on the video playback or the audio stream?

But it's probably most obvious in this one:

In the second exampleyou can clearly see that the sound is playing before the play cursor hits an active step, but the graphic overlay is timed correctly, that seems to point toward the sync issue being intriduced while generating the visuals, and not during editing.

Thanks

[Magic]

Hmm, yes, I think I see what you mean. But, the sync problem seems much worse than the difference between 30 and 29.97.

Have you tried adjusting the Speed parameter in the VideoFile module? If so, I'd be curious to know what value ends up working.

Also, if you're willing to share your original video file with us, I'd be happy to take a look. Maybe you can upload it to Dropbox and then contact us (https://magicmusicvisuals.com/contact) with the link? We'll keep it completely private of course.

Cool videos BTW