Previous topicNext topic

Key - live video

Questions, comments, feedback, etc.
Post Reply
D1gits
Posts: 99
Joined: Thu Feb 25, 2016 11:49 am
Location: Stockholm

Key - live video

Post by D1gits »

Hey guys.

I wonder if there is a "simple way" of having a live video feed and extract the moving objects without green screen. Like taking a picture of the background, without persons in it and use that image as a difference key mask (or what the name is of that effect). I'd love to be able to send keyed video to resolume via MMV. Any Idéa in how to achieve this?

All the best.
Sadler
Posts: 1139
Joined: Sat Aug 02, 2014 7:10 pm
Location: London, UK

Re: Key - live video

Post by Sadler »

If the background plate is absolutely identical. Be difficult though. Components of the foreground might still match the background.

Could you try a depth camera to generate a mask?

Or you could always invest in a Lytro light field camera! ;)
D1gits
Posts: 99
Joined: Thu Feb 25, 2016 11:49 am
Location: Stockholm

Re: Key - live video

Post by D1gits »

Wow yes a Lytro camera would be perfect for the job :)
But I have not heard of a live capture card for them thou...

In my vision I would just take a picture of the stage in its lightning and all just before entering it. That would be the perfect match for the difference key (or difference matte), right? But maybe its to low res or so. But since apple photo booth does it semi good I guess that its possible to make it close to perfect ;)
Magic
Site Admin
Posts: 3440
Joined: Wed Apr 09, 2014 9:28 pm

Re: Key - live video

Post by Magic »

I think you could accomplish this via an ISF shader. I don't have time to try at the moment, but it would work something like this:
1) An "event" parameter would let you capture the current input frame and store it indefinitely as a temporary buffer;
2) Then, an algorithm would compare the stored buffer with every new input frame, and perform a comparison on every pixel. It would be something like this: if the difference between the current pixel and the stored pixel is less than a particular threshold, output nothing (0 alpha); otherwise output the current pixel.

Anyone want to give it a shot? :)
D1gits
Posts: 99
Joined: Thu Feb 25, 2016 11:49 am
Location: Stockholm

Re: Key - live video

Post by D1gits »

If I only knew a bit more of how to program I'd start right away! Hope to se something in this direction in the future : )
D1gits
Posts: 99
Joined: Thu Feb 25, 2016 11:49 am
Location: Stockholm

Re: Key - live video

Post by D1gits »

Hello there again... I wonder if any one could describe a bit more how to achieve the ISF shader that Eric was describing. I got some friends that knows how to code in general. But maybe not ISF. So I need to point out where to begin. Is there a basic shader to start with, that stores a buffer from an input or example or where do I start?

Id love to get this function working for a project I'm doing in July... Or have any of you guys come across any other way to key out live video?


All the best.
Terry Payman
Posts: 710
Joined: Sun Sep 14, 2014 8:15 am
Location: UK
Contact:

Re: Key - live video

Post by Terry Payman »

Some issues I've had when working with the difference between a fixed reference frame and live frames, which are still relevant to Eric's proposed processing:

1) Camera must be rigidly mounted with respect to the subject area, and isolated from vibration (deep bass, people jumping around etc).

2) Camera must be fully manual - No autofocus, no auto exposure, no auto sensitivity, no auto colour.

3) Lighting must be totally invariant. No disco lights! No daylight.

4) Nothing must move in the background (like a flapping screen)

5) Camera and background mustn't see any light from the Magic output.

6) Camera output should have minimal noise.

7) Moving performers must not cast a shadow on the background, or affect the amount of light falling on it indirectly.

Even with awareness of the above, I found it difficult in practice. Camera would drift a bit, performers would move stage props (mic stands, drum stools etc). The reference frame is then spoiled, but during a performance it's hard to get a new frame from an empty stage.

Furthermore, the "difference" output often does not have clean edges and is not a filled solid. As Sadler pointed out "Components of the foreground might still match the background."

Helpful:
a) Illuminate and image in the near infra-red. This gives immunity to interference from projectors and LED lights (sadly not to any tungsten light, or to daylight)

b) Use "Greyscale" on the VideoCapture module, to avoid colour noise. Blue and red channels are much noisier than the green, and "Greyscale" uses mainly the green.

c) Try to have a uniform black background. With infrared lighting even a simple luminance key can give good results.

A much easier setup is to have a reference input that is a long-term average rather than fixed. This shows only edges. Uniform areas disappear, so one doesn't get a solid fill. Nevertheless, it can be very useful.

I'll see if I can implement Eric's suggested processing, passing the live input pixel rather than my previous method which passes the difference between live and reference.

BTW, what OS are you using?

Under Windows 8.1 & 10 the Microsoft Kinect for Xbox One has a depth camera, and interfaces via USB3 so no capture card needed. Not currently supported by Magic, but I'm about to experiment with the sensor in the hope that it will be supported by the expanded video capture facilities that Eric is developing for a future release.
artnik
Posts: 268
Joined: Tue Jan 05, 2016 5:33 pm

Re: Key - live video

Post by artnik »

D1gits wrote: Is there a basic shader to start with, that stores a buffer from an input or example or where do I start?.
Try starting with this Chroma Key ISF: https://www.interactiveshaderformat.com/sketches/642
Magic
Site Admin
Posts: 3440
Joined: Wed Apr 09, 2014 9:28 pm

Re: Key - live video

Post by Magic »

I agree with everything Terry said. Even if you have the perfect algorithm, you will still have many technical difficulties.

I experimented with the following shader a couple months ago, but I couldn't get it to look very good. Maybe you will have better luck :)
RemoveBackground.fs
(896 Bytes) Downloaded 461 times
You can test it like this:
RemoveBG.png
RemoveBG.png (37.24 KiB) Viewed 10373 times
But I recommend that you use a captured image in an Image module rather than the FreezeFrame, because you will want to be able to save the image more permanently.
Terry Payman
Posts: 710
Joined: Sun Sep 14, 2014 8:15 am
Location: UK
Contact:

Re: Key - live video

Post by Terry Payman »

Many thanks for the RemoveBackground module Eric.
I'd just finished implementing a simple threshold version, but your smoothing function hugely improves noise rejection.

Although testing showed that your smoothing worked "perfectly", I had difficulty understanding how this was achieved by
...
alphaMult = (avgDiff-Threshold)/Softness;
...
so I eliminated the conditionals and simplified to
...
float alphaMult = smoothstep(Threshold,(Threshold+Softness),avgDiff);
gl_FragColor = alphaMult * IMG_THIS_PIXEL(fgImage);
...

Did you compare your linear smoothing gradient with smoothstep's curve? To my eye, the results are identical.
Eric wrote:... I recommend that you use a captured image in an Image module rather than the FreezeFrame, because you will want to be able to save the image more permanently.
Really vital! I've often found that freeze frame modules have been re-triggered against my wishes - certainly if they or anything "downstream" is deactivated/re-activated, possibly also with scene changes, as well as the obvious vulnerability if the computer needs to be restarted.

Further help with noise reduction. (I'm puzzled why "Dilate" apparently erodes and "Erode" apparently dilates. Guess it depends on whether one considers the action on blacks or whites, but to me it always seems counter-intuitive with these ISF modules).
EDIT: There is no penalty for motion blurring before the freeze frame. This reduces overall noise by approx 50%. The Dilate-Erode processing slightly reduces image quality, but hugely reduces noise. On balance I need the noise reduction more than the resolution.
Attachments
Reduce noise.jpg
Reduce noise.jpg (89.3 KiB) Viewed 10342 times
Magic
Site Admin
Posts: 3440
Joined: Wed Apr 09, 2014 9:28 pm

Re: Key - live video

Post by Magic »

Did you compare your linear smoothing gradient with smoothstep's curve? To my eye, the results are identical.
Yeah I'm pretty sure I tested with that but I couldn't notice a difference either. I think I decided that the linear equation was faster, but honestly I don't remember :). Either method is perfectly fine.
I'm puzzled why "Dilate" apparently erodes and "Erode" apparently dilates
Hmm I'm not sure, I haven't really looked at those closely. But I'm gonna try out your scene and see how it looks :)


Edit: BTW, if you look at the graph of the smoothstep function, you can see that if you were to compare it with a straight line, it wouldn't be too far off. Like this:
Image
So that's why we can't really notice a difference. But if we were using smoothstep for some other purpose, we might notice it more. For example, if we were "easing" from one sort of thing to another: https://developers.google.com/web/funda ... -of-easing. Easing tends to use a smoothstep-like function for more natural motion. The difference between linear motion is very noticeable when applied to moving objects. You probably know about this already though :)
D1gits
Posts: 99
Joined: Thu Feb 25, 2016 11:49 am
Location: Stockholm

Re: Key - live video

Post by D1gits »

Wow! Thanks for this!
I think this is getting close to the tool Im after... Thanks for the RemoveBackground.fs Eric! And thanks Terry for the nice setup you got running!
Will check it all out with proper cameras instead of my webcam I'm using at the moment.

What a great start of the week : )
Terry Payman
Posts: 710
Joined: Sun Sep 14, 2014 8:15 am
Location: UK
Contact:

Re: Key - live video

Post by Terry Payman »

Many thanks for your detailed answers Eric.
Likewise for the "Easing" link - very interesting. I didn't know the term and hadn't considered all the alternatives. Subsequent searching found this https://www.npmjs.com/package/glsl-easings which might prove useful.

A couple of further thoughts on the image processing.

- Process Alpha separately
- On the alpha layer, if a background pixel is surrounded by foreground pixels (ie background breakthrough) then set it to 1. This should give a solid silhouette for keying. Some sort of threshold and smoothing might be useful, not sure. Foreground must not have such processing, or eg eyes would disappear.

...and another thing to guard against in the background
- Avoid reflective or shiny surfaces, which can mirror movement from almost anywhere outside the camera's field of view.
Sadler
Posts: 1139
Joined: Sat Aug 02, 2014 7:10 pm
Location: London, UK

Re: Key - live video

Post by Sadler »

Just found a webcam app that offers (but doesn't quite deliver for me, YMMV) background removal. I guess given a good camera and the right conditions it might work more consistently than my attempts.

https://www.tridef.com/products/smartcam

The background removal moved the mask too much but it is still quite fun to play with. Only a 7-day trial to try for yourself.
D1gits
Posts: 99
Joined: Thu Feb 25, 2016 11:49 am
Location: Stockholm

Re: Key - live video

Post by D1gits »

Thanks fot the link Sadler : )

I tried it out. But got the same results as you. The mask is quite not right the whole time. But its nice to see that softwares like this is making its move in creating a green screen less key feature! I really look forward to see what the new Depth API for developers in iOS 11 could achieve in this particular challenge...
Sadler
Posts: 1139
Joined: Sat Aug 02, 2014 7:10 pm
Location: London, UK

Re: Key - live video

Post by Sadler »

This is not real-time unfortunately but does look to provide consistent background removal using a clean matt. Hopefully it will come to users fairly soon in some form or another.

https://grail.cs.washington.edu/project ... d-matting/
enuame
Posts: 3
Joined: Fri May 22, 2020 6:38 pm

Re: Key - live video

Post by enuame »

I'm going to put some input into this, as I work in visuals as well as music and programming.

As Terry pointed out, there are a lot of considerations when it comes to "Software" related Chroma Key and to be blunt it doesn't work and never will work in the way of what you are trying to achieve. Even with the latest technology in camera software for hardware Chroma Key is 100% flawed outside of the use of a Green Screen.

There are so many variables you have to consider and not just the few pointed out by Terry and even the best programmers in the industry who work for companies like Logitech who produce Chroma Key Cameras or software houses like Xsplit, have not been able to crack it and even their own software has major flaws in it.

What I'm trying to say and I don't want to dishearten you, but you will never get the same from a Chroma Screen that you would from software, it's not possible and never will be, because of the variables needed. But don't think using a Chroma Screen doesn't have it's flaws, as it does and to use one perfectly, it has to have 0% movement in the screen, otherwise one slight movement in the screen, can kill hours of production.

I know it isn't the solution you want, but trying to rely on software based isn't the solution and will cause more problems than what you can solve in "trying" to achieve it.
Post Reply