Background Removal / Virtual Green-screen & Low-Light Enhance

Background Removal / Virtual Green-screen & Low-Light Enhance v1.1.13

Go to download
Overview Updates (29) Reviews (33) History Discussion
royshilkrot

royshilkrot

Member
royshilkrot submitted a new resource:

Background Removal - Portrait Segmentation - Remove the background in your camera video with the power of neural networks.

This plugin is meant to make it easy to replace the background in portrait images and video. It is using a neural network to predict the mask of the portrait and remove the background pixels. It's easily composable with other OBS plugins to replace the background with e.g. an image or a transparent color.
Click to expand...

Read more about this resource...
 
ASchneider

ASchneider

Member
royshilkrot said:
@ASchneider
i'm working on the Windows version now. Should be done soon.
The trick is to get all the dependency DLLs bundled with the plugin... but i'll work it out.
If you have any SWE cycles - you're welcome to help!
Click to expand...
Hey there! Those skills are away above my league hahah

I know a couple of friends that does live streams interviewing people, they use NDI so the other side can use Skype with its BG remover, use a green screen JPG as their BG and then chrome it locally. It would be nice to just do it locally with their raw video feed.

Thanks a lot mate!
 
ItachiSan

ItachiSan

New Member
Hey sir, I was looking for you! :D

Jokes aside, I was recently looking for a background removal that I could execute locally (I only found greencam but it is a bit heavy as a browser is required).

I can help with both Windows and Linux, not much time available but I can do testing! Lemme know if I can do anything
 
SadMonsterParty

SadMonsterParty

New Member
I'm super interested in this; I'm currently working with this as a "green" screen; I can alpha out the green and my wooden floor fine separately, but my floor is too close to skin tone. If I could 'green' screen out the green parts and 'yellow screen' out the bottom (and wear pants), it'd be fine.

Any suggestions?

Also I hope to be doing this at venues & parties post-quarantine, so a more intelligent or nuanced 'background removal' process would help when space/lighting isn't ideal.
 

Attachments

  • temp_key2.png
    temp_key2.png
    163.8 KB · Views: 318
C

cairotes

New Member
Hy
Very very interested to get a Windows built. I cannot be of much help since I am not familliar with this programing but I can be your tester...
 
A

ajeitler

New Member
NICE Plugin! I already tried some virtual green screen apps that feed their content into OBS. Doing this locally is a big step forward.
The algorithm does not work as fluid as say Zoom's background removal yet, but quite impressive and for my usecase I will replace the other aproaches with it for now :) TANK YOU for your work!
 
royshilkrot

royshilkrot

Member
royshilkrot updated Background Removal - Portrait Segmentation with a new update entry:

Mac, Linux and Windows support and colorspaces

I've sorted out the build on Linux and Windows, so now it supports all 3 major OSs.

The binaries to install it are in the Github Releases page: https://github.com/royshil/obs-backgroundremoval/releases

I also added support for multiple color spaces. Although still facing some problems with YUV 4:2:0 spaces (e.g. NV12 and I420). Other color spaces seem to be working.
Click to expand...

Read the rest of this update entry...
 
ASchneider

ASchneider

Member
Hello! Thanks for compiling the windows version for testing :)

I was not able to make it to work, this is how it looks like here (2080ti + Ryzen 9 3900x)

obs64_8ICt2jrMcx.png


Any idea why this happens? Thanks!
 
royshilkrot

royshilkrot

Member
ASchneider said:
Hello! Thanks for compiling the windows version for testing :)

I was not able to make it to work, this is how it looks like here (2080ti + Ryzen 9 3900x)

View attachment 70533

Any idea why this happens? Thanks!
Click to expand...
this is obviously some color space conversion problem.
it would be cool if you could open an issue for this on GH: https://github.com/royshil/obs-backgroundremoval/issues

but in general, try to see if you can change the color space format to YUV 422.
i can't fully support YUV 420 yet... need to pin down that RGB conversion still.
 
D

dquille

New Member
Finally, an OBS plugin that can do virtual greenscreen! This is looking great so far royshilkrot!

Testing with a c920 on win10, and the background removal is doing a good job. A little pixely at the edges, but not bad and maybe some kind of blur/antialiasing can improve that in the future.

Works with yuy2 video format in obs camera source properties, but frame rates are severely limited for the c920 with that video format at higher resolutions. For the other video formats available for the camera, the filter crashes OBS with mjpeg, and blacks out the picture with h264 like it's trying to work (I can see from the CPU that it's trying to work) but not doing something correctly. I saw this same black out when using a different logitech driver and the I420 color space, so I'm guessing it's a color space support issue like you've already mentioned above.

And related to framerate, this thing is killing my CPU even at the lower framerates I can test it with. Increasing framerates greatly increases CPU usage. Any options that can be exposed in the filter settings that might make it less impactful on the CPU?
 
royshilkrot

royshilkrot

Member
dquille said:
Finally, an OBS plugin that can do virtual greenscreen! This is looking great so far royshilkrot!

Testing with a c920 on win10, and the background removal is doing a good job. A little pixely at the edges, but not bad and maybe some kind of blur/antialiasing can improve that in the future.

Works with yuy2 video format in obs camera source properties, but frame rates are severely limited for the c920 with that video format at higher resolutions. For the other video formats available for the camera, the filter crashes OBS with mjpeg, and blacks out the picture with h264 like it's trying to work (I can see from the CPU that it's trying to work) but not doing something correctly. I saw this same black out when using a different logitech driver and the I420 color space, so I'm guessing it's a color space support issue like you've already mentioned above.

And related to framerate, this thing is killing my CPU even at the lower framerates I can test it with. Increasing framerates greatly increases CPU usage. Any options that can be exposed in the filter settings that might make it less impactful on the CPU?
Click to expand...

I just fixed the colorspace issue for good on the repo (not released a binary yet) using libOBS media-io internal converters.
so color space problem should be fixed, and performance should increase since it's using optimized converters (instead of my conversion hacks).
i will be exploring other segmentation models that have better performace and offer a drop-down selection for the model.
 
D

Daucus

New Member
Not sure if it's related to the color space issues you mentioned above but I thought I'd post my crashlog here in case it exposes any other issues. Thanks!
 

Attachments

  • Crash 2021-05-01 11-56-31.txt
    150 KB · Views: 111
ASchneider

ASchneider

Member
Hello again! Thanks for the latest update and easy to install Windows files.

The color conversion problem was solved but the performance issue remais a big issue here:

1619970115757.png


Also, the media file used in this test was a 1080p30 local recording, the effect changed its quality to maybe a fourth of its original resolution.

Is there something I'm missing here?

Thanks.
 
You must log in or register to reply here.
Top