Question / Help Shadowplay Records Perfectly, OBS does NOT!

So, I am trying to record gameplay for YouTube. I want to use OBS, as Shadowplay not only has shitty audio quality, but it records in VFR. So, Shadowplay is out of the question. However, when I record in OBS, I ALWAYS get the "encoder overload" message, no matter what settings or encoder I use. I would 100% switch to Shadowplay if it weren't for the reasons above, but it just isn't doable. I feel like I have tried every possible thing, and always come out empty handed. ANy ideas? Why does Shadowplay record perfect, yet OBS never does for me?

Shadowplay has the advantage of being able to send the framebuffer directly to the encoder without any transfer over the PCIe bus, or waiting for extra compositing. It's just a straight shot on the card, with compression settings that are known to be within the encoder's capability.

OBS has a lot more capability than shadowplay, but it has to resort to sending that data over the PCIe bus at least twice before it can get to the encoder (more if you're not using the new nvenc encoder). And then... it uses the custom settings that you're free to set.

That's your current problem. You're overloading the nvenc encoder, which really comes down to 2 things -- your resolution/framerate, and your quality setting. For that card, you're not going to be able to use high quality for 1080p60. Just drop the quality down to default and you should be fine... if it still has encoding issues, then you can try dropping down to performance, but that's going to be a noticeable difference in file size to achieve the same quality.