Resource icon

NVIDIA NvEnc Guide

ABOUT THIS GUIDE
The objective of this guide is to help you understand how to use the NVIDIA encoder, NVENC, in OBS. Note: we have simplified some of the concepts to make this guide accessible to a wider audience.

Encoding is all about compressing images. The smaller the size of the image, the less we must compress it and the more quality it keeps. While the same applies for framerate, a viewer can really notice a drop in FPS but not so much in resolution, so we will always try to stream at 60 FPS.

GeForce RTX GPUs have dedicated hardware encoders (NVENC), letting you capture and stream content without impacting GPU or CPU performance. Newer generations of RTX GPU include support for newer, more efficient codecs. The benefit of a more efficient codec can be thought of in one of two ways:
  • Video encoded at same quality with a more efficient codec will occupy less disk space or less streaming bandwidth than a less efficient codec
  • Video encoded with a more efficient codec will be higher quality than a less efficient codec at with the same file size or streaming bandwidth
Relative Compression efficiency of codecs with NVENC:
  • H.264: 1x
  • HEVC: 1.15x
  • AV1: 1.4x
Capabilities per GPU generation:
  • GTX 10 Series: H.264 and HEVC
  • GTX 16 Series: H.264 and HEVC
  • RTX 20 & 30 Series: H.264 and HEVC, and AI powered effects
  • RTX 40 Series: H.264, HEVC, AV1 and AI powered effects
Capabilities per platfomr (as of May 2023):
  • YouTube: H.264, HEVC (beta), AV1 (beta)
  • Twitch: H.264
  • Facebook Gaming: H.264

BITRATE, RESOLUTION AND FRAMERATE

First, run a speed test to determine your upload speed (e.g. Speed Test). We want to use around 75% of your upload speed, as the game and other programs such as Discord will also fight for bandwidth.

Then, we will determine the resolution and FPS that we can use for such bitrate. Most streaming sites have recommendations (Twitch, Youtube, Facebook) on what to use. These are ours:

bitrate.png

Important Note for High Motion Content. If you are going to stream high motion scenes (i.e. Racing games, FPS games, etc.) we highly recommend reducing your resolution. High motion content cannot be compressed as much, and can suffer from more artifacting (encoding errors) that make your stream look “blocky”. If you reduce the resolution, you reduce the data being encoded, and the resulting viewer quality is better. For example, for Fortnite, many streamers decide to stream at 1280x720 60 FPS at 6,000 Kbps, down from 1920x1080.

Note for New and Upcoming Streamers to Twitch. Transcoding allows a viewer to view your video on a different resolution, thus requiring a lower bandwidth. Twitch only offers guaranteed transcoding to Partners; non-partners may receive transcoding depending on server load. This is important if your viewers are on mobile phones or their internet speed is not as fast. You may want to consider streaming at a lower bitrate and resolution to lower the bandwidth required to see your channel.


RECOMMENDED SETTINGS
These are our recommended settings for OBS Studio 29.1 and up. To access the settings, click on the Settings button on the bottom right.

1.png

VIDEO TAB SETTINGS

2.png
  • Base (Canvas) Resolution: Set the resolution you normally play at. That is, your desktop resolution (if you play in borderless mode), or the game resolution you normally use (if you play in full screen).
  • Output (Scaled) Resolution: Enter the resolution appropriate for your Upload Speed and Bitrate, as we discussed in the previous section.
  • Downscale Filter: If your Output Resolution is smaller than the Base Resolution, OBS will use a downscale filter. Selecting a better one will provide a small image sharpness enhancement, at the cost of some encoder workload. NVENC is very efficient and typically runs at low utilization, so we recommend using this with the Lanczos, 36 samples option for the best quality.
  • FPS: Enter the FPS appropriate for your Upload Speed and Bitrate, as we discussed in the previous section.
STREAM TAB SETTINGS

3.png

Select the streaming platform you wish to connect to. You will have the option of signing into your service with your login credentials. Another method is to enter your “Stream Key”.
  • Important Note: Be careful with your stream key! Anyone with it can take over your stream.
Instructions on how to find your stream key can be found on YouTube, Twitch, or Facebook Gaming.

OUTPUT TAB SETTINGS
1683566256432.png

If you want an easy, out of the box configuration, then do the following:
  • Output Mode: Simple
  • Streaming:
    • Bitrate: Enter the Bitrate appropriate for your Upload Speed, as we discussed in the first section.
    • Encoder: Here we will select what hardware will be used to encode, and with what codec.
      • For Twitch, select Hardware (NVENC, H.264).
      • For YouTube, select Hardware (NVENC, AV1) if you have an RTX 40 Series GPU, or Hardware (NVENC, HEVC) otherwise.
    • Encoder Preset: the preset determines the quality of the encode, at a cost of performance for the encoder. Most users should use P6: Slower (Better Quality). If you are trying to execute several encodes at the same time you may want to reduce this present as to not overload the encoder.
  • Recording:
    • Recording Path: This is the directory where the videos will be saved. Make sure the hard drive you select has enough space!
    • Recording Quality: High Quality typically works for most users, but you can change this to Indistinguishable Quality if you have enough disk space or are going to do short videos (about 60 seconds).
    • Recording Format: MKV
    • Encoder: For local recordings you want to make sure that the format you record in is readable by the application you will use the file in. AV1 provides the best quality, followed by HEVC, and last by H.264. But H.264 has the most compatibility with apps, so if you experience any issues with the other 2 you may want to fall back to this one.
OTHER SETTINGS
There are 2 other things you want to configure to ensure a smooth stream:
  • Windows Game Mode: Enable Game Mode. This prioritizes your Game and Streaming Software so they run better.
  • GPU Utilization: If your GPU utilization is above 95% Windows will start prioritizing the game over everything; this can, in some cases, make your stream lag. To solve this, Windows added a new scheduling system called Hardware Accelerated GPU scheduling. This will automatically balance the different workloads on your system so your game, stream and other apps (like a VTubing app) can all work at the same time. Verify or change your setting in your Windows Control Panel under System > Display > Graphics > Default graphics settings.
If for some reason Hardware Accelerated GPU Scheduling does not do the trick, OBS added an option in OBS 24.0.3 to prioritize OBS Studio over everything else. Just run OBS as Admin, and your stream will be silky smooth.

5.png

If for some reason you don't want to run OBS in Admin mode, you can also limit your GPU usage to be below the 95% threshold. To do this, you can:
  • Cap FPS in-game, run the game in Borderless Windowed mode, reduce game graphics or resolution, or turn V-Sync on.
  • Run all assets at 1080p. To do that, double click on the source in OBS and under Resolution select Custom and specify a resolution equal or under 1080p.
And there you have it! We hope this helps you improve your stream quality and reach your goals. Leave us a comment if this worked for you or if you’d like us to update the guide with other info. Happy streaming!


ADVANCED SETTINGS
If you want to mess around with all settings, here are our in-depth recommendations.

Streaming Settings

1683566209303.png
  • Output Mode: Advanced. This gives you access to all the settings. Let’s start!
  • Encoder:
    • For Twitch, select Hardware (NVENC, H.264).
    • For YouTube, select Hardware (NVENC, AV1) if you have an RTX 40 Series GPU, or Hardware (NVENC, HEVC) otherwise.
  • Enforce Streaming Service Encoder Settings: Leave this checked, this will ensure that if you enter a wrong value by mistake it gets corrected.
  • Rate Control: Select CBR. This determines the rate at which frames are going to be encoded.
  • Bitrate: Enter the bitrate appropriate for your Upload Speed, as we discussed in the previous section. Keep in mind that some platforms have a maximum bitrate (i.e. for Twitch it’s currently 6000 Kbps.
  • Keyframe Interval: Set to 2. Streaming platforms may limit what you can select here, and most require a setting of 2.
  • Preset: Select P6: Slower (Better Quality).
  • Tuning: Select High Quality.
  • Multipass Mode: Set to Two Passes (Quarter Resolution).
  • Profile:
    • H.264: Set to High. Profile determines a group of settings in the H.264 Codec. It doesn’t impact performance and gives access to a set of features that are key to streaming, so this should always be set to High.
    • HEVC: Set to main; if you are going to stream in HDR, verify the profile is set to main10.
    • AV1: Set to main (only option).
  • Look-ahead: Checked. This allows the encoder to dynamically select the number of B-Frames, between 0 and the number of B-Frames you specify. B-frames are great because they increase image quality, but they consume a lot of your available bitrate, so they reduce quality on high motion content. Look-ahead enables the best of both worlds. This feature is CUDA accelerated; toggle this off if your GPU utilization is high to ensure a smooth stream.
  • Psycho Visual Tuning: Checked. This enables the Rate Distortion Optimization in the encoder, which greatly optimizes the way you use bitrate, improving image quality on movement.
  • GPU: 0. If you have 2 GPUs in your system, you can select which one is used to encode. This is not recommended, as NVENC is already very efficient and the little gain you can get from using a second card is lost by having to copy the frame to the second GPU.
  • Max B-Frames: Set to 4. If you uncheck the Look-ahead option, reduce this to 2 B-Frames.
Recording Settings

1683566338685.png

  • Type: Standard.
  • Recording Path: This is the directory where the videos will be saved. Make sure the hard drive you select has enough space!
  • Recording Format: MKV
  • Audio Track: Leave it at 1 for default; you can add more audio tracks if you are using more sources.
  • Encoder: NVIDIA NVENC H.264, NVIDIA NVENC HEVC, or NVIDIA NVENC AV1 (RTX 40 series only).
  • Rate Control: We recommend CQP, although VBR will also produce good results.
    • CQ Level (CQ): 15 (you can decrease the number to get higher quality).
    • Bitrate and Max Bitrate (VBR): 40,000 Bitrate; 60,000 Max bitrate. You can increase these to 100,000 and 200,000 (respectively) for higher quality.
  • Keyframe Interval: 0 or 2.
  • Preset: Select P6: Slower (Better Quality).
  • Tuning: Select High Quality.
  • Multipass Mode: Set to Two Passes (Quarter Resolution).
  • Profile:
    • H.264: Set to High. Profile determines a group of settings in the H.264 Codec. It doesn’t impact performance and gives access to a set of features that are key to streaming, so this should always be set to High.
    • HEVC: Set to main; if you are going to capture in HDR, set to main10.
    • AV1: Set to main (only option).
  • Look-ahead: Checked.
  • Psycho Visual Tuning: Checked.
  • GPU: 0. If you have 2 GPUs in your system, you can select which one is used to encode.
  • Max B-Frames: Set to 4. If you uncheck the Look-ahead option, reduce this to 2 B-Frames.

RECORDING AND STREAMING HDR
With OBS version 28 and later, you can record in HDR, and stream HDR to compatible services. Currently you can capture HDR with HEVC and AV1 for local recordings, and stream in HDR to YouTube with HEVC only.

CAPTURING HDR
HDR - or High Dynamic Range - allows you to increase the range of light of your content. You can record and/or stream with HDR.
Note that:
  • HDR content can look washed off in non-HDR displays, so you don’t want to turn this on if your display does not support HDR, or if the platform you are streaming to does not support HDR.
  • HDR requires HEVC; it’s not compatible with H.264.
In order to enable HDR, we’ll need to make sure that:
  1. You have an HDR monitor.
  2. Your Windows and Game settings have HDR turned on.
  3. We configure your full pipeline in OBS to enable HDR.
Windows and Game Settings
Start by making sure that you have HDR turned on for your Display in Windows. Right Click on your desktop > Display Settings, and turn on Use HDR.
1683067218382.png

Next, open the game you want to record or stream, go to the settings and make sure HDR is turned on. In Windows 11 there’s a feature called Auto HDR that enables a virtual HDR mode for any game. This is also a valid option.

OBS Settings
In OBS, start by going to the Advanced tab and changing the Color Format to P010 and the Color Space to Rec. 2100 (PQ).

Next, we need to adjust the Game Capture source. Double click it (or right click it and go to Properties), scroll down find the RGB10A2 Color Space. Make sure this is set to Rec. 2100 (PQ).
1683067218420.png

Your content should now be ready to be captured and/or streamed in HDR! But remember, in order to record and/or stream in HDR you need to change your encoder from H.264 to HEVC. You can visit the How to Record and/or Stream with HEVC section above to adjust this.

HOW TO STREAM HDR WITH HEVC TO YOUTUBE
If you want to stream with HDR to YouTube (only supported service currently), make sure you are using the HEVC encoder, and adjust your Profile to main10.

Finally, navigate to your YouTube Live settings on YouTube.com, and make sure that your Latency setting is set to Low or Normal (do not use Ultra Low as it is not compatible).

ABOUT NVENC
NVENC is NVIDIA’s encoder. It’s a physical section of our GPUs that is dedicated to encoding only. This means that your GPU can operate normally regardless of whether you use this region to stream or record. Other encoders, such as x264, use your CPU to encode, which takes resources away from other programs such as your game. Advanced codecs like AV1 are unable to run on consumer CPUs. This is why using NVENC allows you to play games at a higher framerate and avoid stuttering, giving you and your viewers a better experience.

Codecs efficiency plays an important role in video quality, especially in lower bitrate situations like live streaming. A more efficient codec requires fewer bits (thus lower bandwidth) than a less efficient codec to achieve the same visual quality. The latest AV1 codec is ~40% more efficient than H.264. This also means that more advanced codecs like AV1 will produce higher quality, as if you were streaming with ~40% more bitrate.

What does this mean for live streaming? Simple! Better visual quality can be achieved with the same bandwidth. Upload bandwidth is a variable most streamers have little control over, so codec efficiency is often the biggest player when it comes to stream quality.
Upload bandwidth may be limited by ISP or streaming service caps, falling on average between 6-15 Mbps. Streaming H.264 encoded 4K/60 fps video at this upload speed results in unpleasant blocky, grainy video. Streamed the same video encoded with AV1 results in significant improvements in quality.

NVIDIA has also worked closely with OBS to help optimize OBS Studio for NVIDIA GPUs, improving performance and enabling the latest and greatest features for quality.

One additional advantage of NVENC is that typically, the same version of NVENC is used per GPU generation. For example, a GeForce RTX 4090 and a GeForce RTX 4050 both have the same encoder quality. This is beneficial (and budget friendly!) if you want to get another GPU for your second PC to encode. However, the GeForce RTX 4090, 4080 and 4070 Ti GPUs do offer dual encoders for additional capacity, which is helpful because you can encode at higher resolution and FPS (up to 8K60), or support more simultaneous encodes (e.g. if you want to stream to multiple platforms).

Finally, if you are using an NVIDIA GPU you have access to GeForce Experience’s Game Filters, which allow you to further improve the image quality of your viewers via software by enhancing color, adding sharpness, or introducing cool effects.

ABOUT NVIDIA BROADCAST
NVIDIA Broadcast transforms your gamer den into a home broadcast studio, upgrading standard webcams and microphones into premium, smart devices through the power of AI. Improve the video and audio quality of your livestream through AI capabilities such as virtual background or webcam auto frame, and microphone noise removal. With dedicated AI processors called Tensor Cores on GeForce RTX GPUs, the AI networks are able to run high-quality effects in real-time.

NVIDIA Broadcast features include:
  • Noise Removal: use AI to remove background noise from your microphone feed – be it a loud mechanical keyboard or the AC. The AI network can even be used on incoming audio feeds to mute that one friend who won’t turn on push-to-talk.
  • Virtual Background: use AI to remove the background of your webcam feed and replace it with game footage, a replacement image, or even a subtle blur.
  • Auto Frame: use AI to track your head movement, zooming and automatically cropping on your head so you are always in the frame, even as you move around.
  • Eye Contact: use AI to adjust your eye positioning to make it appear you are staring directly at the camera.
When not broadcasting, these features can also enhance your video conference calls and voice chats, making NVIDIA Broadcast a perfect AI companion for broadcasters and gamers alike.

NVIDIA Broadcast is compatible with a wide range of broadcasting, video conferencing and voice chat apps. These include OBS Studio, Streamlabs, Xsplit, Discord, Skype, Zoom, Webex, Microsoft Teams, Google Meet and more.
Author
findal
Views
171,182
First release
Last update
Rating
4.43 star(s) 7 ratings

More resources from findal

Latest updates

  1. Streaming or Recording with HEVC and HDR

    With OBS version 28, you can stream with HEVC to platforms that support it. HEVC is an improved...
  2. OBS 24.0.3 Admin Mode

    Updated Guide with info about OBS 24.0.3 new solution for high GPU encoding. Forget about...
  3. Windows 10, version 1903 is out

    Updating the guide after the Windows 10 1903 release. For this new version of Windows, the new...

Latest reviews

Please note that you can enable hardware-accelerated GPU scheduling in Windows to have OBS prioritize NVENC encoding so the game rendering doesn't slow your stream down. Running OBS in administrator mode is the other workaround. In Windows 10, search the start menu for "graphics settings", then scroll to the bottom and click "graphics settings" again (smart, Microsoft), then turn hardware-accelerated GPU scheduling on.

Read more here:
https://twitter.com/gerdelgado/status/1496657523700420608
https://twitter.com/search?q=from%3Agerdelgado%20nvenc

And you can find another NVENC guide by the author of the StreamFX plugin here:
https://www.xaymar.com/guides/obs/high-quality-streaming/nvenc/
Hi, i dont know something, more exactly, why obs is giving me an error with nvenc. please help
Perfect guide, appreciate the explanations. Exactly what I'm looking for.
Great guide!
F
findal
Thanks!
Top