Opening NVENC Encoder failed - Randomly

This will probably take a while to narrow this down and it may be similar to another bug posted regarding "GPU fail". Just want to put this in as a placeholder for future updates.

But what I'm seeing is that if I have Effects plugins that use the GPU on the first clip of the work area (that is being encoded), Voukoder sometimes blows.

[16:52:15] ---------------------------------------------

[16:52:15] Opening codec: h264_nvenc with options: 2pass=0|aq-strength=8|b=65000000|bf=2|bluray-compat=1|bufsize=60000000|g=30|gpu=0|init_qpB=51|init_qpI=51|init_qpP=51|level=5.1|maxrate=75000000|nonref_p=1|preset=hq|profile=high|rc=vbr|rc-lookahead=300|spatial-aq=1|strict-gop=0|surfaces=64|temporal-aq=1

[16:52:15] Failed opening codec: h264_nvenc

[16:52:15] Unable to open video encoder: h264_nvenc

[16:52:15] Closing encoders ...

[16:52:15] Opening encoder failed! Aborting ...

[16:52:15] 00000000169A6BA0

OK, finally had the time and the killer Effect to make this happen consistently (at least on one machine).

Attached are the logs. Big takeaway is that the Noise Reduction effect that I'm using is GPU intensive and can utilize it up to 100%. I have it limited to using no more than 50% of video card memory.

No Effect.log = is the work area without an effect

Compile Error.log = example of entry into Media Encoder dialog (from PPro) with minimal GPU activity prior entry/encode

No Exit Message.log = mucking with the Effect (GPU used by Effect), then entry/encode

Total Hang.log = entry into Media Encoder dialog, moving the Current Location bar and getting "Rendering" messages a few times. THEN starting encode.

Interesting on my older machine (GeForce 950 and Intel i7-3770) sometimes it did work just really slowly but on my fancy new GeForce 1080 (2GB) and Amd Ryzen 3900x, consistent failure.

On a related note, is there any way to restrict the amount of GPU memory NVENC sucks up? I foolishly got a card with 2GB (old System had 3GB) and to do further testing on the problem I tried to set it up so I'd have 1st part of the encode without competing usage of the GPU (NVENC get's GPU 0) but that didn't work as the effect needs somewhere > 1GB of GPU ram and NVENC is taking it all.

I combined to issues (sorry), first one is where I've included log files. Under conditions plugin can not see/connect to GPU. Pretty sure there is a bug here somewhere.

2nd is the issue with FFMPEG sucking up GPU memory and not leaving enough for noise reduction plugin that is configured to use NVENC. For this memory sucking, I seem to remember FFMPEG automatically increasing the "Surfaces" (??). Reason I see this as an FFMPEG issue is that I can frameserve the encoding to a 3rd party (TMPGEnc Video Mastering Works) encoder that is configured to use NVENC and it all works fine. I looked at the GPU memory load and it was WAY lower than Voukoder/FFMPEG.

Created a different setup that don't hit the memory issue and it all work. Found my problem ... hidden in the output.

>> BTW, I had a saved preset that I'd transferred from my usual machine to the one I was using for this test. That preset had audio configured to use Fraunhofer but the new machine didn't have the DLL. In the audio settings panel it said I was using regular FFmpeg AAC (no Fran listed) but then I noticed libfdk-aac in the summary panel. End result it that it errored out.