That's really strange, some days ago i compiled Voukoder myself and I had this issue too. But after a re compile this issue was gone.
Maybe you could try this Voukoder Version. It's compiled with ffmpeg 4.3 and Nvenc SDK 10. This Version works for me without this issue (Ignore the Rav1e Codec).
Just replace this dll with the same one in C:\Program Files\Voukoder.
And log.2020-09-10.zip