It's something to do with your resolution. I tried a known working project on my rig (VP15, 3060Ti, Voukoder 13.0.2, Nvidia driver v531.29DCH patched with keylase), and I can make it choke by changing either of the resolution values (width or height) to your values. Tried 2400x3600, 1920x3600, 2400x1080. In all 3 cases, I get the following error in the log when ffmpeg is being initialized:
FFmpeg: InitializeEncoder failed: invalid param (8): Invalid Level.
When setting resolution back to 1920x1080, everything works again.
According to the documentation, NVENC resolution limit for everything from Pascal to Lovelace is 8192x8192, so it shouldn't be an NVENC problem. Either FFmpeg doesn't like your resolution, or it's not receiving the correct syntax from Voukoder.
You could test for an FFmpeg limitation by CPU-rendering a small test file in Vegas and then trying to re-encode the file using FFmpeg and the NVENC encoder directly from command line.