does it work if you select the 'YUV 4:2:0 (8 bit)' template?
Yes that was the problem. I guess I based my templates off of the VoukoderPro YUVA template in Vegas, like you said.
Started fresh with the YUV 4:2:0 8-bit template instead, linked it to my desired Scene, and it seems to render. I'll test it more tomorrow. But it does appear to encode what I told it to do, and we do finally seem to have the speed advantage in rendering.
That being the case, I have a suggestion for your consideration. Once a VoPro template in Vegas is renamed, there is no way to tell what pixel format is being exported from Vegas. And that can lead to confusion, as just witnessed.
So why not have a dropdown menu in the Vegas template itself which allows you to select which pixel format to use? Perhaps accompanied with a helpful hint that if you use YUVA with certain CUDA functions, you're screwed! See below for a mockup of what I mean.
This would serve as both control of the settings, and visual feedback of what the current settings are.
Question: What video/picture format does Vegas pass to Voukoder? Is it raw bitmap, or some form of h.264? If h.264, maybe h264_cuvid would offer an advantage over hwupload_cuda. I found that this was necessary for processing h.264 source material in FFmpeg, seems to be able to cope with a lot of different h.264 flavors. Maybe this wouldn't be useful here. Just a thought. See my command lines in Post #4 above for how I used this.
P.S.: I'll also add the 'hwdownload' filter, but I doubt it makes much sense to upload the frame to gpu, scale it, download it to the cpu, then upload it again to encode it ...
In some specific cases it might make sense. Depends on if you need some oddball filter that NVENC doesn't have, or you might want to balance the workload between your GPU and CPU, or even multiple GPUs with different jobs. There are outlying cases where it makes sense.