Would the keylase patch help? (https://github.com/keylase/nvidia-patch) Obviously this is only a workaround, but sounds like for whatever reason something is not releasing NVENC? If that's the case, this might allow you to at least function.
I've had problems with Voukoder and VoukoderPro not releasing video memory after a render. So there is evidently an ongoing issue somewhere, causing video card resources to not be released in some instances.