How the hell do you only have 16 patrons?! Voukoder is an essential part of my workflow, which would completely fall apart without it! Adobe's h264/h265 encoders absolutely suck, for a while I was rendering out into an intermediate format and then feeding it into ffmpeg. Those were not good days.
It's disappointing that Patreon isn't working out. It's sort of a rule, if I'm using a tool commercially, I pay for it. Voukoder & OBS are the two I can't live without.
I really don't see a clear way to "sell" the program without something bad happening. While gating some features into a trialware model, that sort of sucks from an end-user point. You'd have to gate something really important to push people into a purchase, and that just sort of breeds resentment. Like you say, you really can't go closed-source because of the terrifying licensing fees, but if it remains GPL, some jackass will "fork" a free version somewhere.
It sounds like you've got some ideas, I hope we're all able to work something out so we get a great tool, and we pay your rent!