Took me WAY too much time but I figured it out, sort of, and figured I'd come back here to close it out: YouTube says on its recommended encoding settings page that it prefers and uses the BT.709 colour space in its videos. I've never heard of this one in particular, but I found that rendering to Rec.709 produces a result without any artifacting. No other setting matters - as long as YT doesn't have to convert the colour space (or does so very lightly) artifacting won't be a problem. It is exactly what I was asking for: everything is the same as what I've been using, but with one singular, pesky setting changed under the hood that makes YT retain quality after reencoding. Funny how that works, huh?
TLDR: use the Rec.709 colour space. That's it. Thanks all who helped me noodle around to find this.