I found the issue (or one of them, I wouldn't be surprised if it was multiple), it's in the Intel drivers and/or ffmpeg. vp9_qsv ffmpeg encodes have issues with GoP/I-frames and seeking breaks the playback amongst other things.
I really hope this gets fixed at some point, the hardware acceleration on the new Intel CPUs/iGPUs is excellent, you should look into it if you haven't already. But most of the scene seems to have lost interest in it long ago, because it wasn't working properly. Lavfilters have all but dropped the QSV support. If you read up on the Emby forums e. g. you can see people talking about the latest gen Intel iGPUs easily beating dGPUs in terms of speed and efficiency in transcodes.
So, when this gets fixed in the drivers, there's also gonna be requirements for the parameters that are needed, I'll look into that when the time comes.