If you want, you can separate it visually: by tabs colors or alignment (encoder on the left, others on the right with the gap in the middle) for example. Maybe somehow another way. However, I do not think that it must be. It may, but not must/have to. Also it can increase complexity of UI, depends on implementation. For me as a user do not matter if some options are encoder dependent or independent. For me is more important that all things on the right (logical) place and I can quick navigate to needed options/section or overview options without scroll. And I am not ffmpeg user. Maybe separating options is more important for ffmpeg users? Or you have some other reason? Could it be done by another way?
Additionally, I just found out that most encoders have significantly less encoding options then x264/265. Show only 1-2 option(s) on a separate tab could be a little bit strange. it also should be sorted out somehow.