OK, made some tests.
It seems to only happen if I go under the "Video" section and change the output resolution there. I can easily encode if the output format is the same as the sequence format.
*tries encoding to 720p instead*
Huh. It works. So, that means that it has something to do when using the SD NTSC options which is kind of weird. Even if I use software, it still happens. Even if I use another format than H.264 (NVENC) or H.264 (x.264) or, heck, even VP9.
*tries changing color space*
Oh. I found the issue. So, it happens when you try encoding 480p content with 4:4:4 color space. It then crashes to desktop. But, when I tried 4:2:0, there was no issues and encoded correctly.
Huh.