HDR support in x265 and nvenc HEVC

1. offizieller Beitrag

    Hi Vouk,

    Thank you for your plugin, it will save us a lot of time.

    Following that thread As a user I would like to export with HDR. I tried to encode an HDR video using the x265 encoder but it seems you removed the options for HDR support in version 2.1 (Released Voukoder 2.1).

    On the github, the translation files still have the options but it seems they were removed from the libx265.json. Are there any reason to that ? I'm trying to encode a video using HLG HDR but the metadata put after the encode are not good:

    Code
    Color range                 : Full
Color primaries             : BT.2020
Transfer characteristics    : PQ
Matrix coefficients         : BT.2020 non-constant
Codec configuration box     : hvcC

    It should be:

    Code
    Color range                 : Limited
Color primaries             : BT.2020
Transfer characteristics    : HLG
Matrix coefficients         : BT.2020 non-constant
Codec configuration box     : hvcC

    Is it possible to add those options again in the x265 encoder ? I'm referring to those options:

    Zitat von Vouk

    Code
    Color range                    : Limited
Color primaries                : BT.2020
Transfer characteristics       : SMPTE ST 2084
Matrix coefficients            : BT.2020 constant
Mastering display color primar : R: x=0.708000 y=0.292000, G: x=0.170000 y=0.797000, B: x=0.131000 y=0.046000, White point: x=0.312700 y=0.329000
Mastering display luminance    : min: 0.0000 cd/m2, max: 1000.0000 cd/m2
Maximum Content Light Level    : 1075 cd/m2
Maximum Frame-Average Light Le : 226 cd/m2

    And is it also possible to add HDR support in nvenc ? I'm currently using StaxRip for HDR encoding using NVENC HEVC and it would be nice if it could be supported natively in Voukoder :). It seems StaxRip is using the build from this repository https://github.com/rigaya/NVEnc.

    A last thing if you can, from what I read in R1 there was an option to pipe directly the flow to our own software, is it possible to add this option again ? For debug and custom pipeline it would be very nice.

    Zitat von morphinapg

    Btw, after much testing, I just figured out a solution using 1.2.1 and the piping out functionality (I couldn't find that feature in R2)

    First, the program that I'm piping to is a build of ffmpeg that supports 10bit x265. I used Zeranoe Windows x64, but if it supports 10bit x265 it should work.

    The piping command I used for my project (MaxCLL 1075 and MaxFALL 226) was this:

    Code
    -i - -pix_fmt yuv420p10 -c:v libx265 -x265-params crf=18:colorprim=bt2020:colormatrix=bt2020c:transfer=smpte2084:chromaloc=2:hdr-opt=1:master-display="G(8500,39850)B(6550,2300)R(35400,14600)WP(15635,16450)L(10000000,0)":max-cll="1075,226" D:\Movie.mp4

    You would obviously need to tweak your MaxCLL/MaxFALL and preferably your mastering display values, as well as the output location. This produces only video, but it could probably be tweaked to include audio as well. I found the output location had trouble with longer directory+file names (Such as D:\Users\Andy\Desktop\Movie.mp4) so I decided to stick it on the root of my drive since I wouldn't know where it would go if I just said Movie.mp4.

    For now this will be my solution until you can release an update with built in functionality. Maybe this sample command line will also help you.

    I'm currently using voukoder-2.3beta4 and adobe-premiere-connector-0.9.2.

    Again thank you for your plugin, this is by far the best one that I could have found for encoding in Adobe.

    EDIT: I can help in the developement of the plugin If you have any tutorial or material to introduce me into the Premiere pro plugin dev.

    Einmal editiert, zuletzt von Wickyed (8. November 2019 um 21:18)

    The options are there, they just work differently. You set color space in the regular premiere export window, and then set the other HDR options in the advanced settings.

    The problem is, the color space settings result in a color space conversion that messes up the HDR colors, because voukoder or premiere treats the footage as rec709 SDR.

    However, yes, there's no option for HLG right now, just HDR10 using the options available as they are now. Reverting fine control of those x265 tags would be good, because they allow more options for color space and transfer characteristics, such as --transfer arib-std-b67 for HLG

    However, as per my other post, the color space setting in the Premiere export window should not be performing color space conversions. They should be telling the encoder what color space the footage already is. I realize the problem is Premiere outputs in RGB32. Basically, this needs to be converted to rec709/yuv420, and then encoded to whatever color space we select without further conversion.

    Refer to the ffmpeg command I typed from a previous post for an idea of how this works:

    Code
    -i - -pix_fmt yuv420p10 -c:v libx265 -x265-params crf=18:colorprim=bt2020:colormatrix=bt2020c:transfer=smpte2084:chromaloc=2:hdr-opt=1:master-display="G(8500,39850)B(6550,2300)R(35400,14600)WP(15635,16450)L(10000000,0)":max-cll="1075,226" D:\Movie.mp4

    This interprets the RGB32 video as yuv420p10, then encodes that directly to whatever color space I wanted (in this case, bt2020c). No color conversion is performed.

    Zitat von morphinapg

    The options are there, they just work differently. You set color space in the regular premiere export window, and then set the other HDR options in the advanced settings.

    The problem is, the color space settings result in a color space conversion that messes up the HDR colors, because voukoder or premiere treats the footage as rec709 SDR.

    However, yes, there's no option for HLG right now, just HDR10 using the options available as they are now. Reverting fine control of those x265 tags would be good, because they allow more options for color space and transfer characteristics, such as --transfer arib-std-b67 for HLG

    Yes, I realize that but I think it would be better to have the option directly in the encoder because there are also a lots of other options that could be added in the VUI section https://x265.readthedocs.io/en/default/cli…rmation-options.

    It would also be great to add those options to the NVENC HEVC encoder which support them now.

    Zitat von morphinapg

    However, as per my other post, the color space setting in the Premiere export window should not be performing color space conversions. They should be telling the encoder what color space the footage already is. I realize the problem is Premiere outputs in RGB32. Basically, this needs to be converted to rec709/yuv420, and then encoded to whatever color space we select without further conversion.

    If Premiere outputs only RGB32 I think it is problematic because it would always convert the HDR frame in RGB which would destroy the color space of BT2020 or BT2100.

    However if I refer to this post, Premiere does not outputs in RGB32 or maybe RGB32 -> YUV 444 -> yuv444p16 -> yuv420p10 ?

    Zitat von Vouk

    Yes, This piping feature supported only RGB which is not relevant for the implemented encoders ...

    If you select any non 8bit pixel format in the encoder settings ...

    • Premiere will render it in YUV 4:4:4:4 with 32bit float precision.
    • It will then be converted to yuv444p16 to feed it into FFmpeg/libav.
    • And finally converted to the target format (yuv420p10)

    In my case, I have a Sony AX 700 which outputs those kind of files:

    It is not a 10 bits file and use H264 which supports HLG HDR which I can say in this case is not a "True" HDR file with 10 bits depth so maybe it is not necessary to convert it to 10 bits with yuv420p10.

    The color issue is even here when trying to encode it with H264 of Premiere Pro and HDR enabled. With Voukoder 2.0.8, do you still have this issue or it is working fine ?

    Zitat

    However if I refer to this post, Premiere does not outputs in RGB32 or maybe RGB32 -> YUV 444 -> yuv444p16 -> yuv420p10 ?

    Oh okay, you're right, I was mistakenly remembering the bit about piping.

    Zitat

    In my case, I have a Sony AX 700 which outputs those kind of files:

    This... is an issue, actually. For voukoder HDR output to work, it needs to be sent the unprocessed raw yuv data, and while Premiere may be outputting yuv, it's doing so in a floating point SDR transfer function. So your video will be clipped to 100nit SDR, and outputting to HDR would severely distort the image. This is why I specifically tell my Ninja Inferno to record HDR data in "rec709 mode" which maintains all 10bit source pixels unmodified, but simply flags it as rec709 instead of rec2020. This way, voukoder gets the data unmodified, meaning it gets the correct transfer function and chroma values, even though Premiere thinks its rec709.

    While Premiere supports HDR input, the workflow for it isn't something we can use Voukoder with, since it would require a conversion from floating point SDR (which contains values "above white") to PQ or HLG, which use a very different gamma transfer functions. It's a pretty complicated process that would severely slow down the encoder I'm sure, so it's not optimal. The only way to edit a correctly flagged HDR input file and output it in HDR as well, is to use Premiere's own H265 output encoder, with the HDR flag selected. It's not great, since you can't use CRF and other advanced options, but until Premiere improves on its workflow options, I think that's all you can do.

    To do proper HDR through Voukoder, you're going to need to find a way to convince Premiere that your video is rec709. You may need to reencode using something that preserves the pixel data correctly but doesn't preserve the rec2020/HLG flags, such as avidemux, unless you can find some kind of program that will modify video track flags. Possibly ffmpeg or MP4box? I'm not sure.

    • Offizieller Beitrag

    Currently there is only the the color space dropdown box in premiere:

    This defines in what way I request the framebuffer from premiere. There are two possibilities:

    • rec709 based
    • non-rec709 based (I assume rec601?)

    At this moment it also defines the target colorspace of the exported project:

    For rec601 and rec709 it is quite simple. But the question is: Are we going to have that many of possible combinations of rec2020 that I need to separate this into two parts:

    1. In premiere to chose between rec601 and rec709. This will only define the pixel format that I get back from premiere
    2. In the voukoder dialog add 3 dropdown boxes to select space, trc and primaries.

    I'm not sure if I should do this. As I'd like to keep it as simple as possible. Is it necessary?

    About the VUI flags (x265 options)

    I will add them back to the advanced settings. These are advanced settings anyway and should be set only by users who know what they're doing. Still, what will will happen if the trc is set to SMTPE2084 and the VUI is set to bt2020-10? I don't know.

    It's certainly possible for there to be weird combos, like I've seen 4K Blu-rays with DCI-P3 primaries, even though they all use rec2020 color space. It's also technically possible to use rec709-based trc with rec2020 colors, but this is rare. This would be for example, an SDR video with Wide Color Gamut. And then you also have HLG, which uses the ARIB STD-B67 trc. Some broadcast providers use this format, as well as Wickyed's camera, apparently.

    Maybe what I would suggest is leave the drop down as it is when the user first opens the Premiere Export page, but once they select bt2020nc or bt2020c, then two additional dropdowns appear for transfer fuction and primaries, with st2084 and bt2020 as defaults. I think most people dealing with bt2020 color will have some familiarity with those concepts, but they most likely won't be necessary for rec709 or rec601 formatted videos.

    If you do this, the x265 options for those may not be necessary, but only if you can ensure that no color space conversion is happening. I fear those filters above will still cause a conversion. Changing them to ispace/itrc/iprimaries may fix that. If not, keep the x265 options as a failsafe. I believe that the options on x265 will simply re-interpret what ffmpeg passes to it, without any further conversion. If ffmpeg's filters are doing a conversion, then x265's options won't undo that. They will simply assume the already-converted colors are in the space you specify.

    Also, in the above code, bt2020nc and bt2020c are handled the same. Might want to fix that.

    • Offizieller Beitrag
    Zitat von morphinapg

    Maybe what I would suggest is leave the drop down as it is when the user first opens the Premiere Export page, but once they select bt2020nc or bt2020c, then two additional dropdowns appear for transfer fuction and primaries, with st2084 and bt2020 as defaults. I think most people dealing with bt2020 color will have some familiarity with those concepts, but they most likely won't be necessary for rec709 or rec601 formatted videos.

    I think about adding 3 dropdown boxes to voukoder. The most common values will be preselected. Reason for this that these values are independent from premiere and need also to be set when using an other NLE (i.e. VEGAS)

    Zitat von morphinapg

    I fear those filters above will still cause a conversion.

    Filters will only be applied when the frame data from the NLE differs to the output setting.

    Examples:

    • premiere delivers rec709 limited - output is set to rec709 limited -> No Conversion
    • premiere delivers rec601 limited - output is set to rec2020 something -> Conversion

    Due to the fact that premiere will never deliver rec2020 there will be always a conversion. There needs to be a ruleset defined.

    I've done some test and I think there are also issues with Adobe itself and its HDR definition that does not include HLG. Premiere H264 and HEVC encoders don't seem to handle HLG and the result is very ugly.

    I also encoded directly the flow with StaxRip in x265 which seems to handle correctly the HDR options.

    I let you check the comparison in the PDF and ask if you have any question.

    I'm planning to extend the test with a frameserver in YUV2 and RGB32. The outputs of my AX700 are in YUV 420 8 bits so I think there is some sence to use YUV2 in my case. For the RGB32 I don't think it is really relevant as it will totally destroy the HDR source.

    Also I can't install version 2.0.8, its making my Premiere crash so I will wait for the next version of Voukoder to use again HDR.

    I will stick with non HDR recording for the moment as HLG seems not to be supported natively in Premiere.

    I will also suggest to have a solution that would ease the future integration of other recommendations such as the BT.2100.

    Zitat von Vouk

    I think about adding 3 dropdown boxes to voukoder. The most common values will be preselected. Reason for this that these values are independent from premiere and need also to be set when using an other NLE (i.e. VEGAS)

    Filters will only be applied when the frame data from the NLE differs to the output setting.

    Examples:

    • premiere delivers rec709 limited - output is set to rec709 limited -> No Conversion
    • premiere delivers rec601 limited - output is set to rec2020 something -> Conversion

    Due to the fact that premiere will never deliver rec2020 there will be always a conversion. There needs to be a ruleset defined.

    The issue here is that if you want to be able to output HDR correctly in premiere, then premiere must interpret it as rec709. I really don't think there's a reason for conversion. If you use the ispace, itrc, iprimaries filters then you're defining what the input space is, which is what the main reason for using those options should be. If the user wants to do a conversion, then that should be a separate option on the video filters section of voukoder.

    Perhaps as a compromise, maybe you could keep it the way it is (plus the other two drop downs when bt2020 options are selected) but add "Override Input Color Space" to the video filters section, where the user can override those values for the input, in case something in the NLE is sending the data incorrectly. Also, I would recommend requesting the rec709-based input for anything that isn't rec601 selected.

    Premiere just has a particularly weird way of handling HDR when HDR is flagged correctly. It will never work with voukoder as it is, because it will be sending data in an unclipped rec709 format. Like, there will be values "brighter than white" and the only way to process it correctly would be to perform a manual mathematical conversion from THAT source, into the correctly selected transfer function. Furthermore, I'm pretty sure Adobe is clipping black in this mode as well. It's just a mess on that end and simply won't be sending the data in the way x265 expects, and it will likely stay that way for a while.

    Einmal editiert, zuletzt von morphinapg (9. November 2019 um 21:49)

    • Offizieller Beitrag

    Okay, so I would propose something like this:

    • Im the Video tab in premiere you can only select between rec601 and rec709. The default setting is rec709 as most people will work with HD projects in these days. This would also be the setting to use in your case.
    • In Voukoder you would have additional dropdown boxes for Color Range, Color Space, Color Primaries and Transfer Characteristics. These will be defaulted to the setting in premiere. So if premiere is set to rec709 it will appear as default. Color Range will be defaulted to the premiere setting which is Limited by default.
    • As Voukoder and premiere are set to the same values no color space conversion will be done. In your case you would just set the VUI flags in the x265 configuration to some rec2020 values

    In my opinion this would be the best way. What do you think?

    It seems like a decent way to do it. It's probably a good idea to move those controls to voukoder instead of the premiere window anyway, so you can control them in other programs as easily. This solution would work for x265 (it would effectively do what 2.0.7 did with its HDR settings), but not necessarily for other encoders though if they don't have those individual options, so you may want to eventually consider that "Override Input Color Space" idea for the video filters section. Even if just for the case of having additional control over the conversions happening. Heck, you could even use that to do some funky color effects lol

    I think yeah, if you could implement it there with no conversion, I think that would be ideal and most user friendly. Then you wouldn't need the extra x265 options, as long as all the major formats are included there.

    Zitat von Vouk

    I'll implement it that way. It was really nice discussing that with you to find the best solution for this issue.

    Well done ! I will also test it with NVENC HEVC once released.

    Zitat von Vouk

    They are listed here: https://ffmpeg.org/ffmpeg-filters.html#colorspace

    It seems ffmpeg also supports HLG (https://github.com/FFmpeg/FFmpeg/…2083246ca598fce) but is not listed in the documentation.

    It was added years ago under the value "arib-std-b67" for trc, could you also add it ?

    Again thank you very much for your plugin :)