Full Range washed out

Hi,

This is due to incorrect interpretation of the decoded data. Most likely VirtualDub asks the codec to decode as YV12 (YUV 4:2:0) and interprets the decoded data as limited range YUV, hence when it converts to RGB for display, you get the washed out result (the reason being that VDub uses only the 16-235/16-240 portion of the 0-255 encoded data during conversion, whereas if the encoded data were limited range, it would be correct). EDIT: This was blatantly wrong, see my next post...

The easiest solution is to not use full range YUV. It requires careful consideration and knowing exactly how programs and codecs interpret YUV data and adjust them accordingly.

If you really want to keep full range YUV, there are two options. Either make sure that the conversion to RGB is done by the codec, that way it is not possible to incorrectly interpret the decoded data. This is not always possible to do. The other option is to carefully verify every program you use about how it receives the decoded data (YUV or RGB), and how it interprets it and adjust them accordingly.

To verify the decoded format, hover the mouse over the green [M] icon in the notification area and a tooltip will appear telling you exactly what the codec is decoding. Keep in mind that some programs use the unofficial ffmpeg implementation of the decoder (like Handbrake), and then there won't be any icon there. I also don't have any information whether ffmpeg interprets the data correctly or not (it should, but I haven't tested).

So in summary, your best bet is to disable full range YUV. There really is no real benefit of using it, and it is mainly intended for very special scenarios.

Posted by: Balázs

Hi,

This is due to incorrect interpretation of the decoded data. Most likely VirtualDub asks the codec to decode as YV12 (YUV 4:2:0) and interprets the decoded data as limited range YUV, hence when it converts to RGB for display, you get the washed out result (the reason being that VDub uses only the 16-235/16-240 portion of the 0-255 encoded data during conversion, whereas if the encoded data were limited range, it would be correct).

The easiest solution is to not use full range YUV. It requires careful consideration and knowing exactly how programs and codecs interpret YUV data and adjust them accordingly.

If you really want to keep full range YUV, there are two options. Either make sure that the conversion to RGB is done by the codec, that way it is not possible to incorrectly interpret the decoded data. This is not always possible to do. The other option is to carefully verify every program you use about how it receives the decoded data (YUV or RGB), and how it interprets it and adjust them accordingly.

To verify the decoded format, hover the mouse over the green [M] icon in the notification area and a tooltip will appear telling you exactly what the codec is decoding. Keep in mind that some programs use the unofficial ffmpeg implementation of the decoder (like Handbrake), and then there won't be any icon there. I also don't have any information whether ffmpeg interprets the data correctly or not (it should, but I haven't tested).

So in summary, your best bet is to disable full range YUV. There really is no real benefit of using it, and it is mainly intended for very special scenarios.

So you're saying that when I open the video in VirtualDub that the full range video is interpreted as limited range video? If so, then why is raw capture seemingly interpreted correctly? Is it the case that VD is explicitly converting the full range video to limited range during the capture process?

You say that my best bet is to disable full range YUV, but is it okay to compress a full range signal using limited range YUV? I think that doing this may be giving me color banding issues.

I need more details about the capture pipeline, how is the capture done exactly, and what is the capture format provided by the card to the codec precisely? If it is YUV 4:2:0, then setting full range YUV in the codec settings is only meaningful if the incoming YUV 4:2:0 raw data is full range. If it is not, then the codec will do incorrect conversion to RGB when decoding (as then it assumes the encoded data is full range when it is not). The codec does not use the full/limited setting in any way when data is incoming/decoded as YUV, only YUV<->RGB conversions inside the codec take it into account.

BTW I said it incorrectly last time (sorry, I got confused 🙂 ). The washed out result comes when you assume full range, but have limited range, and not when you assume limited but have full (which would make the image darker than actually is). So the ultimate problem is the codec most likely gets limited YUV 4:2:0, but assumes it is full (because of the settings), and when it decodes as RGB, the result is washed out (as in reality pixels only have limited 16-240 values, but the codec assumes 0-255 when converting during decoding, so you won't have blacks, only grey).

I hope that makes more sense now.

The important things to very carefully examine is where the conversions happen and how does each component treat or assume data when receiving/converting it.

Posted by: Balázs

I need more details about the capture pipeline, how is the capture done exactly, and what is the capture format provided by the card to the codec precisely? If it is YUV 4:2:0, then setting full range YUV in the codec settings is only meaningful if the incoming YUV 4:2:0 raw data is full range. If it is not, then the codec will do incorrect conversion to RGB when decoding (as then it assumes the encoded data is full range when it is not). The codec does not use the full/limited setting in any way when data is incoming/decoded as YUV, only YUV<->RGB conversions inside the codec take it into account.

BTW I said it incorrectly last time (sorry, I got confused 🙂 ). The washed out result comes when you assume full range, but have limited range, and not when you assume limited but have full (which would make the image darker than actually is). So the ultimate problem is the codec most likely gets limited YUV 4:2:0, but assumes it is full (because of the settings), and when it decodes as RGB, the result is washed out (as in reality pixels only have limited 16-240 values, but the codec assumes 0-255 when converting during decoding, so you won't have blacks, only grey).

I hope that makes more sense now.

The important things to very carefully examine is where the conversions happen and how does each component treat or assume data when receiving/converting it.

Capturing is done via VirtualDub, captured with an Elgato 4K60 Pro and using the MagicYUV 420 codec. The capture filter in VD was reporting the signal colorrange as Full (0 - 255) and in the capture filter the output color range was set to bypass (no change). The data format in VirtualDub is YV12. The capture source is a PS4 Pro outputting at 2160p in Full Range RGB.

Maybe it is an issue with the capture card? I used to use a BlackMagic Intensity Pro 4K and Full Range MagicYUV 444 without these issues, but with that card I could use a 24bit RGB data format.

I see, I'm not too much familiar with capturing in VDub, the question I have is who does the conversion to YV12? Is it VDub or does YV12 come directly from the card? Can you check what format does the codec actually receive? You can do that by starting a capture process and hovering the mouse over the red [M] icon in the notification area and looking at the tooltip. Also, can you upload a small sample of both raw 420 and MagicYUV 420 capture?

Posted by: Balázs

I see, I'm not too much familiar with capturing in VDub, the question I have is who does the conversion to YV12? Is it VDub or does YV12 come directly from the card? Can you check what format does the codec actually receive? You can do that by starting a capture process and hovering the mouse over the red [M] icon in the notification area and looking at the tooltip. Also, can you upload a small sample of both raw 420 and MagicYUV 420 capture?

The red M icon say In: YV12 when recording. I'm not sure if it is VirtualDub or the capture card that does the conversion. The capture card does support both YV12 and YUY2 which indicates that it at least doesn't do a hard conversion to YV12.

Below are the two video samples. Note that the file sizes are very large and I plan on deleting them from my Google Drive account once you've downloaded them.

https://drive.google.com/file/d/1otbVVXUqOI9mOqOm6R79WZMgOxuOJtzL/view?usp=sharing
https://drive.google.com/file/d/16tUmSlfnKl7GVM6fKPvBs-JnOlIADhRV/view?usp=sharing

In that case I'm pretty sure it's limited range, unless there is some specific option on the card that specifically indicates it is full range YUV. (You mentioned in your first post that you capture RGB with the RGB range set to full range, where is that setting (screenshot)?)

I'll try to download the files ASAP, but just when I was about to, our internet connection went down (I'm on mobile net now, so can't download large files). It's an issue with the provider, I really hope they'll fix it soon, but it could take up to 72 hours (I darn hope it won't).

Posted by: Balázs

In that case I'm pretty sure it's limited range, unless there is some specific option on the card that specifically indicates it is full range YUV. (You mentioned in your first post that you capture RGB with the RGB range set to full range, where is that setting (screenshot)?)

I'll try to download the files ASAP, but just when I was about to, our internet connection went down (I'm on mobile net now, so can't download large files). It's an issue with the provider, I really hope they'll fix it soon, but it could take up to 72 hours (I darn hope it won't).

The Capture Card doesn't support RGB. I meant that I set my PS4 Pro to output in RGB as opposed to YUV 420 which it supports for outputting 2160p to 4K displays that don't support HDMI 2.0.

Downloaded the files, you can delete them. I'll get back to you shortly.

OK, so looking at the material, it is definitely Limited range, the card provides that.

Full range is very seldom used. If you really want full range for some reason, your only option is to provide RGB data to the codec and let the codec do the conversion. You also have to be very careful during decoding. Normally if the codec decodes as RGB you'll be fine, as then the codec does the conversion, but if it decodes as YUV, you have to make absolutely sure that the receiving application interprets the data correctly.

In the latest version of VirtualDub2, you can change how the data is decoded and interpreted on the Decode Format dialog (Video -> Decode Format menu). By default VirtualDub assumes Rec.601 Limited range for YUV data. So you definitely have to change that to Rec.709 for sure.

Keep in mind that even though you specify Full range in the codec settings when encoding, it ONLY affects YUV<->RGB conversion INSIDE the codec. So if the codec encoded YV12 and VDub also gets YV12 from the codec later, it doesn't matter how the codec was set up.

The check stuff you can use the "Histogram" filter in VirtualDub2. You can also recognize if the color space is wrong (Rec.601/709), it can sometimes show up on the histogram as jagged edges (as I've seen).

Regarding banding, I don't think you can avoid that fully (only HDR can fix that, but that's still not widely supported). If your final target is to upload to Youtube or similar places with h264 codecs, those all use limited range anyhow.

Hope this helps.

Yeah, it seems that this is a capture card or driver issue and not a problem with your codec. I'm going to get in touch with Elgato to try and fix the issues that I'm having.

Thanks for your help, Balázs.

It is not really an 'issue' per se, the card is working correctly, YV12 and YUV in general is limited range by default in 99% of the cases, and Rec.709 if the material is >= 720p.

The only thing you have to watch out for right now is that VirtualDub interprets YUV as Rec.601, so that should be the only thing to set manually.

If you set the codec to limited range when recording, you shouldn't run into any problems.

I originally wanted full range because the game footage looked washed out when playing the games on my monitor during the capture process. However, yesterday I discovered that my monitor actually has an option to operate correctly for limited range inputs. When I capture from my Xbox One in limited range though the footage is washed out in the Virtual Dub preview window and the subsequent captures. Using VDub's histogram shows the color spectrum looking squashed relative to how a normal limited range signal should look.

This seems to be a capture card issue as the game looks fine on my monitor in limited range color but looks incorrect when captured or viewed in VirtualDub. VDub here is set to decode in limited range Rec.709 which is what the capture filter shows is the input video format.

I see, now here's some additional info regarding ranges, just to increase the confusion level a bit 🙂

There is YUV limited range as we discussed, YUV is limited by default, so no problem there.

BUT there is also RGB limited range, which is a whole different story! I don't know too much about that, but as I recall, TV sets might operate in limited range RGB. This is different from monitors, monitors always operate in full range RGB (as well as PCs). It could be that the Xbox is outputting limited RGB. In that case you have to set the capture card to limited range RGB, or the Xbox to output full range RGB. This is just guesswork from my side, but might worth looking around for more info on this.