pl01
Continuation of my observations:
Got EVanlak 8K dummy hdmi plug. Set up the mirroring of it to external display connected to macbook pro 14 m1 pro via DP cable.
This dongle does not corrupt it's edid info (which is over 256 bytes for some reason). Also it has built in support for 120 hz mode and 1440p out of the box.
So it's basically plug and play setup for now.
I do see color banding like the true 8-bit mode without dithering
Whenever I connect it to Windows PC YCbCr 4:4:0 or 4:2:0 is shown in the connection details.
I have a suspicion that using YCbCr dummy and rgb output for external monitor forces 8-bit color internally.
Also tried flashing EDID with YCbCr signals completely removed from EDID. This way mirroring of RGB color space of dummy to RGB color space of external monitor did not help - I've got the same headaches as plugging to it using DP (without dummy mirroring).
Maybe the MacOS's dithering can also be disabled using YCbCr/RGB color space manipulations?
Connecting the monitor (AOC Q27G2S/EU 1440p 165 HZ 8-bit+FRC) using HDMI forces the color space to YCbCr with colors that look off (too darkened). However, the color banding is here the same as with YCbCr to RGB conversion.
Connecting the monitor using HDMI with modified EDID which has all the blocks related to YCbCr support removed results in the same picture 1 to 1 as when connecting using DP 1.4 cable (almost no banding seen) and immediate headaches.
Color reproduction seems the same when mirroring YCbCr hdmi dummy to RGB external display.