- Edited
I previously tried debugging the module's code, and the main decision regarding dithering was applied at this location: https://github.com/torvalds/linux/blob/c2ee9f594da826bea183ed14f2cc029c719bf4da/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crc.c#L237.
I also made some changes in a few other places, including the files you mentioned above. However, it seems I might have overlooked something. I initially got the RX6600 mainly for experimenting with Hackintosh, but those attempts were unsuccessful due to dithering issues. At that time, I didn't focus much on Linux, unlike now with Intel cards. Unfortunately, I've sold the RX6600, so I can't give it another try at the moment.
P.S. I think I'll build a new PC next year based on an AMD CPU. If it has an iGPU, I'll give it another try.