Rikl Add a little BetterDisplay
@ledstrain2024 More precisely, BetterDisplay Screen Streaming (this feature requires the Pro version or the 14 day trial FYI). This enables you to render a true non-Retina "sharp and pixelated-style" desktop to your Retina display.
Guide: https://ledstrain.org/d/2686-i-disabled-dithering-on-apple-silicon-introducing-stillcolor-macos-m1m2m3/1006
For me, using this "True Non-Retina Streaming" method dramatically increased comfort (even more, given that Stillcolor had already helped me so much already) as I no longer feel overstimulated by the amount of detail and "lack of any sharp pixels to lock onto" that usually makes it tiring for me to process text or very information-dense UI on Retina / HiDPI displays.
Unlike simply choosing a "non-retina" resolution in default macOS which causes the GPU to apply a "smoothing" scaling filter that looks blurry and ugly… the Streaming method actually is capable of mapping exactly every four physical pixels to one "sharp" virtual pixel.
This means you can finally "dumb your Retina display down" into a "pixelated" non-Retina display reminiscent of older monitors with visible pixels, while entirely avoiding the "blurry" effect you'd usually get from trying to do this.
BTW, if the screen starts feeling less comfortable to you I'd suggest disabling the contrast adjustment you mentioned. every time I tried adjusting contrast on my m1air instead of leaving color adjustments at default, it immediately started feeling less comfortable to me. Default contrast is noticeably better IMO.
For what it's worth Night Shift also caused issues for me too (even with Stillcolor), so what I ended up doing instead to make the screen feel "less bright" is buy a physical matte privacy screen protector that both eliminates glare and significantly dims the screen (a side effect of the privacy viewing angle reduction feature). See my post history for details on which one I bought. Not sure from your post if the screen filter you're already using is matte or glossy.
Finally, running this command (needs BetterDisplay installed and running) in Terminal and keeping the Terminal window minimized in the background improved the display even more for me in brighter conditions. This command disables "ambient light auto-contrast adjustment" (an obscure feature that is separate from auto-brightness. Unlike auto-brightness, this auto-contrast feature doesn't have an "off switch" in Settings. This command forces it to disable by setting it to zero every 2 seconds)
while true; do /Applications/BetterDisplay.app/Contents/MacOS/BetterDisplay set -namelike=built -framebufferNumericProperty=0x0 -specifier=IOMFBContrastEnhancerStrength; sleep 2; done