With the described setup that I had been using for several weeks, I experienced strong eye strain after an Nvidia driver upgrade 2 days ago. Which could be coincidence, but in the process I have learned how to downgrade, which today I have successfully done. I would like to share the instructions, as suddenly receiving a worse driver could possibly happen to anyone, even when using a non-rolling-release distro like Debian or Ubuntu.
How to downgrade Nvidia drivers via apt:
The most important thing is to save a copy of /var/log/apt/history.log. It contains information about the recent unwanted upgrade process.
Using this file, spot all Nvidia-related packages that had been upgraded. They can be recognized by looking at their versions, which for Nvidia packages follows a scheme like "(470.70.103.01... 470.129...)". The first, smaller version in the parentheses is the old one that got upgraded. Using this older versions, create a text file that has the following structure:
libxnvctrl0=470.94-1~bpo11+1
libnvidia-rtcore=470.103.01-1~bpo11+1
...
The scheme is package=version.
The version numbers, "=xxx.xxx.xx...", must be the old versions that we want to downgrade to.
To remove everything Nvidia-related:
$ sudo apt purge '*nvidia*'
There is a chance it might remove a little to much, especially on Ubuntu, but I use Debian and have no further information on that.
Pretending the text file created earlier was named "downgrade.txt", type:
$ sudo apt install $(cat downgrade.txt | tr "\n" " ")
It should, without complains, explicitly downgrade the packages.
Afterwards, put the Nvidia packages "on hold" to prevent future upgrades:
$ sudo apt-mark hold '*nvidia*'
When you now do "sudo apt update", the new Nvidia packages will be found, but if afterwards you try "sudo apt upgrade", they will be held back and won't get upgraded.
To ever undo the hold, type
$ sudo apt-mark unhold '*nvidia*'
What has not been mentioned yet is that disabling temporal dithering via the NVIDIA X Server Settings may be desirable. To automatically load the dithering state, it is necessary to execute the user command "nvidia-settings --load-config-only" after each desktop login. In Xfce, this can be done by going into Applications - Settings - Session and Startup - Application Autostart and creating, and marking as "checked", a new entry that executes that command.