I don't have a "solution" .. but here's some thoughts that might be useful:
GTX260 does not have any on-board audio capabilities; it is normally a DVI output only card and DVI does not carry audio. As this is the era before nVidia figured out how to do HDMI properly the card shipped with a cable that connects the motherboard soundcard digital output (S/PDIF) to unused pins in the DVI connector which a custom nVidia DVI to HDMI adaptor uses to present "HDMI" audio to a TV or AVR. HDMI and S/PDIF send the same signal, so this allows the card to send full 8-channel audio via HDMI (over S/PDIF the bandwidth is reduced and the receiver will not support more than 6-channel, i.e. 5.1 audio). For this to work in Windows an nVidia audio driver must be used .. so likely there is something similar under linux. If you are using DVI not HDMI i'm wondering what might happen if this cable is disconnected and/or any nvidia audio modules are blacklisted to prevent them from being used.
The kodi.log only shows "nVidia" audio devices being presented by ALSA, yet some fiddling with pulse (which reads hardware capabilities from the kernel in addition to being aware of ALSA devices) managed to get some audio out. I suspect via you've managed to route audio to the internal card; either by ignoring the nVidia devices or via snoop/dmix sending data to all devices. When fiddling with the config in alsamixer did you save/store the active configuration? .. this is not automatic.
In the past (when we supported mk1 AppleTV devices which I maintained) the PCM output volume needed to be 100% before S/PDIF output worked; any less and the signal strength wasn't enough to be correctly carried (S/PDIF is digital but the signal still needs to be clean/strong enough for comms to work). The description of audio needing to be zero is the exact opposite, but makes me wonder if anything above zero results in additional audio data being mixed into the audio output and disrupting the digital signal.