In short: You want to use a USB audio interface, connect it to the AVR over S/PDIF, and play native DSD.
This setup is generally possible with LE.
Theory: I think the problem is the proprietary ASIO driver of your USB audio interface. Because it's proprietary, not all codecs are usable by LE. That means, LE offers all codecs, including hi-res native DSD, but the incomplete Linux driver only accepts lo-res DSD.
Suggestion a): Use a USB audio interface, which is fully supported by Linux.
Suggestion b): Use an RPi with HiFiBerry Digi2 Pro, as mentioned at post #8.
DSD doesn't travel over regular 44.1-48k/16bit stereo SPDIF does it? (Though higher sample rate 192k/24bit SPDIF may carry some of the lower bitrate DSD stuff via DoP?)
DSD64 (as used by SACD) is based on 2.8824MHz 1-bit Delta Sigma sampling (so needs 2.884Mb/s connectivity - which is more than the 1.536Mb/s of 48k/16bit stereo that regular SPDIF supports?)
DSD512 is based on 22.5792MHz delta sigma 1-bit sampling - which is nearly 15x the data rate of basic SPDIF.
AIUI there are USB standards for both DSD native transport, and DSD-over-PCM aka DoP (where DSD audio is kept as DSD format data but presented as if it's PCM at a high sample rate like 768kHz and sent using USB PCM Audio - in a standard that compatible devices then re-package back to DSD allowing native transport of the DSD Delta Signal bitstream. This requires the DSD to DoP packaging software/drivers to be DSD DoP-aware?)
The OP is - I think - asking if it's possible to connect a USB Native DSD DAC to LibreElec to send DSD audio from LE to a USB-connected DAC which then outputs analogue - without transcoding that audio to PCM format audio for DAC decoding. This is a widely used route with smartphones, tablets, Windows and Mac PCs.
I don't know what the Linux support is for DSD64-512 native audio transport over USB - either as DSD Native or carried as DSD-over-PCM.
NB I'm not making any claims for DSD audio being better or worse than PCM audio, nor am I making any claims that DSD512 is better than 48k/16bit audio - but I know for some people it's important...
Display More1.
As far as I understand XMOS is supported and native DSD support possible as long as the hardware is recognised by her USB ID provided in that "C" file above.
2.
Since my DSD Files are converted to 44.1kHz PCM, I have to asume, that my YAMAHA Amplifier is not registered in that list by his USB ID.
Is there any easy way to provide LibreElec with the YAMAHA USB ID without the need to compile the whole distribution?
What Yamaha amplifier do you have ?
I take it from your post that it has a USB connector that will appear to connected devices as a USB DAC for both PCM and DSD audio? If you find out the USB IDs for it that may help?